# MinMax Mean-Field Team Approach for a Leader-Follower Network: A Saddle-
Point Strategy

Mohammad M. Baharloo, Jalal Arabneydi and Amir G. Aghdam This work has been
supported in part by the Natural Sciences and Engineering Research Council of
Canada (NSERC) under Grant RGPIN-262127-17, and in part by Concordia
University under Horizon Postdoctoral Fellowship.Mohammad M. Baharloo, Jalal
Arabneydi, and Amir G. Aghdam are with the Department of Electrical and
Computer Engineering, Concordia University, 1455 de Maisonneuve Blvd. West,
Montreal, QC, Canada, Postal Code: H3G 1M8. Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper investigates a soft-constrained MinMax control problem of a leader-
follower network. The network consists of one leader and an arbitrary number
of followers that wish to reach consensus with minimum energy consumption in
the presence of external disturbances. The leader and followers are coupled in
the dynamics and cost function. Two non-classical information structures are
considered: mean-field sharing and intermittent mean-field sharing, where the
mean-field refers to the aggregate state of the followers. In mean-field
sharing, every follower observes its local state, the state of the leader and
the mean field while in the intermittent mean-field sharing, the mean-field is
only observed at some (possibly no) time instants. A social welfare cost
function is defined, and it is shown that a unique saddle-point strategy
exists which minimizes the worst-case value of the cost function under mean-
field sharing information structure. The solution is obtained by two scalable
Riccati equations, which depend on a prescribed attenuation parameter, serving
as a robustness factor. For the intermittent mean-field sharing information
structure, an approximate saddle-point strategy is proposed, and its converges
to the saddle-point is analyzed. Two numerical examples are provided to
demonstrate the efficacy of the obtained results.

Proceedings of IEEE IEEE Control Systems Letters, 2020.

##  I Introduction

Recently, there has been a surge of interest in the application of networked
control systems in various engineering problems such as sensor networks, swarm
robotics and cooperative coordination of unmanned aerial vehicles, to name
only a few. In this type of system, it is desired to achieve a global
objective (such as consensus or flocking) using local control laws with
limited information exchange. Many practical problems need to be taken into
consideration in the design of a real-world networked-control system. For
instance, multi-agent networks are often subject to external disturbances,
which means that a practical control strategy needs to be robust to unwanted
disturbances. However, in the theoretical analysis, such practical issues are
usually neglected for simplicity.

Different robust control design techniques are introduced in the literature
such as $H_{\infty}$-control, risk-sensitive control and MinMax control, each
of which has its own strengths and weaknesses. For example, in risk-sensitive
asset management in the financial market, a risk factor is utilized in order
to capture the randomness of the system; this suits applications with unknown
disturbances. In MinMax control approach, on the other hand, the external
disturbances are modeled as an adversarial player attempting to maximize the
cost of the system. In general, there are two types of formulations for the
MinMax control problem: (a) hard-constrained formulation, where an upper bound
is set on the magnitude of the disturbance, and (b) soft-constrained
formulation that penalizes the disturbance by a negative quadratic cost
function [1]. It is demonstrated in [1] that the hard-constraint formulation
is much more complex and less tractable than the soft-constraint formulation.
Therefore, in this paper we focus on the soft-constraint formulation only.

There are two main challenges concerning a MinMax control setting in the
leader-follower problem. The first one is the computational complexity that
increases with the number of followers (curse of dimensionality). The second
challenge is that it is not always feasible to assume that the states of all
followers are available, specially when the number of followers is large. In
such cases, a decentralized information structure is more desirable but it
leads to a discrepancy in the followers’ information. To address the above
challenges, mean-field models are introduced in the literature to provide a
tractable approximate solution to large-scale symmetric problems [2, 3]. In
the control setting, mean-field-type game [4, 5, 6] studies an infinite-
population model wherein the control optimization problem reduces to a one-
body McKean-Vlasov optimization problem due to the simplification offered by
the negligible effect in the infinite-population. In contrast, mean-field team
[7] focuses on the finite-population model, where the effect of each
individual player is not necessarily negligible. For linear quadratic mean-
field teams, a transformation-based technique was first proposed in [8] and
further extended in [7, 9, 10], that is similar in spirit to the completion-
of-square approach in mean-field-type game [11, 12]. In general, mean-field-
type game may be viewed as a special case of mean-field team, where the
information structure is mean-field sharing and the number of players is
infinite.

In the mean-field games literature, many researchers consider a leaderless
network with a large number of homogeneous followers in the presence of an
adversarial disturbance (player). For example, the authors in [13] study a
MinMax mean-field-type game problem in the context of social networks. The
proposed solution is an approximate robust mean-field equilibrium that is
formulated in terms of two coupled forward-backward partial differential
equations (i.e., the Hamilton-Jacobi-Isaacs and Fokker-Planck-Kolmogorov
equations). In [14], a Stackelberg-like mean-field game problem is considered,
where an adversarial disturbance moves first and its worst-case value is
computed irrespective of the actions of followers. Given the worst-case
disturbance, an approximate robust Nash equilibrium is obtained for the
followers using two coupled forward-backward stochastic differential
equations. A similar Stackelberg-like mean-field problem is investigated in
[15] with social cost function, where the variational analysis and person-by-
person optimality principle are employed to derive an approximate robust mean-
field-type strategy in terms of two coupled forward-backward stochastic
differential equations. The authors in [16] integrate a leader player to the
above setting in a sequential fashion, where the method is to first solve a
MinMax control problem for the leader against its disturbance (irrespective of
the actions of followers), and then solve a MinMax control problem for the
followers that wish to track a convex combination of the mean-field and the
state of the leader in the presence of disturbances. Unlike the above results
in which the solution is presented in a semi-explicit manner, a leaderless
MinMax mean-field-type game solution is computed explicitly in [12] in terms
of two Riccati equations, where an existence condition for the solution in the
case of scalar time-invariant coefficients is proposed.

In this paper, a leader-follower MinMax mean-field team problem with
multivariate time-varying coefficients and an arbitrary number of homogeneous
followers is investigated. In contrast to [13, 14, 15, 16], we obtain the
unique explicit saddle-point strategy under mean-field sharing information
structure, and then give an approximate solution under intermittent mean-field
sharing. It is to be noted that the solution concept of a saddle-point
strategy (that simultaneously solves convex and concave optimization
problems), is different from a Stackelberg strategy (that solves the two
problems sequentially). Also, the Riccati equations obtained in [12, Equation
29] may be viewed as a special case of the continuous-time counterparts of the
Riccati equations derived in this paper, where the coefficients associated
with the leader are zero.

The remainder of the paper is organized as follows. The problem is defined and
formulated in Section II. The main results are subsequently presented in
Section III along with the required assumptions. In Section IV, two numerical
examples are given to demonstrate the effectiveness of the results, and
finally in Section V some concluding remarks are provided.

##  II Problem Formulation

In this article, $\mathbb{R}$ and $\mathbb{N}$ denote, respectively, the sets
of real and natural numbers. For any $k\in\mathbb{N}$, $\mathbb{N}_{k}$
denotes the finite set $\\{1,\ldots,k\\}$, and $x_{1:k}$ is short-hand
notation for $\\{x_{1},\ldots,x_{k}\\}$. $\mathbb{E}(\bm{\cdot})$ is the
expectation of an event, $\operatorname{Cov}(\bm{\cdot})$ is the covariance
matrix of a random vector, $\operatorname{Tr}(\bm{\cdot})$ is the trace of a
matrix, and $\mathbf{I}$ and $\mathbf{0}$ are, respectively, the identity and
zero matrices.

Consider a multi-agent system consisting of one leader and $n\in\mathbb{N}$
homogeneous followers. Let $x^{0}_{t}\in\mathbb{R}^{\ell_{x}}$,
$u^{0}_{t}\in\mathbb{R}^{\ell_{u}}$, $d^{0}_{t}\in\mathbb{R}^{\ell_{x}}$ and
$w^{0}_{t}\in\mathbb{R}^{\ell_{x}}$ denote, respectively, the state, action,
disturbance and noise of the leader at time $t\in\mathbb{N}$, where
$\ell_{x},\ell_{u}\in\mathbb{N}$. Analogously, denote by
$x^{i}_{t}\in\mathbb{R}^{\ell_{x}}$, $u^{i}_{t}\in\mathbb{R}^{\ell_{u}}$,
$d^{i}_{t}\in\mathbb{R}^{\ell_{x}}$ and $w^{i}_{t}\in\mathbb{R}^{\ell_{x}}$,
the state, action, disturbance and noise of follower $i\in\mathbb{N}_{n}$ at
time $t\in\mathbb{N}$. In addition, define the aggregate state and aggregate
action of followers as:

$\bar{x}_{t}:=\frac{1}{n}\sum_{i=1}^{n}x^{i}_{t},\quad\bar{u}_{t}:=\frac{1}{n}\sum_{i=1}^{n}u^{i}_{t}.$
(1)

The dynamics of the leader at time $t\in\mathbb{N}$ is influenced by the
aggregate state $\bar{x}_{t}$, the disturbance signal $d_{t}^{0}$ and noise
$w_{t}^{0}$ as follows:

$x^{0}_{t+1}=A_{t}^{0}x^{0}_{t}+B_{t}^{0}u^{0}_{t}+S_{t}^{0}\bar{x}_{t}+d^{0}_{t}+w_{t}^{0},$
(2)

where $A^{0}_{t}$, $B^{0}_{t}$ and $S^{0}_{t}$ are matrices of appropriate
dimensions. Furthermore, the dynamics of follower $i$ at time $t$ is affected
by the state of the leader $x^{0}_{t}$, aggregate state $\bar{x}_{t}$, local
disturbance $d^{i}_{t}$ and local noise $w_{t}^{i}$ as described below:

$x^{i}_{t+1}=A_{t}x^{i}_{t}+B_{t}u^{i}_{t}+S_{t}\bar{x}_{t}+E_{t}x_{t}^{0}+d_{t}^{i}+w_{t}^{i},\hskip
8.5359pti\in\mathbb{N}_{n},\hskip 2.84544ptt\in\mathbb{N},$ (3)

where $A_{t}$, $B_{t}$, $S_{t}$ and $E_{t}$ are matrices of appropriate
dimensions. Let $T\in\mathbb{N}$ denote the control horizon, and assume that
the primitive random variables
$\\{x^{0}_{1},\\{x^{i}_{1}\\}_{i\in\mathbb{N}_{n}},w^{0}_{1},\\{w^{i}_{1}\\}_{i\in\mathbb{N}_{n}},\ldots,w^{0}_{T},\\{w^{i}_{T}\\}_{i\in\mathbb{N}_{n}}\\}$
are mutually independent. In addition, it is assumed that the local noises of
followers and the noise of the leader have zero mean and finite covariance
matrices.

To be consistent with the terminology of the mean-field teams literature [7],
the aggregate state of followers is called _mean-field_ in this work. It is to
be noted that the term mean-field has a slightly different meaning in mean-
field games, where it refers to the aggregate state of an infinite population
(as opposed to a finite population) of followers.

In this paper, we consider two non-classical information structures: mean-
field sharing (MFS) and intermittent mean-field sharing (IMFS). In MFS
information structure, the leader has access to its local state as well as the
mean-field at any time $t$, i.e.,

$u^{0}_{t}=g^{0}_{t}(x^{0}_{t},\bar{x}_{t}),$ (4)

where $g^{0}_{t}:\mathbb{R}^{2\ell_{x}}\rightarrow\mathbb{R}^{\ell_{u}}$.
Furthermore, each follower $i\in\mathbb{N}_{n}$ has access to its local state
as well as the state of the leader and the mean-field at time $t$, i.e.,

$u^{i}_{t}=g^{i}_{t}(x^{i}_{t},\bar{x}_{t},x^{0}_{t}),$ (5)

where $g^{i}_{t}:\mathbb{R}^{3\ell_{x}}\rightarrow\mathbb{R}^{\ell_{u}}$. In
IMFS information structure, the mean-field is observed intermittently, i.e.,

$u^{0}_{t}=g^{0}_{t}(x^{0}_{t},z_{t}),\quad
u^{i}_{t}=g^{i}_{t}(x^{i}_{t},z_{t},x^{0}_{t}),\quad\forall
i\in\mathbb{N}_{n},$ (6)

where $z_{t}:=\bar{x}_{t}$ during the time when the mean-field is observed and
$z_{t}:=\mathtt{blank}$ during the time when the mean-field is not observed.
In practice, IMFS information structure is useful when the number of followers
is neither that small (so that the mean-field can be shared at each time
instant) nor is very large (such that the strong law of large numbers can be
applied to the mean-field). In such a case, it is feasible to obtain the mean-
field intermittently such that at some time instants the information structure
is observed while at some others it is not. When the number of followers is
large, IMFS may be reduced to no-mean-field sharing, where the mean-field is
not observed at all, i.e.,

$u^{0}_{t}=g^{0}_{t}(x^{0}_{t}),\quad
u^{i}_{t}=g^{i}_{t}(x^{i}_{t},x^{0}_{t}),\quad\forall i\in\mathbb{N}_{n}.$ (7)

The set of all control laws
$\mathbf{g}:=\\{g^{0}_{1:T},g^{1}_{1:T},\ldots,g^{n}_{1:T}\\}$ is called the
strategy of the network.

###  II-A Problem statement

Let the set $\mathbf{d}$ be defined as
$\\{d^{0}_{1:T},\\{d^{i}_{1:T}\\}_{i\in\mathbb{N}_{n}}\\}$, and $\gamma>0$ be
a given attenuation parameter. Then the cost function of the system is defined
as follows:

$\displaystyle
J_{n}^{\gamma}(\mathbf{g},\mathbf{d})=\mathbb{E}(\sum_{t=1}^{T}\Big{[}\frac{1}{n}\sum_{i=1}^{n}\big{[}(x^{i}_{t})^{\intercal}Q_{t}x^{i}_{t}+(u^{i}_{t})^{\intercal}R_{t}u^{i}_{t}-\gamma^{2}(d_{t}^{i})^{\intercal}d^{i}_{t}\big{]}$
$\displaystyle\quad+(x^{0}_{t})^{\intercal}Q^{0}_{t}x^{0}_{t}+(u^{0}_{t})^{\intercal}R^{0}_{t}u^{0}_{t}-\gamma^{2}(d^{0}_{t})^{\intercal}d^{0}_{t}$
$\displaystyle\quad+(\bar{x}_{t}-x^{0}_{t})^{\intercal}F_{t}(\bar{x}_{t}-x^{0}_{t})+\bar{x}_{t}^{\intercal}P_{t}\bar{x}_{t}+\bar{u}_{t}^{\intercal}H_{t}\bar{u}_{t}\big{]}\Big{]}),$
(8)

where $Q_{t},Q^{0}_{t},R_{t},R^{0}_{t},F_{t},P_{t}$ and $H_{t}$ are symmetric
matrices of appropriate dimensions. Note that the value of $\gamma$ determines
the relative importance of reaching consensus versus rejecting disturbance.

###### Problem 1

Find the saddle-point strategy $\mathbf{g}$ under mean-field sharing
information structure such that

$J^{\gamma,\ast}_{n}=\inf_{\mathbf{g}}\sup_{\mathbf{d}}J_{n}^{\gamma}(\mathbf{g},\mathbf{d}).$
(9)

###### Problem 2

Find an approximate saddle-point strategy $\mathbf{g}_{\varepsilon}$ under
intermittent mean-field sharing information structure such that
$|\sup_{\mathbf{d}}J_{n}^{\gamma}(\mathbf{g}_{\varepsilon},\mathbf{d})-J^{\gamma,\ast}_{n}|\leq\varepsilon(n)$,
where $\varepsilon(n)\geq 0$ and $\lim_{n\rightarrow\infty}\varepsilon(n)=0$.

###### Remark 1

Problems 1 and 2 can be extended to the case where each follower has an
individual weight under the condition presented in [7, Assumption 3.6]. □

The main contributions of the paper are outlined below.

  1. 1.

We present a unique saddle-point strategy in an explicit manner for a leader-
follower network under mean-field sharing information structure, where the
number of followers is not necessarily large, and hence the action of a single
follower can not be neglected (note that the analysis of non-negligible
players is more complex than the case with negligible players, in general).

  2. 2.

We propose an approximate saddle-point strategy under intermittent mean-field
sharing information structure, whose performance is different from that of
mean-field sharing for any finite number of followers.

##  III Main Results

Define the following matrices at any time $t\in\mathbb{N}_{T}$:

$\bar{A}_{t}:=\left[\begin{array}[]{cc}A^{0}_{t}&S^{0}_{t}\\\
E_{t}&A_{t}+S_{t}\end{array}\right],\quad\bar{B}_{t}:=\left[\begin{array}[]{cc}B^{0}_{t}&\mathbf{0}_{\ell_{x}\times\ell_{u}}\\\
\mathbf{0}_{\ell_{x}\times\ell_{u}}&B_{t}\end{array}\right],$ (10)
$\bar{Q}_{t}:=\left[\begin{array}[]{cc}Q^{0}_{t}+F_{t}&-F_{t}\\\
-F_{t}&Q_{t}+P_{t}+F_{t}\end{array}\right],\bar{R}_{t}:=\left[\begin{array}[]{cc}R^{0}_{t}&\mathbf{0}_{\ell_{u}\times\ell_{u}}\\\
\mathbf{0}_{\ell_{u}\times\ell_{u}}&H_{t}+R_{t}\end{array}\right].$ (11)

###### Assumption 1

At any time $t\in\mathbb{N}_{T}$, matrices $Q_{t}$ and $\bar{Q}_{t}$ are
positive semi-definite, and matrices $R_{t}$ and $\bar{R}_{t}$ are positive
definite.

It will be shown later that Problem 1 under Assumption 1 can be cast as a
strictly convex optimization problem with respect to the control actions of
the leader and followers, and a strictly concave optimization problem with
respect to the disturbances. Using the transformation technique introduced in
[7], define the following variables:
$\breve{x}^{i}_{t}:=x^{i}_{t}-\bar{x}_{t}$,
$\breve{u}^{i}_{t}:=u^{i}_{t}-\bar{u}_{t}$,
$\breve{d}^{i}_{t}:=d^{i}_{t}-\bar{d}_{t}$ and
$\breve{w}^{i}_{t}:=w^{i}_{t}-\bar{w}_{t}$, where
$\bar{d}_{t}:=\frac{1}{n}\sum_{i=1}^{n}d^{i}_{t}$ and
$\bar{w}_{t}:=\frac{1}{n}\sum_{i=1}^{n}w^{i}_{t}$. It follows from (3) that:

$\displaystyle\bar{x}_{t+1}$
$\displaystyle=(A_{t}+S_{t})\bar{x}_{t}+B_{t}\bar{u}_{t}+E_{t}x_{t}^{0}+\bar{d}_{t}+\bar{w}_{t},$
$\displaystyle\breve{x}^{i}_{t+1}$
$\displaystyle=A_{t}\breve{x}^{i}_{t}+B_{t}\breve{u}^{i}_{t}+\breve{d}_{t}^{i}+\breve{w}_{t}^{i}.$
(12)

Note that
$\frac{1}{n}\sum_{i=1}^{n}\breve{x}^{i}_{t}=\mathbf{0}_{\ell_{x}\times 1}$,
$\frac{1}{n}\sum_{i=1}^{n}\breve{u}^{i}_{t}=\mathbf{0}_{\ell_{u}\times 1}$,
$\frac{1}{n}\sum_{i=1}^{n}\breve{d}^{i}_{t}=\mathbf{0}_{\ell_{x}\times 1}$ and
$\frac{1}{n}\sum_{i=1}^{n}\breve{w}^{i}_{t}=\mathbf{0}_{\ell_{x}\times 1}$.

Define the following variables backward in time:

$\begin{cases}\breve{M}_{t}:=Q_{t}+A_{t}\breve{M}_{t+1}\breve{\Delta}_{t}^{-1}A_{t}^{\intercal},\\\
\bar{M}_{t}:=\bar{Q}_{t}+\bar{A}_{t}\bar{M}_{t+1}\bar{\Delta}_{t}^{-1}\bar{A}_{t}^{\intercal},\\\
\breve{\Delta}_{t}:=\mathbf{I}_{\ell_{x}\times\ell_{x}}+B_{t}R_{t}^{-1}B_{t}^{\intercal}\breve{M}_{t+1}-\gamma^{-2}\breve{M}_{t+1},\\\
\bar{\Delta}_{t}:=\mathbf{I}_{2\ell_{x}\times
2\ell_{x}}+\bar{B}_{t}\bar{R}_{t}^{-1}\bar{B}_{t}^{\intercal}\bar{M}_{t+1}-\gamma^{-2}\bar{M}_{t+1},\\\
\breve{c}^{i}_{t}:=\breve{c}^{i}_{t+1}+\operatorname{Tr}(\breve{M}_{t+1}\operatorname{Cov}{\breve{w}^{i}_{t}}),\\\
\bar{c}_{t}:=\bar{c}_{t+1}+\operatorname{Tr}(\bar{M}_{t+1}\operatorname{Cov}([w^{0}_{t},\bar{w}_{t}])),\end{cases}$
(13)

where $\breve{M}_{T+1}:=\mathbf{0}_{\ell_{x}\times\ell_{x}}$,
$\bar{M}_{T+1}:=\mathbf{0}_{2\ell_{x}\times 2\ell_{x}}$, $\bar{c}_{T+1}:=0$
and $\breve{c}^{i}_{T+1}:=0$, $i\in\mathbb{N}_{n}$.

###### Theorem 1

Let Assumption 1 hold. Then:

  1. 1.

Given any $\gamma>0$, Problem 1 admits a unique feedback saddle-point strategy
if matrices $\gamma^{2}\mathbf{I}_{\ell_{x}\times\ell_{x}}-\breve{M}_{t+1}$
and $\gamma^{2}\mathbf{I}_{2\ell_{x}\times 2\ell_{x}}-\bar{M}_{t+1}$ are
positive definite for every $t\in\mathbb{N}_{T-1}$, where $\breve{M}_{t+1}$
and $\bar{M}_{t+1}$ are given by (13).

  2. 2.

The saddle-point strategy of the leader is described by:

$u^{0}_{t}=\bar{L}^{1,1}_{t}x^{0}_{t}+\bar{L}^{1,2}_{t}\bar{x}_{t},$ (14)

and for every follower $i\in\mathbb{N}_{n}$:

$u^{i}_{t}=\breve{L}_{t}x^{i}_{t}+\bar{L}_{t}^{2,1}x^{0}_{t}+(\bar{L}_{t}^{2,2}-\breve{L}_{t})\bar{x}_{t},$
(15)

where $\left[\begin{array}[]{c c}\bar{L}^{1,1}_{t}&\bar{L}^{1,2}_{t}\\\
\bar{L}^{2,1}_{t}&\bar{L}^{2,2}_{t}\end{array}\right]:=-\bar{B}_{t}\bar{M}_{t+1}\bar{\Delta}_{t}^{-1}\bar{A}_{t}$
and $\breve{L}_{t}:=-B_{t}\breve{M}_{t+1}\breve{\Delta}_{t}^{-1}A_{t}$.

  3. 3.

The performance under the saddle-point strategy is given by:
$J^{\gamma,\ast}_{n}=\frac{1}{n}\sum_{i=1}^{n}[\operatorname{Tr}(\breve{M}_{1}\operatorname{Cov}(\breve{x}^{i}_{1}))+\breve{c}^{i}_{1}]+\operatorname{Tr}(\bar{M}_{1}\operatorname{Cov}([x^{0}_{1},\bar{x}_{1}]))+\bar{c}_{1}.$

□

###### Proof

The proof is presented in Appendix A. ■

We now impose the following two assumptions on the model.

###### Assumption 2

The initial states and local noises of the followers are i.i.d. random
variables and independent of those of the leader.

###### Assumption 3

All matrices in the dynamic equations (2) and (3) and cost function (II-A) as
well as the covariance matrices are independent of the number of followers.

Let $\hat{m}_{1}$ be the expected value of the initial states of the
followers, and $\hat{m}_{t}$ denote an estimate of the mean-field
$\bar{x}_{t}$ at time $t$ such that if $z_{t+1}=\mathtt{blank}$ at time $t+1$:

$\hat{m}_{t+1}:=(A_{t}+S_{t}+B_{t}\bar{L}^{2,2}_{t})\hat{m}_{t}+(B_{t}\bar{L}^{2,1}_{t}+E_{t})x^{0}_{t}+\bar{d}_{t},$
(16)

and if $z_{t+1}=\bar{x}_{t+1}$, $\hat{m}_{t+1}:=\bar{x}_{t+1}$. Under
Assumptions 2 and 3, it can be shown that $\hat{m}_{t+1}$ almost surely
converges to $\bar{x}_{t+1}$ at every time instant, as $n\rightarrow\infty$,
due the strong law of large numbers, on noting that the dynamics of the mean-
field under the saddle-point strategy is described by:

$\bar{x}_{t+1}=(A_{t}+S_{t}+B_{t}\bar{L}^{2,2}_{t})\bar{x}_{t}+(B_{t}\bar{L}^{2,1}_{t}+E_{t})x^{0}_{t}+\bar{d}_{t}+\bar{w}_{t}.$
(17)

We now replace the mean-field $\bar{x}_{t}$ in the saddle-point strategy of
Theorem 1 with the estimate $\hat{m}_{t}$ (that is measurable under IMFS
information structure)111Note that the worse-case value of $\bar{d}_{t}$ is
given by (119) in the Appendix. to construct the following approximate saddle-
point strategy:

$\begin{cases}v^{0}_{t}=\bar{L}^{1,1}_{t}x^{0}_{t}+\bar{L}^{1,2}_{t}\hat{m}_{t},\\\
v^{i}_{t}=\breve{L}_{t}x^{i}_{t}+\bar{L}_{t}^{2,1}x^{0}_{t}+(\bar{L}_{t}^{2,2}-\breve{L}_{t})\hat{m}_{t},\quad
i\in\mathbb{N}_{n}.\end{cases}$ (18)

Since the dynamic equations (2) and (3), cost function (II-A) and the saddle-
point strategies (14), (15) and (18) are bounded and continuous in
$\bar{x}_{t}$, it results that (18) is an approximate saddle-point strategy.
The reader is referred to [10] for a detailed proof in the context of optimal
control, which is similar, to a great extent, to the convergence proof of
MinMax control problem considered in this subsection, but note that the
Riccati equations here are different and the relative errors defined in [10]
are of intermittent nature. However, these differences do not add much
complexity to the convergence proof because the Riccati equations (13) do not
depend on the number of followers according to Assumption 3. Hence, the rate
of convergence with respect to the number of followers is $1/n$, similar to
[10, Theorem 2]. This leads to the following theorem.

###### Theorem 2

Let Assumptions 1–3 hold. The strategy (18) is an approximate saddle-point
strategy for Problem 2. □

##  IV Numerical Examples

In this section, two numerical examples are provided to illustrate the
efficacy of the obtained results.

Example 1. Consider a multi-agent network consisting of one leader and $100$
identical followers whose dynamics are described by equations (2) and (3),
respectively, with the following numerical parameters: $A^{0}_{t}=0.85$,
$B^{0}_{t}=0.15$, $A_{t}=0.85$, $B_{t}=0.85$, $S^{0}_{t}=0.03$, $S_{t}=0.1$,
$E_{t}=0.01$, $w^{i}_{t}\sim\mathcal{N}(0,0.3)$, $\forall i\in\mathbb{N}_{n}$,
$T=20.$

Let the initial state of the leader be $x_{1}^{0}=30$ and the initial states
of the followers be chosen randomly (with uniform distribution) in the
interval $[0,20]$. The followers are exposed to an external disturbance given
by: $d_{t}^{i}=0.6sin(t)$, $t\in\mathbb{N}_{20}$, $i\in\mathbb{N}_{100}$. The
objective of the leader and followers is to minimize the cost function (II-A)
under the worst-case disturbance, where at any time $t\in\mathbb{N}_{20}$:
$R_{t}=70$, $Q_{t}=8$, $F_{t}=11$, $P_{t}=0.4$, $R_{t}^{0}=50$,
$Q^{0}_{t}=0.5$, $H_{t}=0.1$.

Sample trajectories of the leader and followers are depicted in Figure 1. It
is shown that as the attenuation parameter $\gamma$ increases, the
fluctuations of the mean-field decrease which means better disturbance
rejection.

Figure 1:  Sample trajectories of the leader and followers in Example 1, where
thin colored curves are the states of the followers, thick blue curve is the
mean-field, and thick black curve is the state of the leader.

Example 2. Consider $100$ followers with identical dynamics that wish to track
a reference signal, which may be viewed as a virtual leader with constant
state $x^{0}_{t}=10$, $\forall t\in\mathbb{N}_{30}$. The initial states of the
followers are randomly chosen in the interval $[0,8]$ with a uniform
distribution. The dynamics of the leader and followers are expressed by the
following parameters: $A^{0}_{t}=A_{t}=1$, $B^{0}_{t}=0$,
$w^{i}_{t}\sim\mathcal{N}(0,0.3)$, $\forall i\in\mathbb{N}_{100}$,
$S^{0}_{t}=0$, $S_{t}=0.04$, $B_{t}=1$, $T=30$, $E_{t}=0.001.$ The followers
are subject to local external disturbances given below: $d_{t}^{i}=0.4sin(t)$,
$i\in\mathbb{N}_{100}$, $t\in\mathbb{N}_{30}$. The weight matrices in the cost
function are given by: $R_{t}=0.11$, $Q_{t}=0.01$, $F_{t}=0.07$,
$P_{t}=0.001$, $R_{t}^{0}=10^{-4}$, $Q^{0}_{t}=10^{-4}$, $H_{t}=1$.

In Figure 2, three sample trajectories of the states of followers are
displayed for three different values of the attenuation parameter. It is
observed that as the attenuation parameter increases, the disturbance is
rejected more strongly at the cost of prolonging the consensus process.

Figure 2: The trajectories of followers for different values of the
attenuation parameter $\gamma$ in Example 2.

##  V Conclusions

In this paper, a robust control strategy was proposed for a class of leader-
follower networks. Two decentralized information structures were studied. For
mean-field sharing structure, it was shown that a unique saddle-point strategy
exists under some mild assumptions. For intermittent mean-field sharing, the
proposed strategy was shown to converge to the saddle-point strategy as the
number of followers tends to infinity. The main feature of the obtained
results is the fact that the solutions are identified by two Riccati equations
that are not in the form of forward-backward equations, and their dimensions
do not increase with the number of followers. In addition, it was numerically
verified that the disturbance rejection property of the solution outweighs the
consensus-reaching behaviour when the attenuation parameter is large.

## References

  * [1] WA Van Den Broek, JC Engwerda and Johannes M Schumacher  “Robust equilibria in indefinite linear-quadratic differential games”  In _Journal of Optimization Theory and Applications_ 119.3 Springer, 2003, pp. 565–595
  * [2] P.. Caines, M. Huang and R.. Malhamé  “Mean field games”  In _Handbook of Dynamic Game Theory_ Springer International Publishing, 2018, pp. 345–372
  * [3] Olivier Guéant, Jean-Michel Lasry and Pierre-Louis Lions  “Mean field games and applications”  In __in_ Paris-Princeton Lectures on Mathematical Finance 2010. Lecture Notes in Mathematics_ 2003 Springer,Berlin, Heidelber, 2011, pp. 205–266
  * [4] Alain Bensoussan, Jens Frehse and Phillip Yam  “Mean field games and mean field type control theory”  Springer-Verlag New York, 2013
  * [5] René Carmona, François Delarue and Aimé Lachapelle  “Control of McKean–Vlasov dynamics versus mean field games”  In _Mathematics and Financial Economics_ 7.2 Springer, 2013, pp. 131–166
  * [6] Boualem Djehiche, Alain Tcheukam and Hamidou Tembine  “Mean-Field-Type Games in Engineering”  In _AIMS Electronics and Electrical Engineering_ 1, 2017, pp. 18–73
  * [7] Jalal Arabneydi  “New Concepts in Team Theory: Mean Field Teams and Reinforcement Learning”, 2016
  * [8] J. Arabneydi and A. Mahajan  “Team-Optimal Solution of Finite Number of Mean-Field Coupled LQG Subsystems”  In _Proceedings of the 54 IEEE Conference on Decision and Control_ , 2015, pp. 5308–5313
  * [9] Jalal Arabneydi, Mohammad Baharloo and Amir G. Aghdam  “Optimal Distributed Control for Leader-Follower Networks: A Scalable Design”  In _Proceedings of the 31 Canadian Conference on Electrical and Computer Engineering_ , 2018, pp. 1–4
  * [10] Mohammad Baharloo, Jalal Arabneydi and Amir G. Aghdam  “Near-Optimal Control Strategy in Leader-Follower Networks: A Case Study for Linear Quadratic Mean-Field Teams”  In _Proceedings of the 57 IEEE Conference on Decision and Control_ , 2018, pp. 3288–3293
  * [11] Robert Elliott, Xun Li and Yuan-Hua Ni  “Discrete time mean-field stochastic linear-quadratic optimal control problems”  In _Automatica_ 49.11 Elsevier, 2013, pp. 3222–3233
  * [12] Tyrone Duncan and Hamidou Tembine  “Linear–Quadratic Mean-Field-Type Games: A Direct Method”  In _Games_ 9.1 Multidisciplinary Digital Publishing Institute, 2018, pp. 7
  * [13] Dario Bauso, Hamidou Tembine and Tamer Başar  “Opinion dynamics in social networks through mean-field games”  In _SIAM Journal on Control and Optimization_ 54.6 SIAM, 2016, pp. 3225–3257
  * [14] Jianhui Huang and Minyi Huang  “Robust Mean Field Linear-Quadratic-Gaussian Games with Unknown L2-Disturbance”  In _SIAM Journal on Control and Optimization_ 55.5 SIAM, 2017, pp. 2811–2840
  * [15] Bing-Chang Wang and Jianhui Huang  “Social optima in robust mean field LQG control”  In _Proceedings of the 11 IEEE Asian Control Conference_ , 2017, pp. 2089–2094
  * [16] Mohammad Amin Tajeddini, Hamed Kebriaei and Luigi Glielmo  “Robust decentralised mean field control in leader following multi-agent systems”  In _IET Control Theory & Applications_ 11.16 IET, 2017, pp. 2707–2715
  * [17] Tamer Başar and Pierre Bernhard  “H-infinity optimal control and related minimax design problems: A dynamic game approach”  Birkhaüser Basel, 2008

##  Appendix A Proof of Theorem 1

Rewrite the cost function defined in (II-A) in terms of the new variables as:

$J_{n}^{\gamma}(\mathbf{g},\mathbf{d})=\mathbb{E}(\sum_{t=1}^{T}\Big{[}\frac{1}{n}\sum_{i=1}^{n}(\breve{x}^{i}_{t})^{\intercal}Q_{t}\breve{x}^{i}_{t}+(\breve{u}^{i}_{t})^{\intercal}R_{t}\breve{u}^{i}_{t}-\gamma^{2}(\breve{d}^{i}_{t})^{\intercal}\breve{d}^{i}_{t}\Big{]}\\\
+\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]^{\intercal}\bar{Q}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]^{\intercal}\bar{R}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]-\gamma^{2}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]).$ (19)

At any time $t$, define the following augmented vectors:
$\mathbf{x}_{t}:=[{\breve{x}^{1}_{t}}{}^{\intercal},\ldots,{\breve{x}^{n}_{t}}{}^{\intercal},{x^{0}_{t}}^{\intercal},{\bar{x}_{t}}^{\intercal}]^{\intercal}$,
$\mathbf{u}_{t}:=[{\breve{u}^{1}_{t}}{}^{\intercal},\ldots,{\breve{u}^{n}_{t}}{}^{\intercal},{u^{0}_{t}}^{\intercal},{\bar{u}_{t}}^{\intercal}]^{\intercal}$,
$\mathbf{d}_{t}:=[{\breve{d}^{1}_{t}}{}^{\intercal},\ldots,{\breve{d}^{n}_{t}}{}^{\intercal},{d^{0}_{t}}^{\intercal},{\bar{d}_{t}}^{\intercal}]^{\intercal}$.
Suppose for now that $\mathbf{x}_{t}$ is known, and solve the corresponding
Isaacs’ equation according to [17, Theorem 3.2], i.e., the cost-to-go function
at the terminal time $T$ is:
$V_{T}(\mathbf{x}_{T})=\frac{1}{n}\sum_{i=1}^{n}(\breve{x}^{i}_{T})^{\intercal}\breve{M}_{T}\breve{x}^{i}_{T}+\left[\begin{array}[]{c}x^{0}_{T}\\\
\bar{x}_{T}\end{array}\right]^{\intercal}\bar{M}_{T}\left[\begin{array}[]{c}x^{0}_{T}\\\
\bar{x}_{T}\end{array}\right]+\frac{1}{n}\sum_{i=1}^{n}\breve{c}^{i}_{T}+\bar{c}_{T}$.
Suppose that the cost-to-go function takes the following form at time $t+1$:

$V_{t+1}(\mathbf{x}_{t+1})=\frac{1}{n}\sum_{i=1}^{n}(\breve{x}^{i}_{t+1})^{\intercal}\breve{M}_{t+1}\breve{x}^{i}_{t+1}\\\
+\left[\begin{array}[]{c}x^{0}_{t+1}\\\
\bar{x}_{t+1}\end{array}\right]^{\intercal}\bar{M}_{t+1}\left[\begin{array}[]{c}x^{0}_{t+1}\\\
\bar{x}_{t+1}\end{array}\right]+\frac{1}{n}\sum_{i=1}^{n}\breve{c}^{i}_{t+1}+\bar{c}_{t+1}.$
(20)

It is now desired to show that (20) holds at time $t$ as well. It follows from
Isaacs’ equation that:

$\displaystyle
V_{t}(\mathbf{x}_{t})=\sup_{\mathbf{d}_{t}}\inf_{\mathbf{u}_{t}}(\frac{1}{n}\sum_{i=1}^{n}\big{[}(\breve{x}_{t}^{i})^{\intercal}Q_{t}\breve{x}_{t}^{i}+(\breve{u}_{t}^{i})^{\intercal}R_{t}\breve{u}_{t}^{i}-\gamma^{2}(\breve{d}_{t}^{i})^{\intercal}\breve{d}_{t}^{i}]$
$\displaystyle+\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]^{\intercal}\bar{Q}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]^{\intercal}\bar{R}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]-\gamma^{2}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]$ (33)
$\displaystyle+\mathbb{E}\big{[}V_{t+1}(\mathbf{x}_{t+1})\mid\mathbf{x}_{t},\mathbf{u}_{t},\mathbf{d}_{t}\big{]}).$
(34)

From (III), (20) and (A), one arrives at:

$\displaystyle
V_{t}(\mathbf{x}_{t})=\sup_{\mathbf{d}_{t}}\inf_{\mathbf{u}_{t}}(\frac{1}{n}\sum_{i=1}^{n}\big{[}(\breve{x}_{t}^{i})^{\intercal}Q_{t}\breve{x}_{t}^{i}+(\breve{u}_{t}^{i})^{\intercal}R_{t}\breve{u}_{t}^{i}-\gamma^{2}(\breve{d}_{t}^{i})^{\intercal}\breve{d}_{t}^{i}]$
$\displaystyle+\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]^{\intercal}\bar{Q}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]^{\intercal}\bar{R}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]-\gamma^{2}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]$ (47)
$\displaystyle+\mathbb{E}[\frac{1}{n}\sum_{i=1}^{n}\big{[}(A_{t}\breve{x}_{t}^{i}+B_{t}\breve{u}_{t}^{i}+\breve{d}_{t}^{i}+\breve{w}_{t}^{i})^{\intercal}\breve{M}_{t+1}$
$\displaystyle\quad\times(A_{t}\breve{x}_{t}^{i}+B_{t}\breve{u}_{t}^{i}+\breve{d}_{t}^{i}+\breve{w}_{t}^{i})\big{]}+\frac{1}{n}\sum_{i=1}^{n}\breve{c}^{i}_{t+1}+\bar{c}_{t+1}$
$\displaystyle+(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]+\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]+\left[\begin{array}[]{c}w^{0}_{t}\\\
\bar{w}_{t}\end{array}\right])^{\intercal}\bar{M}_{t+1}$ (56)
$\displaystyle\quad\times(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]+\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]+\left[\begin{array}[]{c}w^{0}_{t}\\\
\bar{w}_{t}\end{array}\right])]).$ (65)

This yields:

$\displaystyle
V_{t}(\mathbf{x}_{t})=\sup_{\mathbf{d}_{t}}\inf_{\mathbf{u}_{t}}(\frac{1}{n}\sum_{i=1}^{n}\big{[}(\breve{x}_{t}^{i})^{\intercal}Q_{t}\breve{x}_{t}^{i}+(\breve{u}_{t}^{i})^{\intercal}R_{t}\breve{u}_{t}^{i}-\gamma^{2}(\breve{d}_{t}^{i})^{\intercal}\breve{d}_{t}^{i}]$
$\displaystyle+\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]^{\intercal}\bar{Q}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]^{\intercal}\bar{R}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right]-\gamma^{2}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]$ (78)
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}[(A_{t}\breve{x}_{t}^{i}+B_{t}\breve{u}_{t}^{i})^{\intercal}\breve{M}_{t+1}(A_{t}\breve{x}_{t}^{i}+B_{t}\breve{u}_{t}^{i})+(\breve{d}^{i}_{t})^{\intercal}\breve{M}_{t+1}\breve{d}^{i}_{t}$
$\displaystyle+2(\breve{d}^{i}_{t})^{\intercal}\breve{M}_{t+1}(A_{t}\breve{x}^{i}_{t}+B_{t}\breve{u}^{i}_{t})+2\breve{w}^{i}_{t}\breve{M}_{t+1}(A_{t}\breve{x}^{i}_{t}+B_{t}\breve{u}^{i}_{t}+\breve{d}^{i}_{t})$
$\displaystyle+2(\breve{d}^{i}_{t})^{\intercal}\breve{M}_{t+1}\breve{w}^{i}_{t}+\operatorname{Tr}(\breve{M}_{t+1}\operatorname{Cov}(\breve{w}^{i}_{t}))]+\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\bar{M}_{t+1}\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]$ (83)
$\displaystyle+(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right])^{\intercal}\bar{M}_{t+1}(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right])$ (92)
$\displaystyle+2\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\bar{M}_{t+1}(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right])+\bar{c}_{t+1}$ (99)
$\displaystyle+2\left[\begin{array}[]{c}\breve{w}^{0}_{t}\\\
\bar{w}_{t}\end{array}\right]^{\intercal}\bar{M}_{t+1}(\bar{A}_{t}\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]+\bar{B}_{t}\left[\begin{array}[]{c}u^{0}_{t}\\\
\bar{u}_{t}\end{array}\right])+\frac{1}{n}\sum_{i=1}^{n}\breve{c}^{i}_{t+1}$
(106) $\displaystyle+2\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\bar{M}_{t+1}\left[\begin{array}[]{c}\breve{w}^{0}_{t}\\\
\bar{w}_{t}\end{array}\right]+\operatorname{Tr}(\bar{M}_{t+1}\operatorname{Cov}([w^{0}_{t},\bar{w}_{t}]))).$
(111)

Given any disturbance vector $\mathbf{d}_{t}$, we now compute the gradient
vector with respect to $\mathbf{u}_{t}$ and set each component to zero in
order to obtain the optimal actions of the leader and followers, i.e., one
has:

$2(\breve{u}_{t}^{i})^{\intercal}R_{t}+2(\breve{u}_{t}^{i})^{\intercal}B_{t}^{\intercal}\breve{M}_{t+1}B_{t}+2(\breve{x}_{t}^{i})^{\intercal}A_{t}^{\intercal}\breve{M}_{t+1}B_{t}\\\
+2(\breve{d}_{t}^{i})^{\intercal}\breve{M}_{t+1}B_{t}=\mathbf{0}_{1\times\ell_{u}},\quad\forall
i\in\mathbb{N}_{n},$ (112)
$2[{u^{0}_{t}}^{\intercal},{\bar{u}_{t}}^{\intercal}]\bar{R}_{t}+2[{u^{0}_{t}}^{\intercal},{\bar{u}_{t}}^{\intercal}]\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t}\\\
+2\left[\begin{array}[]{c}x^{0}_{t}\\\
\bar{x}_{t}\end{array}\right]^{\intercal}\bar{A}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t}+2\left[\begin{array}[]{c}d^{0}_{t}\\\
\bar{d}_{t}\end{array}\right]^{\intercal}\bar{M}_{t+1}\bar{B}_{t}=\mathbf{0}_{1\times
2\ell_{u}},$ (113)

which leads to:

$\breve{u}_{t}^{i*}=-(R_{t}+B_{t}^{\intercal}\breve{M}_{t+1}B_{t})^{-1}(B_{t}^{\intercal}\breve{M}_{t+1}A_{t})\breve{x}_{t}^{i}\\\
-(R_{t}+B_{t}^{\intercal}\breve{M}_{t+1}B_{t})^{-1}(B_{t}^{\intercal}\breve{M}_{t+1})\breve{d}_{t}^{i},$
(114)

and

$\left[\begin{array}[]{c}u_{t}^{0,\ast}\\\
\bar{u}^{\ast}_{t}\end{array}\right]=-(\bar{R}_{t}+\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t})^{-1}(\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{A}_{t})\left[\begin{array}[]{c}x_{t}^{0}\\\
\bar{x}_{t}\end{array}\right]\\\
-(\bar{R}_{t}+\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t})^{-1}(\bar{B}_{t}^{\intercal}\bar{M}_{t+1})\left[\begin{array}[]{c}d_{t}^{0}\\\
\bar{d}_{t}\end{array}\right].$ (115)

It is observed that the Hessian matrix is diagonal, with matrices
$R_{t}+B_{t}^{\intercal}\breve{M}_{t+1}B_{t}$ and
$\bar{R}_{t}+\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t}$ as its diagonal
terms that are positive definite according to Assumption 1. Hence, the cost
function is strictly convex in the newly defined control actions.

By incorporating the optimal strategies (114) and (115) into (A), calculating
the gradient with respect to $\mathbf{d}_{t}$ and setting each component to
zero, one arrives at the following $n+1$ equations:

$-\breve{M}_{t+1}B_{t}(R_{t}+B_{t}^{\intercal}\breve{M}_{t+1}B_{t})^{-1}B_{t}^{\intercal}\breve{M}_{t+1}\breve{d}_{t}^{i}\\\
-\breve{M}_{t+1}B_{t}(R_{t}+B_{t}^{\intercal}\breve{M}_{t+1}B_{t})^{-1}B_{t}^{\intercal}\breve{M}_{t+1}A_{t}\breve{x}_{t}^{i}\\\
-\gamma^{2}\breve{d}_{t}^{i}+\breve{M}_{t+1}\breve{d}_{t}^{i}+\breve{M}_{t+1}A_{t}\breve{x}_{t}^{i}=\mathbf{0}_{\ell_{x}\times
1},\quad\forall i\in\mathbb{N}_{n},$ (116)

and

$-\bar{M}_{t+1}\bar{B}_{t}(\bar{R}_{t}+\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t})^{-1}\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\left[\begin{array}[]{c}d_{t}^{0}\\\
\bar{d}_{t}\end{array}\right]\\\
-\bar{M}_{t+1}\bar{B}_{t}(\bar{R}_{t}+\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{B}_{t})^{-1}\bar{B}_{t}^{\intercal}\bar{M}_{t+1}\bar{A}_{t}\left[\begin{array}[]{c}x_{t}^{0}\\\
\bar{x}_{t}\end{array}\right]\\\
-\gamma^{2}\left[\begin{array}[]{c}d_{t}^{0}\\\
\bar{d}_{t}\end{array}\right]+\bar{M}_{t+1}\left[\begin{array}[]{c}d_{t}^{0}\\\
\bar{d}_{t}\end{array}\right]+\bar{M}_{t+1}\bar{A}_{t}\left[\begin{array}[]{c}x_{t}^{0}\\\
\bar{x}_{t}\end{array}\right]=\mathbf{0}_{2\ell_{x}\times 1}.$ (117)

After some manipulations, the worst-case disturbances can be obtained as:

$\breve{d}_{t}^{i*}=\gamma^{-2}\breve{M}_{t+1}\breve{\Delta}_{t}^{-1}A_{t}\breve{x}_{t}^{i},$
(118)

and

$\left[\begin{array}[]{c}d_{t}^{0,\ast}\\\
\bar{d}^{\ast}_{t}\end{array}\right]=\gamma^{-2}\bar{M}_{t+1}\bar{\Delta}_{t}^{-1}\bar{A}_{t}\left[\begin{array}[]{c}x_{t}^{0}\\\
\bar{x}_{t}\end{array}\right],$ (119)

where $\breve{\Delta}_{t}$ and $\bar{\Delta}_{t}$ are given by (13). In
addition, we find the Hessian matrix which is diagonal with matrices
$-\gamma^{2}\mathbf{I}_{\ell_{x}\times\ell_{x}}+\breve{M}_{t+1}$ and
$-\gamma^{2}\mathbf{I}_{2\ell_{x}\times 2\ell_{x}}+\bar{M}_{t+1}$ as its
diagonal terms. Therefore, if these matrices are negative definite, it is
concluded that the cost function is strictly concave with respect to
disturbances. The recursion (13) is finally obtained by incorporating the
worst-case disturbances (118) and (119) into the optimal strategies (114) and
(115) and comparing the expressions (20) and (A) at times $t+1$ and $t$,
respectively, which completes the proof.


††institutetext: International Centre for Theoretical Sciences (ICTS-TIFR)
Tata Institute of Fundamental Research
Shivakote, Hesaraghatta, Bangalore 560089, India.

# Small corrections to semiclassical gravity and their role in information
paradoxes

Joydeep Chakravarty<EMAIL_ADDRESS>
###### Abstract

In this essay, we qualitatively demonstrate how small non-perturbative
corrections are a necessary addition to semiclassical gravity’s path integral.
We use this to discuss implications for Hawking’s information paradox and the
bags of gold paradox.

(Essay written for the Gravity Research Foundation 2021 Awards for Essays on
Gravitation.)

##  1 Introduction

This work’s primary goal is to shed light on the origin of small non-
perturbative corrections to semiclassical gravity’s path integral, and how
they can be helpful in understanding certain information paradoxes involving
the black hole interior Maldacena:2001kr ; Barbon:2004ce ; Mathur:2012np . The
focus of this work is to address these issues from the bulk viewpoint,
complementary to interesting developments summarized in Mathur:2012np ;
Almheiri:2020cfm ; Raju:2020smc  and references therein. To understand the
necessity of small corrections, we restrict ourselves to asymptotically AdS
spacetime, where such developments are pretty concretely understood
Maldacena:1997re ; Witten-ads-and-holography ; Gubser:1998bc .

Let us briefly discuss what exactly we are interested in, beginning with
Hawking’s information paradox. By studying quantum fields over a fixed black
hole background, it can be shown that the outgoing Hawking radiation is
thermal, a feature that arises due to the presence of the horizon. This leads
to the following question: how is the time evolution of an initially pure
state, say a collapsing star, to the final thermal state of Hawking radiation
consistent with quantum mechanics (QM)? The second situation we are interested
in regards bags of gold spacetimes Marolf:2008tx . Within the framework of
effective field theory (EFT), we can create a vast number of spatially
separated semiclassical excitations in the black hole interior and thus arrive
at an entropy much larger than the Bekenstein Hawking entropy. Besides, such
configurations also violate black holes’ expected spectral properties
Chakravarty:2020wdm ; CR .

It has long been argued that Hawking’s version of the paradox is not well-
posed, as small corrections to a thermal density matrix can purify it, a fact
which was overlooked in Hawking’s original calculation Raju:2020smc . A more
robust reformulation of this statement purely within a statistical framework
is that correlators over typical states in a statistical ensemble mimic
thermal correlators upto small corrections lloyd2013pure . In this work, we
demonstrate the origin of such small corrections and how they provide a basis
for these statistical statements. We also revisit the arguments in
Chakravarty:2020wdm ; CR  why the same small corrections play a role in
resolving the bags of gold paradox in a different light.

##  2 Semiclassical gravity path integral

To explicitly demonstrate small corrections, it is convenient to rewrite the
semiclassical path integral as a path integral over coherent states. To build-
up, let us first look at coherent state path integrals in QM.

###  2.1 Coherent state path integrals in QM

Consider a one-dimensional simple harmonic oscillator. Let $\ket{z}$ be the
coherent state defined by $\ket{z}=e^{za^{\dagger}}\ket{0}$, where $\ket{0}$
denotes the vacuum. The inner product between two coherent states is given by
$\braket{z}{z^{\prime}}=e^{\bar{z}z^{\prime}}$. The lifting of functions from
the phase space to operators acting on the Hilbert space is performed using
the Sudarshan-Mehta P representation:

$\hat{f}=\int\frac{d^{2}z}{2\pi i}\,e^{-z\bar{z}}f(z)\ket{z}\bra{z}$ (1)

Defining $\braket{z}{\psi}=\psi(\bar{z})$, we derive the coherent state path
integral by inserting the identity operator given by (1) in the discretization
of the matrix element $\braket{f}{e^{-iTH}}{i}$, where $H$ denotes the
Hamiltonian. The formal expression for the path integral is given by

$\braket{f}{\,e^{-iH(t_{f}-t_{i})}\,}{i}=\int\mathcal{D}z\mathcal{D}\bar{z}\,\,e^{iS+\frac{1}{2}\left(\absolutevalue{z_{i}}^{2}+\absolutevalue{z_{f}}^{2}\right)}\,\bar{\psi}_{f}(z_{f})\,\psi_{i}(\bar{z}_{i})$
(2)

where $S$ denotes the action
$S=\int_{t_{i}}^{t_{f}}dt\left(\frac{1}{2i}\left(z\,\partial_{t}\bar{z}-\bar{z}\,\partial_{t}z\right)-H(z,\bar{z})\right)$.
The other terms in (2) arise by expressing the initial and the final states in
the basis of coherent states. We will move onto coherent states in free field
theory and finally to the case with gravity turned on in the next subsection.

###  2.2 Coherent state path integrals in free field theory and gravity

Within EFT, we can similarly define normalized coherent states over a vacuum
$\ket{\Omega}$ by

$\ket{\vec{z}}=\exp\Big(-\sum_{i}\frac{\absolutevalue{z_{i}}^{2}}{2}+a_{i}^{\dagger}z_{i}\Big{missing})\ket{\Omega},$

where $a_{i}$ are annihilation operators. The absolute value of the inner
product between two such states is given by
$\absolutevalue{\braket{z}{z^{\prime}}}=e^{-\frac{\absolutevalue{z-z^{\prime}}^{2}}{2}}$.
We again lift functions from a $2n$-dimensional phase space to operators
acting on the Hilbert space using the operator

$\hat{f}=\int\frac{d^{2n}z}{(2\pi i)^{n}}f(z)\ket{z}\bra{z},$ (3)

Taking the expectation value of the operator in (3) over a coherent state
$\ket{\alpha}$, we obtain

$\braket{\alpha\,}{\hat{f}\,}{\alpha}=\int\frac{d^{2n}z}{(2\pi
i)^{n}}\,f(z)\,\exp\left(-\absolutevalue{z-\alpha}^{2}\right).$ (4)

It is straightforward to use the identity operator from (3) to write the
coherent state path integral for an EFT Hamiltonian $H$, as done in (2) for
the QM system. Analogously, we can also write the coherent states and the path
integral for semiclassical gravity as well.

We will work within the linearized gravity approximation, where we fix a
background metric $g^{0}_{\mu\nu}$ and consider small linearized perturbations
$h_{\mu\nu}$ about the same. We perform ADM decomposition of the metric and
quantize the system by lifting the Poisson algebra between the metric on the
spacelike slice and its’ conjugate momenta to commutators. Due to
linearization, this lifting defines canonical commutators between the creation
and the annihilation operators.

Within this context, we study the coherent states of linearized gravity (again
denoted by $\ket{z}$) over a vacuum $\ket{\Omega}$ Papadodimas:2015jra ;
Papadodimas:2015xma . Then the classical values of the metric $g$ and the
scalar field $\phi$ are given by $\braket{z}{\hat{g}}{z}=g$ and
$\braket{z}{\hat{\phi}}{z}=\phi$, where $g=g^{0}+h$. We again play the same
game for gravity and rewrite the path integral. Let us formally denote the
coherent state partition function, which has periodic boundary conditions in
imaginary time as

$Z(\beta)=\int\mathcal{D}z\mathcal{D}\bar{z}\,e^{-S(z,\bar{z},\beta)}.$ (5)

Here $S(z,\bar{z},\beta)$ includes the Einstein Hilbert action coupled to the
scalar field and contributions arising due to the coherent state construction.

##  3 Origin of small corrections and implications for information paradoxes

So far, we have performed non-controversial operations within EFT. We now
discuss why such a lifting of functions from the phase space to Hilbert space
is not correct in gravity. Within ordinary QFT the expected fall-off of
coherent states is given by
$\absolutevalue{\braket{z}{z^{\prime}}}=e^{-\frac{\absolutevalue{z-z^{\prime}}^{2}}{2}}$
for large enough $\absolutevalue{z-z^{\prime}}$. Thus coherent states with
classical expectation values separated far apart in the phase space have
vanishingly small overlaps. However, this does not hold for gravity’s case.
Within the formalism of AdS-CFT, using the dual CFT, it can be shown that
states separated far apart in the phase space do not have vanishingly small
overlaps as expected, but the overlaps saturate at
O$\left(e^{-\mathcal{N}}\right)$, where $G_{N}\propto\frac{1}{\mathcal{N}}$
Papadodimas:2015jra ; Papadodimas:2015xma ; Cotler:2016fpe ; Jafferis:2017tiu
.

We will briefly describe how the gravitational path integral receives small
corrections. While discretizing the partition function to rewrite it as a
coherent state partition function in (5), we have utilized the identity
operator in (3), with $\hat{f}=1$. However as we just discussed, for
$z\gg\alpha$, the inner product between the coherent states saturate at
$\absolutevalue{\braket{z}{\alpha}}=\text{O}\left(e^{-\mathcal{N}}\right)$.
Consequently, equation (4) needs to be appropriately modified, and thus (3) as
well. We will now study the implications of these small corrections for black
holes.

###  3.1 Implications for Hawking’s paradox

Consider the $k$-point correlator given by
$\braket{\psi_{BH}}{\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})}{\psi_{BH}},$ where
$\ket{\psi_{BH}}$ denotes the state of a black hole. Here we assume that
$k\ll\mathcal{N}$ so that the gravitational perturbation theory is well
defined, and the operator insertions have energies peaked about $\omega_{i}$
such that $\omega_{i}\ll\mathcal{N}$ so as to ignore backreaction. A standard
result in EFT follows that $k$-point correlators are thermal:

$\braket{\psi_{BH}}{\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})}{\psi_{BH}}=\text{Tr}\left(\rho\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})\right)$
(6)

where $\rho$ denotes the thermal density matrix. Since in a theory of gravity
we do not have local diffeomorphism-invariant operators, we restrict our
discussion to approximately local operators, defined such that $O(x)$ is
smeared over a small region around $x$. When operator insertions at
$x_{1},x_{2},\dots\,x_{k}$ are sufficiently close to each other, then the
chief contribution to the correlator comes from coherent states situated close
enough to each other, where the EFT inner product
$\absolutevalue{\braket{z}{z^{\prime}}}=e^{-\frac{\absolutevalue{z-z^{\prime}}^{2}}{2}}$
holds. However the path integral goes over the space of all coherent states,
including the ones far away. Consequently the corrections to the correlators
due to these faraway coherent states is
$\text{O}\left(e^{-\mathcal{N}}\right)$, i.e. the deviation from the expected
vanishing inner product. Thus the $k$ point correlator in (6) is modified upto
small corrections by

$\braket{\psi_{BH}}{\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})}{\psi_{BH}}=\text{Tr}\left(\rho\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})\right)+\text{O}\left(e^{-\mathcal{N}}\right)$
(7)

On a related note, from a statistical viewpoint, Hawking’s paradox is resolved
since almost all states in a statistical system closely mimic thermality
lloyd2013pure . These states obey a surprising property: the $k$ point
correlator over such states differ from thermal correlators by a small
correction in the microcanonical ensemble:

$\braket{O(x_{1})\,O(x_{2})\dots\,O(x_{k})}=\text{Tr}\left(\rho_{\Omega}\,O(x_{1})\,O(x_{2})\dots\,O(x_{k})\right)+\text{O}\left(e^{-S}\right)$
(8)

where $\rho_{\Omega}$ denotes the microcanonical density matrix and $S$
denotes the entropy of the statistical system. Our analysis of coherent states
culminating in equation (7) therefore provides a justification why such a
statistical (more precisely, kinematical) statement works in the first place.
Further, the corrections in (8) are consistent with our description since for
a black hole $S\propto\mathcal{N}$. We note here that these small corrections
are of a different nature as compared to corrections arising due to
backreaction. In contrast to backreaction, our small corrections can give an
O$(1)$ contribution to physical observables under certain circumstances as
shown in the next subsection.

###  3.2 Implications for bags of gold paradox

Figure 1: The bags of gold paradox for eternal BH, where bumps denote
excitations spatially separated far apart from each other on an interior slice
whose volume increases with boundary time $t$.

For correlators with operator insertions at points $x_{1},x_{2},\dots\,x_{k}$
spaced far apart from each other, there is a significant contribution to the
path integral due to the non-decaying inner products between far-separated
coherent states, since they represent dominant contributions. Such corrections
can potentially add up to give an O$(1)$ contribution to correlators. Indeed
it was shown that the spectral form factor and two-point correlator
$\braket{O(t)O(0)}$ obey the usually expected decay at a small-time $t$;
however, they increase and plateau off at a sufficiently large time
Cotler:2016fpe .

Now consider a spacelike slice within the interior of a large $AdS$ black hole
which has an increasingly large volume as the slice is time evolved. We place
semiclassical excitations each with small support far apart from each other on
these slices such that their spatial overlap is essentially zero, as shown in
Figure 1. Within EFT, these states have a vanishing inner product when the
distance between them is sufficiently large. Therefore, an estimation of the
black hole’s entropy using EFT leads to a substantial overcounting, far more
significant than the Bekenstein-Hawking entropy. Further, such an EFT
treatment of excitations leads to a violation of the spectral observables such
as the spectral form factor as well as of the two-point correlator
$\braket{O(t)O(0)}$ Chakravarty:2020wdm .

These inconsistencies are resolved by accounting for small inner products
between far-separated coherent states in the presence of gravity
Chakravarty:2020wdm ; CR . Without invoking our present discussion of path
integrals, it was shown there that the entropy, spectral form factor, and
other thermodynamic observables do not give wayward estimates when such small
corrections are adequately taken into account. Our present discussion of path
integrals is a reformulation of this picture, which provides a straightforward
and intuitive understanding of how such puzzles originate starting from first
principles.

##  4 Conclusion

Thus, we have provided a qualitative sketch of how small corrections can help
resolve various paradoxes involving black holes from the bulk viewpoint. A
significant problem is understanding why the coherent states in the presence
of gravity have a fat tail of overlaps from the bulk perspective. Other
important direction is to address the role of small corrections in the context
of thermalization, and to understand how much of our analysis carries over to
flat space. We will leave these questions to future work.

###### Acknowledgements.

I thank Diksha Jain, R. Loganayagam, Akhil Sivakumar and especially Suvrat
Raju for various important discussions leading to this work. I acknowledge
gratitude to the people of India for their steady and generous support to
research in basic sciences.

## References

  * (1) J. M. Maldacena, Eternal black holes in anti-de Sitter, JHEP 04 (2003) 021, [hep-th/0106112].
  * (2) J. L. F. Barbon and E. Rabinovici, Long time scales and eternal black holes, NATO Sci. Ser. II 208 (2006) 255–263, [hep-th/0403268].
  * (3) S. D. Mathur, The information paradox: conflicts and resolutions, Pramana 79 (2012) 1059–1073, [arXiv:1201.2079].
  * (4) A. Almheiri, T. Hartman, J. Maldacena, E. Shaghoulian, and A. Tajdini, The entropy of Hawking radiation, arXiv:2006.06872.
  * (5) S. Raju, Lessons from the Information Paradox, arXiv:2012.05770.
  * (6) J. M. Maldacena, The Large N limit of superconformal field theories and supergravity, Int. J. Theor. Phys. 38 (1999) 1113–1133, [hep-th/9711200].
  * (7) E. Witten, Anti-de Sitter space and holography, Adv. Theor. Math. Phys. 2 (1998) 253–291, [hep-th/9802150].
  * (8) S. Gubser, I. R. Klebanov, and A. M. Polyakov, Gauge theory correlators from noncritical string theory, Phys. Lett. B 428 (1998) 105–114, [hep-th/9802109].
  * (9) D. Marolf, Black Holes, AdS, and CFTs, Gen. Rel. Grav. 41 (2009) 903–917, [arXiv:0810.4886].
  * (10) J. Chakravarty, Overcounting of interior excitations: A resolution to the bags of gold paradox in AdS, JHEP 02 (2021) 027, [arXiv:2010.03575].
  * (11) J. Chakravarty and S. Raju, How do black holes manage to look larger from the inside than the outside?, Unpublished Essay (2020).
  * (12) S. Lloyd, Pure state quantum statistical mechanics and black holes, arXiv:1307.0378.
  * (13) K. Papadodimas and S. Raju, Remarks on the necessity and implications of state-dependence in the black hole interior, Phys. Rev. D 93 (2016), no. 8 084049, [arXiv:1503.08825].
  * (14) K. Papadodimas and S. Raju, Local Operators in the Eternal Black Hole, Phys. Rev. Lett. 115 (2015), no. 21 211601, [arXiv:1502.06692].
  * (15) J. S. Cotler, G. Gur-Ari, M. Hanada, J. Polchinski, P. Saad, S. H. Shenker, D. Stanford, A. Streicher, and M. Tezuka, Black Holes and Random Matrices, JHEP 05 (2017) 118, [arXiv:1611.04650]. [Erratum: JHEP 09, 002 (2018)].
  * (16) D. L. Jafferis, Bulk reconstruction and the Hartle-Hawking wavefunction, arXiv:1703.01519.