# Robust Decentralized Control of Coupled Systems via Risk Sensitive Control
of Decoupled or Simple Models with Measure Change

Zachary Selk111Department of Mathematics and Statistics, Queen’s University.
<EMAIL_ADDRESS>Serdar Yüksel222Department of Mathematics and
Statistics, Queen’s University<EMAIL_ADDRESS>
###### Abstract

Decentralized stochastic control problems with local information involve
problems where multiple agents and subsystems which are coupled via dynamics
and/or cost are present. Typically, however, the dynamics of such couplings is
complex and difficult to precisely model, leading to questions on robustness
in control design. Additionally, when such a coupling can be modeled, the
problem arrived at is typically challenging and non-convex, due to
decentralization of information. In this paper, we develop a robustness
framework for optimal decentralized control of interacting agents, where we
show that a decentralized control problem with interacting agents can be
robustly designed by considering a risk-sensitive version of non-interacting
agents/particles. This leads to a tractable robust formulation where we give a
bound on the value of the cost function in terms of the risk-sensitive cost
function for the non-interacting case plus a term involving the “strength” of
the interaction as measured by relative entropy. We will build on Gaussian
measure theory and an associated variational equality. A particular
application includes mean-field models consisting of (a generally large number
of) interacting agents which are often hard to solve for the case with small
or moderate numbers of agents, leading to an interest in effective
approximations and robustness. By adapting a risk-sensitivity parameter, we
also robustly control a non-symmetrically interacting problem with mean-field
cost by one which is symmetric with a risk-sensitive criterion, and in the
limit of small interactions, show the stability of optimal solutions to
perturbations.

##  1 Introduction

Decentralized stochastic control involves problems where multiple agents and
subsystems are coupled via dynamics and/or cost and agents have local
information. Typically, the dynamics of such couplings is difficult to
precisely model, leading to questions on robustness. Additionally, when such a
coupling can be modelled, the problem arrived at is typically non-convex and
difficult to solve due to decentralization in information for which classical
tools in stochastic control, such as dynamic programming are no longer
directly applicable (see e.g. [42, 44]).

In this paper, we develop a framework for risk-sensitive control for the study
of decentralized stochastic control problems with uncertain interactions among
a finite or infinite collection of interacting particles.

Risk-sensitive stochastic control has been shown to be an effective paradigm
for robust stochastic control, with intrinsic connections between large
deviations and relative entropy; seminal studies in the stochastic domain
include [15, 31, 1]. Rather than minimizing an average “risk neutral” cost
$N(\gamma):=E[C(\gamma)]$, one considers the “risk sensitive” cost
$S(\gamma):=\log E[e^{C(\gamma)}]$. By punishing deviations more harshly
through exponentiation, minimizing $S$ leads to policies that are “robust” to
perturbations of the model. More precisely, for each control $\gamma$, and
under suitable conditions on $C$, we have the variational equality

$\displaystyle\log
E_{\mu_{0}}[e^{C(\gamma)}]=\sup_{\mu\sim\mu_{0}}\\{E_{\mu}[C(\gamma)]-D_{KL}(\mu||\mu_{0})\\},$
(1)

where $D_{KL}$ is the KL-divergence or relative entropy and $\sim$ denotes
equivalence (mutual absolute continuity) of measures. In [15], the authors
related risk neutral control of a system driven by “true noise” with a risk
sensitive control of a system driven by “nominal noise”. That is, if $\mu_{0}$
is the law of some “nominal” noise driving the system and $\mu$ is the law of
the actual noise of the system, then we have the bound

$E_{\mu}[C(\gamma)]\leq\log E_{\mu_{0}}[e^{C(\gamma)}]+D_{KL}(\mu||\mu_{0}).$

We note that for risk sensitive control problems, there is also a direct
relationship with a zero-sum formulation between the controller and
disturbance [24, 4, 5]. These approaches (to robustness) are complementary;
the former uses a variational approach and the latter typically builds on the
direct equivalence of the associated Hamilton-Jacobi-Bellman equations.

We study decentralized stochastic control problems, where the (sometimes
poorly modeled) interaction among agents leads to two types of challenges: (i)
When the mathematical model is known precisely, lack of convexity arising from
decentralized information structure makes the analysis challenging [13, 18,
23, 36, 42, 46]), and (ii) when the mathematical model is not known precisely,
a natural robustness question arises.

In the context of our paper, a particular setup involves mean-field theory.
Mean field theory [12, 19, 20, 25, 27] models control problems when there are
(a generally large number of) interacting agents. One key observation in mean
field theory is that, under some conditions, for large enough systems of
agents one can approximate all interactions through a mean field coupling term
- that is, each agent interacts with the average behavior of all agents.
However, due to this coupling, mean field control problems are often very hard
to solve when the number of agents is not large or when the agents apply non-
identical policies which prohibits concentration results for the
asymptotically large number of agents regime (see [38, 39, 40] on possible
non-optimality of symmetric policies for finitely many agents but their
optimality in the infinite limit). This leads to an interest in effective
approximations and robust solutions.

Our approach builds on defining equivalent information structures via change
of measure arguments, but unlike those in the literature starting with [6], we
approach the equivalent information structures with the aim of attaining
robustness to solutions of complex decentralized stochastic control problems.
The common thread for change of measure arguments (see [37] for a detailed
review) is an absolute continuity condition involving a family of conditional
probabilities on the measurement variables (given the past history or state
realizations) with respect to a common reference measure. By a different
approach and motivation, in this paper, we develop a framework for risk-
sensitive control for the study of decentralized stochastic control problems
with uncertain interactions among a finite or infinite collection of
interacting particles. We will utilize tools from Gaussian measure theory and
an associated variational equality.

In the decentralized information setup, we note that the robustness
relationship presented in our paper cannot be obtained directly through an
equivalence associated with a Hamilton Jacobi Bellman equation arrived at via
a game theoretic formulation, as such an optimality condition is currently not
available due to the decentralized information structure and does not
typically arise (though optimality conditions can be obtained for convex
teams) in team theoretic problems under information constraints when there are
finitely many agents.

Our paper also motivates the study of risk-sensitive decentralized stochastic
control, which has been studied in the recent literature [7, 30, 34, 35, 41].

Main Results. The variational equality (1) can also be applied where the
nominal reference measure $\mu_{0}$ is the law of non-interacting agents and
$\mu$ is the law of interacting agents. In this paper, we propose to use this
variational equality to approximate a risk neutral interacting agent mean
field control problem by a risk sensitive non-interacting control problem.
Theorem 3.6 presents a robustness result under a more relaxed energy condition
but with a mean-field coupling, and Theorem 3.8 presents the robustness result
for decentralized systems under arbitrary coupling but with a more stringent
energy condition.

By adapting a risk-sensitivity parameter, we also (i) robustly control a non-
symmetrically interacting problem with mean-field cost by one which is
symmetric with a risk-sensitive criterion, and (ii) in the limit of small
interactions, show the stability of optimal solutions to perturbations.

We also discuss optimal risk sensitivity. By introducing a parameter
$\alpha\in\mathbb{R}$ which quantifies the “amount” of risk aversion, we can
get the bound

$E_{\mu}[C(\gamma)]\leq\frac{1}{\alpha}\log E_{\mu_{0}}[e^{\alpha
C(\gamma)}]+\frac{1}{\alpha}D_{KL}(\mu||\mu_{0}).$ (2)

By optimizing over $\alpha$ we obtain Theorem 4.2.

As a corollary, we show how we can recover the true model in the small
interaction limit by selecting $\alpha$ appropriately. We additionally show
how we can approximate a non-symmetric problem by a symmetric problem. See
Section 4 for details and additional implications, including on stability of
risk-sensitive optimal solutions to small interactions.

##  2 Problem Setup

We consider two system models, one with a perturbation of the measure on the
agents and the other with a perturbation on the noise (similar to [15]).

System Model A. Consider a system of interacting controlled differential
systems (particles) where each $X^{m}_{t};t\in[0,T],m=1,\cdots,N$ is
$\mathbb{R}^{n}$-valued

$\displaystyle d\tilde{X}^{1}_{t}$
$\displaystyle=(b_{1}(\tilde{X}^{1}_{t},U^{1}_{t})+v_{t}^{1})dt+\sigma_{1}(\tilde{X}^{1}_{t})dB^{1}_{t}$
$\displaystyle\vdots$ $\displaystyle d\tilde{X}^{N}_{t}$
$\displaystyle=(b_{N}(\tilde{X}_{t}^{N},U^{N}_{t})+v_{t}^{N})dt+\sigma_{N}(\tilde{X}^{N}_{t})dB^{N}_{t},$
(3)

with $X^{[1,N]}_{0}=x^{[1,N]}_{0}$ and where the $v^{i}$ (which may be
incorrectly modelled) depend on the entire ensemble
$(\tilde{X}^{1},...,\tilde{X}^{N})$ so that $v^{i}_{t}$ is
$\sigma(\\{X^{[1,N]}_{[0,t]}\\})$-measurable, and satisfies
$E\left[\int_{0}^{T}v_{t}^{2}dt\right]<\infty.$ With $\mathbb{U}$ a compact
metric space, $U^{m}_{t}$ is $\mathbb{U}$ valued for each $m=1,\cdots,N$ and
$t\in[0,T]$

System Model B. Consider a system of interacting controlled differential
systems (particles)

$\displaystyle d\tilde{X}^{1}_{t}$
$\displaystyle=(b_{1}(\tilde{X}^{1}_{t},U^{1}_{t})+\sigma_{1}(\tilde{X}^{1}_{t})v_{t}^{1})dt+\sigma_{1}(\tilde{X}^{1}_{t})dB^{1}_{t}$
$\displaystyle\vdots$ $\displaystyle d\tilde{X}^{N}_{t}$
$\displaystyle=(b_{N}(\tilde{X}_{t}^{N},U^{N}_{t})+\sigma_{N}(\tilde{X}^{N}_{t})v_{t}^{N})dt+\sigma_{N}(\tilde{X}^{N}_{t})dB^{N}_{t},$
(4)

with $X^{[1,N]}_{0}=x^{[1,N]}_{0}$ and where the $v^{i}$ depend on the entire
noise $(B^{1},...,B^{N})$ so that $v^{i}_{t}$ is
$\sigma(\\{B^{[1,N]}_{[0,t]}\\})$-measurable, and satisfies
$E\left[\int_{0}^{T}v_{t}^{2}dt\right]<\infty.$

Here $B^{i}$ are i.i.d. Brownian motions, $U^{i}_{t}$ are control action
variable processes generated under control policies $\gamma^{i}\in\Gamma^{i}$,
where $\Gamma^{i}$ is the set of admissible policies:

In the setup to be considered the admissible policies may be decentralized or
centralized. In the decentralized setup, we view each $X^{i}$, $1\leq i\leq
N$, to be controlled locally (subject to the coupling), such that at each time
$U^{i}_{t}$ is a possibly ${\cal P}(\mathbb{U})$-valued
$\sigma(X^{i}_{[0,t]})$-measurable control. In the centralized setup, we
assume that $U^{i}_{t}$ is a possibly ${\cal P}(\mathbb{U})$-valued
$\sigma(X^{1,N}_{[0,t]})$-measurable control.

Our focus will be on the decentralized setup, as the results are more
consequential given the difficulties associated with optimal decentralized
stochastic control with local information.

As noted in [3, Chapter 2] (see also [33, Section 4]) under such
(decentralized or centralized) feedback policies, the question on the
existence of strong solutions leads to several subtleties. Accordingly,
instead of feedback policies, one may also consider policies which are
probability measure-valued and adapted (as measure-valued controls) to the
local driving noise so that the policies are non-anticipative: thus, one
requires that $\\{(U^{i}_{s},B^{i}_{s}),s\leq t\\}$ be independent of
$B^{i}_{r}-B^{i}_{t},r>t$ for all $t\in[0,T]$, and also independent of
$B^{m}_{t},t\in[0,T]$ for $i\neq m,m=1,2,\cdots,N$. Accordingly, in the above,
standard Lipschitz growth conditions [3, Theorem 2.2.4] (see also [33, Section
4]) are to be imposed on $b_{i},\sigma_{i}$ to ensure the existence of strong
solutions to the stochastic differential equations above under the policies
considered. This is needed for change of measure arguments to be made in our
analysis (so that policies can also be expressed as a function of the driving
noise validating the change of measure equations to be studied). Accordingly,
we impose the following assumptions on the drift $b_{i}$ and the diffusion
matrices $\sigma_{i}$.

  * (A1)

_Lipschitz continuity:_ The function
$\sigma_{i}:\mathbb{R}^{d}\to\mathbb{R}^{d\times d}$,
$b_{i}:\mathbb{R}^{d}\times\mathbb{U}^{i}\to\mathbb{R}^{d}$ are Lipschitz
continuous in $x$ (uniformly with respect to the control action for $b$). In
particular, for some constant $C>0$ depending on $R>0$, we have

$\|b_{i}(x,u)-b_{i}(y,u)\|^{2}+\|\sigma_{i}(x)-\sigma_{i}(y)\|^{2}\,\leq\,C\,\|x-y\|^{2}$

for all $x,y\in\mathbb{R}^{n}$ and $u\in\mathbb{U}^{i}$, where
$\|\sigma_{i}\|:=\sqrt{\mathrm{Trace}(\sigma_{i}\sigma_{i}^{T})}$. We also
assume that $b$ is jointly continuous in $(x,u)$.

  * (A2)

_Affine growth condition:_ $b_{i}$ and $\sigma_{i}$ satisfy a global growth
condition of the form

$\sup_{u\in\mathbb{U}^{i}}|\langle
b_{i}(x,u),x\rangle|+\|\sigma_{i}(x)\|^{2}\leq C_{0}(1+\|x\|^{2})\qquad\forall
x\in\mathbb{R}^{n},$

for some constant $C_{0}>0$.

We consider the following cost criteria.

General Cost with $N<\infty$.

$J_{N}(\underline{\gamma}_{N})=E^{\underline{\gamma}_{N}}_{\mu_{0}^{N}}\left[c_{N}(X^{[1,N]},U^{[1,N]})\right],$

with

$c_{N}(X^{[1,N]},U^{[1,N]})=\int_{0}^{T}g(X_{s}^{[1,N]},U_{s}^{[1,N]})ds$ (5)

where $\underline{\gamma}_{N}=(\gamma^{1},...,\gamma^{N})$ with
$U^{i}_{t}=\gamma^{i}(X^{i}_{[0,t]})$.

We will also consider a more specialized mean field symmetric cost. This will
allow for symmetric interactions in our system and more relaxed conditions.

Mean-Field Symmetric Cost with $N<\infty$ or $N=\infty$.

Consider

$J_{N}(\underline{\gamma}_{N})=E^{\underline{\gamma}_{N}}_{\mu_{0}^{N}}\left[c_{N}(X^{[1,N]},U^{[1,N]})\right],$

with

$c_{N}(X^{[1,N]},U^{[1,N]})=\frac{1}{N}\sum_{i=1}^{N}c\left(X^{i},U^{i},\frac{1}{N}\sum_{m=1}^{N}X^{m},\frac{1}{N}\sum_{m=1}^{N}U^{m}\right)$
(6)

where

$c\left(X^{i},U^{i},\frac{1}{N}\sum_{m=1}^{N}X^{m},\frac{1}{N}\sum_{m=1}^{N}U^{m}\right)=\int_{0}^{T}g(X^{i}_{s},U^{i}_{s},\frac{1}{N}\sum_{m=1}^{N}X^{m}_{s},\frac{1}{N}\sum_{m=1}^{N}U^{m}_{s})ds$

where $\underline{\gamma}_{N}=(\gamma^{1},...,\gamma^{N})$ with
$U^{i}_{t}=\gamma^{i}(X^{i}_{[0,t]})$. We are also interested in the limit

$J(\underline{\gamma})=\limsup_{N\to\infty}J_{N}(\underline{\gamma}).$ (7)

We note that the cost can be made more general: Let for Borel
$B\in\mathbb{U}$,

$\kappa^{N}(B)=\frac{1}{N}\sum_{p=1}^{{N}}\delta_{U^{i}}(B)$

be the empirical measure involving control actions, and for Borel
$D\in\mathbb{R}^{n}$

$\zeta^{N}(B)=\frac{1}{N}\sum_{p=1}^{{N}}\delta_{X^{i}}(D)$

be the measure for local states. We could have $c$ above be of the form
$c(X^{i},U^{i},\zeta^{N},\kappa^{N})$ as well.

The above will be called either true model A or true model B and be associated
with measure $\mu$ under control policy $\underline{\gamma}_{N}$.

A Nominal (Reference) Model with Non-Interacting Agents. Consider now the
nominal system of non-interacting agents given by the collection of stochastic
differential equations

$\begin{cases}dX^{1}_{t}&=b_{1}(X^{1}_{t},U^{1}_{t})dt+\sigma_{1}(X^{1}_{t})dB^{1}_{t}\\\
&...\\\
dX^{N}_{t}&=b_{N}(X^{N}_{t},U^{N}_{t})dt+\sigma_{N}(X^{N}_{t})dB^{N}_{t},\end{cases}$

with $X^{[1,N]}_{0}=x^{[1,N]}_{0}$. The same assumptions on either of the true
models apply here.

The rest of this paper is dedicated to showing a robustness theorem for
$J(\underline{\gamma})$ where risk-neutral stochastic optimal control with
weakly interacting agents may be bounded by risk-sensitive stochastic optimal
control with non interacting agents.

##  3 Robust Decentralized Control Design for Coupled Systems via Risk
Sensitive Control of Non-Interacting Systems

We will consider two classes of models, in the first one the coupling will be
via mean-field effects and in the second one the coupling will be more
general.

In the former setup, a total energy constraint will be absent unlike the
latter setup. In particular, we are interested in proving a robustness bound
for $J$ without the Novikov assumption (8) to be presented below. This will be
achieved by utilizing the symmetry in the model, and thus avoid the total
energy constraint. Later on, we will consider a more general coupling
structure under a more restrictive moment constraint.

First we present a variational formula involving Cameron-Marton spaces and a
formula for relative entropy.

###  3.1 A general variational formula and relative entropy

First, we have the following result for the Cameron-Martin (see e.g. [17,
Chapter 3]) space of countable product spaces.

######  Theorem 3.1.

Let $\mu_{n},n\in\mathbb{N}$ be centered Gaussian measures on the countable
collection of Banach spaces $\mathcal{B}_{n}$ with Cameron-Martin spaces
$\mathcal{H}_{\mu_{n}}$. Consider the Fréchet space
$\mathcal{F}=\Pi_{n=1}^{\infty}\mathcal{B}_{n}$ with product topology and
product measure $\mu$. Then the Cameron-Martin space is

$\mathcal{H}_{\mu}=\\{\mathbf{v}=\\{v_{n}\\}_{n\in\mathbb{N}}:\|\mathbf{v}\|_{\mu}^{2}=\sum_{n=1}^{\infty}\|v_{n}\|_{\mu_{n}}^{2}<\infty\\}.$

###### Proof.

The dual space, $\mathcal{F}^{\ast}$ can be identified with elements of
$\Pi_{n=1}^{\infty}\mathcal{B}_{n}^{\ast}$ with finite support, with
$\mathbf{v}^{\ast}\in\Pi_{n=1}^{\infty}\mathcal{B}_{n}^{\ast}$ being given by

$\mathbf{v}(\mathbf{x})=\sum_{n=1}^{\infty}v_{n}(x_{n})$

for $\mathbf{x}\in\Pi_{n=1}^{\infty}\mathcal{B}_{n}^{\ast}$. Let
$j:\mathcal{F}^{\ast}\to L^{2}(\mathcal{F},\mu)$ be the canonical embedding.
Then $\mathcal{H}_{\mu}$ is the $L^{2}$ closure of $j(\mathcal{F})$ under
$\mu$ with norm
$\|\mathbf{v}\|_{\mu}^{2}=\|j(\mathbf{v})\|_{\mu}^{2}=\sum_{n=1}^{\infty}\|v_{n}\|_{\mu_{n}}^{2}$.
∎

######  Remark 3.2.

Given any measure on Fréchet space $\mathcal{F}$ there exists a compactly
embedded Banach space $\mathcal{B}$ of full measure. Therefore, we may
restrict to that compactly embedded Banach space. See e.g. [9], Theorem 3.5.
Therefore any results stated in terms of Banach spaces hold in this setting.

######  Theorem 3.3.

Let $\mu_{n}$ be centered Gaussian measures on the countable collection of
Banach spaces $\mathcal{B}_{n}$ with Cameron-Martin spaces
$\mathcal{H}_{\mu_{n}}$. Consider the Fréchet space
$\mathcal{F}=\Pi_{n=1}^{\infty}\mathcal{B}_{n}$ with product topology, product
measure $\mu$ (which may be reduced to a Banach space, given Remark 3.2) and
Cameron-Martin space

$\mathcal{H}_{\mu}=\\{\mathbf{v}=\\{v_{n}\\}_{n\in\mathbb{N}}:\|\mathbf{v}\|_{\mu}^{2}=\sum_{n=1}^{\infty}\|v_{n}\|_{\mu_{n}}^{2}<\infty\\}.$

Suppose that $\mathbf{v}$ is a sequence of (potentially random) elements so
that the Novikov condition

$E_{\mu}\left[e^{\frac{1}{2}\|\mathbf{v}\|_{\mu}^{2}}\right]=E_{\mu}\left[e^{\frac{1}{2}\sum_{n=1}^{\infty}\|v_{n}\|_{\mu_{n}}^{2}}\right]<\infty$
(8)

holds. Consider a set of admissible policies
$\Gamma=\prod_{n=1}^{\infty}\Gamma^{n}$. Let
$C:\mathcal{F}\times\Gamma\to\mathbb{R}$ be a map so that for each
$\gamma\in\Gamma$, the functional $C(\cdot,\gamma)$ satisfies the finite
entropy assumption $E_{\mu}\left[e^{C}|C|\right]<+\infty$. Then we have that

$\displaystyle\log
E_{\mu}\left[e^{C(\cdot,\gamma)}\right]=\sup_{\nu\sim\mu}\left\\{E_{\nu}[C]-D_{KL}(\nu||\mu)\right\\},$
(9)

and the supremum is achieved at $\mu^{\ast,\gamma}$ which has density

$\frac{d\mu^{\ast,\gamma}}{d\mu}=\frac{e^{C(\cdot,\gamma)}}{E_{\mu}[e^{C(\cdot,\gamma)}]}.$

###### Proof.

Given the setup, the result follows directly from [8]. ∎

We next present a supporting result on relative entropy.

######  Lemma 3.4.

Let $v_{s}^{n}$ be a sequence of progressively measurable processes so that
$E\left[e^{\frac{1}{2}\sum_{n=1}^{\infty}\int_{0}^{T}(v_{s}^{n})^{2}ds}\right]<+\infty$,
and let $\mu$ be the law of the shifted process
$(B_{t}^{1}+\int_{0}^{t}v_{s}^{1}ds,...)$, then

$D_{KL}(\mu||\mu_{0})=\frac{1}{2}\sum_{n=1}^{\infty}E_{\mu}\left[\int_{0}^{T}(v_{s}^{n})^{2}ds.\right]$
(10)

###### Proof.

By Girsanov, we know that the density for $\mu$ is

$\frac{d\mu}{d\mu_{0}}=\exp\left(\sum_{n=1}^{\infty}\int_{0}^{T}v_{s}^{n}dB_{s}^{n}-\frac{1}{2}\sum_{n=1}^{\infty}\int_{0}^{T}(v_{s}^{n})^{2}ds\right).$

Taking $\log$ and expectation with respect to $\mu$ gives

$D_{KL}(\mu||\mu_{0})=E_{\mu}\left(\sum_{n=1}^{\infty}\int_{0}^{T}v_{s}^{n}dB_{s}^{n}-\frac{1}{2}\sum_{n=1}^{\infty}\int_{0}^{T}(v_{s}^{n})^{2}ds\right).$

By Girsanov, we have that $B=\tilde{B}+\int v$ where $\tilde{B}$ is a Brownian
motion under $\mu$. Therefore we conclude that

$D_{KL}(\mu||\mu_{0})=E_{\mu}\left(\sum_{n=1}^{\infty}\left(\int_{0}^{T}v_{s}^{n}d\tilde{B}_{s}^{n}+\int_{0}^{T}(v_{s}^{n})^{2}ds\right)-\frac{1}{2}\sum_{n=1}^{\infty}\int_{0}^{T}(v_{s}^{n})^{2}ds\right),$

leading to (10).

∎

The above supporting results will be utilized in the following two
subsections.

###  3.2 Systems with mean-field coupling

In this section, we consider System Model A. First, we obtain an expression
for the relative entropy of path measures on solutions to SDEs. This has been
investigated previously in e.g. [21, 22, 28, 29].

######  Lemma 3.5.

Let $\mu_{0}^{N}$ be the law of
$\tilde{X}^{[1,N]}=(\tilde{X}^{1}(t),...,\tilde{X}^{N}(t))$ and let $\mu^{N}$
be the law of $X^{[1,N]}=(X^{1}(t),...,X^{N}(t))$. Then

$\displaystyle D_{KL}(\mu^{N}||\mu_{0}^{N})$
$\displaystyle=\sum_{n=1}^{N}\frac{1}{2}E_{\mu^{N}}\left[\int_{0}^{T}\left|\frac{v_{n}(s)}{\sigma_{n}}\right|^{2}\right].$

Suppose that $\sigma_{i}$ are equal and we have symmetric interaction terms
$v_{n}$. Then

$\displaystyle\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})=\frac{1}{2}E_{\mu^{N}}\left[\int_{0}^{T}v_{1}^{2}(s)ds\right].$
(11)

###### Proof.

By Girsanov we know that the Radon-Nikodym derivative is

$\frac{d\mu^{N}}{d\mu_{0}^{N}}=\exp\left(\left(\int_{0}^{T}\left(\frac{v_{1}(s)}{\sigma_{1}(s)},...,\frac{v_{n}(s)}{\sigma_{N}(s)}\right)\cdot
dB(s)-\frac{1}{2}\int_{0}^{T}\left\|\left(\frac{v_{1}(s)}{\sigma_{1}(s)},...,\frac{v_{n}(s)}{\sigma_{n}(s)}\right)\right\|^{2}ds\right)\right).$

In the constant case, the above simplifies as

$\frac{d\mu^{N}}{d\mu_{0}^{N}}=\exp\left(\left(\int_{0}^{T}\left(\frac{v_{1}(s)}{\sigma_{1}},...,\frac{v_{n}(s)}{\sigma_{N}}\right)\cdot
dB(s)-\frac{1}{2}\int_{0}^{T}\left\|\left(\frac{v_{1}(s)}{\sigma_{1}},...,\frac{v_{n}(s)}{\sigma_{n}}\right)\right\|^{2}ds\right)\right).$

Therefore the relative entropy (i.e., the Kullback-Leibler divergence) is

$\displaystyle D_{KL}(\mu^{N}||\mu_{0}^{N})$
$\displaystyle=E_{\mu^{N}}\left[\left(\int_{0}^{T}\left(\frac{v_{1}(s)}{\sigma_{1}(s)},...,\frac{v_{n}(s)}{\sigma_{N}(s)}\right)\cdot
dB(s)-\frac{1}{2}\int_{0}^{T}\left\|\left(\frac{v_{1}(s)}{\sigma_{1}(s)},...,\frac{v_{n}(s)}{\sigma_{n}(s)}\right)\right\|^{2}ds\right)\right]$
$\displaystyle=\sum_{n=1}^{N}E_{\mu^{N}}\left[\int_{0}^{T}\frac{v_{n}(s)}{\sigma_{n}(s)}dB^{n}(s)-\frac{1}{2}\int_{0}^{T}\left|\frac{v_{n}(s)}{\sigma_{n}(s)}\right|^{2}ds\right]$
$\displaystyle=\sum_{n=1}^{N}\frac{1}{2}E_{\mu^{N}}\left[\int_{0}^{T}\left|\frac{v_{n}(s)}{\sigma_{n}(s)}\right|^{2}ds\right],$

where the last step is because
$E_{\mu^{N}}\left[\int_{0}^{T}\frac{v_{n}(s)}{\sigma_{n}(s)}dB^{n}(s)\right]=E_{\mu^{N}}\left[\int_{0}^{T}\left|\frac{v_{n}(s)}{\sigma_{n}(s)}\right|^{2}ds\right]$.
∎

We have the following robustness result.

######  Theorem 3.6.

Consider the cost functional $J$ defined in (7) with cost criterion (6) and
fixed $\gamma\in\Gamma$. Consider a progressively measurable process $v_{t}\in
L^{2}([0,T]\times\Omega)$ satisfying Novikov’s condition and the measure
$\mu_{\gamma}$ given by the law of $\tilde{X}(t)=(\tilde{X}^{1}(t),...)$ and
$\mu_{\gamma}^{N}$ the law of
$\tilde{X}^{[1,N]}=(\tilde{X}^{1}(t),...,\tilde{X}^{N}(t))$. Let the measure
$\mu_{0,\gamma}$ given by the law of $X(t)=(X^{1}(t),...)$ and
$\mu_{0,\gamma}^{N}$ the law of $X^{[1,N]}=(X^{1}(t),...,X^{N}(t))$.

we have that

$\displaystyle\inf_{\underline{\gamma}\in\Gamma}\limsup_{N\to\infty}E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]$
$\displaystyle\leq\inf_{\underline{\gamma}\in\Gamma}\limsup_{N\to\infty}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0,\gamma}}\left[e^{Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]+\limsup_{N\to\infty}\frac{1}{N}D_{KL}(\mu_{\gamma}^{N}||\mu_{0,\gamma}^{N}).$
(12)

With identical $E[\int_{0}^{T}(v_{s}^{n})^{2}]$ for each $n$ under symmetric
interaction and constant $\sigma_{n}$ that are bounded away from zero as
$|\sigma_{n}|\geq\sigma>0$, the above leads to

$\displaystyle\inf_{\underline{\gamma}\in\Gamma}E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]$
$\displaystyle\leq\inf_{\underline{\gamma}\in\Gamma}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0,\gamma}}\left[e^{Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]+\frac{1}{2\sigma^{2}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}v_{s}^{2}ds\right]\bigg{)}.$
(13)

###### Proof.

Using Donsker-Varadhan (see [8]), for every policy $\gamma\in\Gamma$, we have
the variational expression (suppressing the dependence of the measures on
$\gamma$)

$\displaystyle L(\gamma)$ $\displaystyle:=\limsup_{N\to\infty}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]$
$\displaystyle=\limsup_{N\to\infty}\sup_{\mu^{N}\sim\mu_{0}^{N}}\left\\{E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]-\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})\right\\}.$
(14)

Therefore, for each fixed sequence $\mu^{N}\sim\mu_{0}^{N}$,

$\displaystyle L(\gamma)$ $\displaystyle\geq$
$\displaystyle\limsup_{N\to\infty}\left\\{E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]-\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})\right\\}.$
(15) $\displaystyle\geq$
$\displaystyle\limsup_{N\to\infty}\left\\{\left\\{E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]\right\\}-\limsup_{N\to\infty}\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})\right\\}$

This leads to, for each $\gamma\in\Gamma$

$\limsup_{N\to\infty}\left\\{E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]\right\\}\leq
L(\gamma)+\limsup_{N\to\infty}\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})$

where we again note that $\mu_{\gamma}$ depends on the policy. As the above
applies for each admissible $\gamma$, we have (3.6).

∎

Accordingly, one could optimize $L(\gamma)$ and apply the resulting policy to
the right hand term of (3.6) to obtain a further upper bound.

By a simple adaptation of Theorem 3.6 we have the corollary for the case of
System Model B.

######  Corollary 3.7.

Consider the setup of Theorem 3.6. Let $M_{0}^{N}$ be the law of
$(B^{1},...,B^{N})$, $M_{0}$ be the law of $(B^{1},...)$, $M^{N}$ be the law
of $(B^{1}+\int v^{1},...,B^{N}+\int v^{N})$ and let $M$ be the law of
$(B^{1}+\int v_{1},...)$. Then we have the robustness bound

$\displaystyle\inf_{\underline{\gamma}\in\Gamma}\limsup_{N\to\infty}E^{\underline{\gamma}}_{M}[c_{N}(X^{[1,N]},U^{[1,N]})]$
$\displaystyle\leq\inf_{\underline{\gamma}\in\Gamma}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\log
E^{\underline{\gamma}}_{M_{0}}\left[e^{Nc_{N}(X^{[1,N]},U^{[1,N]})}\right]+\frac{1}{2}E^{\underline{\gamma}}_{M}\left[\int_{0}^{T}v_{s}^{2}ds\right]\bigg{)}.$
(16)

###### Proof.

Apply Theorem 3.6 with $\tilde{v}_{t}^{i}=\sigma_{i}(X^{i})v_{t}^{i}$. ∎

###  3.3 Systems with arbitrary coupling: Measures on Infinite product space
and total energy constraint on coupling

In this section, under the general cost criterion (5), we consider the setup
where the noise of the true system is a perturbation of the noise of the
nominal system directly on the infinite product space arbitrarily, though
under a finite total energy condition. In particular this finite energy
condition prohibits symmetric interaction. However, this is at the benefit of
a general cost criterion.

Building on the analysis above, and Lemma 3.4, we obtain the following.

######  Theorem 3.8.

Let $J$ be the functional defined in equation (7) under cost criterion (5).
Then for fixed $\gamma\in\Gamma$ and for $\mu\sim\mu_{0}$ ($\mu$ corresponding
to System Model B) where $\mu_{0}$ is the law of the infinite collection of
i.i.d. Brownian motions, we have that

$\displaystyle E^{\underline{\gamma}}_{\mu}[c_{N}(X^{[1,N]},U^{[1,N]})]$
$\displaystyle\leq\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{c_{N}(X^{[1,N]},U^{[1,N]})}\right]+D_{KL}(\mu||\mu_{0})$
$\displaystyle=\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{c_{N}(X^{[1,N]},U^{[1,N]})}\right]+\frac{1}{2}\sum_{n=1}^{\infty}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v_{s}^{n})^{2}ds\right],$

implying, via Lemma 3.4, that

$\inf_{\underline{\gamma}\in\Gamma}E^{\underline{\gamma}}_{\mu}[c_{N}(X^{[1,N]},U^{[1,N]})]\leq\inf_{\underline{\gamma}\in\Gamma}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{c_{N}(X^{[1,N]},U^{[1,N]})}\right]+\frac{1}{2}\sum_{n=1}^{\infty}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v_{s}^{n})^{2}ds\right].$

##  4 Refinements: Optimizing Sensitivity, Robustness via Symmetric Models,
Stability to Small Couplings and Large Mean-Field Problems

In this section, we obtain several refinements and implications, including on
optimizing sensitivity parameter in the risk sensitive formulation, and
further structural and quantitative properties.

###  4.1 Optimizing sensitivity

Given $\alpha\in\mathbb{R}$ and functional $C$ we can consider the expression

$F(\alpha,\underline{\gamma}):=\frac{1}{\alpha}\log
E_{\mu_{0}}\left[\exp\left(\alpha C(\underline{\gamma})\right)\right].$

In light of the following proposition, $\alpha$ can be seen to represent the
“amount” of risk sensitivity.

######  Proposition 4.1 ([16], Proposition 3.1).

If $C$ is essentially bounded, we have that

$\displaystyle\lim_{\alpha\to\infty}F(\alpha,\underline{\gamma})$
$\displaystyle=\operatorname{esssup}[C(\underline{\gamma})]$
$\displaystyle\lim_{\alpha\to 0}F(\alpha,\underline{\gamma})$
$\displaystyle=E_{\mu_{0}}[C(\underline{\gamma})]$
$\displaystyle\lim_{\alpha\to-\infty}F(\alpha,\underline{\gamma})$
$\displaystyle=\operatorname{essinf}[C(\underline{\gamma})],$

The limits representing the infinite risk-averse (where only the worst case
matters), risk-neutral and infinite risk-seeking (where only the best case
matters) cases, respectively. In this sense, $\alpha$ parameterizes the
“amount” of risk sensitivity. If we apply Donsker-Varadhan to $F$, we have

$F(\alpha,\underline{\gamma})=\sup_{\mu\sim\mu_{0}}\left\\{E_{\mu}[C(\underline{\gamma})]-\frac{1}{\alpha}D_{KL}(\mu||\mu_{0})\right\\},$

and for fixed $\mu\sim\mu_{0}$ we have the robustness bound

$E_{\mu}[C(\underline{\gamma})]\leq
F(\alpha,\underline{\gamma})+\frac{1}{\alpha}D_{KL}(\mu||\mu_{0})=:\psi(\alpha,\underline{\gamma}),$
(17)

giving an extra degree of freedom $\alpha$. The following proposition gives a
qualitative expression for the optimal $\alpha$.

######  Theorem 4.2.

For fixed $\underline{\gamma}$ if $\alpha^{\ast}$ is a minimizer of
$\psi(\alpha,\underline{\gamma})$ the bound (17) reduces to

$E_{\mu}[C]\leq E_{\hat{\mu}}[C],$

where

$\frac{d\hat{\mu}}{d\mu_{0}}=\frac{1}{E_{\mu_{0}}[e^{\alpha^{\ast}C}]}e^{\alpha^{\ast}C}$

is the solution to the variational problem

$\sup_{\nu\sim\mu_{0}}\left\\{E_{\nu}[C(\underline{\gamma})]-\frac{1}{\alpha}D_{KL}(\nu||\mu_{0})\right\\}.$

Additionally, we have that

$D_{KL}(\hat{\mu}||\mu_{0})=D_{KL}(\mu||\mu_{0}).$

###### Proof.

$\psi$ is differentiable in $\alpha$, with derivative

$\displaystyle\psi_{\alpha}(\alpha,\underline{\gamma})$
$\displaystyle=\frac{1}{\alpha}\frac{1}{E_{\mu_{0}}[e^{\alpha
C}]}E_{\mu_{0}}[e^{\alpha C}C]-\frac{1}{\alpha^{2}}\log E_{\mu_{0}}[e^{\alpha
C}]-\frac{1}{\alpha^{2}}D_{KL}(\mu||\mu_{0})$
$\displaystyle=\frac{1}{\alpha}\left(E_{\hat{\mu}}[C]-\psi(\alpha,\underline{\gamma})\right),$

where

$\frac{d\hat{\mu}}{d\mu_{0}}=\frac{1}{E_{\mu_{0}}[e^{\alpha^{\ast}C}]}e^{\alpha^{\ast}C}$

is the solution to the variational problem

$\sup_{\nu\sim\mu_{0}}\left\\{E_{\nu}[C(\underline{\gamma})]-\frac{1}{\alpha}D_{KL}(\nu||\mu_{0})\right\\}.$

If $\alpha^{\ast}=\alpha^{\ast}(\underline{\gamma})$ is a minima it satisfies

$\psi_{\alpha}(\alpha^{\ast},\underline{\gamma})=\frac{1}{\alpha^{\ast}}\left(E_{\hat{\mu}}[C]-\psi(\alpha^{\ast},\underline{\gamma})\right)=0.$

In this case, the robustness bound (17) becomes

$E_{\mu}[C]\leq E_{\hat{\mu}}[C].$

Additionally, plugging in $\alpha^{\ast}$ to $\psi$ gives

$\displaystyle\psi(\alpha^{\ast},\underline{\gamma})$
$\displaystyle=\sup_{\nu\sim\mu_{0}}\left\\{E_{\nu}[C(\underline{\gamma})]-\frac{1}{\alpha^{\ast}}D_{KL}(\nu||\mu_{0})\right\\}+\frac{1}{\alpha^{\ast}}D_{KL}(\mu||\mu_{0})$
$\displaystyle=E_{\hat{\mu}}[C(\underline{\gamma})]-\frac{1}{\alpha^{\ast}}D_{KL}(\hat{\mu}||\mu_{0})+\frac{1}{\alpha^{\ast}}D_{KL}(\mu||\mu_{0})$
$\displaystyle=\psi(\alpha^{\ast},\underline{\gamma})-\frac{1}{\alpha^{\ast}}D_{KL}(\hat{\mu}||\mu_{0})+\frac{1}{\alpha^{\ast}}D_{KL}(\mu||\mu_{0}),$

implying that

$D_{KL}(\hat{\mu}||\mu_{0})=D_{KL}(\mu||\mu_{0}).$

∎

###  4.2 Risk-sensitive control of non-symmetrically interacting particles
with one that is symmetrically coupled

As a corollary of the above, we have the following.

######  Corollary 4.3.

Let $\mu$ correspond to a system model A or B with non-symmetric interaction
terms $v_{t}^{i}$ but with a symmetric mean-field cost $C$. Then we can bound
the non-symmetric problem
$\inf_{\underline{\gamma}}E_{\mu}[C(\underline{\gamma})]$ by a symmetric
problem $\inf_{\underline{\gamma}}E_{\hat{\mu}}[C(\underline{\gamma})]$, where
$\frac{d\hat{\mu}}{d\mu_{0}}\propto e^{\alpha^{\ast}C(\underline{\gamma})}$
for some $\alpha^{\ast}\in\mathbb{R}$.

The benefit of such a result is that the classical mean-field theory is now
applicable as both the interaction terms and the cost terms are now symmetric
and coupled via mean-field under the nominal model; this would not have been
possible if the dynamics were not symmetrically coupled.

###  4.3 Stability of risk-sensitive solutions to small interactions

Optimal risk-sensitivity can lead to convergence of the robustness bound (17)
to an equality. By choosing $\alpha$ appropriately, we can recover the risk-
neutral problem in the small interaction limit.

######  Proposition 4.4.

Consider the setup of Theorem 3.6 or 3.8 with
$v_{t,n}^{i}=\varepsilon_{n}u_{t}^{i}$ and $\varepsilon_{n}\to 0$. Then there
is a choice of $\alpha_{n}$ so that relation (17) is an equality in the limit
as $n\to\infty$.

###### Proof.

Using Lemma 3.5, (17) becomes

$E_{\mu^{\varepsilon_{n}}}[C(\underline{\gamma})]\leq
F(\alpha,\underline{\gamma})+\frac{\varepsilon_{n}^{2}}{\alpha}D_{KL}(\mu^{1}||\mu_{0}),$
(18)

where $\mu^{\varepsilon}$ is the measure corresponding to system model A or B
with $v_{t}^{i}=\varepsilon u_{t}^{i}$ and $\mu^{1}$ is the law of system
model A or B with $\varepsilon=1$. In this case, we may let $\alpha_{n}$ be so
that $\lim_{n\to\infty}\frac{\varepsilon_{n}^{2}}{\alpha_{n}}=0$ and
$\lim_{n\to\infty}\alpha_{n}=0$. Taking the limit $n\to\infty$ in equation
(18) concludes the proof. ∎

###  4.4 The mean-field limit as the nominal model

For a mean-field model, in the limit of large $N$, if we have that the agents
apply symmetric policies, then it is known that the strategic measure
converges, in some sense, to a collection of independent product measures [21,
22, 28, 29]. Accordingly, it may be more appropriate, for some applications,
to consider such a mean-field limit as the reference model. Here, the
robustness can be regarded with regard to an uncertainty on the number of
agents in the model but with the information that the number is not small.

Notably, consider the model:

$d\tilde{X}^{i}_{t}=(b(\tilde{X}^{i}_{t},\zeta^{N}_{t},U^{i}_{t})+\sigma(\tilde{X}^{i}_{t})v_{t}^{1})dt+\sigma(\tilde{X}^{i}_{t})dB^{i}_{t},\quad
i=1,\cdots,N$

where we assume that $X^{i}_{0}$ are i.i.d. random variables with finite
second moments. We write the above as:

$d\tilde{X}^{i}_{t}=b(\tilde{X}^{i}_{t},\mu_{t},U^{i}_{t})dt+\bigg{(}b(\tilde{X}^{i}_{t},\zeta^{N}_{t},U^{i}_{t})-b(\tilde{X}^{i}_{t},\mu_{t},U^{i}_{t})\bigg{)}dt+\sigma(\tilde{X}^{i}_{t})dB^{i}_{t},\quad
i=1,\cdots,N$

which is now to be studied as an instance of the model considered earlier in
(4) with

$v_{t}^{i}=\bigg{(}b(\tilde{X}^{i}_{t},\zeta^{N}_{t},U^{i}_{t})-b(\tilde{X}^{i}_{t},\mu_{t},U^{i}_{t})\bigg{)},$

so that we have the decoupled model

$dX^{i}_{t}=b(X^{i}_{t},\mu_{t},U^{i}_{t})dt+\sigma(X^{i}_{t})dB^{i}_{t},\quad
i=1,\cdots,N$ (19)

Note that, here if $\mu_{t}$ is the probability measure defined by $X^{i}_{t}$
under an identical Markov policy applied by each agent, then (19) would reduce
to what is known as the McKean-Vlasov dynamics. Thus, models of the type above
are known to be related to solutions of centralized McKean-Vlasov control
problems under an apriori symmetry in control policies [26]). It is known that
$\zeta^{N}$ concentrates around $\mu$ as $N$ increases with explicit
convergence rates.

Thus, the deviation in the empirical mean from the asymptotic limit of the
empirical mean serves as the coupling term. In view of this, our risk-
sensitivity bound writes as, by adapting (3.6) and (17), for any $\alpha>0$
and any policy $\underline{\gamma}$:

$\begin{split}&\limsup_{N\to\infty}E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]\\\
&\leq\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]+\frac{1}{2\sigma^{2}}\frac{1}{\alpha}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v^{1}_{s})^{2}ds\right].\end{split}$
(20)

We will focus on the first term on the right hand side in equation (20) and
vary $\alpha$. We note that $\mu_{t}$ will also change depending on the
parameters as the control policies will change.

Now, consider

$\lim_{\alpha\to 0}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]\bigg{)},$

where the limit is well-defined as the expression is monotonically increasing
in $\alpha$. By interchanging limit and $\inf$ we get the lower bound

$\displaystyle\inf_{\underline{\gamma}\in\Gamma}\lim_{\alpha\to
0}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]\bigg{)}$
$\displaystyle\geq\limsup_{\alpha\to
0}\inf_{\underline{\gamma}\in\Gamma}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]\bigg{)}.$ (21)

If we have a policy sequence $\underline{\gamma}_{n}$ which attains the
infimum for a sequence of $\alpha_{n}\downarrow 0$, this implies that

$\limsup_{\alpha_{n}\to
0}\inf_{\underline{\gamma}\in\Gamma}\bigg{(}\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha_{n}}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha_{n}Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]\bigg{)}$

may provide a bound on the optimal cost provided that the relative entropy
term in (20) can be cautiously studied via a measure concentration analysis as
$N\to\infty$.

Let

$J_{\alpha}(\underline{\gamma}):=\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]$

and

$\inf_{\underline{\gamma}}J_{\alpha^{n}}(\underline{\gamma})=J_{\alpha^{n}}(\underline{\gamma}^{n}),$

be so that $\underline{\gamma}^{n}$ is optimal for $\alpha_{n}$.

We will consider a problem in which the systems are only coupled through the
dynamics and not via the cost. Accordingly, we consider instead of (6) the
cost function

$c_{N}(X^{[1,N]},U^{[1,N]})=\frac{1}{N}\sum_{i=1}^{N}c\left(X^{i},U^{i}\right)$
(22)

For such problems, for any $\alpha_{n}$, since the dynamics are decoupled, an
optimal policy for $J_{\alpha}(\underline{\gamma})$ will be Markovian (see
e.g. Section 5). This will be true even if we allow for policies that are
centralized, that is $\sigma(X^{1}_{[0,t]},\cdots,X^{N}_{[0,t]})$-measurable.
We also assume the following.

######  Assumption 4.5.

Uniform over $x^{i},u^{i}$, we have

$|b(x^{i},\zeta^{N},u^{i})-b(x^{i},\mu,u^{i})|\leq
CW_{1}(\zeta^{N}_{t},\mu_{t}),$

where $W_{1}$ is the Wasserstein metric of order $1$ and $C<\infty$.

######  Proposition 4.6.

Let $U^{i}$ be Markovian policies only depending on $X^{i}$ and let $\sigma$
be constant. Let $b$ be so that

$\sup_{t\geq 0,x\neq
y,m\in\mathcal{P}(\mathbb{R}),\underline{\gamma}\in\Gamma}\left\\{\frac{|b(t,x,U^{i},m)-b(t,y,U^{i},m)|}{|x-y|}\right\\}<\infty,$
(23)

and assume Assumption 4.5. Assume that $b$ is twice differentiable with
respect to $m$ with Lipschitz derivatives uniform in the other variables. Also
assume that $\mu$ has greater than $4$ moments. Then we have that

$\frac{1}{2\sigma^{2}}\frac{1}{\alpha_{n}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v^{1}_{s})^{2}ds\right]\leq
K\frac{1}{2\sqrt{N}\alpha_{n}\sigma^{2}},$ (24)

where $K$ is independent of $U$ and $N$.

###### Proof.

Define $b_{U}(t,x,m)=b(t,x,U,m)$. Then we have by the assumptions on $b$ and
[14, Theorem 3.8] that

$\displaystyle\frac{1}{2\sigma^{2}}\frac{1}{\alpha_{n}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v^{1}_{s})^{2}ds\right]$
$\displaystyle\leq\frac{1}{2\sigma^{2}}\frac{1}{\alpha_{n}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(W_{1}(\zeta_{t}^{N},\mu_{t}))^{2}ds\right]$
$\displaystyle\leq\frac{C}{2\sigma^{2}}\frac{1}{\alpha_{n}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(W_{2}(\zeta_{t}^{N},\mu_{t}))^{2}ds\right]$
$\displaystyle\leq\frac{CK_{U}^{+}T}{2\sigma^{2}}\frac{1}{\alpha_{n}\sqrt{N}},$

where $K_{U}^{+}$ is independent of $N$. By Jensen’s inequality, $W_{2}\geq
W_{1}$ in the above. By the assumptions on $b_{U}$, we have that
$K:=\sup_{U}CTK_{U}^{+}<\infty$. ∎

Accordingly, for any symmetric policy, $\alpha_{n}$ (depending on $N$), can be
designed so that

$\frac{1}{2\sigma^{2}}\frac{1}{\alpha_{n}}E^{\underline{\gamma}}_{\mu}\left[\int_{0}^{T}(v^{1}_{s})^{2}ds\right]\to
0,$

due to measure concentration involving copies of i.i.d. stochastic flows.

Thus, if we can show that $\underline{\gamma}_{n}$ has a converging
subsequence and the cost is lower semi-continuous, then we will be able to
establish the optimality of symmetric policies. Now, we have that on the space
of Markov policies the map $\underline{\gamma}\mapsto X_{[0,T]}$ is weakly
continuous when the policy space is endowed with the Borkar topology [10, 32]
and this applies also for resulting McKean-Vlasov model given in (19) where
$\mu_{t}$ is the law of $X_{t}$ under Assumption 4.5 by [11, Theorem 4.18]
(note that by Jensen’s inequality $W_{2}\geq W_{1}$ thus [11, Theorem 4.18] is
applicable under Assumption 4.5).

Accordingly, for any $\alpha$ the cost is lower semi-continuous in the
control. Furthermore, the space of policies is compact under this topology
[10]. Accordingly,

$\displaystyle\liminf_{\alpha_{n}\downarrow
0}J_{\alpha_{n}}(\underline{\gamma}^{n})$
$\displaystyle\geq\liminf_{\alpha_{n}\downarrow
0}J_{0}(\underline{\gamma}^{n})$ $\displaystyle\geq
J_{0}(\underline{\gamma}^{\infty})$ (25)

for any limit point $\underline{\gamma}^{\infty}$ of the sequence
$\underline{\gamma}^{n}$. The first inequality follows since the cost is
increasing in the risk-parameter $\alpha$ for a given policy. This follows
from the fact that if we view

$\displaystyle F(\alpha,\underline{\gamma})$ $\displaystyle:=$
$\displaystyle\frac{1}{\alpha}\log E_{\mu_{0}}\left[\exp\left(\alpha
C(\underline{\gamma})\right)\right].$
$\displaystyle=\log\bigg{(}E_{\mu_{0}}\left[\bigg{(}\exp\left(C(\underline{\gamma})\right)\bigg{)}^{\alpha}\right]\bigg{)}^{\frac{1}{\alpha}},$

and observing that the $p$-norm of a random variable is non-decreasing in $p$
for $p>0$, which follows from Jensen’s inequality.

The second inequality is due to lower semi-continuity in the control policies
(since the path measure is weakly continuous in the policy as discussed
earlier, and the cost is lower semi-continuous in the path measure).

Furthermore, since $\underline{\gamma}^{\infty}$ is symmetric, the
inequalities in (4.4) are all equalities since we have:

$\displaystyle J_{0}(\underline{\gamma}^{\infty})$
$\displaystyle=\lim_{\alpha_{n}\downarrow
0}J_{\alpha_{n}}(\underline{\gamma}^{\infty})$
$\displaystyle=\liminf_{\alpha_{n}\downarrow
0}J_{\alpha_{n}}(\underline{\gamma}^{\infty})$
$\displaystyle\geq\liminf_{\alpha_{n}\downarrow
0}J_{\alpha_{n}}(\underline{\gamma}^{n})$
$\displaystyle\geq\liminf_{\alpha_{n}\downarrow
0}J_{0}(\underline{\gamma}^{n})$ $\displaystyle\geq
J_{0}(\underline{\gamma}^{\infty})$

Accordingly, given (24), an agent-wise weak limit of $\underline{\gamma}_{n}$,
$\underline{\gamma}^{\infty}$, will satisfy the relation:

$\displaystyle\inf_{\underline{\gamma}}\limsup_{N\to\infty}E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]\leq
J_{0}(\underline{\gamma}^{\infty}).$ (26)

Finally, observe that

$\displaystyle J_{0}(\underline{\gamma}^{\infty})$
$\displaystyle=\lim_{\alpha\to
0}\limsup_{N\to\infty}\frac{1}{N}\frac{1}{\alpha}\log
E^{\underline{\gamma}^{\infty}}_{\mu_{0}}\left[e^{\alpha
Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right]$
$\displaystyle=\limsup_{N\to\infty}\frac{1}{N}E^{\underline{\gamma}^{\infty}}_{\mu_{0}}\left[c_{N}(\omega^{[1,N]},U^{[1,N]})\right]=\limsup_{N\to\infty}\frac{1}{N}E^{\underline{\gamma}^{\infty}}_{\mu}\left[c_{N}(\omega^{[1,N]},U^{[1,N]})\right]$
(27)

since in the limit given a symmetric policy, the attained expected cost values
under the true model and the reference model, are identical.

######  Theorem 4.7.

Consider the model (4.4) and cost function (22), with Assumption 4.5. In the
limit of large $N$, a decentralized policy which only uses local information,
and identical across all agents, solving (19) optimizes the upper bound (20)
(among all admissible, possibly centralized, policies).

The result presented above is related to and consistent with the recent
results in [40], with a robustness angle for finite numbers of agents: In both
discrete-time [37, 38] and continuous-time [26, 40], it has been shown that
for problems of the type with (4.4) and cost function (22), optimal policies
are symmetric and independent. Related risk-sensitive problems have been
considered in [7, 30, 34, 35, 41] in continuous-time.

###  4.5 The discrete-time setup

We note that the same approach can be applied to the discrere-time setup. For
the discrete-time setup, the uncoupled dynamics also exhibit some structural
properties which allow for an easier computation of robust solutions. Several
reduction results to decoupled teams have been studied, see e.g. [45, Chapter
3], where a coupled stochastic team is transformed into a decoupled one by
change of measure arguments generalizing [43].

In particular, an uncoupled risk-sensitive problem may be convex [46] whereas
the coupled model is not. Under convexity and exchangeability, one can show
that optimal policies are symmetric [37].

##  5 Example: Infinite Horizon Average Cost Criterion and Robustness via an
HJB equation

The analysis in the paper motivates the study of the risk-sensitive criterion:

$\inf_{\underline{\gamma}\in\Gamma}\limsup_{N\to\infty}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0}}\left[e^{Nc_{N}(\omega^{[1,N]},U^{[1,N]})}\right].$

We will study an explicit example in the following. Consider the model given
in System Model A with identical particles in (3). Suppose that the goal is to
minimize the mean-field cost:

$J_{N}(\underline{\gamma}_{N})=\limsup_{T\to\infty}\frac{1}{T}E^{\underline{\gamma}_{N}}_{\mu_{0}^{N,T}}\left[c^{T}_{N}(X^{[1,N]},U^{[1,N]})\right],$

with

$c^{T}_{N}(X^{[1,N]},U^{[1,N]})=\frac{1}{N}\sum_{i=1}^{N}c^{T}\left(X^{i},U^{i}\right),$
(28)

where

$c^{T}\left(X^{i},U^{i}\right)=\int_{0}^{T}g(X^{i}_{s},U^{i}_{s})ds$

where $\underline{\gamma}_{N}=(\gamma^{1},...,\gamma^{N})$ with
$U^{i}_{t}=\gamma^{i}(X^{i}_{[0,t]})$. We are also interested in the limit

$J(\underline{\gamma})=\limsup_{N\to\infty}J_{N}(\underline{\gamma}).$ (29)

Note that here we consider a system where there is coupling in the dynamics
but not in the cost.

Following steps similar to that in the proof of Theorem 3.6, consider the
expression

$L:=\inf_{\underline{\gamma}\in\Gamma}\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0}^{N,T}}\left[e^{Nc^{T}_{N}(\omega^{[1,N]},U^{[1,N]})}\right].$
(30)

Using Donsker-Varadhan (see [8]), we have the variational expression

$L=\inf_{\underline{\gamma}\in\Gamma}\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\sup_{\mu^{N}\sim\mu_{0}^{N}}\left\\{E^{\underline{\gamma}}_{\mu}[c_{N}(\omega^{[1,N]},U^{[1,N]})]-\frac{1}{N}D_{KL}(\mu^{N}||\mu_{0}^{N})\right\\}.$

For each fixed $\gamma\in\Gamma$ and $\mu^{N,T}\sim\mu_{0}^{N,T}$,

$\displaystyle\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0}^{N,T}}\left[e^{Nc^{T}_{N}(\omega^{[1,N]},U^{[1,N]})}\right]$
$\displaystyle\geq\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\left\\{E^{\underline{\gamma}}_{\mu}[c^{T}_{N}(\omega^{[1,N]},U^{[1,N]})]-D_{KL}(\mu^{N,T}||\mu_{0}^{N,T})\right\\}.$
$\displaystyle\geq\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\left\\{E^{\underline{\gamma}}_{\mu}[c^{T}_{N}(\omega^{[1,N]},U^{[1,N]})]\right\\}-\limsup_{T\to\infty}\frac{1}{T}\left\\{\frac{1}{N}D_{KL}(\mu^{N,T}||\mu_{0}^{N,T})\right\\}.$
(31)

This leads to, for each $\gamma\in\Gamma$

$\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\left\\{E^{\underline{\gamma}}_{\mu}[c^{T}_{N}(\omega^{[1,N]},U^{[1,N]})]\right\\}\leq\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\log
E^{\underline{\gamma}}_{\mu_{0}^{N,T}}\left[e^{Nc^{T}_{N}(\omega^{[1,N]},U^{[1,N]})}\right]+\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}D_{KL}(\mu^{N,T}||\mu_{0}^{N,T})$

where we again note that $\mu$ depends on the policy. As the above applies for
each admissible $\gamma$, it implies that

$\inf_{\underline{\gamma}\in\Gamma}\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\left\\{E^{\underline{\gamma}}_{\mu}[c^{T}_{N}(\omega^{[1,N]},U^{[1,N]})]\right\\}\leq
L+\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}D_{KL}(\mu_{\gamma^{*}}^{N,T}||\mu_{0,\gamma^{*}}^{N,T}),$

where $\gamma^{*}$ is a policy attaining $L$.

For this system, due to the independence of the measures under
$\mu_{0}^{N,T}$, the risk-sensitive criterion

$\limsup_{T\to\infty}\frac{1}{T}\frac{1}{N}\log\bigg{(}E^{\gamma}[e^{\sum_{i=1}^{N}\int_{0}^{T}c(X^{i}_{s},U^{i}_{s})ds}]\bigg{)}$

reduces to the minimization (via the properties of the logarithm) of

$\limsup_{T\to\infty}\frac{1}{T}\log\bigg{(}E^{\gamma}[e^{\int_{0}^{T}c(X^{i}_{s},U^{i}_{s})ds}]\bigg{)},$

for each of the decoupled particles. The solution to this minimization, under
growth and regularity conditions on $b$ and $\sigma$, has been reported in [2,
Theorems 4.1 and 4.2], and is obtained via the following HJB equation:

$\displaystyle\min_{u\in\mathbb{U}}\bigg{\\{}{\cal
A}\psi(x,u)+c(x,u)\psi(x)\bigg{\\}}=\lambda^{*}\psi(x),\quad\psi(0)=1,$ (32)

where ${\cal A}$ is the generator of the controlled diffusion given in System
Model A (with identical particle dynamics) and $\lambda^{*}$ is the optimal
cost (known as the principal eigenvalue of the HJB equation above). Let
$\gamma^{*}$ solve the HJB equation. In particular, $\gamma^{*}$ is a
stationary Markov policy.

Accordingly, the robustness bound obtained leads to the following.

######  Theorem 5.1.

Consider the coupled model (3) and cost criterion 29. The following holds

$\inf_{\gamma}J(\underline{\gamma})\leq\lambda^{*}+\frac{1}{2\sigma^{2}}\lim_{T\to\infty}\frac{1}{T}E^{\underline{\gamma}^{*}}_{\mu}\left[\int_{0}^{T}v_{s}^{2}ds\right],$

where $\lambda^{*}$ is given in (32). The inequality holds also for the
control policy obtained via (32) for every $x$.

Thus, we are able to obtain an explicit bound, for all $N$, by approximating a
coupled problem with one that is not coupled.

##  6 Conclusion

In this paper, we showed that a decentralized or mean-field control problem
with interacting agents can be approached by considering a risk-sensitive
version of non-interacting agents/particles or particles with structural
coupling (such as symmetric), leading to a robust formulation.

In our first approach, we perturbed the noise on the infinite product space
directly. This has the benefit of allowing for general cost criteria at the
drawback of the finite energy condition defined in equation (8) that in
particular does not allow for symmetric interaction.

In the second approach, we perturbed the law of the agents on the finite
product space. This eliminates the need for the finite energy condition
defined in equation (8) and allows for symmetric interaction.

We obtained bounds on the value of the cost function in terms of the risk-
sensitive cost function for the noninteracting case plus a term involving the
strength of the interaction characterized by relative entropy. We emphasize
the mathematical flexibility afforded when the interaction is symmetric, where
in the absence of symmetry a bounded energy condition is needed whereas in the
symmetric setup this is not.

## Acknowledgments

We would like to gratefully acknowledge Alex Dunlap for a discussion on
optimal risk sensitivity which led to Section 4.

## References

  * [1] V. Anantharam and V. S. Borkar.  A variational formula for risk-sensitive reward.  SIAM Journal on Control and Optimization, 55(2):961–988, 2017.
  * [2] A. Arapostathis, A. Biswas, and S. Saha.  Strict monotonicity of principal eigenvalues of elliptic operators in rd and risk-sensitive control.  Journal de Mathématiques Pures et Appliquées, 124:169–219, 2019.
  * [3] A. Arapostathis, V. S. Borkar, and M. K. Ghosh.  Ergodic Control of Diffusion Processes, volume 143.  Cambridge University Press, 2012.
  * [4] T. Başar and P. Bernhard.  H-infinity Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach.  Birkhäuser, Boston, MA, 1995.
  * [5] T. Başar.  Robust designs through risk sensitivity: An overview.  Journal of Systems Science and Complexity, 34(5):1634–1665, 2021\.
  * [6] V. E. Beneš.  Existence of optimal stochastic control laws.  SIAM Journal on Control, 9(3):446–472, 1971.
  * [7] A. Bensoussan, B. Djehiche, H. Tembine, and P. Yam.  Risk-sensitive mean-field-type control.  In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 33–38. IEEE, 2017.
  * [8] Joris Bierkens and Hilbert J. Kappen.  Explicit solution of relative entropy weighted control.  Systems Control Lett., 72:36–43, 2014.
  * [9] Vladimir I. Bogachev.  Gaussian measures, volume 62 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, 1998.
  * [10] V. S. Borkar.  A topology for Markov controls.  Applied Mathematics and Optimization, 20(1):55–62, 1989.
  * [11] R. Carmona and F. Delarue.  Probabilistic theory of mean field games with applications I-II.  Springer, 2018.
  * [12] R. Carmona, F. Delarue, and D. Lacker.  Mean field games with common noise.  The Annals of Probability, 44(6):3740–3803, 2016.
  * [13] C. D. Charalambous.  Decentralized optimality conditions of stochastic differential decision problems via Girsanov’s measure transformation.  Mathematics of Control, Signals, and Systems, 28(3):1–55, 2016\.
  * [14] P.-E. C. de Raynal and N. Frikha.  From the backward kolmogorov pde on the wasserstein space to propagation of chaos for mckean-vlasov sdes.  Journal de Mathématiques Pures et Appliquées, 156:1–124, 2021.
  * [15] P. G. Dupuis, M. R. James, and I. Petersen.  Robust properties of risk-sensitive control.  Mathematics of Control, Signals and Systems, 13(4):318–332, 2000\.
  * [16] Hans Föllmer and Thomas Knispel.  Entropic risk measures: coherence vs. convexity, model ambiguity, and robust large deviations.  Stoch. Dyn., 11(2-3):333–351, 2011.
  * [17] Martin Hairer.  An introduction to stochastic pdes, 2009.
  * [18] J. Huang, G. Wang, and W. Wang.  A general linear quadratic stochastic control and information value.  Journal of Mathematical Analysis and Applications, page 126486, 2022\.
  * [19] M. Huang, P. E. Caines, and R. P. Malhamé.  Large population stochastic dynamic games: closed-loop Mckean-Vlasov systems and the Nash certainty equivalence principle.  Communications in Information and Systems, 6:221–251, 2006.
  * [20] M. Huang, P. E. Caines, and R. P. Malhamé.  Large-population cost-coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized $\epsilon$-Nash equilibria.  IEEE Transactions on Automatic Control, 52:1560–1571, 2007.
  * [21] Pierre-Emmanuel Jabin and Zhenfu Wang.  Mean field limit and propagation of chaos for Vlasov systems with bounded forces.  J. Funct. Anal., 271(12):3588–3627, 2016.
  * [22] Pierre-Emmanuel Jabin and Zhenfu Wang.  Quantitative estimates of propagation of chaos for stochastic systems with $W^{-1,\infty}$ kernels.  Invent. Math., 214(1):523–591, 2018.
  * [23] J. Jackson and D. Lacker.  Approximately optimal distributed stochastic controls beyond the mean field setting.  arXiv preprint arXiv:2301.02901, 2023.
  * [24] D. Jacobson.  Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games.  IEEE Transactions on Automatic control, 18(2):124–131, 1973.
  * [25] D. Lacker.  Mean field games via controlled martingale problems: existence of Markovian equilibria.  Stochastic Processes and their Applications, 125(7):2856–2894, 2015\.
  * [26] D. Lacker.  Limit theory for controlled McKean–Vlasov dynamics.  SIAM Journal on Control and Optimization, 55(3):1641–1672, 2017\.
  * [27] J. M. Lasry and P. L. Lions.  Mean field games.  Japanese J. of Mathematics, 2:229–260, 2007.
  * [28] Tau Shean Lim, Yulong Lu, and James H. Nolen.  Quantitative propagation of chaos in a bimolecular chemical reaction-diffusion model.  SIAM J. Math. Anal., 52(2):2098–2133, 2020.
  * [29] F. Malrieu.  Logarithmic Sobolev inequalities for some nonlinear PDE’s.  Stochastic Process. Appl., 95(1):109–132, 2001.
  * [30] J. Moon and T. Başar.  Linear quadratic risk-sensitive and robust mean field games.  IEEE Transactions on Automatic Control, 62(3):1062–1077, 2016.
  * [31] I. Petersen, M. R. James, and P. Dupuis.  Minimax optimal control of stochastic uncertain systems with relative entropy constraints.  IEEE Transactions on Automatic Control, 45(3):398–412, 2000.
  * [32] S. Pradhan and S. Yüksel.  Continuity of cost in Borkar control topology and implications on discrete space and time approximations for controlled diffusions under several criteria.  Electronic Journal of Probability, 29:1–32, 2024.
  * [33] Somnath Pradhan and Serdar Yüksel.  Controlled diffusions under full, partial and decentralized information: Existence of optimal policies and discrete-time approximations.  arXiv preprint arXiv:2311.03254, 2023.
  * [34] N. Saldi, T. Başar, and M. Raginsky.  Approximate markov-nash equilibria for discrete-time risk-sensitive mean-field games.  Mathematics of Operations Research, 45(4):1596–1620, 2020.
  * [35] N. Saldi, T. Başar, and M. Raginsky.  Partially observed discrete-time risk-sensitive mean field games.  Dynamic Games and Applications, pages 1–32, 2022.
  * [36] N. Saldi and S. Yüksel.  Geometry of information structures, strategic measures and associated control topologies.  Probability Surveys, 19:450–532, 2022.
  * [37] S. Sanjari, T. Başar, and S. Yüksel.  Isomorphism properties of optimality and equilibrium solutions under equivalent information structure transformations: Stochastic dynamic games and teams.  SIAM Journal on Control and Optimization, 61(5):3102–3130, 2023\.
  * [38] S. Sanjari, N. Saldi, and S. Yüksel.  Optimality of independently randomized symmetric policies for exchangeable stochastic teams with infinitely many decision makers.  Mathematics of Operations Research, 48(3):1254–1285, 2023.
  * [39] S. Sanjari, N. Saldi, and S. Yüksel.  Optimality of decentralized symmetric policies for stochastic teams with mean-field information sharing.  arXiv preprint arXiv:2404.04957, 2024.
  * [40] Sina Sanjari, Naci Saldi, and Serdar Yüksel.  Decentralized exchangeable stochastic dynamic teams in continuous-time, their mean-field limits and optimality of symmetric policies.  arXiv preprint arXiv:2404.15581, 2024.
  * [41] H. Tembine, Q. Zhu, and T. Başar.  Risk-sensitive mean-field games.  IEEE Transactions on Automatic Control, 59(4):835–850, 2013.
  * [42] H. S. Witsenhausen.  A counterexample in stochastic optimal control.  SIAM J. Contr., 6:131–147, 1968.
  * [43] H. S. Witsenhausen.  Equivalent stochastic control problems.  Math. Control, Signals and Systems, 1:3–11, 1988.
  * [44] S. Yüksel.  A universal dynamic program and refined existence results for decentralized stochastic control.  SIAM Journal on Control and Optimization, 58(5):2711–2739, 2020\.
  * [45] S. Yüksel and T. Başar.  Stochastic Teams, Games, and Control under Information Constraints.  Springer, Cham, 2024.
  * [46] S. Yüksel and N. Saldi.  Convex analysis in decentralized stochastic control, strategic measures and optimal solutions.  SIAM J. on Control and Optimization, 55:1–28, 2017.