# Large population games with interactions through controls and common noise:
convergence results and equivalence between $open$–$loop$ and $closed$–$loop$
controls

Mao Fabrice Djete111Ecole Polytechnique Paris, Centre de Mathématiques
Appliquées<EMAIL_ADDRESS>
###### Abstract

In the presence of a common noise, we study the convergence problems in mean
field game (MFG) and mean field control (MFC) problem where the cost function
and the state dynamics depend upon the joint conditional distribution of the
controlled state and the control process. In the first part, we consider the
MFG setting. We start by recalling the notions of $measure$–$valued$ MFG
equilibria and of approximate $closed$–$loop$ Nash equilibria associated to
the corresponding $N$–player game. Then, we show that all convergent sequences
of approximate $closed$–$loop$ Nash equilibria, when $N\to\infty,$ converge to
$measure$–$valued$ MFG equilibria. And conversely, any $measure$–$valued$ MFG
equilibrium is the limit of a sequence of approximate $closed$–$loop$ Nash
equilibria. In other words, $measure$–$valued$ MFG equilibria are the
accumulation points of the approximate $closed$–$loop$ Nash equilibria.
Previous work has shown that $measure$–$valued$ MFG equilibria are the
accumulation points of the approximate $open$–$loop$ Nash equilibria.
Therefore, we obtain that the limits of approximate $closed$–$loop$ Nash
equilibria and approximate $open$–$loop$ Nash equilibria are the same. In the
second part, we deal with the MFC setting. After recalling the $closed$–$loop$
and $open$–$loop$ formulations of the MFC problem, we prove that they are
equivalent. We also provide some convergence results related to approximate
$closed$–$loop$ Pareto equilibria.

##  1 Introduction

Our primarily goal in this paper is to discuss the convergence problem of
$closed$–$loop$ Nash equilibria in the setting of mean field game of controls
(MFGC) or extended mean field game. Let us briefly explained the mathematical
framework that we consider. The full details explanation are given in Section
2.1. We consider that $N$ players have private state processes
$\mathbf{X}^{N}:=(X^{1},\dots,X^{N})$ given by the stochastic differential
equations (SDEs) system

$\displaystyle\mathrm{d}X^{i}_{t}$
$\displaystyle=b\big{(}t,X^{i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}],\alpha^{i}(t,\mathbf{X}^{N}_{t})\big{)}\mathrm{d}t+\sigma\big{(}t,X^{i}_{t}\big{)}\mathrm{d}W^{i}_{t}+\sigma_{0}\mathrm{d}B_{t},\;t\in[0,T],$
(1.1) $\displaystyle\boldsymbol{\alpha}^{N}$
$\displaystyle:=(\alpha^{1},\cdots,\alpha^{N}),\;\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]:=\frac{1}{N}\sum_{i=1}^{N}\delta_{(X^{i}_{t},\alpha^{i}(t,\mathbf{X}^{N}_{t}))}~{}\mbox{and}~{}\varphi^{N}_{t}[\boldsymbol{\alpha}]:=\frac{1}{N}\sum_{i=1}^{N}\delta_{X^{i}_{t}},$
(1.2)

where $T>0$ is a fixed time horizon, $(B,W^{1},\dots,W^{N})$ are independent
Brownian motions where $B$ is called the common noise, and $\alpha^{i}$ is a
Borel measurable function playing the role of the control of player $i$. An
important feature here is the presence in the dynamics $X^{i}$ of player $i$
of the empirical distribution
$\overline{\varphi}^{N}[\boldsymbol{\alpha}^{N}]$ of states and controls of
all players. Given a strategy $(\alpha^{1},\dots,\alpha^{N}),$ the reward to
the player $i$ is

$\displaystyle
J^{N}_{i}(\alpha^{1},\dots,\alpha^{N}):=\mathbb{E}\bigg{[}\int_{0}^{T}L\big{(}t,X^{i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}],\alpha^{i}(t,\mathbf{X}^{N}_{t})\big{)}\mathrm{d}t+g\big{(}X^{i}_{T},\varphi^{N}_{T}[\boldsymbol{\alpha}]\big{)}\bigg{]}.$

For $\varepsilon_{N}\geq 0,$ the strategy $(\alpha^{1},\dots,\alpha^{N})$ will
be called an $\varepsilon_{N}$–$closed$–$loop$ Nash equilibrium if for any
admissible control $\beta,$ and each $i\in\\{1,\cdots,N\\},$

$\displaystyle J^{N}_{i}(\alpha^{1},\dots,\alpha^{N})\geq
J^{N}_{i}(\alpha^{1},...,\alpha^{i-1},\beta,\alpha^{i+1},\dots,\alpha^{N})-\varepsilon_{N}.$
(1.3)

The presence of the term $closed$–$loop$ indicates the fact that we consider
controls which are Borel maps of $[0,T]\times(\mathbb{R}^{n})^{N}$ into $U$.
The convergence problem here consists in characterizing the Nash equilibria
when the number of players $N$ goes to infinity. It is now well known that,
when $N$ tends to infinity, the Nash equilibria are related to the MFG here
called mean field game of controls (MFGC) or Extended mean field game, which
has the following structure (the precise definition is given in Section
2.2.1): for $\varepsilon\geq 0,$ a $(\sigma\\{B_{s},s\leq
t\\})_{t\in[0,T]}$–predictable measure–valued process
$(\overline{\mu}_{t})_{t\in[0,T]}$ is an $\varepsilon$–strong Markovian MFG
equilibrium (or approximate strong Markovian MFG equilibrium) if for all
$t\in[0,T],$ $\overline{\mu}_{t}={\cal
L}(X_{t},\alpha(t,X_{t},\overline{\mu}_{t})|B),$ where the state process $X$
is governed by

$\displaystyle\mathrm{d}X_{t}$
$\displaystyle=b\big{(}t,X_{t},\overline{\mu}_{t},\alpha(t,X_{t},\overline{\mu}_{t})\big{)}\mathrm{d}t+\sigma\big{(}t,X_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},,~{}~{}t\in[0,T]$
$\displaystyle\mu_{t}$ $\displaystyle:={\cal L}(X_{t}|B),$

and one has

$\displaystyle\begin{cases}\displaystyle\mathbb{E}\bigg{[}\int_{0}^{T}L(t,X_{t},\overline{\mu}_{t},\alpha(t,X_{t},\overline{\mu}_{t}))\mathrm{d}t+g(X_{T},\mu_{T})\bigg{]}\geq\sup_{\alpha^{\prime}}\;\mathbb{E}\bigg{[}\int_{0}^{T}L(t,X^{\prime}_{t},\overline{\mu}_{t},\alpha^{\prime}(t,X^{\prime}_{t},\overline{\mu}_{t}))\mathrm{d}t+g(X^{\prime}_{T},\mu_{T})\bigg{]}-\varepsilon,\\\\[6.99997pt]
\displaystyle\mbox{where
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}
the optimization is over the
solutions}}\;\;\mathrm{d}X^{\prime}_{t}=b\big{(}t,X^{\prime}_{t},\overline{\mu}_{t},\alpha^{\prime}(t,X^{\prime}_{t},\overline{\mu}_{t})\big{)}\mathrm{d}t+\sigma\big{(}t,X^{\prime}_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}.\end{cases}$
(1.4)

##### Discussion about the problematic and related studies

The now well–known MFG was introduced in the seminal works of Lasry and Lions
[45] and Huang, Caines, and Malhamé [35] as a way of studying the $N$–player
game described here above, when the number of players $N$ is very large. Since
these pioneering works, this topic has been the subject of much research in
the field of applied mathematics, in particular for its wide variety of
application (see Carmona and Delarue [13] for examples of applications of
MFG). The initial formulations of MFG did not consider the presence of the
empirical distribution of controls like that formulated above in Equation 1.1.
Only the presence of the empirical distribution of states was envisaged. In
order to deal with some modeling issues occurring in finance for instance, a
natural $``$extension$"$ of MFG known as extended MFG or MFG of controls has
been formulated and studied by many studies these recent years, see Gomes and
Voskanyan [31], Carmona and Lacker [15], Cardaliaguet and Lehalle [10],
Gomes, Patrizi, and Voskanyan [32], Bonnans, Hadikhanloo, and Pfeiffer [8],
Graber [33], Alasseur, Ben Taher, and Matoussi [1], Kobeissi [36]. Our study
in this paper treats of extended MFG (or MFG of controls) by giving some
convergence and equivalence results.

As mentioned in the beginning, our primarily goal is to make a rigorous
connection between the $N$–player game and the extended mean field game. More
precisely, ideally, we want to show two main results. First, the convergence
result i.e. given an $\varepsilon_{N}$–Nash equilibrium
$\boldsymbol{\alpha}^{N}:=(\alpha^{1,N},\cdots,\alpha^{N,N})$, the sequence of
empirical distribution of states and controls
$(\overline{\varphi}^{N}[\boldsymbol{\alpha}^{N}])_{N\in\mathbb{N}^{*}}$
$``$converges$"$ to a solution of the MFG of controls when $N\to\infty,$ with
$\displaystyle\lim_{N\to\infty}\varepsilon_{N}=0$. Second, the $converse$
convergence result i.e. any solution of the MFG of controls is the
$``$limit$"$, when $N\to\infty,$ of a sequence of empirical distribution of
states and controls
$(\overline{\varphi}^{N}[\boldsymbol{\alpha}^{N}])_{N\in\mathbb{N}^{*}}$
associated to an $\varepsilon_{N}$–Nash equilibrium
$\boldsymbol{\alpha}^{N}:=(\alpha^{1,N},\cdots,\alpha^{N,N}),$ for some
$\varepsilon_{N}$ satisfying
$\displaystyle\lim_{N\to\infty}\varepsilon_{N}=0.$

Establishing this type of connection justifies the interpretation of MFG as
the right limit formulation of the $N$–player game. Besides the presence of
the empirical distribution of states and controls, another important feature
of our setting is the consideration of $closed$–$loop$ controls i.e. controls
depending on the position of the players. Indeed, the other type of controls
usually considered is the $open$–$loop$ controls i.e. controls adapted to the
filtration generated by the initial values and the Brownian motions. When
$N$($\geq 2,$ the number of players) is fixed, the situation generated by
these two concepts of equilibrium is very different, see the discussion in
[13, Section 2.1.2].

In the setting of $open$–$loop$ controls, the connection between $N$–player
game and MFG (convergence result and $converse$ convergence result) is now
well–known and established. In the situation without the empirical
distribution of controls, under relatively general assumptions, a complete
picture has been proposed by Fisher [29] and Lacker [38] (for the case with
common noise) using the notions of relaxed MFG equilibrium. For the extended
MFG, while allowing the volatility to be controlled, a similar study to [38]
is provided by Djete [24] thanks to the notion of measure–valued MFG
equilibrium. With stronger assumptions but by providing convergence rates,
Laurière and Tangpi [46] study these convergence problems in the situation
without common noise using notably a notion of backward propagation of chaos.

The convergence problems in the case of $closed$–$loop$ controls turn out to
be much more problematic than in the case of $open$–$loop$ controls. Indeed,
as discussed in [13, Section 2.1.2], $open$–$loop$ Nash equilibrium and
$closed$–$loop$ Nash equilibrium behave differently. Without taking into
account the empirical distribution of states and controls (no extended MFG), a
first answer of the convergence problems is provided by Cardaliaguet, Delarue,
Lasry, and Lions [12, Section 3.7]. They use notably an infinite dimension PDE
associated to the limit problem, the so–called $master\;equation$. An
important work is done to ensure that the master equation is correctly
defined, in particular by using the now well–known Lasry–Lions monotonicity
condition. The master equation appears to be a powerful tools to the study of
MFG see for instance Cardaliaguet [9], Gangbo and Świech [30], Bensoussan and
Yam [4], [31], Carmona and Delarue [14], [36], Delarue, Lacker, and Ramanan
[22, 21], Bayraktar, Cecchin, Cohen, and Delarue [3], Bertucci [5, 6].
Unfortunately, for the convergence problems, the need of uniqueness of the MFG
equilibrium (through Lasry–Lions monotonicity condition) makes its use
unsatisfactory. Indeed, it is well known that there are situations where there
are multiple MFG equilibria. A breakthrough has been made by Lacker [41] for
the convergence problem of $closed$–$loop$ controls without requirement of
uniqueness of MFG equilibrium. In a non–common noise setting while imposing a
non–degenerate volatility $\sigma,$ [41] proves the convergence of
$closed$–$loop$ Nash equilibria to a $weak$ MFG equilibrium. In particular,
using some probability changes by Girsanov’s Theorem. As pointed out by
Cardaliaguet and Rainer [11], to avoid a situation like in Folk Theorem, a
non–degenerate volatility seems to be crucial for this convergence. Although
answering the question of the convergence of Nash equilibria to MFG
equilibria, [41] is not able to give a complete answer for the $converse$
convergence result. Indeed, [41] cannot show that any $weak$ MFG equilibrium
is the limit of a sequence of approximate Nash equilibria. In the same spirit,
Lacker and Flem [42] provide, first, a result relating to the convergence of
$closed$–$loop$ Nash equilibria in a common noise setting. Second, by
considering an extension of the notion of $closed$–$loop$ Nash equilibrium,
unlike [41], they are able to show the $converse$ convergence result. However,
they do not answer completely the shortcoming of [41] regarding the converse
convergence result. Indeed, while being appropriate for the study, their
extension of $closed$–$loop$ Nash equilibria is not what we can naturally
expected (see their own comment [42, Remark 2.11,Theorem 2.12]). In the
situation of MFG of controls without common noise, the only article to our
knowledge dealing with convergence issues for the $closed$–$loop$ controls is
Possamaï and Tangpi [49]. They use the result of backward propagation of chaos
of [46] to deal with this issue among many others. Nevertheless, it seems that
no many cases of non–unique MFG equilibria are considered because of their
assumptions of existence and uniqueness for a McKean–Vlasov BSDE. In this
paper, in a MFG of controls setting while allowing a common noise, we will
give a complete characterization of the $closed$–$loop$ case. The convergence
of Nash equilibria and $converse$ convergence is provided. In particular, we
are able to prove that any $weak$ MFG is the limit of Nash equilibria without
extending the notion of Nash equilibria as in [42].

Our secondarily goal is to provide some equivalence results. More precisely,
equivalence results between $open$–$loop$ and $closed$–$loop$ controls are
showed in both MFG of controls and extended mean field control (MFC). Indeed,
first, we prove that the limit of $open$–$loop$ Nash equilibria and
$closed$–$loop$ Nash equilibria are the same. Even if when the number of
players is finite both equilibria can be different, in the infinite players
setting, they are the same. Second, we show that the $open$–$loop$ formulation
and the $closed$–$loop$ formulation of the MFC problem are the same. This
result appears to be the first of this kind in the literature. Especially, we
are able to prove the equivalence without using any convexity assumptions as
it is usually done in the literature (see for instance Filippov [28], Roxin
[50], El Karoui, Huu Nguyen, and Jeanblanc-Picqué [27], Lacker [39]).

##### Main contributions

In the first part, we begin by investigating the convergence problems in the
setting of extended mean field game or mean field game of controls. Theorem
2.10 is our first result in this framework and shows two main results. First,
in an appropriate space (see Theorem 2.10 for details), the sequence of
empirical distribution
$(\varphi^{N}[\boldsymbol{\alpha}^{N}],\overline{\varphi}^{N}[\boldsymbol{\alpha}^{N}])_{N\in\mathbb{N}^{*}}$
is relatively compact, and when $\varepsilon_{N}\to 0,$ any limit point is a
measure–valued MFG solution (see Definition 2.7). Recall that the notion of
measure–valued MFG solution is similar to the one used in [24] (see
equivalence in Proposition 3.5 and Remark 3.6) to treat the convergence of
$open$–$loop$ Nash equilibria. While being the first result dealing with
convergence of $closed$–$loop$ Nash equilibria for MFG of controls with common
noise, this first result of Theorem 2.10 shows that the limits of
$open$–$loop$ and $closed$–$loop$ Nash equilibria are exactly the same. This
result is achieved by adapting the arguments of [41] in the setting of MFGC
with common noise, and by using a delicate estimate of the regularity of the
fokker planck equation proved by Aronson and Serrin [2, Theorem 4]. It is
worth mentioning that this result contains part of those of [42] in the
classical MFG framework but allow in addition $\sigma$ to be non–constant.
Second, similarly to the convergence of $closed$–$loop$ Nash equilibria,
Theorem 2.10 shows the convergence of approximate strong Markovian MFG
equilibria to the measure–valued MFG equilibria.

The second main result for the MFGC is Theorem 2.12 dealing with the
$converse$ convergence problem. We first show that any measure–valued MFG
equilibrium is the limit of sequence of approximate strong Markovian MFG
equilibria. This result combined with the convergence result of approximate
strong Markovian MFG equilibria mentioned in Theorem 2.10 proves that the
notion of approximate strong Markovian MFG equilibrium is the correct infinite
players formulation equivalent to the notion of measure–valued MFG equilibrium
or more generally the correct infinite players formulation equivalent to any
notion of weak MFG equilibrium. In other words, considering measure–valued MFG
equilibrium (or weak MFG) is equivalent to considering approximate strong
Markovian MFG equilibrium. The approximate strong Markovian MFG equilibrium
has the advantage to be close to the usual notion considered in the
literature. The only difference is for the approximate strong Markovian MFG
equilibrium, the optimality is not an exact one but an
$\varepsilon$–optimality (see above Equation 1.4). In a second time, Theorem
2.12 shows that any measure–valued MFG equilibrium is the limit of approximate
$closed$–$loop$ Nash equilibria. Even in the framework of classical MFG, this
result seems to be the first of this kind. Indeed, the $converse$ convergence
result of [42] needs to be done with a slight extension of the notion of
$closed$–$loop$ Nash equilibria which they call $S$–$closed\;loop$ Nash
equilibria. Our result shows that it is possible to avoid this extension.
Mention that Theorem 2.10 and Theorem 2.12 are done without any use of
convexity assumptions.

In the second part, we treat the case of extended mean field control problem
and obtain Theorem 2.14 and Theorem 2.15. In Theorem 2.14, we show that the
$open$–$loop$ and the $closed$–$loop$ formulation of the extended mean field
control problem are exactly the same. A notable point in this result is the
fact that there is no use of convexity assumptions. This appears to be the
first such result, even for the classical mean field control problem. Next, in
Theorem 2.15, we show some limit theory results, namely: any sequence of
approximate Pareto equilibria (see Definition 2.3) is relatively compact and
any limit point is the limit of sequence of approximate strong Markovian
McKean–Vlasov (see Section 2.3.1).

In the assumptions we use in this paper, the condition: $\sigma_{0}$ is
invertible needs to be underlined. Although we believe that there must be a
way of proving our results without this condition, in this article, this
condition is important. Indeed, under this assumption, the filtration
generated by the conditional distribution of state $(\mu_{t})_{t\in[0,T]}$ is
very $``$close$"$ to the filtration generated by the common noise
$(B_{t})_{t\in[0,T]}$. They are even equal when the control $\alpha$ has some
regularities $\alpha$ (see for instance approximation in Proposition A.8 and
Remark A.9). This fact is actually quite classic. Indeed, we know that when a
volatility $\sigma$ is non–degenerated, the unique strong solution $X$ of
$\mathrm{d}X_{t}=\tilde{b}(t,X_{t})\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}W_{t},$
$X_{0}=0,$ has his natural filtration equal to the filtration generated by the
Brownian motion $W.$ This kind of phenomenon appears in our framework. Making
the presence of a common noise obligatory allows us to prove our results
especially the $converse$ convergence result and the equivalence between
$closed$–$loop$ and $open$–$loop$ formulations for the MFC problem. This
effect of the common noise has been observed by many authors for various
purposes (see for instance Delarue [19], Tchuendom [52], [3], Delarue and
Vasileiadis [20]).

##### Outline of the paper

The paper is structured as follows. After introducing some notations, Section
2 introduces both the MFG and MFC frameworks, defines all the notions and
concepts of equilibrium, and states the main results of the article. Section 3
is devoted to the proofs. Namely, first, Section 3.1 provides, over a
canonical space, an equivalence of our notions appropriate for the proofs.
Then, Section 3.2 characterizes the limit of approximate $closed$–$loop$ Nash
equilibria. Next, Section 3.3 characterizes the limit of approximate strong
Markovian MFG equilibria and provides the $converse$ convergence result.
Finally, Section 3.4 gives the proofs relating to the McKean–Vlasov control
problem.

Notations. $(i)$ Given a metric space $(E,\Delta)$, $p\geq 1,$ we denote by
${\cal P}(E)$ the collection of all Borel probability measures on $E$, and by
${\cal P}_{p}(E)$ the subset of Borel probability measures $\mu$ such that
$\int_{E}\Delta(e,e_{0})^{p}\mu(de)<\infty$ for some $e_{0}\in E$. We equip
${\cal P}_{p}(E)$ with the Wasserstein metric ${\cal W}_{p}$ defined by

${\cal
W}_{p}(\mu,\mu^{\prime})~{}:=~{}\bigg{(}\inf_{\lambda\in\Lambda(\mu,\mu^{\prime})}\int_{E\times
E}\Delta(e,e^{\prime})^{p}~{}\lambda(\mathrm{d}e,\mathrm{d}e^{\prime})\bigg{)}^{1/p},$

where $\Lambda(\mu,\mu^{\prime})$ denote the collection of all probability
measures $\lambda$ on $E\times E$ such that $\lambda(\mathrm{d}e,E)=\mu$ and
$\lambda(E,\mathrm{d}e^{\prime})=\mu^{\prime}(\mathrm{d}e^{\prime})$. Equipped
with ${\cal W}_{p},$ ${\cal P}_{p}(E)$ is a Polish space (see [54, Theorem
6.18]). For any $\mu\in{\cal P}(E)$ and $\mu$–integrable function
$\varphi:E\to\mathbb{R},$ we define

$\displaystyle\langle\varphi,\mu\rangle=\langle\mu,\varphi\rangle:=\int_{E}\varphi(e)\mu(\mathrm{d}e),$

and for another metric space $(E^{\prime},\Delta^{\prime})$, we denote by
$\mu\otimes\mu^{\prime}\in{\cal P}(E\times E^{\prime})$ the product
probability of any $(\mu,\mu^{\prime})\in{\cal P}(E)\times{\cal
P}(E^{\prime})$.

Given a probability space $(\Omega,{\cal F},\mathbb{P})$ supporting a
sub-$\sigma$-algebra ${\cal G}\subset{\cal F}$ then for a Polish space $E$ and
any random variable $\xi:\Omega\longrightarrow E$, both the notations ${\cal
L}^{\mathbb{P}}(\xi|{\cal G})(\omega)$ and $\mathbb{P}^{{\cal
G}}_{\omega}\circ(\xi)^{-1}$ are used to denote the conditional distribution
of $\xi$ knowing ${\cal G}$ under $\mathbb{P}$.

$(ii)$ For any $(E,\Delta)$ and $(E^{\prime},\Delta^{\prime})$ two Polish
spaces, we use $C_{b}(E,E^{\prime})$ to denote the set of continuous functions
$f$ from $E$ into $E^{\prime}$ such that $\sup_{e\in
E}\Delta^{\prime}(f(e),e^{\prime}_{0})<\infty$ for some $e^{\prime}_{0}\in
E^{\prime}$. Let $\mathbb{N}^{*}$ denote the set of positive integers. Given
non–negative integers $m$ and $n$, we denote by $\mathbb{S}^{m\times n}$ the
collection of all $m\times n$–dimensional matrices with real entries, equipped
with the standard Euclidean norm, which we denote by $|\cdot|$ regardless of
the dimensions. We also denote $\mathbb{S}^{n}:=\mathbb{S}^{n\times n}$, and
denote by $0_{m\times n}$ the element in $\mathbb{S}^{m\times n}$ whose
entries are all $0$, and by $\mathrm{I}_{n}$ the identity matrix in
$\mathbb{S}^{n}$. Let $k$ be a positive integer, we denote by
$C^{k}_{b}(\mathbb{R}^{n};\mathbb{R})$ the set of bounded maps
$f:\mathbb{R}^{n}\longrightarrow\mathbb{R}$, having bounded continuous
derivatives of order up to and including $k$. Let
$f:\mathbb{R}^{n}\longrightarrow\mathbb{R}$ be twice differentiable, we denote
by $\nabla f$ and $\nabla^{2}f$ the gradient and Hessian of $f$.

$(iii)$ Let $T>0$, and $(\Sigma,\rho)$ be a Polish space, we denote by
$C([0,T];\Sigma)$ the space of all continuous functions on $[0,T]$ taking
values in $\Sigma$. Then, $C([0,T];\Sigma)$ is a Polish space under the
uniform convergence topology, and we denote by $\|\cdot\|$ the uniform norm.
When $\Sigma=\mathbb{R}^{k}$ for some $k\in\mathbb{N}$, we simply write ${\cal
C}^{k}:=C([0,T];\mathbb{R}^{k}),$ also we shall denote by ${\cal C}^{k}_{{\cal
W}}:=C([0,T];{\cal P}(\mathbb{R}^{k})),$ and for $p\geq 1,$ ${\cal
C}^{k,p}_{{\cal W}}:=C([0,T];{\cal P}_{p}(\mathbb{R}^{k})).$

With a Polish space $E$, we denote by $\mathbb{M}(E)$ the space of all Borel
measures $q(\mathrm{d}t,\mathrm{d}e)$ on $[0,T]\times E$, whose marginal
distribution on $[0,T]$ is the Lebesgue measure $\mathrm{d}t$, that is to say
$q(\mathrm{d}t,\mathrm{d}e)=q(t,\mathrm{d}e)\mathrm{d}t$ for a family
$(q(t,\mathrm{d}e))_{t\in[0,T]}$ of Borel probability measures on $E$. For any
$q\in\mathbb{M}(E)$ and $t\in[0,T],$ we define
$q_{t\wedge\cdot}\in\mathbb{M}(E)$ by

$q_{t\wedge\cdot}(\mathrm{d}s,\mathrm{d}e):=q(\mathrm{d}s,\mathrm{d}e)\big{|}_{[0,t]\times
E}+\delta_{e_{0}}(\mathrm{d}e)\mathrm{d}s\big{|}_{(t,T]\times E},\;\text{for
some fixed $e_{0}\in E$.}$ (1.5)

We will say that a Borel measurable function
$h:[0,T]\times\mathbb{R}^{n}\times C([0,T];\Sigma)\times\mathbb{M}(E)$ is
progressively Borel measurable if it verifies
$h(t,x,\mathsf{a},\mathsf{z})=h(t,x,\mathsf{a}_{t\wedge\cdot},\mathsf{z}_{t\wedge\cdot}),$
for any $(t,x,\mathsf{a},\mathsf{z})\in[0,T]\times\mathbb{R}^{n}\times
C([0,T];\Sigma)\times\mathbb{M}(E).$

##  2 Setup and Main results

In this section, we first introduce a $N$–player game, and the definition of
$\varepsilon_{N}$–Nash and $\varepsilon_{N}$–Pareto equilibria. Next, we
formulate the notions of approximate strong Markovian and measure–valued MFG
equilibria which will be essential to describe the limit of the Nash
equilibria. Finally, we give the $open$–$loop$ and $closed$–$loop$
formulations of the McKean–Vlasov optimal control or MFC and use them to deal
with the limit of Pareto equilibria.

The general assumptions used throughout this paper are now formulated. The
dimension $n\in\mathbb{N}^{*},$ the nonempty Polish space $(U,d)$ and the
horizon time $T>0$ are fixed and ${\cal P}^{n}_{U}$ denotes the space of all
Borel probability measures on $\mathbb{R}^{n}\times U$ i.e. ${\cal
P}^{n}_{U}:={\cal P}(\mathbb{R}^{n}\times U).$ Also, we set $p\geq 2,$
$\nu\in{\cal P}_{p^{\prime}}(\mathbb{R}^{n})$ with $p^{\prime}>(p\vee n),$ and
the probability space $(\Omega,\mathbb{H}:=({\cal H}_{t})_{t\in[0,T]},{\cal
H},\mathbb{P})$222The probability space $(\Omega,\mathbb{H},\mathbb{P})$
contains as many random variables as we want in the sense that: each time we
need a sequence of independent uniform random variables or Brownian motions,
we can find them on $\Omega$ without mentioning an enlarging of the space. .
We give ourselves the following bounded Borel measurable functions

$\big{[}b,L\big{]}:[0,T]\times\mathbb{R}^{n}\times{\cal
P}_{p}(\mathbb{R}^{n}\times U)\times
U\longrightarrow\mathbb{R}^{n}\times\mathbb{R},\;\sigma:[0,T]\times\mathbb{R}^{n}\longrightarrow\mathbb{S}^{n\times
n}\;\mbox{and}\;g:\mathbb{R}^{n}\times{\cal
P}_{p}(\mathbb{R}^{n})\longrightarrow\mathbb{R}.$

######  Assumption 2.1.

$(i)$ $\sigma_{0}\in\mathbb{S}^{n\times n}$ is an invertible constant matrix
and $U\subset\mathbb{R}^{q}$, for $q\in\mathbb{N}^{*},$ is a compact nonempty
set;

$(ii)$ The maps $(b,\sigma)$ are Lipschitz in all their variables. Also, the
maps $(L,g)$ are s.t. for each $t,$ $\mathbb{R}^{n}\times{\cal
P}_{p}(\mathbb{R}^{n}\times U)\times U\times{\cal
P}_{p}(\mathbb{R}^{n})\ni(x,\bar{\nu},u,\nu)\to\big{(}L(t,x,\bar{\nu},u),g(x,\nu)\big{)}\in\mathbb{R}^{n}\times\mathbb{R}\times\mathbb{R}$
is continuous;

$(iii)$ ${\rm\underline{Non\mbox{--}degeneracy\;condition}}$: for some
constant $\theta>0$, one has, for all $(t,x)\in[0,T]\times\mathbb{R}^{n}$,

$\displaystyle~{}\theta\mathrm{I}_{n}\leq~{}\sigma\sigma^{\top}(t,x);$

$(iv)$ ${\rm\underline{Separability\;condition}}$: There exist Borel functions
$(b^{\circ},b^{\star},L^{\circ},L^{\star})$ satisfying

$\displaystyle
b(t,x,\bar{\nu},u):=b^{\star}(t,\bar{\nu})+b^{\circ}(t,x,u)\;\;\mbox{and}\;\;L(t,x,\bar{\nu},u):=L^{\star}(t,x,\bar{\nu})+L^{\circ}(t,x,\nu,u),$

for all $(t,x,\bar{\nu},u)\in[0,T]\times\mathbb{R}^{n}\times{\cal
P}^{n}_{U}\times U$ where $\nu(\mathrm{d}x):=\bar{\nu}(\mathrm{d}x,U).$

######  Remark 2.2.

The previous assumptions are standard in the probabilistic approach of mean
field game and control problems. The
${\rm\underline{separability\;condition}}$ is more specific to the extended
mean field game and control problems $($see Carmona and Lacker [15],
Cardaliaguet and Lehalle [10], Laurière and Tangpi [46], Possamaï and Tangpi
[46], Djete [24]$)$. It is mainly used for technical reasons. Notice that some
conditions can be weakened. But, in order to avoid certain unnecessary
technicalities in the proofs, we have chosen these.

###  2.1 The $N$–player games

Let $N\in\mathbb{N}^{*}.$ We denote by ${\cal A}^{c}_{N}$ the collection of
all progressively Borel measurable functions $\alpha:[0,T]\times({\cal
C}^{n})^{N}\to U.$ On the filtered probability space $(\Omega,\mathbb{H},{\cal
H},\mathbb{P}),$ let $(W^{i})_{i\in\mathbb{N}^{*}}$ be a sequence of
independent $\mathbb{H}$–adapted $\mathbb{R}^{n}$–valued Brownian motions, $B$
be a $\mathbb{R}^{n}$–valued $\mathbb{H}$–adapted Brownian motion and
$(\xi^{i})_{i\in\mathbb{N}^{*}}$ a sequence of iid ${\cal H}_{0}$–random
variables of law $\nu.$ Besides, $(W^{i})_{i\in\mathbb{N}^{*}},$ $B$ and
$(\xi^{i})_{i\in\mathbb{N}^{*}}$ are independent. Then, given the control
rule/strategy $\boldsymbol{\alpha}:=(\alpha^{1},\dots,\alpha^{N})\in({\cal
A}^{c}_{N})^{N}$, denote by
$\mathbf{X}^{\boldsymbol{\alpha}}:=(X^{\boldsymbol{\alpha},1}_{\cdot},\dots,X^{\boldsymbol{\alpha},N}_{\cdot})$
the processes satisfying: for each $i\in\\{1,\dots,N\\},$
$\mathbb{E}\big{[}\|X^{\boldsymbol{\alpha},i}\|^{p}\big{]}<\infty,$

$\displaystyle\mathrm{d}X^{\boldsymbol{\alpha},i}_{t}=b\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}],\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+\sigma\big{(}t,X^{\boldsymbol{\alpha},i}_{t}\big{)}\mathrm{d}W^{i}_{t}+\sigma_{0}\mathrm{d}B_{t},\;X^{\boldsymbol{\alpha},i}_{0}=\xi^{i}$
(2.1)

with

$\varphi^{N}_{t}[\boldsymbol{\alpha}](\mathrm{d}x):=\frac{1}{N}\sum_{i=1}^{N}\delta_{X^{\boldsymbol{\alpha},i}_{t}}(\mathrm{d}x)\;\;\mbox{and}\;\;\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}](\mathrm{d}x,\mathrm{d}u):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\big{(}X^{\boldsymbol{\alpha},i}_{t},\;\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}}(\mathrm{d}x,\mathrm{d}u),~{}\mbox{for
all}~{}~{}t\in[0,T].$

The reward value of player $i$ associated with control rule/strategy
$\boldsymbol{\alpha}:=(\alpha^{1},\dots,\alpha^{N})$ is then defined by

$J_{i}(\boldsymbol{\alpha}):=\mathbb{E}\bigg{[}\int_{0}^{T}L\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}],\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+g\big{(}X^{\boldsymbol{\alpha},i}_{T},\varphi^{N}_{T}[\boldsymbol{\alpha}]\big{)}\bigg{]}.$

Now, we give the precise definition of what we call approximate
$closed$–$loop$ Nash and Pareto equilibria.

######  Definition 2.3.

((approximate) equilibria)

Let $N\in\mathbb{N}^{*}$ and $\varepsilon\in\mathbb{R}_{+}.$ We will say that
a control rule/strategy $(\alpha^{1},\dots,\alpha^{N})\in({\cal
A}^{c}_{N})^{N}$ is an

  * •

$\mathbf{Nash\;equilibrium}$: $\varepsilon$–$closed$–$loop$ Nash equilibrium
if

$J_{i}(\alpha^{1},\dots,\alpha^{N})\geq\sup_{\beta\in{\cal
A}^{c}_{N}}J_{i}\big{(}\alpha^{1},\dots,\alpha^{i-1},\beta,\alpha^{i+1},\dots,\alpha^{N}\big{)}-\varepsilon,\;\mbox{for
each}\;i\in\\{1,\dots,N\\}.$

  * •

$\mathbf{Pareto\;equilibrium}$: $\varepsilon$–closed–loop Pareto equilibrium
if

$\sum_{i=1}^{N}J_{i}(\alpha^{1},\cdots,\alpha^{N})\geq\sup_{(\beta^{1},\dots,\beta^{N})\in({\cal
A}^{c}_{N})^{N}}\sum_{i=1}^{N}J_{i}(\beta^{1},\dots,\beta^{N})-\varepsilon.$

######  Remark 2.4.

$(i)$ Because of the fact that the controls $(\alpha^{1},\cdots,\alpha^{N})$
are only Borel measurable, the strong existence and uniqueness of (2.1) is not
a standard one. We refer to Veretennikov [53] for the well–posedness of this
type of SDE when
$\alpha^{i}(t,\mathsf{x}^{1},\cdots,\mathsf{x}^{N})=\alpha^{i}(t,\mathsf{x}^{1}(t),\cdots,\mathsf{x}^{N}(t)).$
For the general situation, we consider existence and uniqueness in law of
(2.1) by a classical application of the Girsanov’s theorem. Let us mention
that, as weak solution, $\mathbf{X}$ may not be defined only on
$(\Omega,\mathbb{H},\mathbb{P}).$ But for ease of reading and to avoid heavy
notations, we assumed $(\Omega,\mathbb{H},\mathbb{P})$ in order to be able to
define all our variables on this space.

$(ii)$ While the existence of approximate Pareto equilibria is obvious, that
of approximate Nash equilibria is unclear. As a consequence of our results, we
will wee that when $\varepsilon>0,$ the approximate Nash equilibria are
well–defined under our assumptions.

$(iii)$ Given our framework, the natural shape for our controls should be
$Markovian$ that is to say
$\alpha^{i}(t,\mathsf{x}^{1},\cdots,\mathsf{x}^{N})=\alpha^{i}(t,\mathsf{x}^{1}(t),\cdots,\mathsf{x}^{N}(t))$
and not fully path dependent as we consider. In the MFG setting, although the
establishment of the convergence of approximate Markovian Nash equilibria to
MFG can be done with our techniques, we are unfortunately only able to
establish the $converse$ convergence result for fully path dependent controls
$($see Theorem 2.12$)$. Therefore, we make the choice to present our article
only with fully path dependent controls. This technical limitation seems to be
a strange phenomenon highlighted in other articles $($see [42, 41]$)$, and
only appears for the mean field game setting.

###  2.2 The mean field game of controls

We formulate, here, first of all the formulation of the extended mean field
game (or mean field game of controls). We will call this formulation
(approximate) strong Markovian MFG equilibrium. Second, inspired by [24], we
give the notion of measure–valued MFG which generalizes the strong Markovian
MFG equilibrium.

####  2.2.1 $\varepsilon$–strong Markovian MFG equilibrium

Let $(B,W)$ be a $\mathbb{R}^{n}\times\mathbb{R}^{n}$–valued
$\mathbb{H}$–Brownian motion, and $\xi$ be a ${\cal F}_{0}$–random variable
$\xi$ such that ${\cal L}(\xi)=\nu.$ We denote by $\mathbb{F}^{B}:=({\cal
F}^{B}_{t})_{t\in[0,T]}$ the filtration generated by $B$ i.e. ${\cal
F}^{B}_{t}:=\sigma\\{B_{s}:s\leq t\\}.$

######  Definition 2.5.

For each $\varepsilon\geq 0,$ we say that a $\mathbb{F}^{B}$ predictable
process $(\overline{\mu}_{t})_{t\in[0,T]}$ is an $\varepsilon$–strong
Markovian MFG solution if:

  1. $(i)$

$\overline{\mu}_{t}={\cal L}(X_{t},\alpha(t,X_{t},\mu)|{\cal F}^{B}_{t})$
$\mathrm{d}t\otimes\mathrm{d}\mathbb{P}$–a.e. where $\alpha$ is a
progressively Borel measurable function
$\alpha:[0,T]\times\mathbb{R}^{n}\times C([0,T];{\cal P}(\mathbb{R}^{n}))\to
U,$ $X$ the unique strong solution of

$\displaystyle\mathrm{d}X_{t}=b\big{(}t,X_{t},\overline{\mu}_{t},\alpha(t,X_{t},\mu)\big{)}\mathrm{d}t+\sigma\big{(}t,X_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;X_{0}=\xi\;\mbox{and}\;\mu_{t}:={\cal
L}(X_{t}|{\cal F}^{B}_{t});$ (2.2)

  2. $(ii)$

For any progressively Borel measurable function
$\alpha^{\prime}:[0,T]\times\mathbb{R}^{n}\times C([0,T];{\cal
P}(\mathbb{R}^{n}))\to U,$ and $X^{\prime}$ the solution of

$\displaystyle\mathrm{d}X^{\prime}_{t}=b\big{(}t,X^{\prime}_{t},\overline{\mu}_{t},\alpha(t,X^{\prime}_{t},\mu)\big{)}\mathrm{d}t+\sigma\big{(}t,X^{\prime}_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;\;X^{\prime}_{0}=\xi,$
(2.3)

one has

$\displaystyle\mathbb{E}\bigg{[}\int_{0}^{T}L(t,X_{t},\overline{\mu}_{t},\alpha(t,X_{t},\mu))\mathrm{d}t+g(X_{T},\mu_{T})\bigg{]}\geq\mathbb{E}\bigg{[}\int_{0}^{T}L(t,X^{\prime}_{t},\overline{\mu}_{t},\alpha(t,X^{\prime}_{t},\mu))\mathrm{d}t+g(X^{\prime}_{T},\mu_{T})\bigg{]}-\varepsilon.$

######  Remark 2.6.

$(i)$ A reader familiar with MFG of controls would expect another shape for an
$($approximate$)$ optimal control $\alpha.$ Indeed, in our setting, the
expected shape of an $($approximate$)$ optimal control should be
$\alpha(t,X_{t},(\overline{\mu}_{s})_{s\in[0,t]})$ and not
$\alpha(t,X_{t},(\mu_{s})_{s\in[0,t]})$ i.e. controls depending on the time,
the state and the conditional distribution of control and state, and not
controls depending on the time, the state and the conditional distribution of
state. However, under our assumptions, in particular the condition
$\sigma_{0}$ is invertible, these two representations are close. Let us
briefly explain this fact. First, as $(\mu_{s})_{s\in[0,t]}$ is a function of
$(\overline{\mu}_{s})_{s\in[0,t]},$ we can always see a control of the form
$\alpha(t,X_{t},(\mu_{s})_{s\in[0,t]})$ as a control of the form
$\alpha(t,X_{t},(\overline{\mu}_{s})_{s\in[0,t]}).$ Second, using the fact
that $\overline{\mu}$ is $\mathbb{F}^{B}$–adapted, so a progressively Borel
function of $B,$ as we will see in the proof $($ see for instance Proposition
3.11 or Proposition A.8 $)$ , it is always possible to approximate a control
of type $\alpha(t,X_{t},(\overline{\mu}_{s})_{s\in[0,t]})$ by a sequence of
controls of type $\alpha(t,X_{t},(\mu_{s})_{s\in[0,t]}).$

$(ii)$ There are two other facts that make us choose this shape of control.
Firstly, this allows us to have a formulation that fully falls within the
framework of MFG without the $($conditional$)$ law of control. Indeed, for the
classical MFG, the controls are of type
$\alpha(t,X_{t},(\mu_{s})_{s\in[0,t]})$ $($see [12, 14, 41]$)$. Secondly, for
the construction of approximate Nash equilibrium from MFG equilibrium, in MFG
of control setting, this formulation gives a more natural way. Indeed, when
$\alpha$ is regular enough $($this will be our case$)$, for each
$N\in\mathbb{N}^{*},$ an approximate Nash equilibrium is constructed by
defining:
$\alpha^{i}(t,\mathsf{x}^{1},\dots,\mathsf{x}^{N}):=\alpha\big{(}t,\mathsf{x}^{i}(t),\pi^{N}\big{)}\;\mbox{with}\;\pi^{N}(t):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\mathsf{x}^{i}(t)}.$
But if $\alpha$ is of type $\alpha(t,X_{t},(\overline{\mu}_{s})_{s\in[0,t]}),$
a natural construction is
$\alpha^{i}(t,\mathsf{x}^{1},\dots,\mathsf{x}^{N}):=\alpha\big{(}t,\mathsf{x}^{i}(t),\overline{\pi}^{N}\big{)}\;\mbox{with}\;\overline{\pi}^{N}(t):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\mathsf{x}^{i}(t),\alpha^{i}(t,\mathsf{x}^{1},\dots,\mathsf{x}^{N})}.$
Due to the appearance of $\alpha^{i}$ in both side of the equality, it is
unclear that this construction is well defined.

$(iii)$ The well–posedness of this notion of equilibrium is not clear at first
sight. But, as we will see later $($see Proposition 2.9 with Theorem 2.12$)$,
for any $\varepsilon>0,$ there exists an $\varepsilon$–strong Markovian MFG
equilibrium.

####  2.2.2 Measure–valued MFG equilibrium

######  Definition 2.7 (measure–valued equilibrium).

We say that a term

$\big{(}\Omega,{\cal F},\mathrm{P},\mathbb{F}:=({\cal F}_{t})_{0\leq t\leq
T},W,B,X,\Lambda,\mu\big{)}$

is a measure–valued MFG equilibirum if

  1. $(i)$

$(\Omega,\mathbb{F},{\cal F},\mathrm{P})$ is a filtered probability space.
$(W,B)$ is a $\mathbb{R}^{n}\times\mathbb{R}^{n}$–valued $\mathbb{F}$–Brownian
motion, $(X,\mu)$ is a $\mathbb{F}$–adapted $\mathbb{R}^{n}\times{\cal
P}(\mathbb{R}^{n})$–valued continuous process and $(\Lambda_{t})_{t\in[0,T]}$
is a $\mathbb{F}$–predictable ${\cal P}({\cal P}^{n}_{U})$–valued process.

  2. $(ii)$

$X_{0},$ $W$ and $(B,\Lambda)$ are $\mathrm{P}$–independent.

  3. $(iii)$

For each $t\in[0,T],$ $\mu_{t}={\cal L}^{\mathrm{P}}\big{(}X_{t}\big{|}{\cal
G}_{t}\big{)}$ $\mathrm{P}$–a.e. where ${\cal
G}_{t}:=\sigma\\{B_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\\},$
$X$ satisfies

$\displaystyle\mathrm{d}X_{t}=\int_{{\cal
P}^{n}_{U}}\;\int_{U}b\big{(}t,X_{t},\bar{\nu},u\big{)}\bar{\nu}^{X_{t}}(\mathrm{d}u)\;\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma\big{(}t,X_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;\;X_{0}=\xi$

and

$\displaystyle\Lambda_{t}\big{(}\\{\bar{\nu}\in{\cal
P}^{n}_{U}:\;\bar{\nu}(\mathrm{d}x,U)=\mu_{t}(\mathrm{d}x)\\}\big{)}=1,\;\;\mathrm{d}t\otimes\mathrm{d}\mathrm{P}\mbox{--a.e.}$

where for each $\bar{\nu}\in{\cal P}^{n}_{U},$ $\mathbb{R}^{n}\ni
x\to\bar{\nu}^{x}\in{\cal P}(U)$ is Borel measurable and satisfies
$\bar{\nu}(\mathrm{d}x,\mathrm{d}u)=\bar{\nu}^{x}(\mathrm{d}u)\bar{\nu}(\mathrm{d}x,U).$

  4. $(iv)$

For any Borel progressive measurable function
$\alpha^{\prime}:[0,T]\times\mathbb{R}^{n}\times\mathbb{M}({\cal
P}^{n}_{U})\to U,$ and for each $t\in[0,T],$ $\mu^{\prime}_{t}={\cal
L}^{\mathrm{P}}(X^{\prime}_{t}\big{|}{\cal G}_{t})$ $\mathrm{P}$–a.e. where
$X^{\prime}$ the solution of

$\displaystyle\mathrm{d}X^{\prime}_{t}=\int_{{\cal
P}^{n}_{U}}\;b\big{(}t,X^{\prime}_{t},\bar{\nu},\alpha(t,X^{\prime}_{t},\Lambda)\big{)}\;\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma\big{(}t,X^{\prime}_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;\;X^{\prime}_{0}=\xi$
(2.4)

one has

$\displaystyle\mathbb{E}^{\mathrm{P}}\bigg{[}\int_{0}^{T}\int_{{\cal
P}^{n}_{U}}\langle
L(t,\cdot,\bar{\nu},\cdot),\bar{\nu}\rangle\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\langle
g(\cdot,\mu_{T}),\mu_{T}\rangle\bigg{]}$
$\displaystyle\geq\mathbb{E}^{\mathrm{P}}\bigg{[}\int_{0}^{T}\int_{{\cal
P}^{n}_{U}}\langle
L(t,\cdot,\bar{\nu},\alpha(t,\cdot,\Lambda)),\mu^{\prime}_{t}\rangle\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\langle
g(\cdot,\mu_{T}),\mu^{\prime}_{T}\rangle\bigg{]}.$

######  Remark 2.8.

$(i)$ Although formulated differently, the definition of the measure–valued
MFG equilibrium given in this paper is quite equivalent to those introduced in
[24, Definition 2.7] for $open$–$loop$ setting $($see Proposition 3.5 and
Remark 3.6 for details$)$. Moreover, in the classical MFG framework i.e. when
there is no law of control, this notion of measure–valued MFG equilibrium
turns out to be equivalent to the classical notion of weak MFG equilibrium
$($see comparison in [24, Comparison Definition 2.7 …] after Remark 2.8$)$.

$(ii)$ We would like to emphasize the fact that our controls do not need to
depend on Brownian motion $B$ The control processes considering only the fixed
measures $(\mu,\Lambda)$ and not $B$ are more natural. Intuitively, a player
has to consider his position and the distribution of the all population. For
more mathematical justification, refer to [12, 14].

With the equivalence between our notion of measure–valued MFG equilibrium and
the notion considered in [24, Definition 2.7] for $open$–$loop$ framework that
we will prove in Proposition 3.5, thanks to the result proved in [25, Theorem
7.2.4], we have the following result.

######  Proposition 2.9.

Under 2.1, there exists at least one measure–valued MFG equilibrium.

####  2.2.3 Limit theorem and converse limit theorem

We are now ready to formulate the main convergence results of this paper
regarding the extended mean field game. We begin with the convergence of
sequence of approximate Nash equilbria and approximate strong Markovian MFG
equilibria towards the measure–valued MFG equilibria. The proof is given in
Section 3.2.2 and Section 3.3.2.

######  Theorem 2.10 (limit theorem).

Let 2.1 hold true.

  * •

For each $N\in\mathbb{N}^{*}$, let
$\boldsymbol{\alpha}^{N}:=(\alpha^{1,N},\dots,\alpha^{N,N})$ be an
$\varepsilon_{N}$–closed–loop Nash equilibrium, then the sequence
$(\mathrm{P}^{N})_{N\in\mathbb{N}^{*}}$ with
$\mathrm{P}^{N}:=\mathrm{P}^{N}[\boldsymbol{\alpha}^{N}]\in{\cal
P}\big{(}{\cal C}^{n}_{{\cal W}}\times\mathbb{M}({\cal P}^{n}_{U})\big{)}$ is
relatively compact in ${\cal W}_{p}$ where

$\mathrm{P}^{N}[\boldsymbol{\alpha}^{N}]:={\cal
L}^{\mathbb{P}}\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\;\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}\bar{\nu})\mathrm{d}t\Big{)}$

and if $\displaystyle\lim_{N\to\infty}\varepsilon_{N}=0,$ then for each limit
point $\mathrm{P}^{\infty}$ there exists a measure–valued solution
$\big{(}\Omega,{\cal F},\mathrm{P},\mathbb{F},W,B,X,\Lambda,\mu\big{)}$ such
that

$\displaystyle\mathrm{P}^{\infty}={\cal L}^{\mathrm{P}}(\mu,\Lambda).$

  * •

For each $\ell\in\mathbb{N}^{*},$ let $\overline{\mu}^{\ell}$ be an
$\varepsilon_{\ell}$–strong Markovian MFG equilibirum, then
$(\mathrm{P}^{\ell})_{\ell\in\mathbb{N}^{*}}\subset{\cal P}\big{(}{\cal
C}^{n}_{{\cal W}}\times\mathbb{M}({\cal P}^{n}_{U})\big{)}$ is relatively
compact in ${\cal W}_{p}$ where
$\mathrm{P}^{\ell}:=\mathbb{P}\circ(\mu^{\ell},\delta_{\overline{\mu}^{\ell}_{s}}(\mathrm{d}u)\mathrm{d}s)^{-1}$
and if $\displaystyle\lim_{\ell\to\infty}\varepsilon_{\ell}=0,$ then for each
limit point $\mathrm{P}^{\infty}$ there exists a measure–valued solution
$\big{(}\Omega,{\cal F},\mathrm{P},\mathbb{F},W,B,X,\Lambda,\mu\big{)}$ such
that

$\displaystyle\mathrm{P}^{\infty}={\cal L}^{\mathrm{P}}(\mu,\Lambda).$

######  Remark 2.11.

In the framework of extended mean field game, these convergence results seem
to be the first of this type. Unlike the recent paper of [49] where the
setting is without common noise, $\sigma$ constant and strong assumptions, in
our paper, we treat the case with common noise and the assumptions are less
strong.

Now we present the converse convergence result. That is, any measure–valued
MFG equilibrium is the limit of sequence of approximate $closed$–$loop$ Nash
equilibria or sequence of approximate strong Markovian MFG equilibria. The
proof is given in Section 3.3.3 and Section 3.3.4.

######  Theorem 2.12 (Converse limit theorem).

Let 2.1 hold true. For any measure–valued MFG equilibrium $\big{(}\Omega,{\cal
F},\mathrm{P},\mathbb{F},W,B,X,\Lambda,\mu\big{)}$ there exists:

  * •

$(\overline{\mu}^{\ell})_{\ell\in\mathbb{N}^{*}}$ s.t. for each
$\ell\in\mathbb{N}^{*},$ $\overline{\mu}^{\ell}$ is an
$\varepsilon_{\ell}$–strong Markovian MFG equilibrium with
$\varepsilon_{\ell}>0,$
$\displaystyle\lim_{\ell\to\infty}\varepsilon_{\ell}=0,$ and

$\displaystyle{\cal
L}^{\mathrm{P}}(\mu,\Lambda)=\displaystyle\lim_{\ell\to\infty}{\cal
L}^{\mathbb{P}}\big{(}\mu^{\ell},\;\delta_{\overline{\mu}^{\ell}}(\mathrm{d}m)\mathrm{d}t\big{)}\;\mbox{in}\;{\cal
W}_{p}.$

  * •

$(\alpha^{1,N},\cdots,\alpha^{N,N})_{N\in\mathbb{N}^{*}}$ s.t. for each
$N\in\mathbb{N}^{*}$,
$\boldsymbol{\alpha}^{N}:=(\alpha^{1,N},\dots,\alpha^{N,N})$ is an
$\varepsilon_{N}$–$closed$–$loop$ Nash equilibrium with $\varepsilon_{N}>0,$
$\displaystyle\lim_{N\to\infty}\varepsilon_{N}=0$ and

$\displaystyle{\cal
L}^{\mathrm{P}}(\mu,\Lambda)=\displaystyle\lim_{N\to\infty}{\cal
L}^{\mathbb{P}}\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\;\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}m)\mathrm{d}t\Big{)}\;\mbox{in}\;{\cal
W}_{p}.$

######  Remark 2.13.

Under these general assumptions, especially allowing the MFG equilibrium to be
non unique, Theorem 2.12 seems to be the first result about the approximation
of any MFG equilibrium via a sequence of approximate $closed$–$loop$ Nash
equilibria or approximate strong Markovian MFG equilibria. Without counting
the fact that we take into account the empirical distribution of states and
controls, even for the classical MFG framework, this result appears as new in
the literature. As highlighted in [41] and later in [42], in the presence of
multiple MFG equilibria, this kind of converse result is delicate to prove
without extension of the notion of $closed$–$loop$ Nash equilibrium $($see
[42]$)$. To bypass this difficulty, the condition $\sigma_{0}$ is invertible
turns out to be quite useful.

###  2.3 The McKean–Vlasov control problem

####  2.3.1 The $Open$–$loop$ and $closed$–$loop$ formulations

Under 2.1, we will formulate here the $open$–$loop$ and $closed$–$loop$
formulations of the McKean–Vlasov control problem on the probability space
$(\Omega,\mathbb{H},{\cal H},\mathbb{P})$ with the
$\mathbb{R}^{n}\times\mathbb{R}^{n}$–valued $\mathbb{H}$–adapted Brownian
motion $(W,B)$ and the ${\cal F}_{0}$–random variable $\xi$ s.t. ${\cal
L}(\xi)=\nu.$ We denote by $\mathbb{F}:=({\cal F}_{t})_{t\in[0,T]}$ and
$\mathbb{G}:=({\cal G}_{t})_{t\in[0,T]}$ the $\mathbb{P}$–completion
filtration of $\big{(}\sigma\\{\xi,B_{r},W_{r}:\;r\leq
t\\}\big{)}_{t\in[0,T]}$ and $\big{(}\sigma\\{B_{r}:\;r\leq
t\\}\big{)}_{t\in[0,T]}$ respectively.

#####  $Open$–$loop$ formulation

Let ${\cal A}^{o}$ be the collection of all $U$–valued processes
$\alpha=(\alpha_{s})_{0\leq s\leq T}$ which are $\mathbb{F}$–predictable.
Then, given $\alpha\in{\cal A}^{o}$, let $X^{o,\alpha}$ be the unique strong
solution of the SDE:
$\mathbb{E}^{\mathbb{P}}\big{[}\|X^{o,\alpha}\|^{p}\big{]}<\infty,$
$X^{o,\alpha}_{0}=\xi$,

$\displaystyle\mathrm{d}X^{o,\alpha}_{t}=b\big{(}t,X^{o,\alpha}_{t},\overline{\mu}^{o,\alpha}_{t},\alpha_{t}\big{)}\mathrm{d}t+\sigma\big{(}t,X^{o,\alpha}_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;\overline{\mu}^{o,\alpha}_{t}:={\cal
L}(X^{o,\alpha}_{t},\alpha_{t}|{\cal
G}_{t})\;\mbox{and}\;\mu^{o,\alpha}_{t}:={\cal L}(X^{o,\alpha}_{t}|{\cal
G}_{t}).$ (2.5)

We will say that $X^{o,\alpha}$ is an $open$–$loop$ McKean–Vlasov process
associated to $\alpha.$ Let us now introduce the following $open$–$loop$
McKean–Vlasov control problem by

$\displaystyle V^{o}_{S}~{}:=~{}\sup_{\alpha\in{\cal
A}^{o}}\Phi^{o}(\alpha)\;\mbox{where}~{}~{}\Phi^{o}(\alpha):=\mathbb{E}\bigg{[}\int_{0}^{T}L(t,X^{o,\alpha}_{t},\overline{\mu}^{o,\alpha}_{t},\alpha_{t})\mathrm{d}t+g\big{(}X^{o,\alpha}_{T},\mu^{o,\alpha}_{T}\big{)}\bigg{]}.$
(2.6)

#####  $Closed\;loop$ formulation

Let ${\cal A}^{c}$ be the collection of all Borel progressively measurable
maps $\alpha:[0,T]\times\mathbb{R}^{n}\times C([0,T];{\cal
P}(\mathbb{R}^{n}))\ni(t,x,\pi)\to\alpha(t,x,\pi)\in U.$ Then, given
$\alpha\in{\cal A}^{c}$, let $X^{c,\alpha}$ be the process satisfying:
$\mathbb{E}^{\mathbb{P}^{\star}}\big{[}\|X^{c,\alpha}\|^{p}\big{]}<\infty,$
$X^{c,\alpha}_{0}=\xi$, and

$\displaystyle\mathrm{d}X^{c,\alpha}_{t}=b\big{(}t,X^{c,\alpha}_{t},\overline{\mu}^{c,\alpha}_{t},\alpha(t,X^{c,\alpha}_{t},\mu^{c,\alpha})\big{)}\mathrm{d}t+\sigma\big{(}t,X^{c,\alpha}_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}$
(2.7)

$\;\mbox{with}\,\overline{\mu}^{c,\alpha}_{t}:={\cal
L}(X^{c,\alpha}_{t},\alpha(t,X^{c,\alpha}_{t},\mu^{c,\alpha})|{\cal
G}_{t})\;\mbox{and}\;\mu^{c,\alpha}_{t}:={\cal L}(X^{c,\alpha}_{t}|{\cal
G}_{t}).$ We will say that $X^{c,\alpha}$ is an $closed$–$loop$ McKean–Vlasov
process associated to $\alpha.$ The $closed$–$loop$ McKean–Vlasov control
problem is given by

$\displaystyle V^{c}_{S}~{}:=~{}\sup_{\alpha\in{\cal
A}^{c}}\Phi^{c}(\alpha)\;\mbox{,}~{}~{}\Phi^{c}(\alpha):=\mathbb{E}\bigg{[}\int_{0}^{T}L\big{(}t,X^{c,\alpha}_{t},\overline{\mu}^{c,\alpha}_{t},\alpha(t,X^{c,\alpha}_{t},\mu^{c,\alpha})\big{)}\mathrm{d}t+g\big{(}X^{c,\alpha}_{T},\mu^{c,\alpha}_{T}\big{)}\bigg{]}.$
(2.8)

####  2.3.2 Equivalence and limit theory results

We now provide here the equivalence result and the limit theory relating to
the McKean–Vlasov control problem. The proofs are in Section 3.4.1 and Section
3.4.2.

######  Theorem 2.14 (equivalence between $closed$–$loop$ and $open$–$loop$
controls).

Let $X^{o,\alpha}$ be an $open$–$loop$ McKean–Vlasov process associated to
$\alpha\in{\cal A}^{o}.$ There exists a sequence
$(\alpha^{\ell})_{\ell\in\mathbb{N}^{*}}\subset{\cal A}^{c}$ s.t.
$X^{c,\alpha^{\ell}}$ is an $closed$–$loop$ McKean–Vlasov process associated
to $\alpha^{\ell},$ and one has

$\displaystyle\mathbb{P}\circ\Big{(}\mu^{o,\alpha},\delta_{\overline{\mu}^{o,\alpha}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\Big{)}^{-1}=\displaystyle\lim_{\ell\to\infty}\mathbb{P}\circ\Big{(}\mu^{c,\alpha^{\ell}},\delta_{\overline{\mu}^{c,\alpha^{\ell}}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\Big{)}^{-1}\;\mbox{in}\;{\cal
W}_{p}.$

Consequently,

$\displaystyle V_{S}^{o}=V_{S}^{c}.$

######  Theorem 2.15 (Limit theory).

For each $N\in\mathbb{N}^{*},$ let $\varepsilon_{N}\in\mathbb{R}_{+}$ and
$\boldsymbol{\alpha}^{N}:=(\alpha^{1,N},\dots,\alpha^{N,N})$ be an
$\varepsilon_{N}$–$closed$–$loop$ Pareto equilibrium. Then, the sequence
$(\mathrm{P}^{N})_{N\in\mathbb{N}^{*}}$ with
$\mathrm{P}^{N}:=\mathrm{P}^{N}[\boldsymbol{\alpha}^{N}]\in{\cal
P}\big{(}{\cal C}^{n}_{{\cal W}}\times\mathbb{M}({\cal P}^{n}_{U})\big{)}$ is
relatively compact in ${\cal W}_{p}$ where

$\mathrm{P}^{N}[\boldsymbol{\alpha}^{N}]:={\cal
L}^{\mathbb{P}}\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\;\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}\bar{\nu})\mathrm{d}t\Big{)}$

and if $\displaystyle\lim_{N\to\infty}\varepsilon_{N}=0,$ then for each limit
point $\mathrm{P}^{\infty}$ there exists a sequence
$(\alpha^{\star,N})_{N\in\mathbb{N}^{*}}\subset{\cal A}^{c}$ s.t if we define
for each $N\in\mathbb{N}^{*},$
$\boldsymbol{\alpha}^{\star,N}:=(\alpha^{\star,1,N},\dots,\alpha^{\star,N,N})$
where

$\displaystyle\alpha^{\star,i,N}(t,\mathsf{x}^{1},\dots,\mathsf{x}^{N}):=\alpha^{\star,N}\big{(}t,\mathsf{x}^{i}(t),\pi^{N}\big{)}\;\mbox{with}\;\pi^{N}(t):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\mathsf{x}^{i}(t)},\;\mbox{for}\;(\mathsf{x}^{1},\dots,\mathsf{x}^{N})\in({\cal
C}^{n})^{N}\;\mbox{and}\;i\in\\{1,\dots,N\\}$

then

$\displaystyle\mathrm{P}^{\infty}=\displaystyle\lim_{N\to\infty}{\cal
L}^{\mathbb{P}}\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{\star,N}],\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{\star,N}]}(\mathrm{d}\bar{\nu})\mathrm{d}t\Big{)}\;\mbox{in}\;{\cal
W}_{p},\;\mbox{therefore}\;V_{S}^{o}=V_{S}^{c}=\displaystyle\lim_{N\to\infty}\;\sup_{\boldsymbol{\alpha}\in({\cal
A}^{c}_{N})^{N}}\;\frac{1}{N}\sum_{1=1}^{N}J_{i}(\boldsymbol{\alpha}).$

######  Remark 2.16.

This type of results has been evoked by some authors, see for instance [39,
47, 26, 43, 23]. However, two points are new here: the absence of convexity
assumptions and the presence of the law of controls. Indeed, first, with
convexity assumptions, without the law of controls, these results have been
established by some authors. Here, we are able to avoid this restriction.
Second, in the presence of the law of controls, even with convexity
assumptions, not many authors provide this kind of equivalence. The
equivalence between $closed$–$loop$ and $open$–$loop$ is usually proved by
representing the optimal $open$–$loop$ control as a $closed$–$loop$ control.
This representation is done by $``$projecting$"$ the optimal $open$–$loop$
control on the state process $X_{t}$ and then using the convexity assumptions
$($see [39, 43]$)$. The problem is, when there is the law of control, after
$``$projection$"$, it is not possible to recover the law of control. Some
information gets lost along the way. Let us mention that the equivalence
between $closed$–$loop$ and $open$–$loop$ formulations appears to be false
when the coefficients are path–dependent as shown by Yong and Zhang [55].

##  3 Proof of main results

###  3.1 Measure–valued equilibrium: a canonical formulation

In this section, we present equivalent formulations of the measure–valued MFG
equilibrium and McKean–Vlasov control problem over a canonical space. These
formulations turn out to be those used in [24] and [23] to manage the
$open$–$loop$ framework. Therefore, we will see from our proofs that the set
of limits of $open$–$loop$ and $closed$–$loop$ controls is the same. These
formulations have the advantage to facilitate the presentation of the proofs.

####  3.1.1 Measure-valued control rules

Denote by $\mathbb{M}:=\mathbb{M}\big{(}{\cal P}^{n}_{U}\big{)}$ the
collection of all finite (Borel) measures $q(\mathrm{d}t,\mathrm{d}e)$ on
$[0,T]\times{\cal P}^{n}_{U}$, whose marginal distribution on $[0,T]$ is the
Lebesgue measure $\mathrm{d}s$, i.e.
$q(\mathrm{d}s,\mathrm{d}e)=q(s,\mathrm{d}e)\mathrm{d}s$ for a measurable
family $(q(s,\mathrm{d}e))_{s\in[0,T]}$ of Borel probability measures on
${\cal P}^{n}_{U}.$ Let $\Lambda$ be the canonical element on $\mathbb{M}$. We
then introduce a canonical filtration $\mathbb{F}^{\Lambda}=({\cal
F}^{\Lambda}_{t})_{0\leq t\leq T}$ on $\mathbb{M}$ by

${\cal F}^{\Lambda}_{t}:=\sigma\big{\\{}\Lambda(C\times[0,s])~{}:\forall s\leq
t,\;C\in{\cal B}({\cal P}^{n}_{U})\big{\\}}.$

For each $q\in\mathbb{M}$, one has the disintegration property:
$q(\mathrm{d}t,\mathrm{d}e)=q(t,\mathrm{d}e)\mathrm{d}t$, and there is a
version of the disintegration such that $(t,q)\mapsto q(t,\mathrm{d}e)$ is
$\mathbb{F}^{\Lambda}$-predictable.

The canonical element on $\overline{\Omega}:={\cal C}^{n}_{{\cal
W}}\times{\cal C}^{n}_{{\cal W}}\times\mathbb{M}\times\mathbb{M}\times{\cal
C}^{n}$ is denoted by $(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda,B).$ Then,
the canonical filtration $\overline{\mathbb{F}}=(\overline{{\cal
F}}_{t})_{t\in[0,T]}$ is defined by: for all $t\in[0,T]$

$\overline{{\cal
F}}_{t}:=\sigma\big{\\{}\mu^{\prime}_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda^{\prime}_{t\wedge\cdot},\Lambda_{t\wedge\cdot},B_{t\wedge\cdot}\big{\\}},$

with $\Lambda^{\prime}_{t\wedge\cdot}$ and $\Lambda_{t\wedge\cdot}$ denote the
restriction of $\Lambda^{\prime}$ and $\Lambda$ on $[0,t]\times{\cal
P}^{n}_{U}$ (see definition 1.5). Notice that we can choose a version of the
disintegration
$\Lambda(\mathrm{d}\bar{\nu},\mathrm{d}t)=\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t$
(resp
$\Lambda^{\prime}(\mathrm{d}\bar{\nu},\mathrm{d}t)=\Lambda^{\prime}_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t$)
such that $(\Lambda_{t})_{t\in[0,T]}$ (resp
$(\Lambda^{\prime}_{t})_{t\in[0,T]}$) is a ${\cal P}({\cal P}^{n}_{U})$–valued
$\overline{\mathbb{F}}$–predictable process. Let us also introduce the
filtration $(\overline{{\cal G}}_{t})_{t\in[0,T]}$ by

$\overline{{\cal
G}}_{t}:=\sigma\big{\\{}\mu_{t\wedge\cdot},\;\Lambda_{t\wedge\cdot},\;B_{t\wedge\cdot}\big{\\}}.$

We consider ${\cal L}$ the following generator: for
$(t,x,\bar{\nu},u)\in[0,T]\times\mathbb{R}^{n}\times{\cal P}^{n}_{U}\times U$,
and $\varphi\in C^{2}(\mathbb{R}^{n})$

$\displaystyle{\cal L}_{t}\varphi(x,\bar{\nu},u)$ $\displaystyle:={\cal
L}^{\circ}_{t}\varphi(x,u)+b^{\star}(t,\bar{\nu})^{\top}\nabla\varphi(x)$
(3.1)

where

$\displaystyle{\cal L}^{\circ}_{t}\varphi(x,u)$
$\displaystyle:=\frac{1}{2}\text{Tr}\big{[}\sigma\sigma^{\top}(t,x)\nabla^{2}\varphi(x)\big{]}+b^{\circ}(t,x,u)^{\top}\nabla\varphi(x).$
(3.2)

Also, for every $f\in C^{2}(\mathbb{R}^{n}),$ let us define
$N_{t}(f):=N_{t}[\mu^{\prime},\mu,\Lambda^{\prime},\Lambda](f)$ by

$\displaystyle N_{t}[\mu^{\prime},\mu,\Lambda^{\prime},\Lambda](f):=\langle
f(\cdot-\sigma_{0}B_{t}),\mu^{\prime}_{t}\rangle-\langle
f,\mu^{\prime}_{0}\rangle$ $\displaystyle-\int_{0}^{t}\bigg{[}\int_{{\cal
P}^{n}_{U}}\int_{\mathbb{R}^{n}}b^{\star}(r,\bar{\nu})^{\top}\nabla
f(x-\sigma_{0}B_{t})\mu^{\prime}_{r}(\mathrm{d}x)\Lambda_{r}(\mathrm{d}\bar{\nu})$
(3.3) $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}-\int_{{\cal
P}^{n}_{U}}\langle{\cal
L}^{\circ}_{r}[f(\cdot-\sigma_{0}B_{r})](\cdot,\cdot),\bar{\nu}\rangle\Lambda^{\prime}_{r}(\mathrm{d}\bar{\nu})\bigg{]}\mathrm{d}r,$
(3.4)

and for each $\pi\in{\cal P}(\mathbb{R}^{n}),$ the Borel set
$\mathbb{Z}_{\pi}$ by

$\displaystyle\mathbb{Z}_{\pi}:=\Big{\\{}\bar{\nu}\in{\cal
P}^{n}_{U}:\bar{\nu}(\mathrm{d}x,U)=\pi(\mathrm{d}x)\Big{\\}}.$

######  Definition 3.1 (measure–valued control rule).

We say that $\mathrm{P}\in{\cal P}(\overline{\Omega})$ is a measure–valued
control rule if:

  1. $(i)$

$\mathrm{P}\big{(}\mu^{\prime}_{0}=\nu\big{)}=1$.

  2. $(ii)$

$(B_{t})_{t\in[0,T]}$ is a $(\mathrm{P},\overline{\mathbb{F}})$ Wiener process
starting at zero and for $\mathrm{P}$–almost every
$\omega\in\overline{\Omega}$, $N_{t}(f)=0$ for all $f\in
C^{2}_{b}(\mathbb{R}^{n})$ and every $t\in[0,T].$

  3. $(iii)$

$(\Lambda^{\prime}_{t})_{t\in[0,T]}$ is a
${\overline{\mathbb{G}}}$–predictable process.

  4. $(iv)$

For $\mathrm{d}\mathrm{P}\otimes\mathrm{d}t$ almost every
$(t,\omega)\in[0,T]\times\overline{\Omega}$,
${\Lambda}^{\prime}_{t}\big{(}\mathbb{Z}_{\mu^{\prime}_{t}}\big{)}=1.$

We shall denote $\overline{\cal P}_{V}$ the set of all measure–valued control
rules.

######  Remark 3.2.

The set of measure–valued control rules $\overline{\cal P}_{V}$ that we
introduced is the same as the one used in [24]. However, as we will see later
$($see Proposition 3.5 and Remark 3.6$)$, for the definition of measure–valued
MFG equilibrium, only the case where $B$ is
$(\sigma\\{\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\\})_{t\in[0,T]}$–adapted
matters.

The next result is one of the key step to understand the measurability
property satisfies by the Brownian motion $B.$

######  Lemma 3.3.

For any $\mathrm{P}\in\overline{\cal P}_{V},$ there exists a continuous
function $\varphi:{\cal C}^{n,\ell}_{{\cal W}}\times{\cal C}^{n,\ell}_{{\cal
W}}\times\mathbb{M}\times\mathbb{M}\to{\cal C}^{n}$ $\mbox{for}\;\;\ell\geq 1$
s.t.

$\mathrm{P}\big{(}B_{t}=\varphi_{t}(\mu^{\prime}_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda^{\prime}_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T]\big{)}=1.$

###### Proof.

Let $(\widetilde{\Omega},\widetilde{\mathbb{F}},\widetilde{\mathrm{P}})$ be an
extension of $(\overline{\Omega},\overline{\mathbb{F}},\mathrm{P})$ supporting
a $\mathbb{R}^{n}$–valued $\mathbb{F}$–Brownian motion $W,$ and a
$\widetilde{{\cal F}}_{0}$–random variable $\xi$ s.t. ${\cal
L}^{\widetilde{\mathrm{P}}}(\xi)=\nu.$ Besides, $W,$ $\xi$ and
$\overline{{\cal G}}_{T}$ are independent. Given
$(\mu,\Lambda,\Lambda^{\prime}),$ let $X^{\prime}$ be the solution of

$\displaystyle\mathrm{d}X^{\prime}_{t}=\int_{{\cal
P}^{n}_{U}}\int_{U}b(t,X^{\prime}_{t},\bar{\nu},u)\bar{\nu}^{\prime
X^{\prime}_{t}}(\mathrm{d}u)\Lambda^{\prime}_{t}(\mathrm{d}\bar{\nu}^{\prime})\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma(t,X^{\prime}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;X^{\prime}_{0}=\xi,\;\widetilde{\mathrm{P}}\mbox{--a.e.}$

By the uniqueness of the previous equation, we can check that ${\cal
L}^{\widetilde{\mathrm{P}}}(X^{\prime}_{t}|\overline{{\cal
G}}_{t})=\mu^{\prime}_{t},$ $\widetilde{\mathrm{P}}$–a.e. for all $t\in[0,T]$
(see similar arguments in Proposition A.7 ). By taking the conditional
expectation, we find that

$\displaystyle
B_{t}=\sigma_{0}^{-1}\bigg{[}\int_{\mathbb{R}^{n}}x\mu^{\prime}_{t}(\mathrm{d}x)-\int_{\mathbb{R}^{n}}x\mu^{\prime}_{0}(\mathrm{d}x)-\int_{0}^{t}\int_{({\cal
P}^{n}_{U})^{2}}\;\int_{\mathbb{R}^{n}\times
U}b\big{(}s,x^{\prime},\bar{\nu},u^{\prime}\big{)}\bar{\nu}^{\prime}(\mathrm{d}x^{\prime},\mathrm{d}u^{\prime})\Lambda^{\prime}_{s}(\mathrm{d}\bar{\nu}^{\prime})\Lambda_{s}(\mathrm{d}\bar{\nu})\mathrm{d}s\bigg{]}.$

Therefore, the function $\varphi$ is defined by

$\displaystyle\varphi_{t}(\pi^{\prime},\pi,q^{\prime},q):=\sigma_{0}^{-1}\bigg{[}\int_{\mathbb{R}^{n}}x\pi^{\prime}_{t}(\mathrm{d}x)-\int_{\mathbb{R}^{n}}x\pi^{\prime}_{0}(\mathrm{d}x)-\int_{0}^{t}\int_{({\cal
P}^{n}_{U})^{2}}\;\int_{\mathbb{R}^{n}\times
U}b\big{(}s,x,\bar{\nu},u\big{)}\bar{\nu}^{\prime}(\mathrm{d}u,\mathrm{d}x)q^{\prime}_{s}(\mathrm{d}\bar{\nu}^{\prime})q_{s}(\mathrm{d}\bar{\nu})\mathrm{d}s\bigg{]}.$

This is enough to conclude. ∎

Let $\mathrm{P}\in\overline{\cal P}_{V}$ s.t.
$\mathrm{P}[\mu=\mu^{\prime},\Lambda=\Lambda^{\prime}]=1$ and
$\mathrm{P}^{\prime}\in\overline{\cal P}_{V}$ satisfying ${\cal
L}^{\mathrm{P}}(\mu,\Lambda,B)={\cal L}^{\mathrm{P}^{\prime}}(\mu,\Lambda,B).$
By the previous Lemma, we deduce that
$\mathrm{P}\big{(}B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T]\big{)}=1,$
and also that
$\mathrm{P}^{\prime}\big{(}B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T]\big{)}=1.$
In other words, $B$ is a function of $(\mu,\Lambda).$ Let
$(\widetilde{\Omega},\widetilde{\mathbb{F}},\widetilde{\mathrm{P}}^{\prime})$
be an extension of
$(\overline{\Omega},\overline{\mathbb{F}},\mathrm{P}^{\prime})$ supporting a
$\mathbb{R}^{n}$–valued $\mathbb{F}$–Brownian motion $W,$ and a
$\widetilde{{\cal F}}_{0}$–random variable $\xi$ s.t. ${\cal L}(\xi)=\nu.$
Besides, $W,$ $\xi$ and $\overline{{\cal G}}_{T}$ are independent.

The next Lemma is essentially an application of Proposition A.6. It basically
indicates that we can replace $\Lambda^{\prime},$ which plays the role of a
control, by a sequence of more regular controls. This fact will be useful to
show the canonical formulation of the measure–valued MFG equilibrium (see
Proposition 3.5) and to deal with the convergence of Nash equilibria (see
Proposition 3.9).

######  Lemma 3.4.

There exists a sequence of continuous functions
$(\widehat{\beta}^{j})_{j\in\mathbb{N}^{*}}$ satisfying: for each
$j\in\mathbb{N}^{*},$

$\displaystyle\widehat{\beta}^{j}:[0,T]\times\mathbb{R}^{n}\times{\cal
C}^{n,1}_{{\cal W}}\times\mathbb{M}\times[0,1]\to
U\;\mbox{with}\;\widehat{\beta}^{j}(t,x,\pi,q)=\widehat{\beta}^{j}(t,x,\pi_{t\wedge\cdot},q_{t\wedge\cdot}),$

s.t. if we define $X^{\prime j}$ the solution of

$\displaystyle\mathrm{d}X^{\prime j}_{t}=\int_{{\cal
P}^{n}_{U}}b\big{(}t,X^{\prime
j}_{t},\bar{\nu},\widehat{\beta}^{j}(t,X^{\prime
j}_{t},\mu,\Lambda)\big{)}\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma(t,X^{\prime
j}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;X^{\prime
j}_{0}=\xi\;\;\;\;\;\widetilde{\mathrm{P}}^{\prime}\mbox{--a.s.}$ (3.5)

then

$\displaystyle\displaystyle\lim_{j\to\infty}\widetilde{\mathrm{P}}^{\prime}\circ\big{(}\mu^{\prime
j},\mu,\Lambda^{\prime j},\Lambda,B\big{)}^{-1}=\mathrm{P}\;\;\mbox{in}\;{\cal
W}_{p},$

with $\mu^{\prime j}_{t}={\cal L}^{\mathrm{P}}\big{(}X^{\prime
j}_{t}\big{|}\overline{{\cal G}}_{t}\big{)},$ $\overline{\mu}^{\prime
j}_{t}={\cal L}^{\mathrm{P}}\big{(}X^{\prime
j}_{t},\widehat{\beta}^{j}(t,X^{\prime
j}_{t},\mu,\Lambda)\big{|}\overline{{\cal G}}_{t}\big{)},$ and
$\Lambda^{\prime j}=\delta_{\overline{\mu}^{\prime
j}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t.$

###### Proof.

It is enough to apply Proposition A.6 and using the information that
$\mathrm{P}^{\prime}\big{(}B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T]\big{)}=1$
i.e. $B$ is a continuous function of $(\mu,\Lambda).$ ∎

Now, using the measure–valued control rules, we give an equivalent definition
of the measure–valued MFG equilibrium.

####  3.1.2 MFG equilibrium on the canonical space

Let us introduce the set of measure–valued equilibrium

$\displaystyle{\cal
S}^{\star}:=\big{\\{}\mathbb{P}\circ(\mu,\Lambda)^{-1}:\;\mbox{where}\;(\Omega,{\cal
F},\mathbb{P},\mathbb{F},W,B,X,\Lambda,\mu)\;\mbox{is a measure--valued MFG
equilibrium}\big{\\}}.$

For all $(\pi^{\prime},\pi,q^{\prime},q)\in{\cal C}^{n}_{{\cal W}}\times{\cal
C}^{n}_{{\cal W}}\times\mathbb{M}\times\mathbb{M},$ one defines

$\displaystyle
J\big{(}\pi^{\prime},\pi,q^{\prime},q\big{)}:=\int_{0}^{T}\bigg{[}\int_{{\cal
P}^{n}_{U}}\langle
L^{\circ}\big{(}t,\cdot,\cdot\big{)},\bar{\nu}^{\prime}\rangle
q^{\prime}_{t}(\mathrm{d}\bar{\nu}^{\prime})+\int_{{\cal P}^{n}_{U}}\langle
L^{\star}\big{(}t,\cdot,\bar{\nu}\big{)},\pi^{\prime}_{t}\rangle
q_{t}(\mathrm{d}\bar{\nu})\bigg{]}\mathrm{d}t+\langle
g(\cdot,\pi_{T}),\pi^{\prime}_{T}\rangle.$ (3.6)

######  Proposition 3.5.

$\mathrm{Q}^{\star}$ belongs to ${\cal S}^{\star}$ if and only if
$\mathrm{Q}^{\star}=\mathrm{P}^{\star}\circ(\mu,\Lambda)^{-1}$ where
$\mathrm{P}^{\star}\in\overline{\cal P}_{V}$, and for every
$\mathrm{P}\in\overline{\cal P}_{V}$ such that ${\cal
L}^{\mathrm{P}^{\star}}\big{(}\mu,\Lambda,B\big{)}={\cal
L}^{\mathrm{P}}\big{(}\mu,\Lambda,B\big{)}$, one has

$\displaystyle\mathbb{E}^{\mathrm{P}^{\star}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}\geq\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]},$
(3.7)

and for $\mathrm{P}^{\star}$–almost every $\omega\in\overline{\Omega},$

$\displaystyle\Lambda^{\prime}_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t=\Lambda_{t}\big{(}\mathrm{d}\bar{\nu}\big{)}\mathrm{d}t\;\mbox{and}\;\mu^{\prime}=\mu.$
(3.8)

######  Remark 3.6.

This definition of equilibrium is exactly the one proposed in [24] for the
$open$–$loop$ case. But as mention in Lemma 3.4 $($see also Remark 3.2$),$ as
$\mathrm{P}^{\star}[\mu=\mu^{\prime},\Lambda=\Lambda^{\prime}]=1,$ $B$ is in
fact
$(\sigma\\{\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\\})_{t\in[0,T]}$–adapted.
Therefore, for the definition of measure–valued MFG equilibrium, we only need
to focus on measure–valued control rules s.t. $\Lambda^{\prime}$ is
$(\sigma\\{\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable.

###### Proof.

Let $(\Omega,{\cal F},\mathrm{P},\mathbb{F},W,B,X,\Lambda,\mu)$ be a
measure–valued MFG equilibrium. Let us check that $\mathrm{P}^{\star}:={\cal
L}^{\mathbb{P}}\big{(}\mu,\mu,\Lambda,\Lambda,B\big{)}$ satisfies the desired
properties (3.7) and (3.8). Recall that, for each $t\in[0,T],$ $\mu_{t}={\cal
L}^{\mathrm{P}}\big{(}X_{t}\big{|}{\cal G}_{t}\big{)}$ $\mathrm{P}$–a.e. where
${\cal
G}_{t}:=\sigma\\{\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot},B_{t\wedge\cdot}\\}$
and $X$ is a weak solution of

$\displaystyle\mathrm{d}X_{t}=\int_{{\cal
P}^{n}_{U}}\;\int_{U}b\big{(}t,X_{t},\bar{\nu},u\big{)}\bar{\nu}^{X_{t}}(\mathrm{d}u)\;\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma\big{(}t,X_{t}\big{)}\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;\;X_{0}=\xi$

and $\Lambda_{t}(\mathbb{Z}_{\mu_{t}})=1$
$\mathrm{d}\mathbb{P}\otimes\mathrm{d}t$ a.e. By definition, the property
(3.8) is obviously verified. It is straightforward that
$\mathrm{P}^{\star}\in\overline{\cal P}_{V}.$ Consequently, by Lemma 3.3,
$B_{t}$ is a continuous function of
$(\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}).$ This means there exists a
continuous function $\varphi:{\cal C}^{n,1}_{{\cal W}}\times\mathbb{M}\to{\cal
C}^{n}$ s.t.
$\mathrm{P}\big{(}B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T]\big{)}=1.$

Let $\mathrm{P}\in\overline{\cal P}_{V}$ with
$\mathrm{P}\circ(\mu,\Lambda,B)^{-1}=\mathrm{P}^{\star}\circ(\mu,\Lambda,B)^{-1},$
by Lemma 3.4, there exists
$(\mathrm{P}^{j})_{j\in\mathbb{N}^{*}}\subset\overline{\cal P}_{V}$ s.t for
all $j\in\mathbb{N}^{*},$
$\mathrm{P}^{j}\circ(\mu,\Lambda,B)^{-1}=\mathrm{P}^{\star}\circ(\mu,\Lambda,B)^{-1},$
$\mathrm{P}^{j}:=\mathbb{P}^{\star}\circ\big{(}\mu^{\prime
j},\mu,\Lambda^{\prime j},\Lambda,B\big{)}^{-1}$ where $(\mu^{\prime
j},\mu,\Lambda^{\prime j},\Lambda)$ is defined in Lemma 3.4 and

$\displaystyle\displaystyle\lim_{j\to\infty}\mathbb{E}^{\mathrm{P}^{j}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}=\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

As $(\Omega,{\cal F},\mathrm{P},\mathbb{F},W,B,X,\Lambda,\mu)$ is a
measure–valued MFG equilibrium, one has the second property i.e. the
optimality

$\displaystyle\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}=\displaystyle\lim_{j\to\infty}\mathbb{E}^{\mathrm{P}^{j}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}\leq\mathbb{E}^{\mathrm{P}^{\star}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

This is enough to conclude one part.

Now, let $\mathrm{Q}^{\star}\in\overline{\cal P}_{V}$ satisfying (3.7) and
(3.8). Using Lemma 3.3, we can verify that $B$ is a continuous function of
$(\mu,\Lambda).$ Let $\mu,$ $\Lambda$ and $B$ be random variables on
$(\Omega,\mathbb{H},\mathbb{P})$ s.t. ${\cal
L}^{\mathbb{P}}(\mu,\Lambda,B,W,\xi)={\cal
L}^{\mathrm{Q}^{\star}}(\mu,\Lambda,B)\otimes{\cal L}^{\mathbb{P}}(W,\xi).$
Let $X$ be the solution of:

$\displaystyle\mathrm{d}X_{t}=\int_{{\cal
P}^{n}_{U}}\int_{U}b(t,X_{t},\bar{\nu},u)\bar{\nu}^{X_{t}}(\mathrm{d}u)\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},\;X_{0}=\xi,\;\mathbb{P}\mbox{--a.e.}$

then by uniqueness $\mu_{t}={\cal L}^{\mathbb{P}}(X_{t}|\mu,\Lambda)={\cal
L}^{\mathbb{P}}(X_{t}|\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),$
$\mathbb{P}$–a.e. for all $t\in[0,T].$ It is then easy to verify that
$(\Omega,{\cal F},\mathbb{P},\mathbb{F},W,B,X,\Lambda,\mu)$ is a
measure–valued MFG equilibrium and ${\cal L}^{\mathbb{P}}(\mu,\Lambda)={\cal
L}^{\mathrm{Q}}(\mu,\Lambda)$. ∎

Next, using the same measure–valued control rules, we define an equivalent
formulation of the McKean–Vlasov optimal control.

####  3.1.3 McKean–Vlasov optimal control on the canonical space

Let us define

$\displaystyle{\cal V}:=\big{\\{}\mathrm{P}\in\overline{\cal
P}_{V}:\;\;\mathrm{P}\big{(}\mu=\mu^{\prime},\Lambda=\Lambda^{\prime}\big{)}=1\big{\\}},$

${\cal K}:=\big{\\{}\mathrm{P}\in\overline{\cal
P}_{V}:\exists\;\gamma:[0,T]\times{\cal C}^{n}\to{\cal
P}(\mathbb{R}^{n})\;\mbox{s.t.}\;\;\mathrm{d}\mathrm{P}\otimes\mathrm{d}t\mbox{--a.e.}\;(t,\omega),\mu_{t}=\gamma(t,B_{t\wedge\cdot})\big{\\}}$
and

$\displaystyle{\cal V}^{c}:=\Big{\\{}\mathrm{P}\in{\cal V}\cap{\cal
K}:\exists\;\alpha\in{\cal
A}^{c}\;\mbox{s.t.}\;\mathrm{d}\mathrm{P}\otimes\mathrm{d}t\mbox{--a.e.}\;(t,\omega),\;\Lambda_{t}(\omega)\big{(}\bar{\nu}:\;\bar{\nu}=\delta_{\alpha(t,x,\mu(\omega))}(\mathrm{d}u)\mu_{t}(\omega)(\mathrm{d}x)\big{)}=1\Big{\\}}$

######  Proposition 3.7.

We have the following reformulation of $V^{c}_{S}$

$\displaystyle V^{c}_{S}=\sup_{\mathrm{P}\in{\cal
V}^{c}}\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda,\Lambda^{\prime})\big{]}.$

where $J$ is defined in Equation 3.6. Besides, under 2.1, one has

$\displaystyle V^{o}_{S}=\sup_{\mathrm{P}\in{\cal
V}}\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda,\Lambda^{\prime})\big{]}.$

###### Proof.

The first result is just a reformulation. The second one comes from [23,
Theorem 3.1]. ∎

###  3.2 Limit of Nash equilibria

This section is devoted to the analysis of the behavior of sequence of
$closed$–$loop$ Nash equilibria when the number of players goes to infinity.
At the end of this section, we will show that any limit point of sequence of
Nash equilibria is a measure–valued MFG equilibrium.

####  3.2.1 Technical results

Recall that $(\Omega,{\cal H},\mathbb{P})$ is a filtered probability space
supporting a ${\cal H}_{0}$–random variable $\xi$ s.t. ${\cal
L}^{\mathbb{P}}(\xi)=\nu,$ and a $\mathbb{R}^{n}\times\mathbb{R}^{n}$–valued
$\mathbb{H}$–Brownian motion $(W,B).$ Besides, let $\mu$ be a
$\mathbb{H}$–adapted ${\cal P}(\mathbb{R}^{n})$–valued continuous process and
$\Lambda$ be a ${\cal P}({\cal P}^{n}_{U})$–valued $\mathbb{H}$ predictable
process s.t. $B$ is $\mathbb{G}:=({\cal G}_{t})_{t\in[0,T]}$–adapted where
${\cal G}_{t}:=\sigma\\{\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\\}.$
Besides, $(\mu,\Lambda)$ is $\mathbb{P}$–independent of $(\xi,W).$ For a
continuous function

$\displaystyle\beta:[0,T]\times\mathbb{R}^{n}\times{\cal C}^{n,1}_{{\cal
W}}\times\mathbb{M}\to
U,\;\mbox{satisfying}\;\beta(t,x,\pi,q)=\beta(t,x,\pi_{t\wedge\cdot},q_{t\wedge\cdot}),$

let $X$ be the unique strong solution of:
$\mathbb{E}^{\mathbb{P}}[\|X\|^{p^{\prime}}]<\infty$ with $p^{\prime}>p,$ for
all $t\in[0,T]$

$\displaystyle\mathrm{d}X^{\prime}_{t}=\int_{{\cal
P}^{n}_{U}}b(t,X^{\prime}_{t},\bar{\nu},\beta(t,X^{\prime}_{t},\mu,\Lambda))\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+\sigma(t,X^{\prime}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;X_{0}=\xi.$
(3.9)

Also, we denote for all $t\in[0,T],$ $\mu^{\prime}_{t}:={\cal
L}(X^{\prime}_{t}|{\cal G}_{t}),$ $\overline{\mu}^{\prime}_{t}:={\cal
L}(X^{\prime}_{t},\beta(t,X^{\prime}_{t},\mu,\Lambda)|{\cal G}_{t})$ a.e. and
$\Lambda^{\prime}:=\delta_{\overline{\mu}^{\prime}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t.$

Notice that the process (3.9) is exactly the one used for the approximation of
measure–valued control rule in Lemma 3.4. In what follows, we will show that
it is possible to approximate this type of process by a sequence of
interacting processes when a certain condition (see Equation 3.11) is
satisfied (see Lemma 3.8).

For each $N\in\mathbb{N}^{*},$ and any $N$–progressively Borel measurable
functions $\boldsymbol{\alpha}:=(\alpha^{1},\cdots,\alpha^{N})$ s.t.
$\alpha^{i}:[0,T]\times({\cal C}^{n})^{N}\to U$ and define

$\displaystyle\beta^{\boldsymbol{\alpha},i}(t,\mathbf{x}):=\beta\big{(}t,\mathsf{x}^{i}(t),\pi[\mathbf{x}],q^{\boldsymbol{\alpha}}[\mathbf{x}]\big{)},$
(3.10)

where

$\displaystyle\mathbf{x}:=(\times^{1},\cdots,\times^{N}),\;\pi[\mathbf{x}]:=\frac{1}{N}\sum_{j=1}^{N}\delta_{\mathsf{x}^{j}},\;\overline{m}^{\boldsymbol{\alpha}}(t,\mathbf{x}):=\frac{1}{N}\sum_{j=1}^{N}\delta_{\big{(}\mathsf{x}^{j}(t),\;\alpha^{j}(t,\mathbf{x})\big{)}},\;\mbox{and}\;q^{\boldsymbol{\alpha}}[\mathbf{x}]:=\delta_{\overline{m}^{\boldsymbol{\alpha}}(t,\mathbf{x})}(\mathrm{d}\bar{\nu})\mathrm{d}t.$

######  Lemma 3.8.

Let 2.1 hold true, and a sequence $(\alpha^{i})_{i\in\mathbb{N}^{*}}$ s.t. for
each $N\in\mathbb{N}^{*},$
$\boldsymbol{\alpha}^{N}:=(\alpha^{1},\dots,\alpha^{N})\in({\cal
A}^{c}_{N})^{N}$ and

$\displaystyle\displaystyle\lim_{N\to\infty}\mathbb{P}\circ\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\delta_{\big{(}\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]\big{)}}(\mathrm{d}\bar{\nu}^{\prime})\mathrm{d}t,B\Big{)}^{-1}=\mathbb{P}\circ(\mu,\Lambda,B)^{-1}\;\;\mbox{in}\;{\cal
W}_{p}.$ (3.11)

Then

$\displaystyle\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{\boldsymbol{\alpha}^{N},i},\alpha^{i+1},\cdots,\alpha^{N}\big{)}=\mathbb{E}^{\mathbb{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

###### Proof.

The proof is largely inspired by [41, Proof of Proposition 5.6].We use some
probability changes on $(\Omega,\mathbb{H},\mathbb{P}).$ Given
$\boldsymbol{\alpha}:=(\alpha^{1},\dots,\alpha^{N})\in({\cal A}^{c}_{N})^{N},$
$\mathbf{X}^{\boldsymbol{\alpha}}:=(X^{\boldsymbol{\alpha},1},\dots,X^{\boldsymbol{\alpha},N})$
satisfies (2.1). For each $i\in\\{1,\cdots,N\\},$ we define

$\boldsymbol{\alpha}^{i}:=(\boldsymbol{\alpha}^{[-i]},\beta^{\boldsymbol{\alpha},i})\;\;\mbox{and}\;\;\mathbf{Y}^{i}:=(Y^{i,1},\cdots,Y^{i,N}):=(X^{\boldsymbol{\alpha}^{i},1},\dots,X^{\boldsymbol{\alpha}^{i},N}).$

Notice that, $Y^{i,i}$ satisfies

$\displaystyle\mathrm{d}Y^{i,i}_{t}=b\big{(}t,Y^{i,i}_{t},\overline{\mu}^{i,N}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{Y}^{i})\big{)}\mathrm{d}t+\sigma\big{(}t,Y^{i,i}_{t}\big{)}\mathrm{d}W^{i}_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;Y^{i,i}_{0}=X^{i}_{0}$

with

$\mu^{i,N}_{t}(\mathrm{d}x):=\frac{1}{N}\sum_{j=1}^{N}\delta_{Y^{i,j}_{t}}(\mathrm{d}x),\;\;\overline{\mu}^{i,N}_{t}(\mathrm{d}x,\mathrm{d}u):=\frac{1}{N}\big{(}\sum_{j\neq
i}^{N}\delta_{(Y^{i,j}_{t},\;\alpha^{j}(t,\mathbf{Y}^{i}))}(\mathrm{d}x,\mathrm{d}u)+\delta_{(Y^{i,i}_{t},\;\beta^{\boldsymbol{\alpha},i}(t,\mathbf{Y}^{i}))}(\mathrm{d}x,\mathrm{d}u)\big{)}$

$\mbox{and}\;\;\Lambda^{i,N}:=\delta_{\overline{\mu}^{i,N}_{t}}(\mathrm{d}\nu)\mathrm{d}t.$

Now, let us introduce

$\displaystyle
Z^{i}_{t}:=\exp\bigg{\\{}\int_{0}^{t}\phi^{i}_{r}\mathrm{d}W^{i}_{r}-\frac{1}{2}\int_{0}^{t}|\phi^{i}_{r}|^{2}\mathrm{d}r\bigg{\\}}\;\mbox{for
all}\;t\in[0,T],\;\mbox{and}\;\frac{\mathrm{d}\mathbb{Q}^{i}}{\mathrm{d}\mathbb{P}}:=Z^{i}_{T}$

with

$\displaystyle\phi^{i}_{t}:=\sigma(t,X^{\boldsymbol{\alpha},i}_{t})^{-1}\Big{(}b\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\zeta}^{i,N,\beta}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}-b\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}],\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\Big{)}$

and

$\displaystyle\overline{\zeta}^{i,N,\beta}_{t}:=\frac{1}{N}\big{(}\sum_{j\neq
i}^{N}\delta_{(X^{\boldsymbol{\alpha},j}_{t},\;\alpha^{j}(t,\mathbf{X}^{\boldsymbol{\alpha}}))}(\mathrm{d}x,\mathrm{d}u)+\delta_{(X^{\boldsymbol{\alpha},i}_{t},\;\beta^{\boldsymbol{\alpha},i}(t,\mathbf{X}^{\boldsymbol{\alpha}}))}(\mathrm{d}x,\mathrm{d}u)\big{)}.$

By uniqueness, we can check that ${\cal
L}^{\mathbb{Q}^{i}}(\mathbf{X}^{\alpha},B)={\cal
L}^{\mathbb{P}}(\mathbf{Y}^{i},B)$ for each $i.$ Notice that

$\displaystyle
J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{\boldsymbol{\alpha},i},\alpha^{i+1},\cdots,\alpha^{N}\big{)}$
$\displaystyle=\mathbb{E}^{\mathbb{P}}\bigg{[}\int_{0}^{T}L\big{(}t,Y^{i,i}_{t},\overline{\mu}^{i,N}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{Y}^{i})\big{)}\mathrm{d}t+g\big{(}Y^{i,i}_{T},\mu^{i,N}_{T}\big{)}\bigg{]}$
$\displaystyle=\mathbb{E}^{\mathbb{Q}^{i}}\bigg{[}\int_{0}^{T}L\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\zeta}^{i,N,\beta}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+g\big{(}X^{\boldsymbol{\alpha},i}_{T},\varphi^{N}_{T}[\boldsymbol{\alpha}]\big{)}\bigg{]}$
$\displaystyle=\mathbb{E}^{\mathbb{P}}\bigg{[}Z^{i}_{T}\bigg{(}\int_{0}^{T}L\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\zeta}^{i,N,\beta}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+g\big{(}X^{\boldsymbol{\alpha},i}_{T},\varphi^{N}_{T}[\boldsymbol{\alpha}]\big{)}\bigg{)}\bigg{]}.$

Next, let us consider the sequence
$(\mathrm{Q}^{N})_{N\in\mathbb{N}^{*}}\subset{\cal P}\Big{(}{\cal
P}\big{(}{\cal C}^{n}\times{\cal C}^{n}\times{\cal C}^{1}\times{\cal
C}^{n}_{{\cal W}}\times\mathbb{M}({\cal P}^{n}_{U}\times U)\times{\cal
C}^{n}\big{)}\times{\cal C}^{n}_{{\cal W}}\times\mathbb{M}({\cal
P}^{n}_{U})\times{\cal C}^{n}\Big{)}$

$\displaystyle\mathrm{Q}^{N}:=\mathbb{P}\circ\bigg{(}\frac{1}{N}\sum_{i=1}^{N}\delta_{\big{(}X^{\boldsymbol{\alpha},i}-\sigma_{0}B_{t},\;W^{i},\;Z^{i},\;\varphi^{N}[\boldsymbol{\alpha}],\;\Phi^{i},\;\gamma^{N},\;B\big{)}},\;\;\varphi^{N}[\boldsymbol{\alpha}],\;\gamma^{N},\;\;B\bigg{)}^{-1}$

where

$\displaystyle\Phi^{i}:=\delta_{\big{(}\overline{\zeta}^{i,N,\beta}_{t},\;\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}}(\mathrm{d}m,\mathrm{d}u)\mathrm{d}t\;\mbox{and}\;\gamma^{N}:=\delta_{{\overline{\varphi}}^{N}_{t}[\boldsymbol{\alpha}]}(\mathrm{d}\bar{\nu})\mathrm{d}t.$

By using the fact that $(b,\sigma)$ is bounded and the non–degeneracy of
$\sigma,$ with similar arguments to [16, Proposition A.1, Proposition A.2,
Proposition B.1], it is straightforward to check that the sequence
$(\mathrm{Q}^{N})_{N\in\mathbb{N}^{*}}$ is relatively compact in ${\cal
W}_{p}$ (recall that ${\cal L}^{\mathbb{P}}(\xi)=\nu\in{\cal
P}_{p^{\prime}}(\mathbb{R}^{n})$ with $p^{\prime}>p$). Denote
$\mathrm{Q}^{\infty}$ the limit of a sub–sequence, for sake of simplicity, we
will use the same notation for the sequence and its sub–sequence. We now want
to identify the limit. Mention that $Z^{i}$ verifies:
$\mathrm{d}Z^{i}_{t}=Z^{i}_{t}\phi^{i}_{t}\mathrm{d}W^{i}_{t}$ with
$Z^{i}_{0}=1.$ For any function twice differentiable
$f:\mathbb{R}^{n}\times\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R},$ let us
define

$\displaystyle{\cal A}f(t,x,w,h,z,\pi,q,m,u)$
$\displaystyle:=\nabla_{x}f(x,w,z)b\big{(}t,x+\sigma_{0}h,m,a\big{)}+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{x}f(x,w,z)\sigma(t,x+\sigma_{0}h)\sigma(t,x+\sigma_{0}h)^{\top}\big{]}$
$\displaystyle+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{w}f(x,w,z)\big{]}+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{z}f(x,w,z)\big{|}\sigma(t,x+\sigma_{0}h)^{-1}\big{(}b\big{(}t,x+\sigma_{0}h,m,\beta(t,x+\sigma_{0}h,\pi,q)\big{)}-b\big{(}t,x+\sigma_{0}h,m,u\big{)}\big{)}\big{|}^{2}\big{]}|z|^{2}$
$\displaystyle+\nabla_{w}\nabla_{x}f(x,w,z)\sigma(t,x+\sigma_{0}h)+\nabla_{z}\nabla_{x}f(x,w,z)z\big{(}b\big{(}t,x+\sigma_{0}h,m,\beta(t,x+\sigma_{0}h,\pi,q)\big{)}-b\big{(}t,x+\sigma_{0}h,m,u\big{)}\big{)}$
$\displaystyle+\nabla_{z}\nabla_{w}f(x,w,z)z\sigma(t,x+\sigma_{0}h)^{-1}\big{(}b\big{(}t,x+\sigma_{0}h,m,\beta(t,x+\sigma_{0}h,\pi,q)\big{)}-b\big{(}t,x+\sigma_{0}h,m,u\big{)}\big{)}$

and for
$(t,\mathsf{x},\mathsf{w},\mathsf{z},\mathsf{b},\pi,q,\widehat{\mathbf{q}})\in[0,T]\times{\cal
C}^{n}\times{\cal C}^{n}\times{\cal C}^{1}\times{\cal C}^{n}\times{\cal
C}^{n}_{{\cal W}}\times\mathbb{M}({\cal P}^{n}_{U})\times\mathbb{M}({\cal
P}^{n}_{U}\times U)$

$\displaystyle\mathrm{M}f[t,\mathsf{x},\mathsf{w},\mathsf{b},\mathsf{z},\pi,q,\widehat{\mathbf{q}}]:=f(\mathsf{x}(t),\mathsf{w}(t),\mathsf{z}(t))-\int_{0}^{t}\int_{{\cal
P}^{n}_{U}\times U}{\cal
A}f(s,\mathsf{x}(s),\mathsf{w}(s),\mathsf{z}(s),\mathsf{b}(s),\pi,\mathbf{q},m,u)\widehat{\mathbf{q}}_{s}(\mathrm{d}m,\mathrm{d}u)\mathrm{d}s.$

We denote by $\widehat{\Omega}:={\cal C}^{n}\times{\cal C}^{n}\times{\cal
C}^{n}\times{\cal C}^{1}\times{\cal C}^{n}_{{\cal W}}\times\mathbb{M}({\cal
P}^{n}_{U}\times U),$
$(\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\widehat{\mu},\widehat{\Phi},\widehat{\gamma})$
the canonical processes, and $\widehat{\mathbb{F}}$ its canonical filtration.
Also, we note $(\Pi,\mu,\gamma,B)$ the canonical element of $\Omega:={\cal
P}(\widehat{\Omega})\times{\cal C}^{n}_{{\cal W}}\times\mathbb{M}({\cal
P}^{n}_{U})\times{\cal C}^{n}.$ Let $0\leq s\leq t\leq T,$ and a continuous
bounded function $h:\widehat{\Omega}\to\mathbb{R},$ with

$\displaystyle
h_{s}:=h(\widehat{X}_{s\wedge\cdot},\widehat{W}_{s\wedge\cdot},\widehat{B}_{s\wedge\cdot},\widehat{Z}_{s\wedge\cdot},\widehat{\mu}_{s\wedge\cdot},\widehat{\Phi}_{s\wedge\cdot}).$

Let
$\widehat{X}^{\boldsymbol{\alpha},i}=X^{\boldsymbol{\alpha},i}-\sigma_{0}B.$
By Itô formula, we know that

$\displaystyle\mathrm{d}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})$
$\displaystyle=\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\sigma(t,X^{\boldsymbol{\alpha},i}_{t})\mathrm{d}W^{i}_{t}+\nabla_{w}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\mathrm{d}W^{i}_{t}+\nabla_{z}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})Z^{i}_{t}\phi^{i}_{t}\mathrm{d}W^{i}_{t}$
$\displaystyle+\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})b\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}],\alpha^{i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\sigma(t,X^{\boldsymbol{\alpha},i}_{t})\sigma(t,X^{\boldsymbol{\alpha},i}_{t})^{\top}\big{]}\mathrm{d}t$
$\displaystyle+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{w}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\big{]}\mathrm{d}t+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{z}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\phi^{i}_{t}(\phi^{i}_{t})^{\top}\big{]}|Z^{i}_{t}|^{2}\mathrm{d}t$
$\displaystyle+\nabla_{w}\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\sigma(t,X^{\boldsymbol{\alpha},i}_{t})\mathrm{d}t+\nabla_{z}\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})\sigma(t,X^{\boldsymbol{\alpha},i}_{t})Z^{i}_{t}\phi^{i}_{t}\mathrm{d}t$
$\displaystyle+\nabla_{z}\nabla_{w}f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})Z^{i}_{t}\phi^{i}_{t}\mathrm{d}t.$

Let us introduce
$h^{i}_{s}:=h(\widehat{X}^{\boldsymbol{\alpha},i}_{s\wedge\cdot},W^{i}_{s\wedge\cdot},Z^{i}_{s\wedge\cdot},\varphi^{N}_{s\wedge\cdot}[\boldsymbol{\alpha}],\Phi^{i}_{s\wedge\cdot}),$
and

$\displaystyle\mathrm{M}^{i}_{t}f$
$\displaystyle:=f(\widehat{X}^{\boldsymbol{\alpha},i}_{t},W^{i}_{t},Z^{i}_{t})$
$\displaystyle-\int_{0}^{t}\bigg{[}\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})b\big{(}r,X^{\boldsymbol{\alpha},i}_{r},\overline{\varphi}^{N}_{r}[\boldsymbol{\alpha}],\alpha^{i}(r,X^{\boldsymbol{\alpha}})\big{)}+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})\sigma(r,X^{\boldsymbol{\alpha},i}_{r})\sigma(r,X^{\boldsymbol{\alpha},i}_{r})^{\top}\big{]}$
$\displaystyle\;\;+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{w}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})\big{]}+\frac{1}{2}\mathrm{Tr}\big{[}\nabla^{2}_{z}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})\phi^{i}_{r}(\phi^{i}_{r})^{\top}\big{]}|Z^{i}_{r}|^{2}$
$\displaystyle\;\;+\nabla_{w}\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})\sigma(r,X^{\boldsymbol{\alpha},i}_{r})+\nabla_{z}\nabla_{x}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})\sigma(r,X^{\boldsymbol{\alpha},i}_{r})Z^{i}_{r}\phi^{i}_{r}+\nabla_{z}\nabla_{w}f(\widehat{X}^{\boldsymbol{\alpha},i}_{r},W^{i}_{r},Z^{i}_{r})Z^{i}_{r}\phi^{i}_{r}\bigg{]}\mathrm{d}r.$

Notice that for $\mathrm{d}t\otimes\mathrm{d}\mathbb{P}$–a.e.

$\displaystyle\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}{\cal
W}_{p}\Big{(}\overline{\zeta}^{i,N,\beta}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}]\Big{)}=0.$
(3.12)

Then, in the sub–sequence $(\mathrm{Q}^{N})_{N\in\mathbb{N}^{*}},$ we can use
$\overline{\zeta}^{i,N,\beta}_{t}$ or
$\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}]$ without affected the limit
$\mathrm{Q}^{\infty}.$ With all the previous observations, one can check that

$\displaystyle\mathbb{E}^{\mathrm{Q}^{\infty}}\Big{[}\Big{|}\mathbb{E}^{\Pi}\Big{[}\Big{(}\mathrm{M}f[t,\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\widehat{\mu},\widehat{\gamma},\widehat{\Phi}]-\mathrm{M}f[s,\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\widehat{\mu},\widehat{\gamma},\widehat{\Phi}]\Big{)}h_{s}\Big{]}\Big{|}^{2}\Big{]}$
$\displaystyle=\displaystyle\lim_{N}\mathbb{E}^{\mathrm{Q}^{N}}\Big{[}\Big{|}\mathbb{E}^{\Pi}\Big{[}\Big{(}\mathrm{M}f[t,\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\widehat{\mu},\widehat{\gamma},\widehat{\Phi}]-\mathrm{M}f[s,\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\widehat{\mu},\widehat{\gamma},\widehat{\Phi}]\Big{)}h_{s}\Big{]}\Big{|}^{2}\Big{]}=\displaystyle\lim_{N}\mathbb{E}^{\mathbb{P}}\Big{[}\Big{|}\frac{1}{N}\sum_{i=1}^{N}\big{(}\mathrm{M}^{i}_{t}f-\mathrm{M}^{i}_{s}f\big{)}h^{i}_{s}\Big{|}^{2}\Big{]}$
$\displaystyle=\displaystyle\lim_{N}\frac{1}{N^{2}}\sum_{i=1}^{N}\mathbb{E}^{\mathbb{P}}\Big{[}\Big{|}\big{(}\mathrm{M}^{i}_{t}f-\mathrm{M}^{i}_{s}f\big{)}h^{i}_{s}\Big{|}^{2}\Big{]}=0.$

Notice that the character bounded of coefficients $(b,\sigma)$ and test
functions $(f,h)$ is important for passing to the limit. This is true for all
$h,$ then we can deduce that $\mathrm{Q}^{\infty}$ $\omega$–a.e., for each $f$
bounded twice differentiable, $\mathrm{M}f$ is a
$(\Pi(\omega),\widehat{\mathbb{F}})$–martingale (see similar method in [39,
Proof of Proposition 5.1], [41, Proof of Proposition 5.6], [26, roposition
4.17]). Also, using Equation 3.12, it is easy to check that

$\displaystyle\Pi(\omega)\big{[}\widehat{\mu}=\mu(\omega),\;\widehat{\gamma}=\gamma(\omega),\;\widehat{B}=B(\omega)\big{]}=1,\;\mathrm{Q}^{\infty}\;\omega\mbox{--a.e.}$

Consequently, $\mathrm{Q}^{\infty}\;\omega\mbox{--a.e.},$ on an extension
$\big{(}\widehat{\Omega},\widehat{\mathbb{F}},\widehat{\mathbb{P}}_{{\omega}}\big{)}:=\big{(}\widehat{\Omega}\times[0,1],(\widehat{{\cal
F}}_{t}\otimes{\cal B}([0,1]))_{t\in[0,T]},\Pi({\omega})\otimes\lambda\big{)}$
of $\big{(}\widehat{\Omega},(\widehat{{\cal
F}}_{t})_{t\in[0,T]},\Pi({\omega})\big{)},$ there exists $\widehat{M}$ a
$(\widehat{\mathbb{F}},\widehat{\mathbb{P}}_{{\omega}})$–martingale measure
with quadratic variation $\widehat{\Phi}$ s.t.
$\widehat{\mathbb{P}}_{\omega}$–a.e.

$\displaystyle\mathrm{d}\widehat{X}_{t}=\int_{{\cal
P}^{n}_{U}}b^{\star}(t,\bar{\nu})\gamma_{t}(\omega)(\mathrm{d}\bar{\nu})\mathrm{d}t+\int_{U}b^{\circ}(t,\widetilde{X}_{t},u)\widehat{\Phi}_{t}({\cal
P}^{n}_{U},\mathrm{d}u)\mathrm{d}t+\sigma(t,\widetilde{X}_{t})\mathrm{d}\widehat{W}_{t},\;\widehat{W}_{t}=\widehat{M}({\cal
P}^{n}_{U}\times U\times[0,t]),$
$\displaystyle\mathrm{d}\widehat{Z}_{t}=\int_{{\cal P}^{n}_{U}\times
U}\widehat{Z}_{t}\;\sigma(t,\widetilde{X}_{t})^{-1}\Big{(}b\big{(}t,\widetilde{X}_{t},\bar{\nu},\beta(t,\widetilde{X}_{t},\mu(\omega),\gamma(\omega))\big{)}-b\big{(}t,\widetilde{X}_{t},\bar{\nu},u\big{)}\Big{)}\widehat{M}(\mathrm{d}\bar{\nu},\mathrm{d}u,\mathrm{d}t)\;\;\mbox{with}\;\widetilde{X}:=\widehat{X}+\sigma_{0}B(\omega),$

and
$\widehat{\mathbb{P}}_{\omega}\circ\big{(}\widehat{X},\widehat{W},\widehat{B},\widehat{Z},\mu(\omega),\Phi,\gamma(\omega)\big{)}^{-1}=\Pi(\omega).$

Now, we define the following change of probability

$\displaystyle\frac{\mathrm{d}\widehat{\mathbb{Q}}_{\omega}}{\mathrm{d}\widehat{\mathbb{P}}_{\omega}}:=\widehat{Z}_{T}=\exp\Big{\\{}\Delta_{T}-\frac{1}{2}\langle\Delta\rangle_{T}\Big{\\}}$

where

$\displaystyle\Delta_{t}:=\int_{0}^{t}\int_{{\cal P}^{n}_{U}\times
U}\sigma(r,\widetilde{X}_{r})^{-1}\Big{(}b\big{(}r,\widetilde{X}_{r},\bar{\nu},\beta(r,\widetilde{X}_{r},\mu(\omega),\gamma(\omega))\big{)}-b\big{(}r,\widetilde{X}_{r},\bar{\nu},u\big{)}\Big{)}\widehat{M}(\mathrm{d}\bar{\nu},\mathrm{d}u,\mathrm{d}r).$

Let us introduce

$\displaystyle\widehat{V}_{t}:=\widehat{W}_{t}-\int_{0}^{t}\int_{{\cal
P}^{n}_{U}\times
U}\Big{(}b\big{(}r,\widetilde{X}_{r},\bar{\nu},\beta(r,\widetilde{X}_{r},\mu(\omega),\gamma(\omega))\big{)}-b\big{(}r,\widetilde{X}_{r},\bar{\nu},u\big{)}\Big{)}\widehat{M}(\mathrm{d}\bar{\nu},\mathrm{d}u,\mathrm{d}r),$

then, by Girsanov’s Theorem, $\widehat{V}$ is a
$(\widehat{\mathbb{F}},\widehat{\mathbb{Q}}_{\omega})$–Brownian motion, and
$\widehat{X}$ satisfies

$\displaystyle\widehat{X}_{t}=\xi+\int_{0}^{t}\int_{{\cal
P}^{n}_{U}}b\big{(}r,\widehat{X}_{r}+\sigma_{0}B_{r}(\omega),\bar{\nu},\beta(r,\widehat{X}_{r}+\sigma_{0}B_{r}(\omega),\mu(\omega),\gamma(\omega))\big{)}\gamma_{r}(\omega)(\mathrm{d}\bar{\nu})\mathrm{d}r+\int_{0}^{t}\sigma(r,\widehat{X}_{r}+\sigma_{0}B_{r}(\omega))\mathrm{d}\widehat{V}_{r}.$
(3.13)

By Equation 3.12, one finds that

$\displaystyle\mathrm{Q}^{\infty}\circ\big{(}\mu,\gamma,B\big{)}^{-1}=\displaystyle\lim_{N}\mathbb{P}\circ\Big{(}\varphi^{N}[\boldsymbol{\alpha}],\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}]}(\mathrm{d}\bar{\nu})\mathrm{d}t,B\Big{)}^{-1}=\mathbb{P}\circ(\mu,\Lambda,B)^{-1}.$

By uniqueness of (3.13) (and Equation 3.9), we deduce that

$\displaystyle\int_{\Omega}{\cal
L}^{\widehat{\mathbb{Q}}_{\omega}}\big{(}\widehat{X},\mu(\omega),\gamma(\omega),B(\omega)\big{)}\mathrm{Q}^{\infty}(\mathrm{d}\omega)=\mathbb{E}^{\mathbb{P}}\big{[}{\cal
L}^{\mathbb{P}}\big{(}X^{\prime}-\sigma_{0}B,\mu,\Lambda,B\big{|}{\cal
G}_{T}\big{)}\big{]}=\mathbb{P}\circ\big{(}X^{\prime}-\sigma_{0}B,\mu,\Lambda,B\big{)}^{-1}.$

This is true for any limit point $\mathrm{Q}^{\infty},$ then we have the
convergence of the entire sequence $(\mathrm{Q}^{N})_{N\in\mathbb{N}^{*}}.$
Therefore

$\displaystyle\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{\boldsymbol{\alpha},i},\alpha^{i+1},\cdots,\alpha^{N}\big{)}$
$\displaystyle=\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}^{\mathbb{P}}\bigg{[}Z^{i}_{T}\bigg{(}\int_{0}^{T}L\big{(}t,X^{\boldsymbol{\alpha},i}_{t},\overline{\zeta}^{i,N,\beta}_{t},\beta^{\boldsymbol{\alpha},i}(t,\mathbf{X}^{\boldsymbol{\alpha}})\big{)}\mathrm{d}t+g\big{(}X^{\boldsymbol{\alpha},i}_{T},\varphi^{N}_{T}[\boldsymbol{\alpha}]\big{)}\bigg{)}\bigg{]}$
$\displaystyle=\int_{\Omega}\mathbb{E}^{\Pi(\omega)}\bigg{[}\widehat{Z}_{T}\bigg{(}\int_{0}^{T}\int_{{\cal
P}^{n}_{U}}L\big{(}t,\widetilde{X}_{t},\bar{\nu},\beta(t,\widetilde{X}_{t},\mu(\omega),\gamma(\omega)\big{)}\gamma_{t}(\omega)(\mathrm{d}\bar{\nu})\mathrm{d}t+g\big{(}\widetilde{X}_{T},\mu_{T}(\omega)\big{)}\bigg{)}\bigg{]}\mathrm{Q}^{\infty}(\mathrm{d}\omega)$
$\displaystyle=\int_{\Omega}\mathbb{E}^{\widehat{\mathbb{Q}}(\omega)}\bigg{[}\int_{0}^{T}\int_{{\cal
P}^{n}_{U}}L\big{(}t,\widetilde{X}_{t},\bar{\nu},\beta(t,\widetilde{X}_{t},\mu(\omega),\gamma(\omega)\big{)}\gamma_{t}(\omega)(\mathrm{d}\bar{\nu})\mathrm{d}t+g\big{(}\widetilde{X}_{T},\mu_{T}(\omega)\big{)}\bigg{]}\mathrm{Q}^{\infty}(\mathrm{d}\omega)$
$\displaystyle=\mathbb{E}^{\mathbb{P}}\bigg{[}\int_{0}^{T}\int_{{\cal
P}^{n}_{U}}L\big{(}t,X^{\prime}_{t},\bar{\nu},\beta(t,X^{\prime}_{t},\mu,\Lambda\big{)}\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t+g\big{(}X^{\prime}_{T},\mu_{T}\big{)}\bigg{]}.$

∎

Now, by combining the previous proposition and Lemma 3.4, we show that any
measure–valued control rule satisfying a certain condition ( see Equation
3.14) is the limit of a sequence of interacting processes.

######  Proposition 3.9.

Let $\mathrm{P}\in\overline{\cal P}_{V}$ s.t. there exists a continuous
function $\varphi:{\cal C}^{n,1}_{{\cal W}}\times\mathbb{M}\to{\cal C}^{n}$
s.t.
$\mathrm{P}(B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T])=1,$
and a sequence $(\alpha^{i})_{i\in\mathbb{N}^{*}}$ s.t. for each
$N\in\mathbb{N}^{*},$
$\boldsymbol{\alpha}^{N}:=(\alpha^{1},\dots,\alpha^{N})\in({\cal
A}^{c}_{N})^{N}$ and

$\displaystyle\displaystyle\lim_{N\to\infty}\mathbb{P}\circ\Big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\delta_{\big{(}\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]\big{)}}(\mathrm{d}\bar{\nu})\mathrm{d}t,B\Big{)}^{-1}=\mathrm{P}\circ\big{(}\mu,\Lambda,B\big{)}^{-1}.$
(3.14)

Then there exists a sequence
$(\beta^{i,N})_{(i,N)\in\\{1,\cdots,N\\}\times\mathbb{N}^{*}}$ satisfying for
each $(i,N)\in\\{1,\cdots,N\\}\times\mathbb{N}^{*},$ $\beta^{i,N}\in{\cal
A}^{c}_{N},$ and

$\displaystyle\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{i,N},\alpha^{i+1},\cdots,\alpha^{N}\big{)}=\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

###### Proof.

By Lemma 3.4, we can assume that
$\mathrm{P}=\mathbb{P}\circ\big{(}\mu^{\prime},\mu,\Lambda^{\prime},\Lambda,B\big{)}^{-1}$
where for each $t\in[0,T],$ $\mu^{\prime}_{t}={\cal
L}^{\mathbb{P}}\big{(}X^{\prime}_{t}\big{|}{\cal G}_{t}\big{)},$
$\overline{\mu}^{\prime}_{t}={\cal
L}^{\mathbb{P}}\big{(}X^{\prime}_{t},\widehat{\beta}(t,X^{\prime}_{t},\mu,\Lambda)\big{|}{\cal
G}_{t}\big{)}$ and
$\Lambda^{\prime}=\delta_{\overline{\mu}^{\prime}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,$
with

$\displaystyle\mathrm{d}X^{\prime}_{t}=\int_{{\cal
P}^{n}_{U}}b(t,X^{\prime}_{t},\bar{\nu}^{\prime},\widehat{\beta}(t,X^{\prime}_{r},\mu,\Lambda))\Lambda_{r}(\mathrm{d}\bar{\nu}^{\prime})\mathrm{d}t+\sigma(t,X^{\prime}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;X^{\prime}_{0}=\xi$

and $\widehat{\beta}$ is a continuous function satisfying:

$\displaystyle\widehat{\beta}:[0,T]\times\mathbb{R}^{n}\times{\cal
C}^{n,1}_{{\cal W}}\times\mathbb{M}\to
U\;\mbox{with}\;\widehat{\beta}(t,x,\pi,q)=\widehat{\beta}(t,x,\pi_{t\wedge\cdot},q_{t\wedge\cdot}).$

Then by Lemma 3.8, if $(\widehat{\beta})^{\alpha,i}$ is defined in (3.10), one
has

$\displaystyle\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},(\widehat{\beta})^{\boldsymbol{\alpha}^{N},i},\alpha^{i+1},\cdots,\alpha^{N}\big{)}=\mathbb{E}^{\mathbb{P}^{\star}_{u}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

By combining our previous results, we get the proof of the proposition.

∎

####  3.2.2 Proof of Theorem 2.10 (limit set of approximate Nash equilibria)

By using [23, Proposition 5.4.] , one finds
$(\mathrm{P}^{N})_{N\in\mathbb{N}^{*}}$ is relatively compact in ${\cal
W}_{p}$ where

$\displaystyle\mathrm{P}^{N}:=\mathbb{P}^{\star}\circ\Big{(}(\varphi^{N}_{t}[\boldsymbol{\alpha}^{N}])_{t\in[0,T]},(\varphi^{N}_{t}[\boldsymbol{\alpha}^{N}])_{t\in[0,T]},\delta_{\overline{\varphi}^{N}_{s}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}\bar{\nu})\mathrm{d}s,\delta_{\overline{\varphi}^{N}_{s}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}\bar{\nu}^{\prime})\mathrm{d}s,B\Big{)}^{-1},$

and each limit point $\mathrm{P}^{\infty}$ of any sub–sequence belongs to
$\overline{\cal P}_{V}.$ Next, let us show that
$\mathrm{P}^{\infty}\circ(\mu,\Lambda)^{-1}\in{\cal S}^{\star}.$ To simplify,
the sequence $(\mathrm{P}^{N})_{N\in\mathbb{N}^{*}}$ and its sub–sequence
share the same notation. It is straightforward that
$\mathrm{P}^{\infty}[\mu=\mu^{\prime},\Lambda=\Lambda^{\prime}]=1.$ Therefore,
by Lemma 3.3, there exists a continuous function $\varphi:{\cal
C}^{n,1}_{{\cal W}}\times\mathbb{M}\to{\cal C}^{n}$ s.t.
$\mathrm{P}^{\infty}(B_{t}=\varphi_{t}(\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}),\;t\in[0,T])=1.$

Let $\mathrm{P}\in\overline{\cal P}_{V}$ such that ${\cal
L}^{\mathrm{P}}\big{(}\mu,\Lambda,B\big{)}={\cal
L}^{\mathrm{P}^{\infty}}\big{(}\mu,\Lambda,B\big{)}.$ By Proposition 3.9,
there exists a sequence
$(\beta^{i,N})_{(i,N)\in\\{1,\cdots,N\\}\times\mathbb{N}^{*}}$ satisfying for
each $(i,N)\in\\{1,\cdots,N\\}\times\mathbb{N}^{*},$ $\beta^{i,N}\in{\cal
A}^{c}_{N},$ and

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{i,N},\alpha^{i+1},\cdots,\alpha^{N}\big{)}=\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}.$

Therefore

$\displaystyle\mathbb{E}^{\mathrm{P}^{\infty}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}$
$\displaystyle=\displaystyle\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}J_{i}(\boldsymbol{\alpha}^{N})$
$\displaystyle\geq\displaystyle\lim_{N\to\infty}\bigg{(}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}\alpha^{1},\cdots,\alpha^{i-1},\beta^{i,N},\alpha^{i+1},\cdots,\alpha^{N}\big{)}-\varepsilon_{N}\bigg{)}=\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]},$

then
$\mathbb{E}^{\mathrm{P}^{\infty}}\big{[}J(\mu^{\prime},\mu,\Lambda^{\prime},\Lambda)\big{]}\geq\mathbb{E}^{\mathrm{P}}\big{[}J(\mu,\mu^{\prime},\Lambda,\Lambda^{\prime})\big{]},$
for any $\mathrm{P}\in\overline{\cal P}_{V}$ such that ${\cal
L}^{\mathrm{P}}\big{(}\mu,\Lambda,B\big{)}={\cal
L}^{\mathrm{P}^{\infty}}\big{(}\mu,\Lambda,B\big{)}.$ We conclude that
$\mathrm{P}^{\infty}\circ(\mu,\Lambda)^{-1}\in{\cal S}^{\star}.$

###  3.3 Limit set of approximate strong Markovian MFG equilibria and
approximation of measure–valued MFG equilibrium

In this section, first, we will prove the second part of Theorem 2.10 namely
showing that the limit set of approximate strong Markovian MFG solution is the
measure–valued MFG equilibrium. Second, we will prove the results of Theorem
2.12.

To achieve these goals, we need to introduce the notion of approximate
$open$–$loop$ MFG equilibrium. Let $\mathbb{F}^{B}:=({\cal
F}^{B}_{t})_{t\in[0,T]}$ be the natural filtration of $B$ i.e. ${\cal
F}^{B}_{t}:=\sigma\\{B_{s}:s\leq t\\}.$ Let $\alpha$ be a
$(\sigma\\{X_{0},W_{t\wedge\cdot},B_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable
process. We denote by $X$ the unique solution of

$\displaystyle\mathrm{d}X_{t}=b\big{(}t,X_{t},\overline{\mu}_{t},\alpha_{t}\big{)}\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\;\;\mbox{with}\;X_{0}=\xi,\;\mu_{t}:={\cal
L}(X_{t}|{\cal F}^{B}_{t})\;\mbox{and}\;\overline{\mu}_{t}:={\cal
L}(X_{t},\alpha_{t}|{\cal F}^{B}_{t}).$ (3.15)

Given $\overline{\mu},$ let $\alpha^{\prime}$ be another
$(\sigma\\{X_{0},W_{t\wedge\cdot},B_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable
process. let $X^{\prime}$ be the solution of

$\displaystyle\mathrm{d}X^{\prime}_{t}=b\big{(}t,X^{\prime}_{t},\overline{\mu}_{t},\alpha^{\prime}_{t}\big{)}\mathrm{d}t+\sigma(t,X^{\prime}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\;\;\mbox{with}\;X^{\prime}_{0}=\xi.$
(3.16)

We denote $\mu^{\alpha^{\prime}}_{t}:={\cal L}(X^{\prime}_{t}|{\cal
F}^{B}_{t})$ and $\overline{\mu}^{\alpha^{\prime}}_{t}:={\cal
L}(X^{\prime}_{t},\alpha^{\prime}_{t}|{\cal F}^{B}_{t})$ a.e. Let
$\varepsilon\geq 0,$ we say that the McKean–Vlasov process $X$ associated to
the control $\alpha$ is an $\varepsilon$–$open$–$loop$ MFG equilibrium if :

$\displaystyle\mathbb{E}\big{[}J\big{(}\mu,\mu,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}\geq\sup_{\alpha^{\prime}}\mathbb{E}\big{[}J\big{(}\mu^{\alpha^{\prime}},\mu,\delta_{\overline{\mu}^{\alpha^{\prime}}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\varepsilon.$

####  3.3.1 Approximate strong Markovian MFG equilibrium as an approximate
$open$–$loop$ MFG equilibrium

The next Proposition show that any approximate strong Markovian MFG
equilibrium can be seen as an approximate $open$–$loop$ MFG equilibrium.

######  Proposition 3.10.

For any $\varepsilon$–strong Markovian MFG equilibrium
$(\overline{\mu}_{t})_{t\in[0,T]}=({\cal L}(X_{t},\alpha(t,X_{t},\mu)|{\cal
F}^{B}_{t}))_{t\in[0,T]},$ the process $X$ associated to the control
$(\alpha(t,X_{t},\mu))_{t\in[0,T]}$ is a $\varepsilon$–$open$–$loop$ MFG
equilibrium.

###### Proof.

As $X$ is a strong solution of Equation 2.2 and $\mu$ is
$\mathbb{F}^{B}$–adapted, $(\alpha(t,X_{t},\mu))_{t\in[0,T]}$ is a
$(\sigma\\{X_{0},W_{t\wedge\cdot},B_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable
process. To prove our proposition, it suffices to show the optimality
condition. Let $\alpha^{\prime}$ be a
$(\sigma\\{X_{0},W_{t\wedge\cdot},B_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable
process and $X^{\prime}$ solution of Equation 3.16. Let us introduce
$\mu^{\prime}_{t}:={\cal L}(X^{\prime}_{t}|{\cal F}^{B}_{t})$ and
$\overline{\mu}^{\prime}_{t}:={\cal
L}(X^{\prime}_{t},\alpha^{\prime}_{t}|{\cal F}^{B}_{t}).$ We denote by
$R^{\prime}$ the solution of

$\displaystyle\mathrm{d}R^{\prime}_{t}=\int_{U}b\big{(}t,R^{\prime}_{t},\overline{\mu}_{t},u\big{)}\overline{\mu}^{\prime
R^{\prime}_{t}}_{t}\mathrm{d}t+\sigma(t,R^{\prime}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\;\;\mbox{with}\;R^{\prime}_{0}=\xi,$

$(\mu^{\prime x}_{t})_{x\in\mathbb{R}^{n}}$ is the disintegration of
$\overline{\mu}^{\prime}_{t}$ i.e.
$\overline{\mu}^{\prime}_{t}(\mathrm{d}u,\mathrm{d}x)=\overline{\mu}^{\prime
x}_{t}(\mathrm{d}u)\mu_{t}(\mathrm{d}x).$ By uniqueness, we can check that
$\mu^{\prime}_{t}={\cal L}(R^{\prime}_{t}|{\cal F}^{B}_{t})$ a.e. Now, we can
apply Proposition A.6. By Proposition A.6, there exists a sequence of
continuous functions $(\beta^{k})_{k\in\mathbb{N}^{*}}$ satisfying: for each
$k\in\mathbb{N}^{*},$

$\displaystyle\beta^{k}:[0,T]\times\mathbb{R}^{n}\times{\cal
C}^{n}\times\mathbb{M}\to
U\;\mbox{with}\;\beta^{k}(t,x,b,v)=\beta^{k}(t,x,b_{t\wedge\cdot},v_{t\wedge\cdot}).$

such that if we let $X^{\prime k}$ be the unique strong solution of:

$\displaystyle\mathrm{d}X^{\prime k}_{t}=b(t,X^{\prime
k}_{t},\overline{\mu}_{t},\beta^{k}(t,X^{\prime
k}_{t},B,\Lambda))\mathrm{d}t+\sigma(t,X^{\prime
k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;X^{\prime
k}_{0}=\xi,\;\Lambda:=\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,$

with $\mu^{\prime k}_{t}:={\cal L}(X^{\prime k}_{t}|{\cal F}^{B}_{t})$ and
$\overline{\mu}^{\prime k}_{t}:={\cal L}\big{(}X^{\prime
k}_{t},\beta^{k}(t,X^{k}_{t},B,\Lambda)|{\cal F}^{B}_{t}\big{)}$ then

$\displaystyle\displaystyle\lim_{k\to\infty}\big{(}\mu^{\prime
k},\delta_{\overline{\mu}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}=\big{(}\mu^{\prime},\delta_{\overline{\mu}^{\prime}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)},\;\mathbb{P}\mbox{--a.e.},\;\mbox{for
the Wasserstein metric}\;{\cal W}_{p}.$

With the same techniques used in Lemma 3.3, we can show that $B$ is
$(\sigma\\{\mu_{t\wedge\cdots},\Lambda_{t\wedge\cdot}\\})_{t\in[0,T]}$–adapted.
As a result, we can find a progressively Borel measurable function
$\tilde{\beta}^{k}:[0,T]\times\mathbb{R}^{n}\times{\cal C}^{n}_{\cal
W}\times\mathbb{M}\to U$ such that $\beta^{k}(t,X^{\prime
k}_{t},B,\Lambda)=\tilde{\beta}^{k}(t,X^{\prime k}_{t},\mu,\Lambda)$ a.e.
Consequently, as $\overline{\mu}$ is an $\varepsilon$–strong Markovian MFG
equilibrium, one has

$\displaystyle\mathbb{E}\big{[}J\big{(}\mu,\mu,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}$
$\displaystyle\geq\displaystyle\lim_{k\to\infty}\mathbb{E}\big{[}J\big{(}\mu^{\prime
k},\mu,\delta_{\overline{\mu}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\varepsilon$
$\displaystyle\geq\mathbb{E}\big{[}J\big{(}\mu^{\prime},\mu,\delta_{\overline{\mu}^{\prime}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\varepsilon.$

This is enough to conclude. ∎

####  3.3.2 Proof of Theorem 2.10 (limit set of approximate strong Markovian
MFG equilibria)

Let $(\overline{\mu}^{\ell})_{\ell\in\mathbb{N}*}$ be a sequence s.t. for each
$\ell\geq 1,$ $\overline{\mu}^{\ell}$ is an $\varepsilon_{\ell}$–strong
Markovian MFG equilibrium for $\varepsilon_{\ell}\geq 0.$ By Proposition 3.10,
$\overline{\mu}^{\ell}$ can be seen as an $\varepsilon_{\ell}$ $open$–$loop$
MFG equilibrium. Let $(\mathrm{P}^{\ell})_{\ell\in\mathbb{N}^{*}}$ be the
sequence defined by

$\displaystyle\mathrm{P}^{\ell}:=\mathbb{P}\circ(\mu^{\ell},\mu^{\ell},\delta_{\overline{\mu}^{\ell}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{\ell}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,B)^{-1}.$

It is enough to apply [24, Theorem 2.12] to conclude that the sequence
$(\mathrm{P}^{\ell})_{\ell\in\mathbb{N}^{*}}$ of approximate $open$–$loop$ MFG
equilibria is relatively compact in ${\cal W}_{p}$ and that, when
$\lim_{\ell\to\infty}\varepsilon_{\ell}=0,$ each limit point $\mathrm{P}$ is
such that $\mathrm{P}\circ(\mu,\Lambda)^{-1}\in{\cal S}^{\star}.$

####  3.3.3 Approximation of approximate $open$–$loop$ MFG equilibrium by
approximate strong Markovian MFG equilibrium

In this part, we show that any approximate $open$–$loop$ MFG equilibrium is
the limit of a sequence of approximate strong Markovian MFG equilibria.

Let $X$ be an $\varepsilon$–$open$–$loop$ MFG equilibrium associated to the
control $\alpha.$ Recall that $\mu$ and $\overline{\mu}$ are defined in
Equation 3.15.

######  Proposition 3.11.

There exists a sequence of Borel functions $(\beta^{k})_{k\in\mathbb{N}^{*}}$
satisfying: for each $k\in\mathbb{N}^{*},$

$\displaystyle\beta^{k}:[0,T]\times\mathbb{R}^{n}\times C([0,T];{\cal
P}_{p}(\mathbb{R}^{n}))\to
U\;\mbox{with}\;\beta^{k}(t,x,\pi)=\beta^{k}(t,x,\pi_{t\wedge\cdot}).$

such that $\beta^{k}$ is lipschitz in $(x,\pi)$ uniformly in $t,$ and if we
let $X^{k}$ be the unique strong solution of:

$\displaystyle\mathrm{d}X^{k}_{t}=b(t,X^{k}_{t},\overline{\mu}^{k}_{t},\beta^{k}(t,X^{k}_{t},\mu^{k}_{t}))\mathrm{d}t+\sigma(t,X^{k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}$

with $\mu^{k}_{t}:={\cal L}(X^{k}_{t}|{\cal F}^{B}_{t})$ and
$\overline{\mu}^{k}_{t}:={\cal
L}\big{(}X^{k}_{t},\beta^{k}(t,X^{k}_{t},\mu^{k})|{\cal F}^{B}_{t}\big{)}$
then

$\displaystyle\displaystyle\lim_{k\to\infty}\big{(}\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}=\big{(}\mu,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)},\;\mathbb{P}\mbox{--a.e.},\;\mbox{for
the Wasserstein metric}\;{\cal W}_{p}.$

###### Proof.

This is essentially an application of Proposition A.6 and Proposition A.8.
Indeed, by Proposition A.6, there exists a sequence of Lipschitz functions
$(\gamma^{k})_{k\in\mathbb{N}^{*}}$ satisfying: for each $k\in\mathbb{N}^{*},$

$\displaystyle\gamma^{k}:[0,T]\times\mathbb{R}^{n}\times{\cal
C}^{n}\times{\cal C}^{n,p}_{{\cal W}}\times\mathbb{M}\to
U\;\mbox{with}\;\gamma^{k}(t,x,b,\pi,v)=\gamma^{k}(t,x,b_{t\wedge\cdot},\pi_{t\wedge\cdot},v_{t\wedge\cdot}).$

such that if we let $X^{k}$ be the unique strong solution of:

$\displaystyle\mathrm{d}X^{k}_{t}=b(t,X^{k}_{t},\overline{\mu}^{k}_{t},\gamma^{k}(t,X^{k}_{t},B,\mu,\Lambda))\mathrm{d}t+\sigma(t,X^{k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{where}\;\Lambda:=\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,$

with $\mu^{k}_{t}:={\cal
L}(X^{k}_{t}|B_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot})$ and
$\overline{\mu}^{k}_{t}:={\cal
L}\big{(}X^{k}_{t},\gamma^{k}(t,X^{k}_{t},B,\mu,\Lambda)|B_{t\wedge\cdot},\mu_{t\wedge\cdot},\Lambda_{t\wedge\cdot}\big{)}$
then

$\displaystyle\displaystyle\lim_{k\to\infty}\big{(}\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}=\big{(}\mu,\Lambda_{t}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)},\;\mathbb{P}\mbox{--a.e.},\;\mbox{for
the Wasserstein metric}\;{\cal W}_{p}.$

Notice that as $(\mu,\Lambda)$ is $\mathbb{F}^{B}$–predictable, we can write
$\gamma^{k}(t,X^{k}_{t},B,\mu,\Lambda):=\widetilde{\gamma}^{k}(t,X^{k}_{t},B)$
where $\widetilde{\gamma}^{k}:[0,T]\times\mathbb{R}^{n}\times{\cal C}^{n}\to
U$ is a progressively Borel measurable function. By using another
approximation thanks to [18, Theorem 6.4.1], we can take
$\widetilde{\gamma}^{k}$ as a Lipschitz function. We can rewrite $X^{k}$

$\displaystyle\mathrm{d}X^{k}_{t}=b\big{(}t,X^{k}_{t},\delta_{\widetilde{\gamma}^{k}(t,x,B)}(\mathrm{d}u)\mu^{k}_{t}(\mathrm{d}x),\widetilde{\gamma}^{k}(t,X^{k}_{t},B)\big{)}\mathrm{d}t+\sigma(t,X^{k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}.$

Now, we take $k$ as fixed. By Proposition A.6, there exists a sequence of
functions $(\phi^{j})_{j\in\mathbb{N}^{*}}$ s.t. for each
$j\in\mathbb{N}^{*},$ $\phi^{j}:[0,T]\times{\cal C}^{n,p}_{{\cal
W}}\ni(t,\pi)\to\phi^{j}(t,\pi_{t\wedge\cdot})\in{\cal C}^{n}$ with $\phi^{j}$
Lipschitz in $\pi$ uniformly in $t$ satisfying for each $j\in\mathbb{N}^{*}$

$\displaystyle B_{t}=\phi^{j}(t,\zeta^{j})$

where $\zeta^{j}_{t}={\cal L}(X^{j}_{t}|{\cal F}^{B}_{t})={\cal
L}(X^{j}_{t}|{\cal F}^{B}_{T})$ with $X^{j}$ satisfying

$\displaystyle\mathrm{d}X^{j}_{t}=b\big{(}t,X^{j}_{t},\delta_{\widetilde{\gamma}^{k}(t,x,B)}(\mathrm{d}u)\zeta^{j}_{t}(\mathrm{d}x),\widetilde{\gamma}^{k}(t,X^{j}_{t},B)\big{)}\mathrm{d}t+\sigma(t,X^{j}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t},$

and
$\displaystyle\lim_{j\to\infty}\mathbb{E}\bigg{[}\int_{0}^{T}\|\zeta^{j}_{t}-\mu^{k}_{t}\|_{\rm
TV}\;\mathrm{d}t\bigg{]}=0.$ If we define
$\beta^{k}(t,x,\pi):=\widetilde{\gamma}^{k}\big{(}t,x,(\phi^{j}(s,\pi))_{s\in[0,t]}\big{)}.$
This is enough to conclude.

∎

######  Proposition 3.12.

Let us stay in the context in Proposition 3.11. There exists a sequence
$(\varepsilon_{k})_{k\in\mathbb{N}^{*}}\subset(0,\infty)$ s.t.
$\displaystyle\limsup_{k\to\infty}\varepsilon_{k}\in[0,\varepsilon]$ and
$\mu^{k}$ is an $\varepsilon_{k}$–strong Markovian MFG equilibrium.

###### Proof.

Let $X^{\prime k}$ be the solution of

$\displaystyle\mathrm{d}X^{\prime k}_{t}=b(t,X^{\prime
k}_{t},\overline{\mu}^{k}_{t},\beta^{\prime k}(t,X^{\prime
k}_{t},\mu^{k}_{t}))\mathrm{d}t+\sigma(t,X^{\prime
k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;\mu^{\prime
k}_{t}:={\cal L}(X^{\prime k}_{t}|{\cal F}^{B}_{t}),\;\overline{\mu}^{\prime
k}_{t}:={\cal L}(X^{\prime k}_{t},\beta^{\prime k}(t,X^{\prime
k}_{t},\mu^{k}_{t})|{\cal F}^{B}_{t})$

where $(\beta^{k})_{k\in\mathbb{N}^{*}}$ is a sequence s.t.

$\displaystyle\mathbb{E}\big{[}J\big{(}\mu^{\prime
k},\mu^{k},\delta_{\overline{\mu}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\mathbb{E}\big{[}J\big{(}\mu^{k},\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}$
$\displaystyle\geq\sup_{\alpha\in{\cal
A}^{c}}\mathbb{E}\big{[}J\big{(}\mu^{\alpha},\mu^{k},\delta_{\overline{\mu}^{\alpha}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\mathbb{E}\big{[}J\big{(}\mu^{k},\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-1/2^{k}.$

Given $\overline{\mu},$ $\mu^{k}$ and $X^{\prime k},$ let
$\widetilde{X}^{\prime k}$ be the solution of

$\displaystyle\mathrm{d}\widetilde{X}^{\prime k}_{t}=b(t,\widetilde{X}^{\prime
k}_{t},\overline{\mu}_{t},\beta^{\prime k}(t,X^{\prime
k}_{t},\mu^{k}_{t}))\mathrm{d}t+\sigma(t,\widetilde{X}^{\prime
k}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\mbox{with}\;\widetilde{\mu}^{\prime
k}_{t}:={\cal L}(\widetilde{X}^{\prime k}_{t}|{\cal
F}^{B}_{t}),\;\widetilde{\overline{\mu}}^{\prime k}_{t}:={\cal
L}(\widetilde{X}^{\prime k}_{t},\beta^{\prime k}(t,X^{\prime
k}_{t},\mu^{k}_{t})|{\cal F}^{B}_{t}).$

There is a constant $C$ independent of $k$ s.t. for each $t\in[0,T],$

$\displaystyle\mathbb{E}\Big{[}\sup_{s\in[0,t]}|\widetilde{X}^{\prime
k}_{s}-X^{\prime k}_{s}|^{2}\Big{]}\leq
C\;\;\mathbb{E}\bigg{[}\bigg{|}\int_{0}^{t}b^{\star}(s,\overline{\mu}^{k}_{s})\mathrm{d}s-\int_{0}^{t}b^{\star}(s,\overline{\mu}_{s})\mathrm{d}s\bigg{|}^{2}+\int_{0}^{t}\sup_{r\in[0,s]}|\widetilde{X}^{\prime
k}_{r}-X^{\prime k}_{r}|^{2}\mathrm{d}r\bigg{]},$

by applying Grönwall’s lemma and using Proposition 3.11, it is easy to check
that
$\displaystyle\lim_{k\to\infty}\mathbb{E}\Big{[}\sup_{t\in[0,T]}|\widetilde{X}^{\prime
k}_{t}-X^{\prime k}_{t}|\Big{]}=0.$ This leads to

$\displaystyle\lim_{k\to\infty}{\cal W}_{p}\big{(}(\mu^{\prime
k},\delta_{\overline{\mu}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t),(\widetilde{\mu}^{\prime
k},\delta_{\widetilde{\overline{\mu}}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t)\big{)}=0,\;\mathbb{P}\mbox{--a.e.}.$

Then, as $X$ is an $\varepsilon$–$open$–$loop$ MFG equilibrium,

$\displaystyle\displaystyle\lim_{k\to\infty}\mathbb{E}\big{[}J\big{(}\mu^{\prime
k},\mu^{k},\delta_{\overline{\mu}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\mathbb{E}\big{[}J\big{(}\mu^{k},\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}$
$\displaystyle\displaystyle\lim_{k\to\infty}\mathbb{E}\big{[}J\big{(}\widetilde{\mu}^{\prime
k},\mu,\delta_{\widetilde{\overline{\mu}}^{\prime
k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\mathbb{E}\big{[}J\big{(}\mu,\mu,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}\leq\varepsilon.$

To conclude, it is enough to define $(\varepsilon_{k})_{k\in\mathbb{N}^{*}}$
by

$\displaystyle\varepsilon_{k}:=\sup_{\alpha\in{\cal
A}^{c}}\mathbb{E}\big{[}J\big{(}\mu^{\alpha},\mu^{k},\delta_{\overline{\mu}^{\alpha}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}-\mathbb{E}\big{[}J\big{(}\mu^{k},\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}.$

∎

##### Proof of Theorem 2.12 (measure–valued MFG via approximate strong
markovian MFG)

Let $\mathrm{P}\in\overline{\cal P}_{V}$ such that
$\mathrm{P}\circ(\mu,\Lambda)^{-1}\in{\cal S}^{\star}.$ By [24, Theorem 2.13],
there exists a sequence of
$(\sigma\\{X_{0},W_{t\wedge\cdot},B_{t\wedge\cdot}\\})_{t\in[0,T]}$–predictable
processes $(\alpha^{\ell})_{\ell\in\mathbb{N}^{*}}$ such that
$X^{\alpha^{\ell}}$ the McKean–Vlasov process solution of Equation 3.15
associated to $\alpha^{\ell}$ is an $\varepsilon^{\ell}$–$open$–$loop$ MFG
equilibrium with $\varepsilon_{\ell}>0,$
$\displaystyle\lim_{\ell\to\infty}\varepsilon_{\ell}=0,$ and

$\displaystyle\displaystyle\lim_{\ell\to\infty}\mathbb{P}\circ\big{(}\mu^{\ell},\mu^{\ell},\delta_{\overline{\mu}^{\ell}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{\ell}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}^{-1}=\mathrm{P}\;\mbox{in}\;{\cal
W}_{p},\;\mbox{where}\;\mu^{\ell}_{t}:={\cal L}(X^{\alpha^{\ell}}_{t}|{\cal
F}^{B}_{t})\;\mbox{and}\;\overline{\mu}^{\ell}_{t}={\cal
L}(X^{\alpha^{\ell}}_{t},\alpha^{\ell}_{t}|{\cal F}^{B}_{t}).$

To conclude, it is enough to approximate $open$–$loop$ MFG equilibrium by
sequence of approximate strong markovian MFG equilibrium. This is done by
Proposition 3.12.

####  3.3.4 Approximation of approximate strong Markovian MFG equilibrium by
approximate $closed$–$loop$ Nash equilibria

Now, in this part, we show that any approximate strong Markovian MFG
equilibrium is the limit of sequence of approximate $closed$–$loop$ Nash
equilibria.

Let $\overline{\mu}$ be an $\varepsilon$–strong markovian MFG equilibrium
associated to control $\beta$ where
$\beta:[0,T]\times\mathbb{R}^{n}\times{\cal C}^{n,p}_{{\cal
W}}\ni(t,x,\pi)\to\beta(t,x,\pi_{t\wedge\cdot})\in U$ is Lipschitz in
$(x,\pi)$ uniformly in $t.$ We denote by $X$ the McKean–Vlasov process
associated to $\overline{\mu},$

$\displaystyle\mathrm{d}X_{t}=b(t,X_{t},\overline{\mu}_{t},\beta(t,X_{t},\mu))\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}\;\;\mbox{with}\;\;\mu_{t}:={\cal
L}(X_{t}|{\cal F}^{B}_{t}),\;\overline{\mu}_{t}:={\cal
L}\big{(}X_{t},\beta(t,X_{t},\mu)|{\cal F}^{B}_{t}\big{)}.$

For each $N\in\mathbb{N}^{*},$ let us define
$\boldsymbol{\alpha}^{i,N}(t,\mathbf{x}):=\beta(t,\mathsf{x}^{i}(t),\pi[\mathbf{x}])$
for $\mathbf{x}:=(\mathsf{x}^{1},\cdots,\mathsf{x}^{N})\in({\cal C}^{n})^{N}$
with
$\pi_{t}[\mathbf{x}]:=\frac{1}{N}\sum_{i=1}^{N}\delta_{\mathsf{x}^{i}(t)}.$ We
denote $\boldsymbol{\alpha}^{N}:=(\beta^{1,N},\cdots,\beta^{N,N}).$

######  Proposition 3.13.

Under 2.1, one has

$\displaystyle\displaystyle\lim_{N\to\infty}\varphi^{N}[\boldsymbol{\alpha}^{N}]=\mu\;\mbox{in}\;{\cal
W}_{p},\;\mathbb{P}\mbox{--a.e.}.$

###### Proof.

We define

$\displaystyle
b(t,x,\pi):=b(t,x,\delta_{\beta(t,y,\pi)}(\mathrm{d}u)\pi_{t}(\mathrm{d}y),\beta(t,x,\pi))\;\mbox{for
each}\;(t,x,\pi)\in[0,T]\times\mathbb{R}^{n}\times{\cal C}^{n}_{{\cal W}}.$

As $\beta$ is Lipschitz in $(x,\pi)$ uniformly in $t,$ under 2.1, we can apply
[26, Proposition 4.15] (a classical propagation of chaos with common noise,
see also [37]) and find the desired result. ∎

######  Proposition 3.14.

There exists a sequence $(\varepsilon_{N})_{N^{*}}\subset(0,\infty)$ s.t.
$\displaystyle\limsup_{N\to\infty}\varepsilon_{N}\in[0,\varepsilon]$ and
$\boldsymbol{\alpha}^{N}:=(\beta^{1,N},\cdots,\beta^{N,N})$ is an
$\varepsilon_{N}$–$closed$–$loop$ Nash equilibria for each
$N\in\mathbb{N}^{*}$.

###### Proof.

Let us define

$\displaystyle c^{i,N}:=\sup_{\alpha^{\prime}\in{\cal
A}_{N}^{c}}J_{i}\big{(}((\boldsymbol{\alpha}^{N})^{-i},\alpha^{\prime})\big{)}-J_{i}(\boldsymbol{\alpha}^{N}).$

There exists a sequence of controls
$(\kappa^{i,N})_{(i,N)\in\\{1,\dots,N\\}\times\mathbb{N}^{*}}$ satisfying
$J_{i}\big{(}((\boldsymbol{\alpha}^{N})^{-i},\kappa^{i,N})\big{)}-J_{i}(\boldsymbol{\alpha}^{N})\geq
c^{i,N}-2^{-N},$ for each $i\in\\{1,...,N\\}.$ We define

$\boldsymbol{\alpha}^{i,N}:=(\beta^{1,N},\dots,\beta^{,i-1,N},\kappa^{i,N},\beta^{i+1,N},\dots,\beta^{N,N}).$

By using the same technique as in the proof of Lemma 3.8, one has for each
$i\in\\{1,\dots,N\\},$

$\displaystyle\displaystyle\lim_{N\to\infty}\mathbb{E}\big{[}{\cal
W}_{p}(\varphi^{N}[\boldsymbol{\alpha}^{i,N}],\mu)\big{]}.$ (3.17)

Indeed, let
$\mathbf{Y}^{i}:=(Y^{i,1},\cdots,Y^{i,N}):=(X^{\boldsymbol{\alpha}^{i,N},1},\dots,X^{\boldsymbol{\alpha}^{i,N},N}),$
and we introduce

$\displaystyle
Z^{i}_{t}:=\exp\bigg{\\{}\int_{0}^{t}\phi^{i}_{r}\mathrm{d}W^{i}_{r}-\frac{1}{2}\int_{0}^{t}|\phi^{i}_{r}|^{2}\mathrm{d}r\bigg{\\}}\;\mbox{for
all}\;t\in[0,T],\;\mbox{and}\;\frac{\mathrm{d}\mathbb{Q}^{i}}{\mathrm{d}\mathbb{P}}=Z^{i}_{T}$

with

$\displaystyle\phi^{i}_{t}:=\sigma(t,X^{\boldsymbol{\alpha}^{N},i}_{t})^{-1}\Big{(}b\big{(}t,X^{\boldsymbol{\alpha}^{N},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{i,N}],\kappa^{i,N}(t,\mathbf{X}^{\alpha})\big{)}-b\big{(}t,X^{\boldsymbol{\alpha}^{N},i}_{t},\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}],\alpha^{\star,i,N}(t,\mathbf{X}^{\boldsymbol{\alpha}^{N}})\big{)}\Big{)}.$

By uniqueness, we can check that ${\cal
L}^{\mathbb{Q}^{i}}(\mathbf{X}^{\boldsymbol{\alpha}^{N}},B)={\cal
L}^{\mathbb{P}}(\mathbf{Y}^{i},B)$ for each $i.$ Then,

$\displaystyle\displaystyle\lim_{N\to\infty}\mathbb{E}^{\mathbb{P}}\big{[}{\cal
W}_{p}(\varphi^{N}[\boldsymbol{\alpha}^{i,N}],\mu)\big{]}=\displaystyle\lim_{N\to\infty}\mathbb{E}^{\mathbb{P}}\big{[}Z^{i}_{T}{\cal
W}_{p}(\varphi^{N}[\boldsymbol{\alpha}^{N}],\mu)\big{]}\leq\displaystyle\lim_{N\to\infty}\mathbb{E}^{\mathbb{P}}\big{[}|Z^{i}_{T}|^{2}\big{]}^{1/2}\mathbb{E}^{\mathbb{P}}\big{[}{\cal
W}_{p}(\varphi^{N}[\boldsymbol{\alpha}^{N}],\mu)^{2}\big{]}^{1/2}=0$

where we used Proposition 3.13. A consequence of (3.17) is
$\displaystyle\lim_{N\to\infty}\mathbb{E}\big{[}{\cal
W}_{p}(\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{i,N}],\overline{\mu}_{t})\big{]}$
$\mathrm{d}\mathbb{P}\otimes\mathrm{d}t$–a.e. Next, let $\widetilde{Y}^{i,N}$
be the solution of

$\displaystyle\mathrm{d}\widetilde{Y}^{i,N}_{t}=b(t,\widetilde{Y}^{i,N}_{t},\overline{\mu}_{t},\kappa^{i,N}(t,\mathbf{Y}^{i}))\mathrm{d}t+\sigma(t,\widetilde{Y}^{i,N}_{t})\mathrm{d}W_{t}+\sigma_{0}\mathrm{d}B_{t}.$

With the previous results, one has
$\displaystyle\lim_{N\to\infty}\mathbb{E}^{\mathbb{P}}\bigg{[}\sup_{t\in[0,T]}|\widetilde{Y}^{i,N}_{t}-Y^{i,N}_{t}|\bigg{]}=0.$

Therefore, using all previous observations and as $\overline{\mu}$ is an
$\varepsilon$–strong markovian MFG equilibrium, so can be seen as an
$\varepsilon$–$open$–$loop$ MFG equilibrium,

$\displaystyle\varepsilon$
$\displaystyle\geq\Big{(}\displaystyle\limsup_{N\to\infty}\Big{(}\frac{1}{N}\sum_{i=1}^{N}\mathbb{E}\bigg{[}\int_{0}^{T}L\big{(}t,\widetilde{Y}^{i,N}_{t},\overline{\mu}_{t},\kappa^{i}(t,\mathbf{Y}^{i})\big{)}\mathrm{d}t+g\big{(}\widetilde{Y}^{i,N}_{T},\mu\big{)}\bigg{]}-\mathbb{E}\big{[}J\big{(}\mu,\mu,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}\big{]}\Big{)}$
$\displaystyle\geq\displaystyle\limsup_{N\to\infty}\Big{(}\frac{1}{N}\sum_{i=1}^{N}J_{i}\big{(}((\boldsymbol{\alpha}^{N})^{-i},\kappa^{i,N})\big{)}-\frac{1}{N}\sum_{i=1}^{N}J_{i}(\boldsymbol{\alpha}^{N})\Big{)}\geq\displaystyle\limsup_{N\to\infty}\frac{1}{N}\sum_{i=1}^{N}c^{i,N}.$

To obtain the result as formulated, observe that by symmetry
$\varepsilon_{N}:=c^{i,N}=c^{1,N}.$ This is enough to conclude.

∎

##### Proof of Theorem 2.12 (measure–valued MFG via approximate Nash
equilibria)

Let $\mathrm{P}\in\overline{\cal P}_{V}$ such that
$\mathrm{P}\circ(\mu,\Lambda)^{-1}\in{\cal S}^{\star}.$ By Theorem 2.12 (first
part) (see also Proposition 3.12), there exist
$(\varepsilon_{k})_{k\in\mathbb{N}^{*}}\subset(0,\infty)$ with
$\displaystyle\lim_{k\to\infty}\varepsilon_{k}=0$ and a sequence
$(\beta^{k})_{k\in\mathbb{N}^{*}}$ s.t. $\overline{\mu}^{k}$ is an
$\varepsilon_{k}$–strong Markovian MFG equilibrium associated to the control
$\beta^{k}$ with $\beta^{k}$ Lipschitz in $(x,\pi)$ uniformly in $t,$ and

$\displaystyle\displaystyle\lim_{k\to\infty}\mathbb{P}\circ\big{(}\mu^{k},\mu^{k},\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t,\delta_{\overline{\mu}^{k}_{t}}(\mathrm{d}\bar{\nu})\mathrm{d}t\big{)}^{-1}=\mathrm{P}\;\mbox{in}\;{\cal
W}_{p}.$

To conclude, it is enough to approximate approximate strong Markovian MFG
equilibrium by approximate Nash equilibria. This is done by Proposition 3.14.

###  3.4 McKean–Vlasov optimal control

####  3.4.1 Proof of Theorem 2.14

Thanks to Proposition 3.12, we can approximate any $open$–$loop$ McKean–Vlasov
process by a sequence of $closed$–$loop$ McKean–Vlasov processes. Also, by
Proposition 3.7, we can easily verify that $V^{c}_{S}\leq V^{o}_{S}.$ We
therefore find our result and $V^{c}_{S}=V^{o}_{S}.$ We deduce that

$\displaystyle V^{c}_{S}=V^{o}_{S}=\sup_{\mathrm{P}\in{\cal
V}}\mathbb{E}^{\mathrm{P}}\big{[}J(\mu^{\prime},\mu,\Lambda,\Lambda^{\prime})\big{]}.$

####  3.4.2 Proof of Theorem 2.15

Let $(\widetilde{\mathrm{P}}^{N})_{N\in\mathbb{N}^{*}}\subset{\cal P}({\cal
C}^{n}_{{\cal W}}\times\mathbb{M}\times{\cal C}^{n})$ be the sequence defined
by

$\displaystyle\widetilde{\mathrm{P}}^{N}:=\mathbb{P}\circ\big{(}\varphi^{N}[\boldsymbol{\alpha}^{N}],\delta_{\overline{\varphi}^{N}_{t}[\boldsymbol{\alpha}^{N}]}(\mathrm{d}\bar{\nu})\mathrm{d}t,B\big{)}^{-1}.$

If we denote by $(\widetilde{\mu},\widetilde{\Lambda},\widetilde{B})$ the
canonical process of ${\cal C}^{n}_{{\cal W}}\times\mathbb{M}\times{\cal
C}^{n}.$ Notice that
$\mathrm{P}^{N}=\widetilde{\mathrm{P}}^{N}\circ(\widetilde{\mu},\Lambda)^{-1},$
where $\mathrm{P}^{N}$ is the sequence given in Theorem 2.15. By [23,
Proposition 3.4], $(\widetilde{\mathrm{P}}^{N})_{N\in\mathbb{N}^{*}}$ is
relatively compact in ${\cal W}_{p}$, and for each limit point