#  Pontryagin’s Minimum Principle and Forward-Backward Sweep Method for the
System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic
Control

Takehiro Tottori1 and Tetsuya J. Kobayashi1,2,3,4 1Department of Mathematical
Informatics, Graduate School of Information Science and Technology, The
University of Tokyo, Tokyo 113-8654, Japan2Department of Electrical
Engineering and Information Systems, Graduate School of Engineering, The
University of Tokyo, Tokyo 113-8654, Japan3Institute of Industrial Science,
The University of Tokyo, Tokyo 153-8505, Japan4Universal Biology Institute,
The University of Tokyo, Tokyo 113-8654, Japan

###### Abstract

Memory-limited partially observable stochastic control (ML-POSC) is the
stochastic optimal control problem under incomplete information and memory
limitation. In order to obtain the optimal control function of ML-POSC, a
system of the forward Fokker-Planck (FP) equation and the backward Hamilton-
Jacobi-Bellman (HJB) equation needs to be solved. In this work, we firstly
show that the system of HJB-FP equations can be interpreted via the
Pontryagin’s minimum principle on the probability density function space.
Based on this interpretation, we then propose the forward-backward sweep
method (FBSM) to ML-POSC, which has been used in the Pontryagin’s minimum
principle. FBSM is an algorithm to compute the forward FP equation and the
backward HJB equation alternately. Although the convergence of FBSM is
generally not guaranteed, it is guaranteed in ML-POSC because the coupling of
HJB-FP equations is limited to the optimal control function in ML-POSC.

##  I INTRODUCTION

In many practical applications of the stochastic optimal control theory,
several constraints need to be considered. Especially in small devices [1, 2]
and in biological systems [3, 4, 5, 6, 7, 8, 9, 10], incomplete information
and memory limitation become predominant because their sensors are extremely
noisy and their memory resources are severely limited. In order to account
these constraints, memory-limited partially observable stochastic control (ML-
POSC) has recently been proposed [11]. Because ML-POSC formulates the noisy
observation and the limited memory explicitly, ML-POSC can directly take
incomplete information and memory limitation into account in the stochastic
optimal control problem.

However, ML-POSC cannot be solved in the similar way as the conventional
stochastic control, which is also called completely observable stochastic
control (COSC). In COSC, the optimal control function depends only on the
Hamilton-Jacobi-Bellman (HJB) equation, which is a time-backward partial
differential equation given the terminal condition (Fig. 1(a)) [12].
Therefore, the optimal control function of COSC can be obtained by solving the
HJB equation backward in time from the terminal condition, which is called the
value iteration method [13, 14, 15]. In contrast, the optimal control function
of ML-POSC depends not only on the HJB equation but also on the Fokker-Planck
(FP) equation, which is a time-forward partial differential equation given the
initial condition (Fig. 1(b)) [11]. Because the HJB equation and the FP
equation interact with each other through the optimal control function in ML-
POSC, the optimal control function of ML-POSC cannot be obtained by the value
iteration method.

In order to propose an algorithm to ML-POSC, we firstly show that the system
of HJB-FP equations can be interpreted via the Pontryagin’s minimum principle
on the probability density function space. The Pontryagin’s minimum principle
is one of the most representative approaches to the deterministic control,
which converts the optimal control problem into the two-point boundary value
problem of the forward state equation and the backward adjoint equation [16,
17, 18]. We show that the system of HJB-FP equations is an extension of the
system of the state and adjoint equations from the deterministic control to
the stochastic control.

The system of HJB-FP equations also appears in the mean-field stochastic
control (MFSC) [19, 20, 21]. Although the relationship between the system of
HJB-FP equations and the Pontryagin’s minimum principle has been mentioned
briefly in MFSC [22, 23, 24], its details have not yet been investigated. We
resolve this problem by deriving the system of HJB-FP equations in the similar
way as the Pontryagin’s minimum principle.

We then propose the forward-backward sweep method (FBSM) to ML-POSC. FBSM is
an algorithm to compute the forward FP equation and the backward HJB equation
alternately, which can be interpreted as an extension of the value iteration
method. FBSM has been proposed in the Pontryagin’s minimum principle of the
deterministic control, which computes the forward state equation and the
backward adjoint equation alternately [25, 26, 27]. Because FBSM is easy to
implement, it has been used in many applications [28, 29]. However, the
convergence of FBSM is not guaranteed in the deterministic control except for
special cases [30, 31] because the coupling of the state and adjoint equations
is not limited to the optimal control function (Fig. 1(c)). In contrast, we
show that the convergence of FBSM is generally guaranteed in ML-POSC because
the coupling of HJB-FP equations is limited only to the optimal control
function (Fig. 1(b)).

FBSM corresponds to the fixed-point iteration method in MFSC [32, 33, 34, 35].
Although the fixed-point iteration method is the most basic algorithm in MFSC,
its convergence is not guaranteed for the same reason as the deterministic
control (Fig. 1(d)). Therefore, ML-POSC is a special class where FBSM or the
fixed-point iteration method is guaranteed to converge.

This paper is organized as follows: In Sec. II, we formulate ML-POSC. In Sec.
III, we derive the system of HJB-FP equations from the viewpoint of the
Pontryagin’s minimum principle. In Sec. IV, we propose FBSM to ML-POSC and
prove the convergence. In Sec. V, we apply FBSM to the linear-quadratic-
Gaussian (LQG) problem. In Sec. VI, we verify the convergence of FBSM by
numerical experiments. In Sec. VII, we conclude this paper. In APPENDIX I, we
briefly review the Pontryagin’s minimum principle of the deterministic
control. In APPENDIX II, we show the Pontryagin’s minimum principle of MFSC.
In APPENDIX III, we show the proof in the main text.

(a) Completely observable stochastic control (COSC)

(b) Memory-limited partially observable stochastic control (ML-POSC)

(c) Deterministic control

(d) Mean-field stochastic control (MFSC)

Figure 1:  Schematic diagram of the relationship between the backward
dynamics, the optimal control function, and the forward dynamics in (a) the
completely observable stochastic control (COSC), (b) the memory-limited
partially observable stochastic control (ML-POSC), (c) the deterministic
control, and (d) the mean-field stochastic control (MFSC). $w^{*}$, $p^{*}$,
$\lambda^{*}$, and $s^{*}$ are the solutions of the Hamilton-Jacobi-Bellman
(HJB) equation, the Fokker-Planck (FP) equation, the adjoint equation, and the
state equation, respectively. $u^{*}$ is the optimal control function. (a) In
COSC, because the optimal control function $u^{*}$ does not depend on the FP
equation $p^{*}$, the optimal control function $u^{*}$ can be obtained by
solving the HJB equation $w^{*}$ backward in time from the terminal condition,
which is called the value iteration method. (b) In ML-POSC, because the
optimal control function $u^{*}$ depends on the FP equation $p^{*}$ as well as
the HJB equation $w^{*}$, the optimal control function $u^{*}$ cannot be
obtained by the value iteration method. In this paper, we propose the forward-
backward sweep method (FBSM) to ML-POSC, which computes the HJB equation
$w^{*}$ and the FP equation $p^{*}$ alternately. Because the coupling of the
HJB equation $w^{*}$ and the FP equation $p^{*}$ is limited only to the
optimal control function $u^{*}$, the convergence of FBSM is guaranteed in ML-
POSC. (c) In the deterministic control, because the coupling of the adjoint
equation $\lambda^{*}$ and the state equation $s^{*}$ is not limited to the
optimal control function $u^{*}$, the convergence of FBSM is not guaranteed.
(d) In MFSC, because the coupling of the HJB equation $w^{*}$ and the FP
equation $p^{*}$ is not limited to the optimal control function $u^{*}$, the
convergence of FBSM is not guaranteed.

##  II MEMORY-LIMITED PARTIALLY OBSERVABLE STOCHASTIC CONTROL

In this section, we briefly review the formulation of ML-POSC [11], which is
the stochastic optimal control problem under incomplete information and memory
limitation.

###  II-A Problem formulation

In this subsection, we formulate ML-POSC [11]. The state of the system
$x_{t}\in\mathbb{R}^{d_{x}}$ at time $t\in[0,T]$ evolves by the following
stochastic differential equation (SDE):

$\displaystyle dx_{t}$
$\displaystyle=b(t,x_{t},u_{t})dt+\sigma(t,x_{t},u_{t})d\omega_{t},$ (1)

where $x_{0}$ obeys $p_{0}(x_{0})$, $u_{t}\in\mathbb{R}^{d_{u}}$ is the
control, and $\omega_{t}\in\mathbb{R}^{d_{\omega}}$ is the standard Wiener
process. In ML-POSC, because the controller cannot completely observe the
state $x_{t}$, the observation $y_{t}\in\mathbb{R}^{d_{y}}$ is obtained
instead of the state $x_{t}$, which evolves by the following SDE:

$\displaystyle dy_{t}$ $\displaystyle=h(t,x_{t})dt+\gamma(t)d\nu_{t},$ (2)

where $y_{0}$ obeys $p_{0}(y_{0})$, and $\nu_{t}\in\mathbb{R}^{d_{\nu}}$ is
the standard Wiener process. Furthermore, because the controller cannot
completely memorize the observation history
$y_{0:t}:=\\{y_{\tau}|\tau\in[0,t]\\}$, the observation history $y_{0:t}$ is
compressed into the finite-dimensional memory $z_{t}\in\mathbb{R}^{d_{z}}$,
which evolves by the following SDE:

$\displaystyle
dz_{t}=c(t,z_{t},v_{t})dt+\kappa(t,z_{t},v_{t})dy_{t}+\eta(t,z_{t},v_{t})d\xi_{t},$
(3)

where $z_{0}$ obeys $p_{0}(z_{0})$, $v_{t}\in\mathbb{R}^{d_{v}}$ is the
control, and $\xi_{t}\in\mathbb{R}^{d_{\xi}}$ is the standard Wiener process.
The controller determines the state control $u_{t}$ and the memory control
$v_{t}$ based on the memory $z_{t}$ as follows:

$\displaystyle u_{t}=u(t,z_{t}),\ v_{t}=v(t,z_{t}).$ (4)

The objective function of ML-POSC is given by the following expected
cumulative cost function:

$\displaystyle
J[u,v]:=\mathbb{E}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[\int_{0}^{T}f(t,x_{t},u_{t},v_{t})dt+g(x_{T})\right],$
(5)

where $f$ is the cost function, $g$ is the terminal cost function,
$p(x_{0:T},y_{0:T},z_{0:T};u,v)$ is the probability of $x_{0:T}$, $y_{0:T}$,
and $z_{0:T}$ given $u$ and $v$ as parameters, and $\mathbb{E}_{p}[\cdot]$ is
the expectation with respect to the probability $p$.

ML-POSC is the problem to find the optimal control functions $u^{*}$ and
$v^{*}$ that minimize the expected cumulative cost function $J[u,v]$ as
follows:

$\displaystyle u^{*},v^{*}=\mathop{\rm argmin}\limits_{u,v}J[u,v].$ (6)

###  II-B Problem reformulation

Although the formulation of ML-POSC in the previous subsection is intuitive,
it is inconvenient for further mathematical investigations. In order to
resolve this problem, we reformulate ML-POSC in this subsection. The
formulation in this subsection is simpler and more general than that in the
previous subsection.

We first define the extended state $s_{t}$ as follows:

$\displaystyle s_{t}:=\left(\begin{array}[]{c}x_{t}\\\ z_{t}\\\
\end{array}\right)\in\mathbb{R}^{d_{s}},$ (9)

where $d_{s}=d_{x}+d_{z}$. The extended state $s_{t}$ evolves by the following
SDE:

$\displaystyle
ds_{t}=\tilde{b}(t,s_{t},\tilde{u}_{t})dt+\tilde{\sigma}(t,s_{t},\tilde{u}_{t})d\tilde{\omega}_{t},$
(10)

where $s_{0}$ obeys $p_{0}(s_{0})$,
$\tilde{u}_{t}\in\mathbb{R}^{d_{\tilde{u}}}$ is the control, and
$\tilde{\omega}_{t}\in\mathbb{R}^{d_{\tilde{\omega}}}$ is the standard Wiener
process. ML-POSC determines the control
$\tilde{u}_{t}\in\mathbb{R}^{d_{\tilde{u}}}$ based on the memory $z_{t}$ as
follows:

$\displaystyle\tilde{u}_{t}=\tilde{u}(t,z_{t}).$ (11)

The extended state SDE (10) includes the previous SDEs (1), (2), (3) as a
special case because they can be represented as follows:

$\displaystyle ds_{t}=\left(\begin{array}[]{c}b(t,x_{t},u_{t})\\\
c(t,z_{t},v_{t})+\kappa(t,z_{t},v_{t})h(t,x_{t})\\\
\end{array}\right)dt+\left(\begin{array}[]{ccc}\sigma(t,x_{t},u_{t})&O&O\\\
O&\kappa(t,z_{t},v_{t})\gamma(t)&\eta(t,z_{t},v_{t})\\\
\end{array}\right)\left(\begin{array}[]{c}d\omega_{t}\\\ d\nu_{t}\\\
d\xi_{t}\\\ \end{array}\right),$ (19)

where $p_{0}(s_{0})=p_{0}(x_{0})p_{0}(z_{0})$.

The objective function of ML-POSC is given by the following expected
cumulative cost function:

$\displaystyle
J[\tilde{u}]:=\mathbb{E}_{p(s_{0:T};\tilde{u})}\left[\int_{0}^{T}\tilde{f}(t,s_{t},\tilde{u}_{t})dt+\tilde{g}(s_{T})\right].$
(20)

where $\tilde{f}$ is the cost function, and $\tilde{g}$ is the terminal cost
function. It is obvious that this objective function (20) is more general than
that in the previous subsection (5).

ML-POSC is the problem to find the optimal control function $\tilde{u}^{*}$
that minimizes the expected cumulative cost function $J[\tilde{u}]$ as
follows:

$\displaystyle\tilde{u}^{*}=\mathop{\rm
argmin}\limits_{\tilde{u}}J[\tilde{u}].$ (21)

In the following sections, we mainly consider the formulation of this
subsection because it is simpler and more general than that in the previous
subsection. Moreover, we omit $\tilde{\cdot}$ for the notational simplicity.

##  III PONTRYAGIN’S MINIMUM PRINCIPLE

If the control $u_{t}$ is determined based on the extended state $s_{t}$,
i.e., $u_{t}=u(t,s_{t})$, ML-POSC is the same problem with the COSC of the
extended state, and its optimality conditions can be obtained in the
conventional way [12]. In reality, however, because ML-POSC determines the
control $u_{t}$ based only on the memory $z_{t}$, i.e., $u_{t}=u(t,z_{t})$,
its optimality conditions cannot be obtained in the similar way as COSC. In
the previous work [11], the optimality conditions of ML-POSC were obtained by
employing a mathematical technique of MFSC [23, 24].

In this section, we obtain the optimality conditions of ML-POSC by employing
the Pontryagin’s minimum principle [12, 16, 17, 18] on the probability density
function space (Fig. 2). The conventional approach in ML-POSC [11] and MFSC
[23, 24] can be interpreted as the conversion from the Bellman’s dynamic
programming principle to the Pontryagin’s minimum principle on the probability
density function space.

In APPENDIX I, we briefly review the Pontryagin’s minimum principle of the
deterministic control. In this section, we obtain the optimality conditions of
ML-POSC in the similar way as APPENDIX I. Furthermore, in APPENDIX II, we
obtain the optimality conditions of MFSC in the similar way as APPENDIX I.
MFSC is more general than ML-POSC except for the partial observability.
Especially, the expected Hamiltonian is non-linear with respect to the
probability density function in MFSC while it is linear in ML-POSC.

Figure 2:  The relationship between the Bellman’s dynamic programming
principle (top) and the Pontryagin’s minimum principle (bottom) on the state
space (left) and on the probability density function space (right). The left-
hand side corresponds to the deterministic control, which is briefly reviewed
in APPENDIX I. The right-hand side corresponds to the memory-limited partially
observable stochastic control (ML-POSC) and the mean-field stochastic control
(MFSC), which are shown in Sec. III and APPENDIX II, respectively. The
conventional approach in ML-POSC [11] and MFSC [23, 24] can be interpreted as
the conversion from the Bellman’s dynamic programming principle into the
Pontryagin’s minimum principle on the probability density function space.

###  III-A Hamiltonian

Before we show the optimality conditions, we define the Hamiltonian as
follows:

$\displaystyle\mathcal{H}\left(t,s,u,w\right):=f(t,s,u)+\mathcal{L}_{u}w(t,s),$
(22)

where ${}^{\forall}w:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}$, and
$\mathcal{L}_{u}$ is the backward diffusion operator, which is defined as
follows:

$\displaystyle\mathcal{L}_{u}w(t,s)$
$\displaystyle:=\sum_{i=1}^{d_{s}}b_{i}(t,s,u)\frac{\partial w(t,s)}{\partial
s_{i}}+\frac{1}{2}\sum_{i,j=1}^{d_{s}}D_{ij}(t,s,u)\frac{\partial^{2}w(t,s)}{\partial
s_{i}\partial s_{j}},$ (23)

where $D(t,s,u):=\sigma(t,s,u)\sigma^{\top}(t,s,u)$. The extended state SDE
(10) can be converted into the following Fokker-Planck (FP) equation:

$\displaystyle\frac{\partial p(t,s)}{\partial
t}=\mathcal{L}_{u}^{\dagger}p(t,s),$ (24)

where $p(0,s)=p_{0}(s)$, and $\mathcal{L}_{u}^{{\dagger}}$ is the forward
diffusion operator, which is defined as follows:

$\displaystyle\mathcal{L}_{u}^{{\dagger}}p(t,s)$
$\displaystyle:=-\sum_{i=1}^{d_{s}}\frac{\partial(b_{i}(t,s,u)p(t,s))}{\partial
s_{i}}+\frac{1}{2}\sum_{i,j=1}^{d_{s}}\frac{\partial^{2}(D_{ij}(t,s,u)p(t,s))}{\partial
s_{i}\partial s_{j}}.$ (25)

We note that $\mathcal{L}_{u}^{{\dagger}}$ is the conjugate of
$\mathcal{L}_{u}$ as follows:

$\displaystyle\int w(t,s)\mathcal{L}_{u}^{{\dagger}}p(t,s)ds=\int
p(t,s)\mathcal{L}_{u}w(t,s)ds,$ (26)

The following lemma shows the relationship between the objective function
$J[u]$ and the Hamiltonian $\mathcal{H}(t,s,u,w)$, which is significant for
the optimality conditions.

###### Lemma 1

Let ${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{u}}$ and
${}^{\forall}u^{\prime}:[0,T]\times\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{u}}$ be
the arbitrary control functions, and let $p$ and $p^{\prime}$ be the
probability density functions of the extended state driven by $u$ and
$u^{\prime}$, respectively. Then $J[u]-J[u^{\prime}]$ satisfies the following
equation:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime})\right]-\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{\prime},w^{\prime})\right]\right)dt,$
(27)

where $w^{\prime}$ is the solution of the following Hamilton-Jacobi-Bellman
(HJB) equation:

$\displaystyle-\frac{\partial w^{\prime}(t,s)}{\partial
t}=\mathcal{H}\left(t,s,u^{\prime},w^{\prime}\right),$ (28)

where $w^{\prime}(T,s)=g(s)$.

###### Proof:

The proof is shown in APPENDIX III-A. ∎

###  III-B Necessary condition

We show the necessary condition of the optimal control function of ML-POSC
that corresponds to the Pontryagin’s minimum principle on the probability
density function space.

###### Theorem 1

In ML-POSC, the optimal control function $u^{*}$ satisfies the following
equation:

$\displaystyle u^{*}(t,z)=\mathop{\rm
argmin}\limits_{u}\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}\left(t,s,u,w^{*}\right)\right],\
a.s.\ ^{\forall}t\in[0,T],\ ^{\forall}z\in\mathbb{R}^{d_{z}}$ (29)

where $p_{t}^{*}(x|z):=p^{*}(t,s)/\int p^{*}(t,s)dx$ is the conditional
probability density function of the state $x$ given the memory $z$, and
$p^{*}(t,s)$ is the solution of the FP equation:

$\displaystyle\frac{\partial p^{*}(t,s)}{\partial
t}=\mathcal{L}_{u^{*}}^{\dagger}p^{*}(t,s),$ (30)

where $p^{*}(0,s)=p_{0}(s)$. $w^{*}(t,s)$ is the solution of the following HJB
equation:

$\displaystyle-\frac{\partial w^{*}(t,s)}{\partial
t}=\mathcal{H}\left(t,s,u^{*},w^{*}\right),$ (31)

where $w^{*}(T,s)=g(s)$.

###### Proof:

The proof is shown in APPENDIX III-B. ∎

In the Pontryagin’s minimum principle of the deterministic control, the state
equation (87) and the adjoint equation (88) can be expressed by the
derivatives of the Hamiltonian (APPENDIX I). Similarly, the system of HJB-FP
equations (30) and (31) can be expressed by the variations of the expected
Hamiltonian

$\displaystyle\bar{\mathcal{H}}(t,p,u,w):=\mathbb{E}_{p(s)}\left[\mathcal{H}\left(t,s,u,w\right)\right],$
(32)

as follows:

$\displaystyle\frac{\partial p^{*}(t,s)}{\partial t}$
$\displaystyle=\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
w}(s),$ (33) $\displaystyle-\frac{\partial w^{*}(t,s)}{\partial t}$
$\displaystyle=\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
p}(s),$ (34)

where $p^{*}(0,s)=p_{0}(s)$ and $w^{*}(T,s)=g(s)$. Therefore, the system of
HJB-FP equations can be interpreted via the Pontryagin’s minimum principle on
the probability density function space.

###  III-C Sufficient condition

The Pontryagin’s minimum principle becomes a necessary and sufficient
condition if the expected Hamiltonian is convex as follows:

###### Proposition 1

Assume that the expected Hamiltonian $\bar{\mathcal{H}}(t,p,u,w)$ is convex
with respect to $p$ and $u$. If the control function $u^{*}$ satisfies (29),
it is the optimal control function of ML-POSC.

###### Proof:

The proof is shown in APPENDIX III-C. ∎

###  III-D Relationship with the Bellman’s dynamic programming principle

From the Bellman’s dynamic programming principle on the probability density
function space [11], the optimal control function of ML-POSC is given by the
following equation:

$\displaystyle u^{*}(t,z,p)=\mathop{\rm
argmin}\limits_{u}\mathbb{E}_{p(x|z)}\left[\mathcal{H}\left(t,s,u,\frac{\delta
V^{*}(t,p)}{\delta p}(s)\right)\right].$ (35)

More specifically, the optimal control function of ML-POSC is given by
$u^{*}(t,z)=u^{*}(t,z,p^{*})$, where $p^{*}$ is the solution of the FP
equation (30). $V^{*}(t,p)$ is the value function on the probability density
function space, which is the solution of the following Bellman equation:

$\displaystyle-\frac{\partial V^{*}(t,p)}{\partial
t}=\mathbb{E}_{p(s)}\left[\mathcal{H}\left(t,s,u^{*},\frac{\delta
V^{*}(t,p)}{\delta p}(s)\right)\right],$ (36)

where $V^{*}(T,p)=\mathbb{E}_{p(s)}\left[g(s)\right]$. Because the Bellman
equation (36) is a functional differential equation, it cannot be solved even
numerically. In order to resolve this problem, the previous work [11]
converted the Bellman equation (36) into the system of HJB-FP equations (30)
and (31) as follows:

######  Proposition 2 ([11])

We define $w^{*}(t,s)$ from $V^{*}(t,p)$ as follows:

$\displaystyle w^{*}(t,s):=\frac{\delta V^{*}(t,p^{*})}{\delta p}(s),$ (37)

where $p^{*}$ is the solution of FP equation (30). $w^{*}$ satisfies HJB
equation (31) from the Bellman equation (36).

###### Proof:

The proof is shown in [11]. ∎

This approach [11] can be interpreted as the conversion from the Bellman’s
dynamic programming principle to the Pontryagin’s minimum principle.

###  III-E Relationship with the completely observable stochastic control

In the COSC of the extended state, the control $u_{t}$ is determined based on
the extended state $s_{t}$, i.e., $u_{t}=u(t,s_{t})$. In the COSC of the
extended state, the Pontryagin’s minimum principle on the probability density
function space is given by the following proposition, which is a necessary
condition of the optimal control function:

###### Proposition 3

In COSC, the optimal control function $u^{*}$ satisfies the following
equation:

$\displaystyle u^{*}(t,s)=\mathop{\rm
argmin}\limits_{u}\mathcal{H}\left(t,s,u,w^{*}\right),\ a.s.\
^{\forall}t\in[0,T],\ ^{\forall}s\in\mathbb{R}^{d_{s}}$ (38)

where $w^{*}(t,s)$ is the solution of HJB equation (31).

###### Proof:

The proof is almost the same with Theorem 1. ∎

While the optimal control function of ML-POSC depends on the FP equation and
the HJB equation, the optimal control function of COSC depends only on the HJB
equation. From this nice property of COSC, Proposition 3 is a necessary and
sufficient condition without the convexity of the expected Hamiltonian as
follows:

###### Proposition 4

If the control function $u^{*}$ satisfies (38), it is the optimal control
function of COSC.

###### Proof:

The proof is shown in APPENDIX III-D. ∎

Proposition 4 is consistent with the conventional result of COSC [12]. Unlike
ML-POSC and MFSC, COSC can be solved by the Bellman’s dynamic programming
principle on the state space. In COSC, the Pontryagin’s minimum principle on
the probability density function space is equivalent with the Bellman’s
dynamic programming principle on the state space. Because the Bellman’s
dynamic programming principle on the state space is a necessary and sufficient
condition, the Pontryagin’s minimum principle on the probability density
function space also becomes a necessary and sufficient condition.

##  IV FORWARD-BACKWARD SWEEP METHOD

In this section, we propose FBSM to ML-POSC, and then prove the convergence.
While the convergence of FBSM is not guaranteed in the deterministic control
[25, 27, 28, 31] and MFSC [32, 33, 34, 35], it is guaranteed in ML-POSC
because the coupling of HJB-FP equations is limited only to the optimal
control function in ML-POSC.

###  IV-A Forward-backward sweep method

Algorithm 1 Forward-Backward Sweep Method (FBSM)

//— Initial step —//

$k\leftarrow 0$

$p_{0}^{k}(s)\leftarrow p_{0}(s)$

for $t=0$ to $T-dt$ do

Initialize $u_{t}^{k}(z)$

$p_{t+dt}^{k}(s)\leftarrow
p_{t}^{k}(s)+\mathcal{L}_{u_{t}^{k}}^{{\dagger}}p_{t}^{k}(s)dt$

end for

while $J[u_{0:T-dt}^{k}]$ do not converge do

if $k$ is even then

//— Backward step —//

$w_{T}^{k+1}(s)\leftarrow g(s)$

for $t=T-dt$ to $0$ do

$u_{t}^{k+1}(z)\leftarrow\mathop{\rm
argmin}\limits_{u}\mathbb{E}_{p_{t}^{k}(x|z)}\left[\mathcal{H}(t,s,u,w_{t+dt}^{k+1})\right]$

$w_{t}^{k+1}(s)\leftarrow
w_{t+dt}^{k+1}(s)+\mathcal{H}(t,s,u_{t}^{k+1},w_{t+dt}^{k+1})dt$

end for

else

//— Forward step —//

$p_{0}^{k+1}(s)\leftarrow p_{0}(s)$

for $t=0$ to $T-dt$ do

$u_{t}^{k+1}(z)\leftarrow\mathop{\rm
argmin}\limits_{u}\mathbb{E}_{p_{t}^{k+1}(x|z)}\left[\mathcal{H}(t,s,u,w_{t+dt}^{k})\right]$

$p_{t+dt}^{k+1}(s)\leftarrow
p_{t}^{k+1}(s)+\mathcal{L}_{u_{t}^{k+1}}^{{\dagger}}p_{t}^{k+1}(s)dt$

end for

end if

$k\leftarrow k+1$

end while

return $u_{0:T-dt}^{k}$

In this subsection, we propose the forward-backward sweep method (FBSM) to ML-
POSC, which is summarized in Algorithm 1. FBSM is an algorithm to compute the
forward FP equation and the backward HJB equation alternately. More
specifically, in the initial step of FBSM, we initialize the control function
$u_{0:T-dt}^{0}$ and compute the FP equation $p_{0:T}^{0}$ forward in time
from the initial condition. In the backward step, we compute the HJB equation
$w_{0:T}^{1}$ backward in time from the terminal condition and simultaneously
update the control function from $u_{0:T-dt}^{0}$ to $u_{0:T-dt}^{1}$ by
minimizing the conditional expected Hamiltonian. In the forward step, we
compute the FP equation $p_{0:T}^{2}$ forward in time from the initial
condition and simultaneously update the control function from $u_{0:T-dt}^{1}$
to $u_{0:T-dt}^{2}$ in the similar way as the backward step. By iterating the
backward step and the forward step, the objective function $J[u_{0:T-dt}^{k}]$
monotonically decreases and finally converges to the local minimum at which
the control function $u_{0:T-dt}^{k}$ satisfies the Pontryagin’s minimum
principle (Theorem 1).

The Pontryagin’s minimum principle is a necessary condition of the optimal
control function, not a sufficient condition. Therefore, the control function
obtained by FBSM is not necessarily the global optimum except in the case
where the expected Hamiltonian is convex (Proposition 1). However, the control
function obtained by FBSM is expected to be superior to most of control
functions because it is locally optimal.

###  IV-B Convergence

(a) Memory-limited partially observable stochastic control (ML-POSC)

(b) Deterministic control

(c) Mean-field stochastic control (MFSC)

Figure 3:  Schematic diagram of the effect of updating the control function on
the forward and backward dynamics in (a) the memory-limited partially
observable stochastic control (ML-POSC), (b) the deterministic control, and
(c) the mean-field stochastic control (MFSC). $w_{0:T}$, $p_{0:T}$,
$\lambda_{0:T}$, and $s_{0:T}$ are the solutions of the Hamilton-Jacobi-
Bellman (HJB) equation, the Fokker-Planck (FP) equation, the adjoint equation,
and the state equation, respectively. $u_{0:T-dt}$ is the given control
function. (a) In ML-POSC, the update from $u_{t}$ to $u_{t}^{\prime}$ does not
change $p_{0:t}$ and $w_{t+dt:T}$. From this property, the convergence of FBSM
is guaranteed. (b) In the deterministic control, the update from $u_{t}$ to
$u_{t}^{\prime}$ changes $\lambda_{t+dt:T}$ to $\lambda_{t+dt:T}^{\prime}$
because the adjoint equation depends on the state equation. Because FBSM does
not take the change of $\lambda_{t+dt:T}$ into account, the convergence of
FBSM is not guaranteed. (c) In MFSC, the update from $u_{t}$ to
$u_{t}^{\prime}$ changes $w_{t+dt:T}$ to $w_{t+dt:T}^{\prime}$ because the HJB
equation depends on the FP equation. Because FBSM does not take the change of
$w_{t+dt:T}$ into account, the convergence of FBSM is not guaranteed.

FBSM has been used in the deterministic control [25, 27, 28, 31] and MFSC [32,
33, 34, 35]. However, the convergence of FBSM for these cases is not
guaranteed because the backward dynamics depends on the forward dynamics even
without the optimal control function (Fig. 1(c,d)). In contrast, the
convergence of FBSM is guaranteed in ML-POSC because the backward HJB equation
does not depend on the forward FP equation without the optimal control
function (Fig. 1(b)).

We firstly show the following lemma, which plays a key role in the convergence
of FBSM:

###### Lemma 2

Given $u_{0:t-dt,t+dt:T-dt}:=\\{u_{0},...,u_{t-dt},u_{t+dt},...,u_{T-dt}\\}$,
$u_{t}^{*}$ is defined by

$\displaystyle u_{t}^{*}(z):=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}(x|z)}\left[\mathcal{H}\left(t,s,u_{t},w_{t+dt}\right)\right],$
(39)

where $p_{t}(x|z):=p_{t}(s)/\int p_{t}(s)dx$ is the conditional probability
density function of the state $x$ given the memory $z$, and $p_{t}(s)$ is the
solution of the following FP equation:

$\displaystyle p_{t+dt}(s)=p_{t}(s)+\mathcal{L}_{u_{t}}^{\dagger}p_{t}(s)dt,$
(40)

where $p_{0}(s)$. $w_{t}(s)$ is the solution of the following HJB equation:

$\displaystyle
w_{t}(s)=w_{t+dt}(s)+\mathcal{H}\left(t,s,u_{t},w_{t+dt}\right)dt,$ (41)

where $w_{T}(s)=g(s)$. Then $u_{t}^{*}$ satisfies the following equation in
ML-POSC:

$\displaystyle u_{t}^{*}=\mathop{\rm argmin}\limits_{u_{t}}J[u_{0:T-dt}].$
(42)

###### Proof:

This lemma is proved by the time discretization method in APPENDIX III-E and
also by the similar way as the Pontyragin’s minimum principle in APPENDIX
III-F. While the former is more intuitive, the latter is more useful for
comparing ML-POSC with the deterministic control and MFSC. ∎

Importantly, $w_{t+dt}$ does not depend on $u_{t}$ in ML-POSC (Fig. 3(a))
while $\lambda_{t+dt}$ and $w_{t+dt}$ depend on $u_{t}$ in the deterministic
control (Fig. 3(b)) and MFSC (Fig. 3(c)), respectively. Therefore, $u_{t}^{*}$
can be obtained without modifying $w_{t+dt}$ in ML-POSC, which is essentially
different from the deterministic control and MFSC. From this nice property of
ML-POSC, FBSM becomes monotonic as follows:

###### Lemma 3

In FBSM of ML-POSC, the objective function is monotonically non-increasing
with respect to the update of the control function at each time step. More
specifically,

$\displaystyle J[u_{0:t-dt}^{k},u_{t:T-dt}^{k+1}]\leq
J[u_{0:t}^{k},u_{t+dt:T-dt}^{k+1}]$ (43)

is satisfied in the backward step, and

$\displaystyle J[u_{0:t-dt}^{k+1},u_{t:T-dt}^{k}]\geq
J[u_{0:t}^{k+1},u_{t+dt:T-dt}^{k}]$ (44)

is satisfied in the forward step.

###### Proof:

The proof is shown in APPENDIX III-G. ∎

###### Theorem 2

In FBSM of ML-POSC, the objective function is monotonically non-increasing
with respect to the update of the control function at each iteration step.
More specifically,

$\displaystyle J[u_{0:T-dt}^{k+1}]\leq J[u_{0:T-dt}^{k}]$ (45)

is satisfied.

###### Proof:

It is obvious from Lemma 3. ∎

Therefore, if $J[u_{0:T-dt}]$ has a lower bound, FBSM is guaranteed to
converge to the local minimum in ML-POSC. At the local minimum, the
Pontryagin’s minimum principle is satisfied as follows:

###### Theorem 3

Assume that if the candidate of $u_{t}^{k+1}(z)$ includes $u_{t}^{k}(z)$, then
set $u_{t}^{k+1}(z)$ at $u_{t}^{k}(z)$. If

$\displaystyle J[u_{0:T-dt}^{k+1}]=J[u_{0:T-dt}^{k}]$ (46)

holds, $u_{0:T-dt}^{k+1}$ satisfies the Pontryagin’s minimum principle
(Theorem 1), which is the necessary condition of the optimal control function.

###### Proof:

The proof is shown in APPENDIX III-H. ∎

Therefore, unlike the deterministic control and MFSC, in FBSM of ML-POSC, the
objective function $J[u_{0:T-dt}^{k}]$ monotonically decreases and finally
converges to the local minimum at which the control function $u_{0:T-dt}^{k}$
satisfies the Pontryagin’s minimum principle.

##  V LINEAR-QUADRATIC-GAUSSIAN PROBLEM

In this section, we apply FBSM into the LQG problem of ML-POSC [11]. In the
LQG problem of ML-POSC, the system of HJB-FP equations is reduced from partial
differential equations to ordinary differential equations.

###  V-A Problem formulation

In the LQG problem of ML-POSC, the extended state SDE (10) is given as follows
[11]:

$\displaystyle
ds_{t}=\left(A(t)s_{t}+B(t)u_{t}\right)dt+\sigma(t)d\omega_{t},$ (47)

where $s_{0}$ obeys the Gaussian distribution
$p_{0}(s_{0}):=\mathcal{N}\left(s_{0}\left|\mu_{0},\Lambda_{0}\right.\right)$
where $\mu_{0}$ is the mean vector and $\Lambda_{0}$ is the precision matrix.
The objective function (20) is given as follows:

$\displaystyle
J[u]:=\mathbb{E}_{p(s_{0:T};u)}\left[\int_{0}^{T}\left(s_{t}^{\top}Q(t)s_{t}+u_{t}^{\top}R(t)u_{t}\right)dt+s_{T}^{\top}Ps_{T}\right],$
(48)

where $Q(t)\succeq O$, $R(t)\succ O$, and $P\succeq O$. The LQG problem of ML-
POSC is the problem to find the optimal control function $u^{*}$ that minimize
the objective function $J[u]$ as follows:

$\displaystyle u^{*}=\mathop{\rm argmin}\limits_{u}J[u].$ (49)

###  V-B Pontryagin’s minimum principle

In the LQG problem of ML-POSC, the Pontryagin’s minimum principle (Theorem 1)
can be calculated as follows [11]:

######  Proposition 5 ([11])

In the LQG problem of ML-POSC, the optimal control function satisfies the
following equation:

$\displaystyle u^{*}(t,z)=-R^{-1}B^{\top}\left(\Pi
K(\Lambda)(s-\mu)+\Psi\mu\right),$ (50)

where $K(\Lambda)$ is defined as follows:

$\displaystyle
K(\Lambda):=\left(\begin{array}[]{cc}O&\Lambda_{xx}^{-1}\Lambda_{xz}\\\ O&I\\\
\end{array}\right),$ (53)

where $\mu(t)$ and $\Lambda(t)$ are the mean vector and the precision matrix
of the extended state, respectively, which correspond to the solution of the
FP equation (30). We note that
$\mathbb{E}_{p_{t}(z|x)}\left[s\right]=K(\Lambda)(s-\mu)+\mu$ is satisfied.
$\mu(t)$ and $\Lambda(t)$ are the solutions of the following ordinary
differential equations (ODEs):

$\displaystyle\dot{\mu}$
$\displaystyle=\left(A-BR^{-1}B^{\top}\Psi\right)\mu,$ (54)
$\displaystyle\dot{\Lambda}$ $\displaystyle=-\left(A-BR^{-1}B^{\top}\Pi
K(\Lambda)\right)^{\top}\Lambda-\Lambda\left(A-BR^{-1}B^{\top}\Pi
K(\Lambda)\right)-\Lambda\sigma\sigma^{\top}\Lambda,$ (55)

where $\mu(0)=\mu_{0}$ and $\Lambda(0)=\Lambda_{0}$. $\Psi(t)$ and $\Pi(t)$
are the control gain matrices of the deterministic and stochastic extended
state, respectively, which correspond to the solution of the HJB equation
(31). $\Psi(t)$ and $\Pi(t)$ are the solutions of the following ODEs:

$\displaystyle-\dot{\Psi}$ $\displaystyle=Q+A^{\top}\Psi+\Psi A-\Psi
BR^{-1}B^{\top}\Psi,$ (56) $\displaystyle-\dot{\Pi}$
$\displaystyle=Q+A^{\top}\Pi+\Pi A-\Pi
BR^{-1}B^{\top}\Pi+(I-K(\Lambda))^{\top}\Pi BR^{-1}B^{\top}\Pi(I-K(\Lambda)),$
(57)

where $\Psi(T)=\Pi(T)=P$. The ODE of $\Psi$ (56) is the Riccati equation [12,
36], which also appears in the LQG problem of COSC. In contrast, the ODE of
$\Pi$ (57) is the partially observable Riccati equation [11], which appears
only in the LQG problem of ML-POSC.

###### Proof:

The proof is shown in [11]. ∎

The ODE of $\Psi$ (56) can be solved backward in time from the terminal
condition. Using $\Psi$, the ODE of $\mu$ (54) can be solved forward in time
from the initial condition. In contrast, the ODEs of $\Pi$ (57) and $\Lambda$
(55) cannot be solved in the similar way as the ODEs of $\Psi$ (56) and $\mu$
(54) because they interact with each other, which is the similar problem with
the system of HJB-FP equations.

###  V-C FBSM in the LQG problem

Algorithm 2 Forward-Backward Sweep Method (FBSM) in the LQG problem

//— Initial step —//

$k\leftarrow 0$

$\Lambda_{0}^{k}\leftarrow\Lambda_{0}$

for $t=0$ to $T-dt$ do

Initialize $\Pi_{t+dt}^{k}$

$\Lambda_{t+dt}^{k}\leftarrow\Lambda_{t}^{k}+\mathcal{F}(\Lambda_{t}^{k},\Pi_{t+dt}^{k})dt$

end for

while $J[u_{0:T-dt}^{k}]$ do not converge do

if $k$ is even then

//— Backward step —//

$\Pi_{T}^{k+1}\leftarrow P$

for $t=T-dt$ to $0$ do

$\Pi_{t}^{k+1}\leftarrow\Pi_{t+dt}^{k+1}+\mathcal{G}(\Lambda_{t}^{k},\Pi_{t+dt}^{k+1})dt$

end for

else

//— Forward step —//

$\Lambda_{0}^{k+1}\leftarrow\Lambda_{0}$

for $t=0$ to $T-dt$ do

$\Lambda_{t+dt}^{k+1}\leftarrow\Lambda_{t}^{k+1}+\mathcal{F}(\Lambda_{t}^{k+1},\Pi_{t+dt}^{k})dt$

end for

end if

$k\leftarrow k+1$

end while

return $u_{0:T-dt}^{k}$

In the FBSM of the LQG problem, the FP equation (30) and the HJB equation (31)
are reduced to the ODE of $\Lambda$ (55) and the ODE of $\Pi$ (57),
respectively, which is summarized in Algorithm 2. $\mathcal{F}(\Lambda,\Pi)$
and $\mathcal{G}(\Lambda,\Pi)$ are defined by the right-hand sides of (55) and
(57), respectively, as follows:

$\displaystyle\mathcal{F}(\Lambda,\Pi)$
$\displaystyle:=-\left(A-BR^{-1}B^{\top}\Pi
K(\Lambda)\right)^{\top}\Lambda-\Lambda\left(A-BR^{-1}B^{\top}\Pi
K(\Lambda)\right)-\Lambda\sigma\sigma^{\top}\Lambda,$ (58)
$\displaystyle\mathcal{G}(\Lambda,\Pi)$ $\displaystyle:=Q+A^{\top}\Pi+\Pi
A-\Pi BR^{-1}B^{\top}\Pi+(I-K(\Lambda))^{\top}\Pi
BR^{-1}B^{\top}\Pi(I-K(\Lambda)).$ (59)

The following proposition shows that FBSM is reduced from Algorithm 1 to
Algorithm 2 in the LQG problem:

###### Proposition 6

Algorithm 2 is the FBSM of the LQG problem when the control function is
initialized by

$\displaystyle
u^{0}(t,z)=-R^{-1}B^{\top}\left(\Pi^{0}K(\Lambda^{0})(s-\mu)+\Psi\mu\right),$
(60)

where $\Pi^{0}$ is arbitrary and $\Lambda^{0}$ is the solution of
$\dot{\Lambda}^{0}=\mathcal{F}(\Lambda^{0},\Pi^{0})$ given
$\Lambda^{0}(0)=\Lambda_{0}$.

###### Proof:

The proof is shown in APPENDIX III-I. ∎

##  VI NUMERICAL EXPERIMENTS

In this section, we verify the convergence of FBSM in ML-POSC by conducting
the numerical experiments of the LQG and non-LQG problems. The setting of the
numerical experiments is the same with the previous work [11].

###  VI-A LQG problem

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4:  (a,b,c) The elements of the control gain matrix
$\Pi^{k}(t)\in\mathbb{R}^{2\times 2}$ and (d,e,f) those of the precision
matrix $\Lambda^{k}(t)\in\mathbb{R}^{2\times 2}$ obtained by FBSM (Algorithm
2) in the numerical experiment of the LQG problem of ML-POSC. Because
$\Pi_{xz}^{k}(t)=\Pi_{zx}^{k}(t)$ and
$\Lambda_{xz}^{k}(t)=\Lambda_{zx}^{k}(t)$, $\Pi_{zx}^{k}(t)$ and
$\Lambda_{zx}^{k}(t)$ are not visualized. The darkest curve corresponds to the
first iteration $k=0$, and the brightest curve corresponds to the last
iteration $k=50$. $\Pi^{0}(t)$ is initialized by $\Pi^{0}(t)=O$.

(a)

(b)

(c)

(d)

Figure 5:  Performance of FBSM in the numerical experiment of the LQG problem
of ML-POSC. (a) The objective function $J[u^{k}]$ with respect to the
iteration $k$. (b,c,d) Stochastic simulation of the state $x_{t}$ (b), the
memory $z_{t}$ (c), and the cumulative cost (d) for 100 samples. The
expectation of the cumulative cost at $t=10$ corresponds to the objective
function (64). Blue and orange curves correspond to the first iteration $k=0$
and the last iteration $k=50$, respectively.

In this subsection, we verify the convergence of FBSM in ML-POSC by conducting
the numerical experiment of the LQG problem. We consider the state
$x_{t}\in\mathbb{R}$, the observation $y_{t}\in\mathbb{R}$, and the memory
$z_{t}\in\mathbb{R}$, which evolve by the following SDEs:

$\displaystyle dx_{t}$ $\displaystyle=\left(x_{t}+u_{t}\right)dt+d\omega_{t},$
(61) $\displaystyle dy_{t}$ $\displaystyle=x_{t}dt+d\nu_{t},$ (62)
$\displaystyle dz_{t}$ $\displaystyle=v_{t}dt+dy_{t},$ (63)

where $x_{0}$ and $z_{0}$ obey the standard Gaussian distributions, $y_{0}$ is
an arbitrary real number, $\omega_{t}\in\mathbb{R}$ and $\nu_{t}\in\mathbb{R}$
are independent standard Wiener processes, $u_{t}=u(t,z_{t})\in\mathbb{R}$ and
$v_{t}=v(t,z_{t})\in\mathbb{R}$ are the controls. The objective function to be
minimized is given as follows:

$\displaystyle
J[u,v]:=\mathbb{E}_{p(x_{0:10},y_{0:10},z_{0:10};u,v)}\left[\int_{0}^{10}\left(x_{t}^{2}+u_{t}^{2}+v_{t}^{2}\right)dt\right].$
(64)

Therefore, the objective of this problem is to minimize the state variance by
the small state and memory controls.

This problem corresponds to the LQG problem, which is defined by (47) and
(48). By defining $s_{t}:=(x_{t},z_{t})\in\mathbb{R}^{2}$,
$\tilde{u}_{t}:=(u_{t},v_{t})\in\mathbb{R}^{2}$, and
$\tilde{\omega}_{t}:=(\omega_{t},\nu_{t})\in\mathbb{R}^{2}$, the SDEs (61),
(62), (63) can be rewritten as follows:

$\displaystyle ds_{t}=\left(\left(\begin{array}[]{cc}1&0\\\ 1&0\\\
\end{array}\right)s_{t}+\tilde{u}_{t}\right)dt+d\tilde{\omega}_{t},$ (67)

which corresponds to (47). Furthermore, the objective function (64) can be
rewritten as follows:

$\displaystyle
J[\tilde{u}]:=\mathbb{E}_{p(s_{0:10};\tilde{u})}\left[\int_{0}^{10}\left(s_{t}^{\top}\left(\begin{array}[]{cc}1&0\\\
0&0\\\
\end{array}\right)s_{t}+\tilde{u}_{t}^{\top}\tilde{u}_{t}\right)dt\right],$
(70)

which corresponds to (48).

We apply the FBSM of the LQG problem (Algorithm 2) to this problem.
$\Pi^{0}(t)$ is initialized by $\Pi^{0}(t)=O$. In order to solve the ODEs of
$\Pi^{k}(t)$ and $\Lambda^{k}(t)$, we use the Runge-Kutta method. Figure 4
shows the control gain matrix $\Pi^{k}(t)\in\mathbb{R}^{2\times 2}$ and the
precision matrix $\Lambda^{k}(t)\in\mathbb{R}^{2\times 2}$ obtained by FBSM.
The color of the curve represents the iteration $k$. The darkest curve
corresponds to the first iteration $k=0$, and the brightest curve corresponds
to the last iteration $k=50$. Importantly, $\Pi^{k}(t)$ and $\Lambda^{k}(t)$
converge with respect to the iteration $k$.

Figure 5(a) shows the objective function $J[u^{k}]$ with respect to the
iteration $k$. The objective function $J[u^{k}]$ monotonically decreases with
respect to the iteration $k$, which is consistent with Theorem 2. The
objective function $J[u^{k}]$ converges, and its $u^{k}$ satisfies the
Pontryagin’s minimum principle from Theorem 3. These monotonicity and
convergence of FBSM are the nice properties of ML-POSC that are not guaranteed
in the deterministic control and MFSC.

Figure 5(b,c,d) compares the performance of the control function $u^{k}$ at
the first iteration $k=0$ and the last iteration $k=50$ by conducting the
stochastic simulation. At the first iteration $k=0$, the distributions of the
state and the memory are unstable, and the cumulative cost diverges. In
contrast, at the last iteration $k=50$, the distributions of the state and the
memory are stable, and the cumulative cost is smaller. This result indicates
that FBSM improves the performance in ML-POSC.

###  VI-B non-LQG problem

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

Figure 6:  The solutions of the HJB equation $w^{k}(t,s)$ (a-j) and the FP
equation $p^{k}(t,s)$ (k-t) at the first iteration $k=0$ (a-e,k-o) and at the
last iteration $k=50$ (f-j,p-t) of FBSM (Algorithm 1) in the numerical
experiment of the non-LQG problem of ML-POSC. $u^{0}(t,z)$ is initialized by
$u^{0}(t,z)=0$.

(a)

(b)

(c)

Figure 7:  Performance of FBSM in the numerical experiment of the non-LQG
problem of ML-POSC. (a) The objective function $J[u^{k}]$ with respect to the
iteration $k$. (b) Stochastic simulation of the state $x_{t}$ for 100 samples.
The black rectangles and cross represent the obstacles and the goal,
respectively. Blue and orange curves correspond to the first iteration $k=0$
and the last iteration $k=50$, respectively. (c) The objective function (74),
which is computed from 100 samples.

In this subsection, we verify the convergence of FBSM in ML-POSC by conducting
the numerical experiment of the non-LQG problem. We consider the state
$x_{t}\in\mathbb{R}$, the observation $y_{t}\in\mathbb{R}$, and the memory
$z_{t}\in\mathbb{R}$, which evolve by the following SDEs:

$\displaystyle dx_{t}$ $\displaystyle=u_{t}dt+d\omega_{t},$ (71)
$\displaystyle dy_{t}$ $\displaystyle=x_{t}dt+d\nu_{t},$ (72) $\displaystyle
dz_{t}$ $\displaystyle=dy_{t},$ (73)

where $x_{0}$ and $z_{0}$ obey the Gaussian distributions
$p_{0}(x_{0})=\mathcal{N}(x_{0}|0,0.01)$ and
$p_{0}(z_{0})=\mathcal{N}(z_{0}|0,0.01)$, respectively, $y_{0}$ is an
arbitrary real number, $\omega_{t}\in\mathbb{R}$ and $\nu_{t}\in\mathbb{R}$
are independent standard Wiener processes, $u_{t}=u(t,z_{t})\in\mathbb{R}$ is
the control. For the sake of simplicity, the memory control is not considered.
The objective function to be minimized is given as follows:

$\displaystyle
J[u]:=\mathbb{E}_{p(x_{0:1},y_{0:1},z_{0:1};u)}\left[\int_{0}^{1}\left(Q(t,x_{t})+u_{t}^{2}\right)dt+10x_{1}^{2}\right],$
(74)

where

$\displaystyle Q(t,x):=\begin{cases}1000&(0.3\leq t\leq 0.6,0.1\leq|x|\leq
2.0),\\\ 0&(others).\end{cases}$ (75)

The cost function is high on the black rectangles in Fig. 7 (b), which
represents the obstacles. In addition, the terminal cost function is the
lowest on the black cross in Fig. 7 (b), which represents the desirable goal.
Therefore, the system should avoid the obstacles and reach the goal with the
small control. Because the cost function is non-quadratic, it is a non-LQG
problem.

We apply FBSM (Algorithm 1) to this problem. $u^{0}(t,z)$ is initialized by
$u^{0}(t,z)=0$. In order to solve the HJB equation and the FP equation, we use
the finite-difference method. Figure 6 shows $w^{k}(t,s)$ and $p^{k}(t,s)$
obtained by FBSM. From the proof of Lemma 2 (APPENDIX III-E), $w^{k}(t,s)$ is
given as follows:

$\displaystyle
w^{k}(t,s)=\mathbb{E}_{p(s_{t+dt:T}|s_{t}=s;u^{k})}\left[\int_{t}^{1}\left(Q(\tau,x_{\tau})+(u_{\tau}^{k})^{2}\right)d\tau+10x_{1}^{2}\right].$
(76)

Because $u^{0}(t,z)=0$, $w^{0}(t,s)$ reflects the cost function corresponding
to the obstacles and the goal (Fig. 6(a-e)). In contrast, because
$u^{50}(t,z)\neq 0$, $w^{50}(t,s)$ becomes more complex (Fig. 6(f-j)).
Especially, while $w^{0}(t,s)$ does not depends on the memory $z$,
$w^{50}(t,s)$ depends on the memory $z$, which indicates that the control
function $u^{50}(t,z)$ is adjusted by the memory $z$. Furthermore, while
$p^{0}(t,s)$ is a unimodal distribution (Fig. 6(k-o)), $p^{50}(t,s)$ is a
bimodal distribution (Fig. 6(p-t)), which can avoid the obstacles.

Figure 7(a) shows the objective function $J[u^{k}]$ with respect to the
iteration $k$. The objective function $J[u^{k}]$ monotonically decreases with
respect to the iteration $k$, which is consistent with Theorem 2. The
objective function $J[u^{k}]$ converges, and its $u^{k}$ satisfies the
Pontryagin’s minimum principle from Theorem 3. These monotonicity and
convergence of FBSM are the nice properties of ML-POSC that are not guaranteed
in the deterministic control and MFSC.

Figure 7(b,c) compares the performance of the control function $u^{k}$ at the
first iteration $k=0$ and the last iteration $k=50$ by conducting the
stochastic simulation. At the first iteration $k=0$, the obstacles cannot be
avoided, which results in the higher objective function. In contrast, at the
last iteration $k=50$, the obstacles can be avoided, which results in the
lower objective function. This result indicates that FBSM improves the
performance in ML-POSC.

##  VII CONCLUSION

In this work, we firstly showed that the system of HJB-FP equations
corresponds to the Pontryagin’s minimum principle on the probability density
function space. Although the relationship between the system of HJB-FP
equations and the Pontryagin’s minimum principle has been mentioned briefly in
MFSC [22, 23, 24], its details have not yet been investigated. We resolved
this problem by deriving the system of HJB-FP equations in a similar way to
the Pontryagin’s minimum principle. We then proposed FBSM to ML-POSC. Although
the convergence of FBSM is generally not guaranteed in the deterministic
control [25, 27, 28, 31] and MFSC [32, 33, 34, 35], we proved the convergence
in ML-POSC by noting the fact that the update of the current control function
does not affect the future HJB equation in ML-POSC. Therefore, ML-POSC is a
special and nice class where FBSM is guaranteed to converge.

The regularized FBSM has recently been proposed in the deterministic control,
which is guaranteed to converge even in the general deterministic control [37,
38]. Our work gives an intuitive reason why the regularized FBSM is guaranteed
to converge. In the regularized FBSM, the Hamiltonian is regularized, which
makes the update of the control function smaller. When the regularization is
sufficiently strong, the effect from the current control function to the
future backward dynamics is negligible. Therefore, the regularized FBSM of the
deterministic control is guaranteed to converge for the similar reason to the
FBSM of ML-POSC. However, the convergence of the regularized FBSM is much
slower because the stronger regularization makes the update of the control
function smaller. The FBSM of ML-POSC does not suffer from such a problem
because the future backward dynamics already does not depend on the current
control function without regularization.

Our work gives a hint about a modification of the fixed-point iteration method
to ensure the convergence in MFSC. Although the fixed-point iteration method
is the most basic algorithm in MFSC, its convergence is not guaranteed [32,
33, 34, 35]. Our work showed that the fixed-point iteration method is
equivalent with the FBSM on the probability density function space. Therefore,
the idea of the regularized FBSM may also be applied to the fixed-point
iteration method. More specifically, the fixed-point iteration method may be
guaranteed to converge by regularizing the expected Hamiltonian.

In FBSM, we solve the HJB equation and the FP equation by the finite-
difference method. However, because the finite-difference method is prone to
the curse of dimensionality, it is difficult to solve high-dimensional ML-
POSC. In order to resolve this problem, we consider two directions. One
direction is the policy iteration method [15, 39, 40]. Although the policy
iteration method is almost the same with FBSM, only the update of the control
function is different. While FBSM updates the system of HJB equations and the
control function simultaneously, the policy iteration method updates them
separately. In the policy iteration method, the system of HJB-FP equations
becomes linear, which can be solved by the sampling method [41, 42, 43].
Because the sampling method is more tractable than the finite-difference
method, the policy iteration method may allow high-dimensional ML-POSC to be
solved. Furthermore, the policy iteration method has recently been studied in
MFSC [44, 45, 46]. However, its convergence is not guaranteed except for
special cases in MFSC. In the similar way as FBSM, the convergence of the
policy iteration method may be guaranteed in ML-POSC.

The other direction is the machine learning. The neural network-based
algorithms have recently been proposed in MFSC, which can solve high-
dimensional problems efficiently [47, 48]. By extending these algorithms,
high-dimensional ML-POSC may be solved efficiently. Furthermore, unlike MFSC,
the coupling of HJB-FP equations is limited only to the optimal control
function in ML-POSC. By exploiting this nice property, more efficient
algorithms may be devised for ML-POSC.

##  APPENDIX I DETERMINISTIC CONTROL

In this section, we review the Pontryagin’s minimum principle in the
deterministic control [12, 16, 17, 18].

###  APPENDIX I-A Problem formulation

In this subsection, we formulate the deterministic control [16, 17, 18]. The
state of the system $s_{t}\in\mathbb{R}^{d_{s}}$ at time $t\in[0,T]$ evolves
by the following ordinary differential equation (ODE):

$\displaystyle\frac{ds_{t}}{dt}=b(t,s_{t},u_{t}),$ (77)

where the initial state is $s_{0}$, and the control is
$u_{t}=u(t)\in\mathbb{R}^{d_{u}}$. The objective function is given by the
following cumulative cost function:

$\displaystyle J[u]:=\int_{0}^{T}f(t,s_{t},u_{t})dt+g(s_{T}),$ (78)

where $f$ is the cost function, and $g$ is the terminal cost function. The
deterministic control is the problem to find the optimal control function
$u^{*}$ that minimizes the cumulative cost function $J[u]$ as follows:

$\displaystyle u^{*}=\mathop{\rm argmin}\limits_{u}J[u].$ (79)

###  APPENDIX I-B Hamiltonian

Before we show the optimality conditions, we define the Hamiltonian as
follows:

$\displaystyle\mathcal{H}\left(t,s,u,\lambda\right):=f(t,s,u)+\lambda^{\top}b(t,s,u),$
(80)

where ${}^{\forall}\lambda\in\mathbb{R}^{d_{s}}$ is the adjoint variable. The
following lemma shows the relationship between the objective function $J[u]$
and the Hamiltonian $\mathcal{H}(t,s,u,\lambda)$, which is significant for the
optimality conditions [12, 16].

######  Lemma 4 ([12, 16])

Let ${}^{\forall}u:[0,T]\to\mathbb{R}^{d_{u}}$ and
${}^{\forall}u^{\prime}:[0,T]\to\mathbb{R}^{d_{u}}$ be the arbitrary control
functions, and let $s$ and $s^{\prime}$ be the states driven by the control
functions $u$ and $u^{\prime}$, respectively. Then $J[u]-J[u^{\prime}]$
satisfies the following equation:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{\prime})-\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})-\left(\frac{\partial\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})}{\partial
s}\right)^{\top}(s_{t}-s_{t}^{\prime})\right)dt$ $\displaystyle\ \ \
+g(s_{T})-g(s_{T}^{\prime})-\left(\frac{\partial g(s_{T}^{\prime})}{\partial
s}\right)^{\top}(s_{T}-s_{T}^{\prime}),$ (81)

where $\lambda_{t}^{\prime}$ is the adjoint variable of $s_{t}^{\prime}$,
which is the solution of the following ODE:

$\displaystyle-\frac{d\lambda_{t}^{\prime}}{dt}=\frac{\partial\mathcal{H}\left(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime}\right)}{\partial
s},$ (82)

where $\lambda_{T}^{\prime}=\partial g(s_{T}^{\prime})/\partial s$.

###### Proof:

$J[u]-J[u^{\prime}]$ can be calculated as follows:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\left[\int_{0}^{T}f(t,s_{t},u_{t})dt+g(s_{T})\right]-\left[\int_{0}^{T}f(t,s_{t}^{\prime},u_{t}^{\prime})dt+g(s_{T}^{\prime})\right]$
$\displaystyle=\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{\prime})-(\lambda_{t}^{\prime})^{\top}b(t,s_{t},u_{t})\right)dt+g(s_{T})\right]$
$\displaystyle\ \ \
-\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})-(\lambda_{t}^{\prime})^{\top}b(t,s_{t}^{\prime},u_{t}^{\prime})\right)dt+g(s_{T}^{\prime})\right]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{\prime})-\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}(\lambda_{t}^{\prime})^{\top}\left(b(t,s_{t},u_{t})-b(t,s_{t}^{\prime},u_{t}^{\prime})\right)dt+g(s_{T})-g(s_{T}^{\prime}).$
(83)

From the state equation (77),

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{\prime})-\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}(\lambda_{t}^{\prime})^{\top}\frac{d(s_{t}-s_{t}^{\prime})}{dt}dt+g(s_{T})-g(s_{T}^{\prime}).$
(84)

From the integration by parts and $s_{0}-s_{0}^{\prime}=0$,

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{\prime})-\mathcal{H}(t,s_{t}^{\prime},u_{t}^{\prime},\lambda_{t}^{\prime})\right)dt$
$\displaystyle\ \ \
+\int_{0}^{T}\left(\frac{d\lambda_{t}^{\prime}}{dt}\right)^{\top}(s_{t}-s_{t}^{\prime})dt+g(s_{T})-g(s_{T}^{\prime})-(\lambda_{T}^{\prime})^{\top}(s_{T}-s_{T}^{\prime}).$
(85)

From the adjoint equation (82), (81) is obtained. ∎

###  APPENDIX I-C Necessary condition

We show the Pontryagin’s minimum principle, which is a necessary condition of
the optimal control function as follows [12, 16].

######  Proposition 7 ([12, 16])

In the deterministic control, the optimal control function $u^{*}$ satisfies
the following equation:

$\displaystyle u^{*}(t)=\mathop{\rm
argmin}\limits_{u}\mathcal{H}\left(t,s_{t}^{*},u,\lambda_{t}^{*}\right),\
^{\forall}t\in[0,T],$ (86)

where $s_{t}^{*}$ is the state driven by the optimal control function $u^{*}$,
which is the solution of the following state equation:

$\displaystyle\frac{ds_{t}}{dt}=\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial\lambda},$
(87)

where the initial state is $s_{0}$. Because
$\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})/\partial\lambda=b(t,s_{t}^{*},u_{t}^{*})$,
(87) is consistent with (77). $\lambda_{t}^{*}$ is the adjoint variable of
$s_{t}^{*}$, which is the solution of the following adjoint equation:

$\displaystyle-\frac{d\lambda_{t}^{*}}{dt}=\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s},$ (88)

where $\lambda_{T}^{*}=\partial g(s_{T}^{*})/\partial s$.

###### Proof:

We define the control function

$\displaystyle u^{\varepsilon}(t):=\begin{cases}u^{*}(t)&t\in[0,T]\backslash
E_{\varepsilon},\\\ u(t)&t\in E_{\varepsilon},\\\ \end{cases}$ (89)

where $E_{\varepsilon}:=[t^{\prime},t^{\prime}+\varepsilon]\subseteq[0,T]$,
and ${}^{\forall}u:[0,T]\to\mathbb{R}^{d_{u}}$. From the Lemma 4,
$J[u^{\varepsilon}]-J[u^{*}]$ can be calculated as follows:

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t}^{\varepsilon},u_{t}^{\varepsilon},\lambda_{t}^{*})-\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})-\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}^{\varepsilon}-s_{t}^{*})\right)dt$ $\displaystyle\ \ \
+g(s_{T}^{\varepsilon})-g(s_{T}^{*})-\left(\frac{\partial
g(s_{T}^{*})}{\partial s}\right)^{\top}(s_{T}^{\varepsilon}-s_{T}^{*})$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t}^{\varepsilon},u_{t}^{*},\lambda_{t}^{*})-\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})-\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}^{\varepsilon}-s_{t}^{*})\right)dt$ $\displaystyle\ \ \
+g(s_{T}^{\varepsilon})-g(s_{T}^{*})-\left(\frac{\partial
g(s_{T}^{*})}{\partial s}\right)^{\top}(s_{T}^{\varepsilon}-s_{T}^{*})$
$\displaystyle\ \ \
+\int_{E_{\varepsilon}}\left(\mathcal{H}(t,s_{t}^{\varepsilon},u_{t},\lambda_{t}^{*})-\mathcal{H}(t,s_{t}^{\varepsilon},u_{t}^{*},\lambda_{t}^{*})\right)dt.$
(90)

Letting $\varepsilon\to 0$,

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}^{\varepsilon}-s_{t}^{*})-\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}^{\varepsilon}-s_{t}^{*})\right)dt$ $\displaystyle\ \ \
+\left(\frac{\partial g(s_{T}^{*})}{\partial
s}\right)^{\top}(s_{T}^{\varepsilon}-s_{T}^{*})-\left(\frac{\partial
g(s_{T}^{*})}{\partial s}\right)^{\top}(s_{T}^{\varepsilon}-s_{T}^{*})$
$\displaystyle\ \ \
+\left(\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}},\lambda_{t^{\prime}}^{*})-\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}}^{*},\lambda_{t^{\prime}}^{*})\right)dt$
$\displaystyle=\left(\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}},\lambda_{t^{\prime}}^{*})-\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}}^{*},\lambda_{t^{\prime}}^{*})\right)dt.$
(91)

Because $u^{*}$ is the optimal control function, the following inequality is
satisfied:

$\displaystyle 0\leq J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\left(\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}},\lambda_{t^{\prime}}^{*})-\mathcal{H}(t^{\prime},s_{t^{\prime}}^{*},u_{t^{\prime}}^{*},\lambda_{t^{\prime}}^{*})\right)dt.$
(92)

Therefore, (86) is obtained. ∎

###  APPENDIX I-D Sufficient condition

The Pontryagin’s minimum principle becomes a necessary and sufficient
condition if the Hamiltonian and the terminal cost function are convex as
follows [12, 16]:

######  Proposition 8 ([12, 16])

Assume that $\mathcal{H}(t,s,u,\lambda)$ is convex with respect to $s$ and $u$
and $g(s)$ is convex with respect to $s$. If the control function $u^{*}$
satisfies (86), then it is the optimal control function of the deterministic
control.

###### Proof:

We define the arbitrary control function
${}^{\forall}u:[0,T]\to\mathbb{R}^{d_{u}}$. From the Lemma 4, $J[u]-J[u^{*}]$
is given by the following equation:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{*})-\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})-\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}-s_{t}^{*})\right)dt$ $\displaystyle\ \ \
+g(s_{T})-g(s_{T}^{*})-\left(\frac{\partial g(s_{T}^{*})}{\partial
s}\right)^{\top}(s_{T}-s_{T}^{*}).$ (93)

Because $\mathcal{H}(t,s,u,\lambda)$ is convex with respect to $s$ and $u$ and
$g(s)$ is convex with respect to $s$, the following inequalities are
satisfied:

$\displaystyle\mathcal{H}(t,s_{t},u_{t},\lambda_{t}^{*})$
$\displaystyle\geq\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})+\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
s}\right)^{\top}(s_{t}-s_{t}^{*})+\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
u}\right)^{\top}(u_{t}-u_{t}^{*}),$ (94) $\displaystyle g(s_{T})$
$\displaystyle\geq g(s_{T}^{*})+\left(\frac{\partial g(s_{T}^{*})}{\partial
s}\right)^{\top}(s_{T}-s_{T}^{*}).$ (95)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle\geq\int_{0}^{T}\left(\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
u}\right)^{\top}(u_{t}-u_{t}^{*})dt.$ (96)

Because $u^{*}$ satisfies (86), the following stationary condition is
satisfied:

$\displaystyle\frac{\partial\mathcal{H}(t,s_{t}^{*},u_{t}^{*},\lambda_{t}^{*})}{\partial
u}=0.$ (97)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$ $\displaystyle\geq 0.$ (98)

Therefore, $u^{*}$ is the optimal control function. ∎

###  APPENDIX I-E Relationship with the Bellman’s dynamic programming
principle

From the Bellman’s dynamic programming principle, the optimal control function
$u^{*}$ is given as follows [12, 16]:

$\displaystyle u^{*}(t,s)=\mathop{\rm
argmin}\limits_{u}\mathcal{H}\left(t,s,u,\frac{\partial V^{*}(t,s)}{\partial
s}\right).$ (99)

More specifically, the optimal control function is given by
$u^{*}(t)=u^{*}(t,s_{t}^{*})$. $V^{*}(t,s)$ is the value function, which is
the solution of the following Hamilton-Jacobi-Bellman (HJB) equation:

$\displaystyle-\frac{\partial V^{*}(t,s)}{\partial
t}=\mathcal{H}\left(t,s,u^{*},\frac{\partial V^{*}(t,s)}{\partial s}\right),$
(100)

where $V^{*}(T,s)=g(s)$. The following proposition converts the Bellman’s
dynamic programming principle into the Pontryagin’s minimum principle [12,
16]:

######  Proposition 9 ([12, 16])

We define $\lambda_{t}^{*}$ from $V^{*}(t,s)$ as follows:

$\displaystyle\lambda_{t}^{*}:=\frac{\partial V^{*}(t,s_{t}^{*})}{\partial
s},$ (101)

where $s_{t}^{*}$ is the solution of the state equation (87).
$\lambda_{t}^{*}$ satisfies the adjoint equation (88) from the HJB equation
(100).

###### Proof:

We firstly define

$\displaystyle\Lambda^{*}(t,s):=\frac{\partial V^{*}(t,s)}{\partial s}.$ (102)

Differentiating the HJB equation (100) with respect to $s$, the following
equation is obtained:

$\displaystyle-\frac{\partial\Lambda^{*}(t,s)}{\partial
t}=\frac{\partial\mathcal{H}\left(t,s,u^{*},\Lambda^{*}\right)}{\partial
s}+\left(\frac{\partial\Lambda^{*}(t,s)}{\partial
s}\right)^{\top}b(t,s^{*},u^{*}),$ (103)

where $\Lambda^{*}(T,s)=\partial g(s)/\partial s$. Then the derivative of
$\lambda_{t}^{*}=\Lambda^{*}(t,s_{t}^{*})$ with respect to $t$ can be
calculated as follows:

$\displaystyle\frac{d\lambda_{t}^{*}}{dt}=\frac{\partial\Lambda^{*}(t,s_{t}^{*})}{\partial
t}+\left(\frac{\partial\Lambda^{*}(t,s_{t}^{*})}{\partial
s}\right)^{\top}\frac{ds_{t}^{*}}{dt}.$ (104)

By substituting (103) into (104), the following equation is obtained:

$\displaystyle-\frac{d\lambda_{t}^{*}}{dt}=\frac{\partial\mathcal{H}\left(t,s,u^{*},\lambda^{*}\right)}{\partial
s}+\left(\frac{\partial\Lambda^{*}(t,s_{t}^{*})}{\partial
s}\right)^{\top}\underbrace{\left(b(t,s^{*},u^{*})-\frac{ds_{t}^{*}}{dt}\right)}_{(*)}.$
(105)

From the state equation (87), $(*)=0$ is satisfied. Therefore,
$\lambda^{*}(t)$ satisfies the adjoint equation (88). ∎

##  APPENDIX II MEAN-FIELD STOCHASTIC CONTROL

In this section, we show that the system of HJB-FP equations in MFSC
corresponds to the Pontryagin’s minimum principle on the probability density
function space. Although the relationship between the system of HJB-FP
equations and the Pontryagin’s minimum principle has been mentioned briefly in
MFSC [22, 23, 24], its details have not yet been investigated. In this
section, we resolve this problem by deriving the system of HJB-FP equations in
a similar way as APPENDIX I.

###  APPENDIX II-A Problem formulation

In this subsection, we formulate MFSC [19, 20, 21]. The state of the system
$s_{t}\in\mathbb{R}^{d_{s}}$ at time $t\in[0,T]$ evolves by the following
stochastic differential equation (SDE):

$\displaystyle
ds_{t}=b(t,s_{t},p_{t},u_{t})dt+\sigma(t,s_{t},p_{t},u_{t})d\omega_{t},$ (106)

where $s_{0}$ obeys $p_{0}(s_{0})$, $p_{t}(s):=p(t,s)$ is the probability
density function of the state $s$, $u_{t}(s):=u(t,s)\in\mathbb{R}^{d_{u}}$ is
the control, and $\omega_{t}\in\mathbb{R}^{d_{\omega}}$ is the standard Wiener
process. The objective function is given by the following expected cumulative
cost function:

$\displaystyle
J[u]:=\mathbb{E}_{p(s_{0:T};u)}\left[\int_{0}^{T}f(t,s_{t},p_{t},u_{t})dt+g(s_{T},p_{T})\right],$
(107)

where $f$ is the cost function, $g$ is the terminal cost function,
$p(s_{0:T};u)$ is the probability of $s_{0:t}:=\\{s_{\tau}|\tau\in[0,t]\\}$
given $u$ as a parameter, and $\mathbb{E}_{p}[\cdot]$ is the expectation with
respect to the probability $p$. MFSC is the problem to find the optimal
control function $u^{*}$ that minimizes the expected cumulative cost function
$J[u]$ as follows:

$\displaystyle u^{*}=\mathop{\rm argmin}\limits_{u}J[u].$ (108)

###  APPENDIX II-B Hamiltonian

Before we show the optimality conditions, we define the Hamiltonian as
follows:

$\displaystyle\mathcal{H}\left(t,s,p,u,w\right):=f(t,s,p,u)+\mathcal{L}_{u}w(t,s),$
(109)

where ${}^{\forall}w:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}$, and
$\mathcal{L}_{u}$ is the backward diffusion operator, which is defined as
follows:

$\displaystyle\mathcal{L}_{u}w(t,s)$
$\displaystyle:=\sum_{i=1}^{d_{s}}b_{i}(t,s,p,u)\frac{\partial
w(t,s)}{\partial
s_{i}}+\frac{1}{2}\sum_{i,j=1}^{d_{s}}D_{ij}(t,s,p,u)\frac{\partial^{2}w(t,s)}{\partial
s_{i}\partial s_{j}},$ (110)

where $D(t,s,p,u):=\sigma(t,s,p,u)\sigma^{\top}(t,s,p,u)$. We also define the
expected Hamiltonian and the expected terminal cost function as follows:

$\displaystyle\bar{\mathcal{H}}(t,p,u,w)$
$\displaystyle:=\mathbb{E}_{p(s)}\left[\mathcal{H}\left(t,s,p,u,w\right)\right],$
(111) $\displaystyle\bar{g}(p)$
$\displaystyle:=\mathbb{E}_{p(s)}\left[g(s,p)\right].$ (112)

The state SDE (106) can be converted into the following Fokker-Planck (FP)
equation:

$\displaystyle\frac{\partial p(t,s)}{\partial
t}=\mathcal{L}_{u}^{\dagger}p(t,s),$ (113)

where $p(0,s)=p_{0}(s)$, and $\mathcal{L}_{u}^{{\dagger}}$ is the forward
diffusion operator, which is defined as follows:

$\displaystyle\mathcal{L}_{u}^{{\dagger}}p(t,s)$
$\displaystyle:=-\sum_{i=1}^{d_{s}}\frac{\partial(b_{i}(t,s,p,u)p(t,s))}{\partial
s_{i}}+\frac{1}{2}\sum_{i,j=1}^{d_{s}}\frac{\partial^{2}(D_{ij}(t,s,p,u)p(t,s))}{\partial
s_{i}\partial s_{j}}.$ (114)

We note that $\mathcal{L}_{u}^{{\dagger}}$ is the conjugate of
$\mathcal{L}_{u}$ as follows:

$\displaystyle\int w(t,s)\mathcal{L}_{u}^{{\dagger}}p(t,s)ds=\int
p(t,s)\mathcal{L}_{u}w(t,s)ds.$ (115)

The following lemma shows the relationship between the objective function
$J[u]$ and the expected Hamiltonian $\bar{\mathcal{H}}(t,p,u,w)$, which is
significant for the optimality conditions.

###### Lemma 5

Let ${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}^{d_{u}}$ and
${}^{\forall}u^{\prime}:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}^{d_{u}}$ be
the arbitrary control functions, and let $p$ and $p^{\prime}$ be the
probability density functions of the state driven by the control functions $u$
and $u^{\prime}$, respectively. Then $J[u]-J[u^{\prime}]$ satisfies the
following equation:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{\prime})-\bar{\mathcal{H}}(t,p^{\prime},u^{\prime},w^{\prime})-\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{\prime},u^{\prime},w^{\prime}\right)}{\delta
p}(s)\left(p(t,s)-p^{\prime}(t,s)\right)ds\right)dt$ $\displaystyle\ \ \
+\bar{g}(p)-\bar{g}(p^{\prime})-\int\frac{\delta\bar{g}(p^{\prime})}{\delta
p}(s)\left(p(T,s)-p^{\prime}(T,s)\right)ds,$ (116)

where $w^{\prime}$ is the solution of the following Hamilton-Jacobi-Bellman
(HJB) equation:

$\displaystyle-\frac{\partial w^{\prime}(t,s)}{\partial
t}=\frac{\delta\bar{\mathcal{H}}\left(t,p^{\prime},u^{\prime},w^{\prime}\right)}{\delta
p}(s),$ (117)

where $w^{\prime}(T,s)=(\delta\bar{g}(p^{\prime})/\delta p)(s)$.

###### Proof:

$J[u]-J[u^{\prime}]$ can be calculated as follows:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\mathbb{E}_{p(s_{0:T})}\left[\int_{0}^{T}f(t,s_{t},p_{t},u_{t})dt+g(s_{T},p_{T})\right]-\mathbb{E}_{p^{\prime}(s_{0:T})}\left[\int_{0}^{T}f(t,s_{t},p_{t}^{\prime},u_{t}^{\prime})dt+g(s_{T},p_{T}^{\prime})\right]$
$\displaystyle=\mathbb{E}_{p(s_{0:T})}\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t},p_{t},u_{t},w^{\prime})-\mathcal{L}_{u_{t}}w^{\prime}(t,s_{t})\right)dt+g(s_{T},p_{T})\right]$
$\displaystyle\ \ \
-\mathbb{E}_{p^{\prime}(s_{0:T})}\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t},p_{t}^{\prime},u_{t}^{\prime},w^{\prime})-\mathcal{L}_{u_{t}^{\prime}}w^{\prime}(t,s_{t})\right)dt+g(s_{T},p_{T}^{\prime})\right]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{\prime})-\bar{\mathcal{H}}(t,p^{\prime},u^{\prime},w^{\prime})\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{L}_{u}w^{\prime}(t,s)\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{L}_{u^{\prime}}w^{\prime}(t,s)\right]\right)dt+\bar{g}(p)-\bar{g}(p^{\prime}).$
(118)

Because $\mathcal{L}_{u_{t}}$ and $\mathcal{L}_{u_{t}^{\prime}}$ are the
conjugates of $\mathcal{L}_{u_{t}}^{{\dagger}}$ and
$\mathcal{L}_{u_{t}^{\prime}}^{{\dagger}}$, respectively,

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{\prime})-\bar{\mathcal{H}}(t,p^{\prime},u^{\prime},w^{\prime})\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\int\left(\mathcal{L}_{u}^{{\dagger}}p(t,s)-\mathcal{L}_{u^{\prime}}^{{\dagger}}p^{\prime}(t,s)\right)w^{\prime}(t,s)dsdt+\bar{g}(p)-\bar{g}(p^{\prime}).$
(119)

From the FP equation (113),

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{\prime})-\bar{\mathcal{H}}(t,p^{\prime},u^{\prime},w^{\prime})\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\int\frac{\partial\left(p(t,s)-p^{\prime}(t,s)\right)}{\partial
t}w^{\prime}(t,s)dsdt+\bar{g}(p)-\bar{g}(p^{\prime}).$ (120)

From the integration by parts and
$p(0,s)-p^{\prime}(0,s)=p_{0}(s)-p_{0}(s)=0$,

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{\prime})-\bar{\mathcal{H}}(t,p^{\prime},u^{\prime},w^{\prime})\right)dt$
$\displaystyle\ \ \
+\int_{0}^{T}\int\left(p(t,s)-p^{\prime}(t,s)\right)\frac{\partial
w^{\prime}(t,s)}{\partial t}dsdt$ $\displaystyle\ \ \
+\bar{g}(p)-\bar{g}(p^{\prime})-\int\left(p(T,s)-p^{\prime}(T,s)\right)w^{\prime}(T,s)ds.$
(121)

From the HJB equation (117), (116) is obtained. ∎

###  APPENDIX II-C Necessary condition

We show the necessary condition of the optimal control function of MFSC that
corresponds to the Pontryagin’s minimum principle on the probability density
function space.

###### Theorem 4

In MFSC, the optimal control function $u^{*}$ satisfies the following
equation:

$\displaystyle u^{*}(t,s)=\mathop{\rm
argmin}\limits_{u}\mathcal{H}\left(t,s,p^{*},u,w^{*}\right),\ a.s.\
^{\forall}t\in[0,T],\ ^{\forall}s\in\mathbb{R}^{d_{s}}$ (122)

where $p^{*}(t,s)$ is the solution of the following FP equation:

$\displaystyle\frac{\partial p^{*}(t,s)}{\partial
t}=\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta w}(s),$ (123)

where $p^{*}(0,s)=p_{0}(s)$. Because
$(\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})/\delta
w)(s)=\mathcal{L}_{u^{*}}^{\dagger}p^{*}(t,s)$, (123) is consistent with
(113). $w^{*}(t,s)$ is the solution of the following HJB equation:

$\displaystyle-\frac{\partial w^{*}(t,s)}{\partial
t}=\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta p}(s),$ (124)

where $w^{*}(T,s)=(\delta\bar{g}(p^{*})/\delta p)(s)$.

###### Proof:

We define the control function

$\displaystyle
u^{\varepsilon}(t,z):=\begin{cases}u^{*}(t,s)&(t,s)\in([0,T]\times\mathbb{R}^{d_{s}})\backslash(E_{\varepsilon_{1}}\times
F_{\varepsilon_{2}}),\\\ u(t,s)&(t,s)\in E_{\varepsilon_{1}}\times
F_{\varepsilon_{2}},\\\ \end{cases}$ (125)

where
$E_{\varepsilon_{1}}:=[t^{\prime},t^{\prime}+\varepsilon_{1}]\subseteq[0,T]$,
$F_{\varepsilon_{2}}:=[s^{\prime},s^{\prime}+\varepsilon_{2}]\subseteq\mathbb{R}^{d_{s}}$,
and ${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}^{d_{u}}$. From
the Lemma 5, $J[u^{\varepsilon}]-J[u^{*}]$ can be calculated as follows:

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p^{\varepsilon},u^{\varepsilon},w^{*})-\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})-\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{*},u^{*},w^{*}\right)}{\delta
p}(s)\left(p^{\varepsilon}(t,s)-p^{*}(t,s)\right)ds\right)dt$ $\displaystyle\
\ \
+\bar{g}(p^{\varepsilon})-\bar{g}(p^{*})-\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)\left(p^{\varepsilon}(T,s)-p^{*}(T,s)\right)ds$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p^{\varepsilon},u^{*},w^{*})-\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})-\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{*},u^{*},w^{*}\right)}{\delta
p}(s)\left(p^{\varepsilon}(t,s)-p^{*}(t,s)\right)ds\right)dt$ $\displaystyle\
\ \
+\bar{g}(p^{\varepsilon})-\bar{g}(p^{*})-\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)\left(p^{\varepsilon}(T,s)-p^{*}(T,s)\right)ds$ $\displaystyle\ \ \
+\int_{E_{\varepsilon_{1}}}\int_{F_{\varepsilon_{2}}}\left(\mathcal{H}(t,s,p^{\varepsilon},u,w^{*})-\mathcal{H}(t,s,p^{\varepsilon},u^{*},w^{*})\right)p^{\varepsilon}(t,s)dsdt.$
(126)

Letting $\varepsilon_{1}\to 0$ and $\varepsilon_{2}\to 0$,

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{*},u^{*},w^{*}\right)}{\delta
p}(s)\left(p^{\varepsilon}(t,s)-p^{*}(t,s)\right)ds\right.$ $\displaystyle\ \
\
\left.-\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{*},u^{*},w^{*}\right)}{\delta
p}(s)\left(p^{\varepsilon}(t,s)-p^{*}(t,s)\right)ds\right)dt$ $\displaystyle\
\ \ +\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)\left(p^{\varepsilon}(T,s)-p^{*}(T,s)\right)ds-\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)\left(p^{\varepsilon}(T,s)-p^{*}(T,s)\right)ds$ $\displaystyle\ \ \
+\left(\mathcal{H}(t^{\prime},s^{\prime},p^{*},u,w^{*})-\mathcal{H}(t^{\prime},s^{\prime},p^{*},u^{*},w^{*})\right)p^{*}(t^{\prime},s^{\prime})dsdt$
$\displaystyle=\left(\mathcal{H}(t^{\prime},s^{\prime},p^{*},u,w^{*})-\mathcal{H}(t^{\prime},s^{\prime},p^{*},u^{*},w^{*})\right)p^{*}(t^{\prime},s^{\prime})dsdt.$
(127)

Because $u^{*}$ is the optimal control function, the following inequality is
satisfied:

$\displaystyle 0\leq J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\left(\mathcal{H}(t^{\prime},s^{\prime},p^{*},u,w^{*})-\mathcal{H}(t^{\prime},s^{\prime},p^{*},u^{*},w^{*})\right)p^{*}(t^{\prime},s^{\prime})dsdt.$
(128)

Therefore, (122) is obtained. ∎

###  APPENDIX II-D Sufficient condition

The Pontryagin’s minimum principle becomes a necessary and sufficient
condition if the expected Hamiltonian and the expected terminal cost function
are convex as follows:

###### Proposition 10

Assume that the expected Hamiltonian $\bar{\mathcal{H}}(t,p,u,w)$ is convex
with respect to $p$ and $u$ and the expected terminal cost function
$\bar{g}(p)$ is convex with respect to $p$. If the control function $u^{*}$
satisfies (122), it is the optimal control function of MFSC.

###### Proof:

We define the arbitrary control function
${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}^{d_{u}}$. From the
Lemma 5, $J[u]-J[u^{*}]$ is given by the following equation:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\bar{\mathcal{H}}(t,p,u,w^{*})-\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})-\int\frac{\delta\bar{\mathcal{H}}\left(t,p^{*},u^{*},w^{*}\right)}{\delta
p}(s)\left(p(t,s)-p^{*}(t,s)\right)ds\right)dt$ $\displaystyle\ \ \
+\bar{g}(p)-\bar{g}(p^{*})-\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)\left(p(T,s)-p^{*}(T,s)\right)ds.$ (129)

Because $\bar{\mathcal{H}}(t,p,u,w)$ is convex with respect to $p$ and $u$ and
$\bar{g}(p)$ is convex with respect to $p$, the following inequalities are
satisfied:

$\displaystyle\bar{\mathcal{H}}(t,p,u,w^{*})$
$\displaystyle\geq\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})+\int\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
p}(s)(p(t,s)-p^{*}(t,s))ds$ $\displaystyle\ \ \
+\int\left(\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
u}(s)\right)^{\top}(u(t,s)-u^{*}(t,s))ds,$ (130) $\displaystyle\bar{g}(p)$
$\displaystyle\geq\bar{g}(p^{*})+\int\frac{\delta\bar{g}(p^{*})}{\delta
p}(s)(p(T,s)-p^{*}(T,s))ds.$ (131)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle\geq\int_{0}^{T}\mathbb{E}_{p^{*}(t,s)}\left[\left(\frac{\partial\mathcal{H}(t,s,p^{*},u^{*},w^{*})}{\partial
u}\right)^{\top}(u(t,s)-u^{*}(t,s))\right]dt.$ (132)

Because $u^{*}$ satisfies (122), the following stationary condition is
satisfied:

$\displaystyle\frac{\partial\mathcal{H}(t,s,p^{*},u^{*},w^{*})}{\partial
u}=0.$ (133)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$ $\displaystyle\geq 0$ (134)

Therefore, $u^{*}$ is the optimal control function. ∎

###  APPENDIX II-E Relationship with the Bellman’s dynamic programming
principle

From the Bellman’s dynamic programming principle on the probability density
function space [49, 50, 51], the optimal control function of MFSC is given by
the following equation:

$\displaystyle u^{*}(t,s,p)=\mathop{\rm
argmin}\limits_{u}\mathcal{H}\left(t,s,p,u,\frac{\delta V^{*}(t,p)}{\delta
p}(s)\right).$ (135)

More specifically, the optimal control function is given by
$u^{*}(t,s)=u^{*}(t,s,p^{*})$, where $p^{*}(t,s)$ is the solution of the FP
equation (123). $V^{*}(t,p)$ is the value function on the probability density
function space, which is the solution of the following Bellman equation:

$\displaystyle-\frac{\partial V^{*}(t,p)}{\partial
t}=\mathbb{E}_{p(s)}\left[\mathcal{H}\left(t,s,p,u^{*},\frac{\delta
V^{*}(t,p)}{\delta p}(s)\right)\right],$ (136)

where $V^{*}(T,p)=\mathbb{E}_{p(s)}\left[g(s)\right]$. Because the Bellman
equation (136) is a functional differential equation, it cannot be solved even
numerically. In order to resolve this problem, the previous works [23, 24]
converted the Bellman equation (136) into the system of HJB-FP equation (123)
and (124) as follows:

######  Proposition 11 ([23, 24])

We define $w^{*}(t,s)$ from $V^{*}(t,p)$ as follows:

$\displaystyle w^{*}(t,s):=\frac{\delta V^{*}(t,p^{*})}{\delta p}(s),$ (137)

where $p^{*}(t,s)$ is the solution of the FP equation (123). $w^{*}(t,s)$
satisfies the HJB equation (124) from the Bellman equation (136).

###### Proof:

The proof is shown in [23, 24]. ∎

This approach can be interpreted as the conversion from the Bellman’s dynamic
programming principle into the Pontryagin’s minimum principle.

##  APPENDIX III PROOF

###  APPENDIX III-A Proof of Lemma 1

$J[u]-J[u^{\prime}]$ can be calculated as follows:

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\mathbb{E}_{p(s_{0:T})}\left[\int_{0}^{T}f(t,s_{t},u_{t})dt+g(s_{T})\right]-\mathbb{E}_{p^{\prime}(s_{0:T})}\left[\int_{0}^{T}f(t,s_{t},u_{t}^{\prime})dt+g(s_{T})\right]$
$\displaystyle=\mathbb{E}_{p(s_{0:T})}\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t},w^{\prime})-\mathcal{L}_{u_{t}}w^{\prime}(t,s_{t})\right)dt+g(s_{T})\right]$
$\displaystyle\ \ \
-\mathbb{E}_{p^{\prime}(s_{0:T})}\left[\int_{0}^{T}\left(\mathcal{H}(t,s_{t},u_{t}^{\prime},w^{\prime})-\mathcal{L}_{u_{t}^{\prime}}w^{\prime}(t,s_{t})\right)dt+g(s_{T})\right]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime})\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{H}(t,s,u^{\prime},w^{\prime})\right]\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{L}_{u}w^{\prime}(t,s)\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{L}_{u^{\prime}}w^{\prime}(t,s)\right]\right)dt+\mathbb{E}_{p(T,s)}\left[g(s)\right]-\mathbb{E}_{p^{\prime}(T,s)}\left[g(s)\right].$
(138)

Because $\mathcal{L}_{u_{t}}$ and $\mathcal{L}_{u_{t}^{\prime}}$ are the
conjugates of $\mathcal{L}_{u_{t}}^{{\dagger}}$ and
$\mathcal{L}_{u_{t}^{\prime}}^{{\dagger}}$, respectively,

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime})\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{H}(t,s,u^{\prime},w^{\prime})\right]\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\int\left(\mathcal{L}_{u}^{{\dagger}}p(t,s)-\mathcal{L}_{u^{\prime}}^{{\dagger}}p^{\prime}(t,s)\right)w^{\prime}(t,s)dsdt+\mathbb{E}_{p(T,s)}\left[g(s)\right]-\mathbb{E}_{p^{\prime}(T,s)}\left[g(s)\right].$
(139)

From the FP equation (24),

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime})\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{H}(t,s,u^{\prime},w^{\prime})\right]\right)dt$
$\displaystyle\ \ \
-\int_{0}^{T}\int\frac{\partial\left(p(t,s)-p^{\prime}(t,s)\right)}{\partial
t}w^{\prime}(t,s)dsdt+\mathbb{E}_{p(T,s)}\left[g(s)\right]-\mathbb{E}_{p^{\prime}(T,s)}\left[g(s)\right].$
(140)

From the integration by parts and
$p(0,s)-p^{\prime}(0,s)=p_{0}(s)-p_{0}(s)=0$,

$\displaystyle J[u]-J[u^{\prime}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime})\right]-\mathbb{E}_{p^{\prime}(t,s)}\left[\mathcal{H}(t,s,u^{\prime},w^{\prime})\right]\right)dt$
$\displaystyle\ \ \
+\int_{0}^{T}\int\left(p(t,s)-p^{\prime}(t,s)\right)\frac{\partial
w^{\prime}(t,s)}{\partial t}dsdt$ $\displaystyle\ \ \
+\mathbb{E}_{p(T,s)}\left[g(s)\right]-\mathbb{E}_{p^{\prime}(T,s)}\left[g(s)\right]-\int\left(p(T,s)-p^{\prime}(T,s)\right)w^{\prime}(T,s)ds.$
(141)

From the HJB equation (28), (27) is obtained.

###  APPENDIX III-B Proof of Theorem 1

We define the control function

$\displaystyle
u^{\varepsilon}(t,z):=\begin{cases}u^{*}(t,z)&(t,z)\in([0,T]\times\mathbb{R}^{d_{z}})\backslash(E_{\varepsilon_{1}}\times
F_{\varepsilon_{2}}),\\\ u(t,z)&(t,z)\in E_{\varepsilon_{1}}\times
F_{\varepsilon_{2}},\\\ \end{cases}$ (142)

where
$E_{\varepsilon_{1}}:=[t^{\prime},t^{\prime}+\varepsilon_{1}]\subseteq[0,T]$,
$F_{\varepsilon_{2}}:=[z^{\prime},z^{\prime}+\varepsilon_{2}]\subseteq\mathbb{R}^{d_{z}}$,
and ${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{u}}$. From
the Lemma 1, $J[u^{\varepsilon}]-J[u^{*}]$ can be calculated as follows:

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p^{\varepsilon}(t,s)}\left[\mathcal{H}(t,s,u^{\varepsilon},w^{*})\right]-\mathbb{E}_{p^{\varepsilon}(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]\right)dt$
$\displaystyle=\int_{E_{\varepsilon_{1}}}\int_{F_{\varepsilon_{2}}}\left(\mathbb{E}_{p_{t}^{\varepsilon}(x|z)}\left[\mathcal{H}(t,s,u,w^{*})\right]-\mathbb{E}_{p_{t}^{\varepsilon}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]\right)p_{t}^{\varepsilon}(z)dzdt.$
(143)

Letting $\varepsilon_{1}\to 0$ and $\varepsilon_{2}\to 0$,

$\displaystyle J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\left(\mathbb{E}_{p_{t^{\prime}}^{*}(x^{\prime}|z^{\prime})}\left[\mathcal{H}(t^{\prime},s^{\prime},u,w^{*})\right]-\mathbb{E}_{p_{t^{\prime}}^{*}(x^{\prime}|z^{\prime})}\left[\mathcal{H}(t^{\prime},s^{\prime},u^{*},w^{*})\right]\right)p_{t^{\prime}}^{*}(z^{\prime})dzdt.$
(144)

Because $u^{*}$ is the optimal control function, the following inequality is
satisfied:

$\displaystyle 0\leq J[u^{\varepsilon}]-J[u^{*}]$
$\displaystyle=\left(\mathbb{E}_{p_{t^{\prime}}^{*}(x^{\prime}|z^{\prime})}\left[\mathcal{H}(t^{\prime},s^{\prime},u,w^{*})\right]-\mathbb{E}_{p_{t^{\prime}}^{*}(x^{\prime}|z^{\prime})}\left[\mathcal{H}(t^{\prime},s^{\prime},u^{*},w^{*})\right]\right)p_{t^{\prime}}^{*}(z^{\prime})dzdt.$
(145)

Therefore, (29) is obtained.

###  APPENDIX III-C Proof of Proposition 1

We define the arbitrary control function
${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{u}}$. From the
Lemma 1, $J[u]-J[u^{*}]$ is given by the following equation:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{*})\right]-\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]\right)dt.$
(146)

Because $\bar{\mathcal{H}}(t,p,u,w)$ is convex with respect to $p$ and $u$,
the following inequality is satisfied:

$\displaystyle\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{*})\right]$
$\displaystyle=\bar{\mathcal{H}}(t,p,u,w^{*})$
$\displaystyle\geq\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})+\int\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
p}(s)(p(t,s)-p^{*}(t,s))ds$ $\displaystyle\ \ \
+\int\left(\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
u}(z)\right)^{\top}(u(t,z)-u^{*}(t,z))dz.$ (147)

Because

$\displaystyle\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
p}(s)$ $\displaystyle=\left.\frac{\delta}{\delta p}\left(\int
p(s)\mathcal{H}(t,s,u^{*},w^{*})ds\right)\right|_{p=p^{*}}=\mathcal{H}(t,s,u^{*},w^{*}),$
(148) $\displaystyle\frac{\delta\bar{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta
u}(z)$ $\displaystyle=\left.\frac{\delta}{\delta u}\left(\int
p_{t}^{*}(z)\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u,w^{*})\right]dz\right)\right|_{u=u^{*}}=p_{t}^{*}(z)\frac{\partial\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]}{\partial
u},$ (149)

the above inequality can be calculated as follows:

$\displaystyle\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{*})\right]$
$\displaystyle\geq\int
p^{*}(t,s)\mathcal{H}(t,s,u^{*},w^{*})ds+\int\mathcal{H}(t,s,u^{*},w^{*})(p(t,s)-p^{*}(t,s))ds$
$\displaystyle\ \ \ +\int
p_{t}^{*}(z)\left(\frac{\partial\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]}{\partial
u}\right)^{\top}(u(t,z)-u^{*}(t,z))dz$
$\displaystyle=\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]+\mathbb{E}_{p_{t}^{*}(z)}\left[\left(\frac{\partial\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]}{\partial
u}\right)^{\top}(u(t,z)-u^{*}(t,z))\right].$ (150)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle\geq\int_{0}^{T}\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]+\mathbb{E}_{p_{t}^{*}(z)}\left[\left(\frac{\partial\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]}{\partial
u}\right)^{\top}(u(t,z)-u^{*}(t,z))\right]dt.$ (151)

Because $u^{*}$ satisfies (29), the following stationary condition is
satisfied:

$\displaystyle\frac{\partial\mathbb{E}_{p_{t}^{*}(x|z)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]}{\partial
u}=0.$ (152)

Hence, the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$ $\displaystyle\geq 0$ (153)

Therefore, $u^{*}$ is the optimal control function.

###  APPENDIX III-D Proof of Proposition 4

We define the arbitrary control function
${}^{\forall}u:[0,T]\times\mathbb{R}^{d_{s}}\to\mathbb{R}^{d_{u}}$. From the
Lemma 1, $J[u]-J[u^{*}]$ is given by the following equation:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle=\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{*})\right]-\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]\right)dt.$
(154)

From (38), the following inequality is satisfied:

$\displaystyle J[u]-J[u^{*}]$
$\displaystyle\geq\int_{0}^{T}\left(\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]-\mathbb{E}_{p(t,s)}\left[\mathcal{H}(t,s,u^{*},w^{*})\right]\right)dt=0.$
(155)

Therefore, $u^{*}$ is the optimal control function.

###  APPENDIX III-E Proof of Lemma 2 by the time discretized method

$u_{t}^{\prime}:=\mathop{\rm argmin}\limits_{u_{t}}J[u_{0:T-dt}]$ can be
calculated as follows:

$\displaystyle u_{t}^{\prime}$ $\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}J[u_{0:T-dt}]$ $\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p(s_{0:T};u_{0:T-dt})}\left[\int_{0}^{T}f(\tau,s_{\tau},u_{\tau})d\tau+g(s_{T})\right]$
$\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p(s_{t:T};u_{0:T-dt})}\left[\int_{t}^{T}f(\tau,s_{\tau},u_{\tau})d\tau+g(s_{T})\right]$
$\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p(s_{t:T};u_{0:T-dt})}\left[f(t,s_{t},u_{t})dt+\int_{t+dt}^{T}f(\tau,s_{\tau},u_{\tau})d\tau+g(s_{T})\right]$
$\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}(s_{t})}\left[f(t,s_{t},u_{t})dt+\mathbb{E}_{p(s_{t+dt}|s_{t};u_{t})}\left[w_{t+dt}(s_{t+dt})\right]\right].$
(156)

where $p_{t}(s)$ is the solution of the FP equation (40), and $w_{t+dt}(s)$ is
defined by

$\displaystyle
w_{t+dt}(s):=\mathbb{E}_{p(s_{t+2dt:T}|s_{t+dt}=s;u_{t+dt:T-dt})}\left[\int_{t+dt}^{T}f(\tau,s_{\tau},u_{\tau})d\tau+g(s_{T})\right].$
(157)

Although $p_{t}(s)$ and $w_{t+dt}(s)$ depend on $u_{0:t-dt}$ and
$u_{t+dt:T-dt}$, respectively, they do not depend on $u_{t}$ in ML-POSC. From
the Ito’s lemma,

$\displaystyle u_{t}^{\prime}$ $\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}(s_{t})}\left[f(t,s_{t},u_{t})dt+w_{t+dt}(s_{t})+\mathcal{L}_{u_{t}}w_{t+dt}(s_{t})dt\right]$
$\displaystyle=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}(s)}\left[\mathcal{H}(t,s,u_{t},w_{t+dt})\right].$
(158)

Because the control $u_{t}$ is the function of the memory $z$ in ML-POSC, the
minimization by $u_{t}$ can be exchanged with the expectation by $p_{t}(z)$ as
follows:

$\displaystyle u_{t}^{\prime}(z)=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}(x|z)}\left[\mathcal{H}\left(t,s,u_{t},w_{t+dt}\right)\right]=u_{t}^{*}(z).$
(159)

Therefore, (42) is proved. Finally, we prove that $w_{t}(s)$ is the solution
of the HJB equation (41). $w_{t}(s)$ can be calculated as follows:

$\displaystyle w_{t}(s)$
$\displaystyle=\mathbb{E}_{p(s_{t+dt:T}|s_{t}=s;u_{t:T-dt})}\left[\int_{t}^{T}f(\tau,s_{\tau},u_{\tau})d\tau+g(s_{T})\right]$
$\displaystyle=f(t,s,u_{t})dt+\mathbb{E}_{p(s_{t+dt}|s_{t}=s;u_{t})}\left[w_{t+dt}(s_{t+dt})\right]$
$\displaystyle=f(t,s,u_{t})dt+w_{t+dt}(s)+\mathcal{L}_{u_{t}}w_{t+dt}(s)dt$
$\displaystyle=w_{t+dt}(s)+\mathcal{H}(t,s,u_{t},w_{t+dt})dt,$ (160)

where $w_{T}(s)=g(s)$. Therefore, (41) is proved.

###  APPENDIX III-F Proof of Lemma 2 by the similar way as the Pontyragin’s
minimum principle

From the Lemma 1, the following equality is satisfied:

$\displaystyle
J[u_{0:t-dt},u_{t},u_{t+dt:T-dt}]-J[u_{0:t-dt},u_{t}^{*},u_{t+dt:T-dt}]$
$\displaystyle=\left(\mathbb{E}_{p_{t}(s)}\left[\mathcal{H}(t,s,u_{t},w_{t+dt})\right]-\mathbb{E}_{p_{t}(s)}\left[\mathcal{H}(t,s,u_{t}^{*},w_{t+dt})\right]\right)dt$
$\displaystyle=\mathbb{E}_{p_{t}(z)}\left[\mathbb{E}_{p_{t}(x|z)}\left[\mathcal{H}(t,s,u_{t},w_{t+dt})\right]-\mathbb{E}_{p_{t}(x|z)}\left[\mathcal{H}(t,s,u_{t}^{*},w_{t+dt})\right]\right]dt.$
(161)

From (39), the following inequality is satisfied.

$\displaystyle
J[u_{0:t-dt},u_{t},u_{t+dt:T-dt}]-J[u_{0:t-dt},u_{t}^{*},u_{t+dt:T-dt}]$
$\displaystyle\geq 0.$ (162)

Therefore, $u_{t}^{*}$ satisfies (42).

###  APPENDIX III-G Proof of Lemma 3

We mainly prove the inequality of the forward step (44). We can prove the
inequality of the backward step (43) in a similar way. In the forward step,
$u_{0:t-dt}^{k+1}$ and $u_{t+dt:T-dt}^{k}$ are given, and $u_{t}^{k+1}$ is
defined by $u_{t}^{k+1}:=\mathop{\rm
argmin}\limits_{u_{t}}\mathbb{E}_{p_{t}^{k+1}(x|z)}\left[\mathcal{H}\left(t,s,u_{t},w_{t+dt}^{k}\right)\right]$.
In this case, from Lemma 2, the following inequality is satisfied:

$\displaystyle J[u_{0:t-dt}^{k+1},u_{t}^{k},u_{t+dt:T-dt}^{k}]\geq
J[u_{0:t-dt}^{k+1},u_{t}^{k+1},u_{t+dt:T-dt}^{k}].$ (163)

Therefore, the inequality of the forward step (44) is proved.

###  APPENDIX III-H Proof of Theorem 3

We mainly consider the forward step. The similar discussion is possible for
the backward step. If $J[u_{0:T-dt}^{k+1}]=J[u_{0:T-dt}^{k}]$ holds, then
$J[u_{0:t}^{k+1},u_{t+dt:T-dt}^{k}]=J[u_{0:t-dt}^{k+1},u_{t:T-dt}^{k}]$ holds
from Lemma 3. Because $J[u_{0}^{k+1},u_{dt:T-dt}^{k}]=J[u_{0:T-dt}^{k}]$
holds, $u_{0}^{k+1}=u_{0}^{k}$ holds. Then, because
$J[u_{0}^{k},u_{dt}^{k+1},u_{2dt:T-dt}^{k}]=J[u_{0:T-dt}^{k}]$ holds,
$u_{dt}^{k+1}=u_{dt}^{k}$ holds. Iterating this procedure from $t=0$ to
$t=T-dt$, $u_{0:T-dt}^{k+1}=u_{0:T-dt}^{k}$ holds. Therefore, because the HJB
equation and the FP equation depend on the same control function
$u_{0:T-dt}^{k+1}=u_{0:T-dt}^{k}$, $u_{0:T-dt}^{k+1}$ satisfies the
Pontryagin’s minimum principle (Theorem 1), which is the necessary condition
of the optimal control function.

###  APPENDIX III-I Proof of Proposition 6

We firstly consider the initial step. When the control function is initialized
by (60), the solution of the FP equation is given by the Gaussian distribution
$p_{t}^{0}(s):=\mathcal{N}\left(s\left|\mu,\Lambda^{0}\right.\right)$, where
$\mu$ is the solution of (54) and $\Lambda^{0}$ is the solution of
$\dot{\Lambda}^{0}=\mathcal{F}(\Lambda^{0},\Pi^{0})$ given
$\Lambda^{0}(0)=\Lambda_{0}$.

We then consider the backward step. In the LQG problem of ML-POSC, the
solution of the HJB equation is given by the quadratic function
$w_{t}^{k+1}(s)=s^{\top}\Pi^{k+1}s+(\alpha^{k+1})^{\top}s+\beta^{k+1}$, where
$\Pi^{k+1}$, $\alpha^{k+1}$, and $\beta^{k+1}$ are the solutions of the
following ODEs:

$\displaystyle-\dot{\Pi}^{k+1}$
$\displaystyle=\mathcal{G}(\Lambda^{k},\Pi^{k+1}),$ (164)
$\displaystyle-\dot{\alpha}^{k+1}$
$\displaystyle=(A-BR^{-1}B^{\top}\Pi^{k+1})^{\top}\alpha^{k+1}-2(I-K(\Lambda^{k}))^{\top}\Pi^{k+1}BR^{-1}B^{\top}\Pi^{k+1}(I-K(\Lambda^{k}))\mu,$
(165) $\displaystyle-\dot{\beta}^{k+1}$
$\displaystyle=\tr\left(\Pi^{k+1}\sigma\sigma^{\top}\right)-\frac{1}{4}(\alpha^{k+1})^{\top}BR^{-1}B^{\top}\alpha^{k+1}+\mu^{\top}(I-K(\Lambda^{k}))^{\top}\Pi^{k+1}BR^{-1}B^{\top}\Pi^{k+1}(I-K(\Lambda^{k}))\mu,$
(166)

where $\Pi^{k+1}(T)=P$, $\alpha^{k+1}(T)=0$, and $\beta^{k+1}(T)=0$. The
control function $u_{t}^{k+1}(z)$ is given by the following equation:

$\displaystyle
u_{t}^{k+1}(z)=-R^{-1}B^{\top}\left(\Pi^{k+1}K(\Lambda^{k})(s-\mu)+\Psi\mu\right).$
(167)

The details of these calculations are almost the same with [11].

We finally consider the forward step. When the control function
$u_{t}^{k+1}(z)$ is given by (167), the solution of the FP equation is given
by the Gaussian distribution
$p_{t}^{k+2}(s):=\mathcal{N}\left(s\left|\mu,\Lambda^{k+2}\right.\right)$,
where $\mu$ is the solution of (54) and $\Lambda^{k+2}$ is the solution of
$\dot{\Lambda}^{k+2}=\mathcal{F}(\Lambda^{k+2},\Pi^{k+1})$ given
$\Lambda^{k+2}(0)=\Lambda_{0}$. Therefore, FBSM is reduced from Algorithm 1 to
Algorithm 2 in the LQG problem.

## ACKNOWLEDGMENT

The first author received a JSPS Research Fellowship (Grant No. 21J20436).
This work was supported by JSPS KAKENHI (Grant No. 19H05799) and JST CREST
(Grant No. JPMJCR2011).

## References

  * [1] R. Fox and N. Tishby, “Minimum-information LQG control Part II: Retentive controllers,” in 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 5603–5609, Dec. 2016.
  * [2] R. Fox and N. Tishby, “Minimum-information LQG control part I: Memoryless controllers,” in 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 5610–5616, Dec. 2016.
  * [3] W. Li and E. Todorov, “An Iterative Optimal Control and Estimation Design for Nonlinear Stochastic System,” in Proceedings of the 45th IEEE Conference on Decision and Control, pp. 3242–3247, Dec. 2006\.  ISSN: 0191-2216.
  * [4] W. Li and E. Todorov, “Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic system,” International Journal of Control, vol. 80, pp. 1439–1453, Sept. 2007.
  * [5] T. J. Kobayashi, “Implementation of Dynamic Bayesian Decision Making by Intracellular Kinetics,” Physical Review Letters, vol. 104, p. 228104, June 2010.  Publisher: American Physical Society.
  * [6] T. J. Kobayashi and A. Kamimura, “Dynamics of intracellular information decoding,” Physical Biology, vol. 8, p. 055007, Aug. 2011.  Publisher: IOP Publishing.
  * [7] K. Nakamura and T. J. Kobayashi, “Connection between the Bacterial Chemotactic Network and Optimal Filtering,” Physical Review Letters, vol. 126, p. 128102, Mar. 2021.  Publisher: American Physical Society.
  * [8] K. Nakamura and T. J. Kobayashi, “Optimal sensing and control of run-and-tumble chemotaxis,” Physical Review Research, vol. 4, p. 013120, Feb. 2022.  Publisher: American Physical Society.
  * [9] A. Pezzotta, M. Adorisio, and A. Celani, “Chemotaxis emerges as the optimal solution to cooperative search games,” Physical Review E, vol. 98, p. 042401, Oct. 2018.  Publisher: American Physical Society.
  * [10] F. Borra, M. Cencini, and A. Celani, “Optimal collision avoidance in swarms of active Brownian particles,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2021, p. 083401, Aug. 2021.  Publisher: IOP Publishing.
  * [11] T. Tottori and T. J. Kobayashi, “Memory-Limited Partially Observable Stochastic Control and its Mean-Field Control Approach,” Sept. 2022\.  arXiv:2203.10682 [math].
  * [12] J. Yong and X. Y. Zhou, Stochastic Controls.  New York, NY: Springer New York, 1999.
  * [13] H. J. Kushner and P. G. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time.  New York, NY: Springer US, 1992.
  * [14] W. H. Fleming and H. M. Soner, Controlled Markov processes and viscosity solutions.  No. 25 in Applications of mathematics, New York: Springer, 2nd ed ed., 2006.
  * [15] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming.  Wiley-Interscience, Aug. 2014.
  * [16] R. Vinter, Optimal Control.  Boston: BirkhÃ¤user Boston, 2010.
  * [17] F. L. Lewis, D. Vrabie, and V. L. Syrmos, Optimal Control.  John Wiley & Sons, Mar. 2012.  Google-Books-ID: U3Gtlot_hYEC.
  * [18] L. T. Aschepkov, D. V. Dolgy, T. Kim, and R. P. Agarwal, Optimal Control.  Cham: Springer International Publishing, 2016.
  * [19] A. Bensoussan, J. Frehse, and P. Yam, Mean Field Games and Mean Field Type Control Theory.  Springer Briefs in Mathematics, New York, NY: Springer New York, 2013\.
  * [20] R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications I.  No. volume 83 in Probability theory and stochastic modelling, Cham: Springer Nature, 2018.
  * [21] R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications II, vol. 84 of Probability Theory and Stochastic Modelling.  Cham: Springer International Publishing, 2018.
  * [22] R. Carmona and F. Delarue, “The Master Equation for Large Population Equilibriums,” in Stochastic Analysis and Applications 2014 (D. Crisan, B. Hambly, and T. Zariphopoulou, eds.), vol. 100, pp. 77–128, Cham: Springer International Publishing, 2014.  Series Title: Springer Proceedings in Mathematics & Statistics.
  * [23] A. Bensoussan, J. Frehse, and S. C. P. Yam, “The Master equation in mean field theory,” Journal de MathÃ©matiques Pures et AppliquÃ©es, vol. 103, pp. 1441–1474, June 2015.
  * [24] A. Bensoussan, J. Frehse, and S. C. P. Yam, “On the interpretation of the Master Equation,” Stochastic Processes and their Applications, vol. 127, pp. 2093–2137, July 2017.
  * [25] I. Krylov and F. Chernous’ko, “On a method of successive approximations for the solution of problems of optimal control,” USSR Computational Mathematics and Mathematical Physics, vol. 2, pp. 1371–1382, Jan. 1963.
  * [26] S. K. Mitter, “Successive approximation methods for the solution of optimal control problems,” Automatica, vol. 3, pp. 135–149, Jan. 1966.
  * [27] F. L. Chernousko and A. A. Lyubushin, “Method of successive approximations for solution of optimal control problems,” Optimal Control Applications and Methods, vol. 3, no. 2, pp. 101–114, 1982.
  * [28] S. Lenhart and J. T. Workman, Optimal Control Applied to Biological Models.  New York: Chapman and Hall/CRC, May 2007.
  * [29] J. A. Sharp, K. Burrage, and M. J. Simpson, “Implementation and acceleration of optimal control for systems biology,” Journal of The Royal Society Interface, vol. 18, no. 181, p. 20210241, 2021.  Publisher: Royal Society.
  * [30] W. Hackbusch, “A numerical method for solving parabolic equations with opposite orientations,” Computing, vol. 20, pp. 229–240, Sept. 1978.
  * [31] M. McAsey, L. Mou, and W. Han, “Convergence of the forward-backward sweep method in optimal control,” Computational Optimization and Applications, vol. 53, pp. 207–226, Sept. 2012.
  * [32] E. Carlini and F. J. Silva, “Semi-Lagrangian schemes for mean field game models,” in 52nd IEEE Conference on Decision and Control, pp. 3115–3120, Dec. 2013.  ISSN: 0191-2216.
  * [33] E. Carlini and F. J. Silva, “A Fully Discrete Semi-Lagrangian Scheme for a First Order Mean Field Game Problem,” SIAM Journal on Numerical Analysis, vol. 52, pp. 45–67, Jan. 2014.  Publisher: Society for Industrial and Applied Mathematics.
  * [34] E. Carlini and F. J. Silva, “A semi-Lagrangian scheme for a degenerate second order mean field game system,” Discrete & Continuous Dynamical Systems, vol. 35, no. 9, p. 4269, 2015.  Company: Discrete & Continuous Dynamical Systems Distributor: Discrete & Continuous Dynamical Systems Institution: Discrete & Continuous Dynamical Systems Label: Discrete & Continuous Dynamical Systems Publisher: American Institute of Mathematical Sciences.
  * [35] M. Lauriere, “Numerical Methods for Mean Field Games and Mean Field Type Control,” June 2021.  arXiv:2106.06231 [cs, math].
  * [36] A. Bensoussan, Estimation and Control of Dynamical Systems, vol. 48 of Interdisciplinary Applied Mathematics.  Cham: Springer International Publishing, 2018.
  * [37] Q. Li, L. Chen, C. Tai, and W. E, “Maximum Principle Based Algorithms for Deep Learning,” Journal of Machine Learning Research, vol. 18, no. 165, pp. 1–29, 2018.
  * [38] X. Liu and J. Frank, “Symplectic RungeâKutta discretization of a regularized forwardâbackward sweep iteration for optimal control problems,” Journal of Computational and Applied Mathematics, vol. 383, p. 113133, Feb. 2021.
  * [39] R. E. Bellman, Dynamic Programming.  Princeton, USA: Princeton University Press, 1957.
  * [40] R. A. Howard, Dynamic programming and Markov processes.  Dynamic programming and Markov processes, Oxford, England: John Wiley, 1960.  Pages: viii, 136.
  * [41] H. J. Kappen, “Linear Theory for Control of Nonlinear Stochastic Systems,” Physical Review Letters, vol. 95, p. 200201, Nov. 2005.  Publisher: American Physical Society.
  * [42] H. J. Kappen, “Path integrals and symmetry breaking for optimal control theory,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2005, pp. P11011–P11011, Nov. 2005.  Publisher: IOP Publishing.
  * [43] S. Satoh, H. J. Kappen, and M. Saeki, “An Iterative Method for Nonlinear Stochastic Optimal Control Based on Path Integrals,” IEEE Transactions on Automatic Control, vol. 62, pp. 262–276, Jan. 2017.  Conference Name: IEEE Transactions on Automatic Control.
  * [44] S. Cacace, F. Camilli, and A. Goffi, “A policy iteration method for Mean Field Games,” arXiv:2007.04818 [math], July 2021.  arXiv: 2007.04818.
  * [45] M. LauriÃ¨re, J. Song, and Q. Tang, “Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians,” arXiv:2110.02552 [cs, math], Oct. 2021.  arXiv: 2110.02552.
  * [46] F. Camilli and Q. Tang, “Rates of convergence for the policy iteration method for Mean Field Games systems,” Mar. 2022.  arXiv:2108.00755 [math].
  * [47] L. Ruthotto, S. J. Osher, W. Li, L. Nurbekyan, and S. W. Fung, “A machine learning framework for solving high-dimensional mean field game and mean field control problems,” Proceedings of the National Academy of Sciences, vol. 117, pp. 9183–9193, Apr. 2020.
  * [48] A. T. Lin, S. W. Fung, W. Li, L. Nurbekyan, and S. J. Osher, “Alternating the population and control neural networks to solve high-dimensional stochastic mean-field games,” Proceedings of the National Academy of Sciences, vol. 118, Aug. 2021.
  * [49] M. LauriÃ¨re and O. Pironneau, “Dynamic programming for mean-field type control,” Comptes Rendus Mathematique, vol. 352, pp. 707–713, Sept. 2014\.
  * [50] M. LauriÃ¨re and O. Pironneau, “Dynamic Programming for Mean-Field Type Control,” Journal of Optimization Theory and Applications, vol. 169, pp. 902–924, June 2016.
  * [51] H. Pham and X. Wei, “Bellman equation and viscosity solutions for mean-field stochastic control problem,” ESAIM: Control, Optimisation and Calculus of Variations, vol. 24, pp. 437–461, Jan. 2018.