# A Linear Quadratic Stochastic Stackelberg Differential Game with Time Delay
††thanks: This work is financially supported by the National Key R&D Program
of China (2018YFB1305400), and the National Natural Science Foundations of
China (11971266, 11571205, 11831010).

Weijun Meng, Jingtao Shi School of Mathematics, Shandong University, Jinan
250100, P.R. China, E-mail<EMAIL_ADDRESS>author, School
of Mathematics, Shandong University, Jinan 250100, P.R. China, E-mail:
<EMAIL_ADDRESS>
Abstract: This paper is concerned with a linear quadratic stochastic
Stackelberg differential game with time delay. The model is general, in which
the state delay and the control delay both appear in the state equation,
moreover, they both enter into the diffusion term. By introducing two Pseudo-
Riccati equations and a special matrix equation, the state feedback
representation of the open-loop Stackelberg strategy is derived, under some
assumptions. Finally, two examples are given to illustrate the applications of
the theoretical results.

Keywords: time delay, stochastic Stackelberg differential game, leader and
follower, linear quadratic control, Riccati equation, open-loop Stackelberg
strategy

Mathematics Subject Classification: 93E20, 60H10, 49N10, 91A15, 91A65, 34K35,
34K50

##  1 Introduction

Stackelberg differential games have been investigated in order to obtain
optimal strategies in competitive economics in which there exist leaders and
followers. Stackelberg strategies are rational solutions when there are two
players and the leader knows the rational reaction of the follower and reveals
first his strategy, while the follower does not know the rational reaction of
the leader and has to optimize his criterion for a given control of the
leader. H. von Stackelberg [27] first introduced a hierarchical solution for
markets with leaders and followers in 1934, which is now known as the
Stackelberg equilibrium. The notion of the Stackelberg solution was later
extended to a multiperiod setting, see Simaan and Cruz [26]. Among these, for
the deterministic dynamics, the Stackelberg strategy was studied in
Papavassilopoulos and Cruz [20], Pan and Yong [19], Başar and Olsder [3],
Freiling et al. [9]. And for the stochastic dynamics when random noises are
considered, the Stackelberg strategy was considered in Castanon and Athans
[7], Başar [2], Yong [30], Bensoussan et al. [6], Shi et al. [23, 24, 25] and
the references therein.

In the above results, the states of the controlled systems only depend on the
value of the current time. However, the development of some random phenomena
in the real world depends not only on their current value, but also on their
past history. So far there have been extensive literature to study the
stochastic optimal control problems with time delay, see Mohammed [15, 16],
Øksendal and Sulem [18], Chen and Wu [8], Huang et al. [11], Zhang and Xu [31]
and the references therein. Hence it is of great importance to consider time
delay in the dynamics of Stackelberg differential games. There exist a few
literatures in this topic and let us mention some of them. Harband [10]
studied the existence of monotonic solutions of a nonlinear car-following
equation. Ishida and Shimemura [12] derived the sufficient conditions for
constructing a team-optimal closed-loop Stackelberg strategies in a linear
quadratic differential game with time delay. Ishida and Shimemura [13]
researched necessary and sufficient conditions for the open-loop Stackelberg
strategies in a linear quadratic differential game with time delay, where the
evolution of the game with time delay is described by coupled differential
equations composed of lumped and distributed parameter subsystems. Øksendal et
al. [17] considered explicit formulae for equilibrium prices in a continuous-
time vertical contracting model, in which the information is delayed.
Bensoussan et al. [5] investigated an $N$-player interacting strategic game in
the presence of a (endogenous) dominating player, who gives direct influence
on individual agents, through its impact on their control in the sense of
Stackelberg game, and then on the whole community. But only the leader’s state
delay appears in the model. Xu and Zhang [29] focused on leader-follower
differential games with time delay and overcame the non-causality difficulty
of strategy design caused by the delay by introducing the new co-states which
capture the future information of the control and the new state which contains
the past effects, however the system is deterministic. Bensoussan et al. [4]
was concerned with a linear-quadratic mean-field game between a leader and a
group of followers, but no delay appears in the diffusion term. Xu et al. [28]
obtained the open-loop Stackelberg strategy for the linear quadratic leader-
follower stochastic differential game with time delay only appearing in the
leader’s problem. Li et al. [14] considered a Stackelberg strategy for
deterministic mixed $H_{2}/H_{\infty}$ control problem with time delay in the
leader’s control. Bai et al. [1] addressed a Stackelberg stochastic
differential reinsurance investment game problem and derived the equilibrium
strategy explicitly for the game, however the diffusion term of the follower
does not contain the time delay.

As mentioned above, in the above literatures, either the delays do not both
appear in the state equations of leader and follower, or the state delay and
the control delay do not both enter the diffusion term of the state equation.
This paper aims to propose a general model, which can cover all the above
cases, and gives the state feedback representation of an open-loop Stackelberg
strategy to such a differential game. In summary, the contributions and
innovations of this paper are as follows.

  * •

Our model is general. The state equations of leader and follower both contain
state delay and control delay, moreover, which both enter into the diffusion
terms.

  * •

Our conclusions are simple. The open-loop stackelberg strategies are derived
via two Pseudo-Riccati equations (see (3.5), (4.4)) and a matrix equation
(4.10), which are easier to solve than the Riccati equations (6), (19) and the
matrix equation (25) in [28].

  * •

Our theory is efficient. Two examples are addressed applying the above
theoretical results.

The paper is organized as follows. Section 2 formulates the linear quadratic
stochastic Stackelberg differential game with time delay. The optimization
problem of the follower is discussed in Section 3, and then the optimization
problem of the leader is addressed in Section 4. To illustrate the
applications of the theoretical results, two examples are considered in
Section 5. Finally Section 6 gives some concluding remarks.

##  2 Problem statement

We first introduce some notations which will be used in this paper and then
state the problem.

Throughout the paper, $\mathbf{R}^{n\times m}$ is the Euclidean space of all
$n\times m$ real matrices, $\mathbf{S}^{n}$ is the space of all $n\times n$
symmetric matrices. We simply write $\mathbf{R}^{n\times m}$ as
$\mathbf{R}^{n}$ when $m=1$. The norm in $\mathbf{R}^{n}$ is denoted by
$|\cdot|$ and the inner product is denoted by $\langle\cdot,\cdot\rangle$. The
transpose of vectors or matrices is denoted by the superscript ⊤. $I$ is the
identity matrix with appropriate dimension.

Let $T>0$ be fixed and $\delta>0$ be the time delay, suppose
$(\Omega,\mathcal{F},\mathbb{P})$ is a complete probability space,
$\\{W(t)\\}_{t\geq 0}$ is a one-dimensional Brownian motion, and we define the
filtration $\\{\mathcal{F}_{t}\\}_{t\geq 0}=\sigma\\{W(r);0\leq r\leq t\\}$.
$\mathbb{E}[\cdot]$ denotes the expectation under the probability measure
$\mathbb{P}$ and
$\mathbb{E}^{\mathcal{F}_{t}}[\cdot]:=\mathbb{E}[\cdot|\mathcal{F}_{t}]$
denotes the conditional expectation, for any $t\in(0,T]$.

We then introduce some spaces. For a positive integer $p$, we define

$\displaystyle\begin{aligned} &L^{p}([0,T];\mathbf{R}^{n\times
n}):=\bigg{\\{}\mathbf{R}^{n\times n}\mbox{-valued funciton }\phi(t);\
\int_{0}^{T}|\phi(t)|^{p}dt<\infty\bigg{\\}},\\\
&L^{\infty}([0,T];\mathbf{R}^{n\times n}):=\bigg{\\{}\mathbf{R}^{n\times
n}\mbox{-valued funciton }\phi(t);\ \sup\limits_{0\leq t\leq
T}|\phi(t)|dt<\infty\bigg{\\}},\\\
&C([0,T];\mathbf{R}^{n}):=\bigg{\\{}\mathbf{R}^{n}\mbox{-valued continuous
funciton }\phi(t);\ \sup\limits_{0\leq t\leq T}|\phi(t)|<\infty\bigg{\\}},\\\
&L^{2}_{\mathcal{F}}([0,T];\mathbf{R}^{n}):=\bigg{\\{}\mathbf{R}^{n}\mbox{-valued
}\mathcal{F}_{t}\mbox{-adapted process }\phi(t);\
\mathbb{E}\int_{0}^{T}|\phi(t)|^{2}dt<\infty\bigg{\\}},\\\
&L^{2}_{\mathcal{F}}(\Omega;C([0,T];\mathbf{R}^{n})):=\bigg{\\{}\mathbf{R}^{n}\mbox{-valued
}\mathcal{F}_{t}\mbox{-adapted process }\phi(t);\
\mathbb{E}\Big{[}\sup\limits_{0\leq t\leq
T}|\phi(t)|^{2}\Big{]}<\infty\bigg{\\}}.\end{aligned}$

Consider the following linear controlled system with time delay, which is a
stochastic differential delayed equation (in short, SDDE):

$\left\\{\begin{aligned}
dX^{u_{1},u_{2}}(t)&=\big{[}A(t)X^{u_{1},u_{2}}(t)+\bar{A}(t)X^{u_{1},u_{2}}(t-\delta)+\bar{B}_{1}(t)u_{1}(t-\delta)+\bar{B}_{2}(t)u_{2}(t-\delta)\big{]}dt\\\
&\quad+\big{[}C(t)X^{u_{1},u_{2}}(t)+\bar{C}(t)X^{u_{1},u_{2}}(t-\delta)+\bar{D}_{1}(t)u_{1}(t-\delta)+\bar{D}_{2}(t)u_{2}(t-\delta)\big{]}dW(t),\\\
&\hskip 284.52756ptt\in[0,T],\\\ X^{u_{1},u_{2}}(t)&=\varphi(t),\
u_{1}(t)=\eta_{1}(t),\ u_{2}(t)=\eta_{2}(t),\quad
t\in[-\delta,0],\end{aligned}\right.$ (2.1)

and the cost functionals for the leader and the follower are as follows,
respectively:

$\displaystyle
J_{i}(u_{1}(\cdot),u_{2}(\cdot))=\mathbb{E}\bigg{[}\int_{0}^{T}\big{[}X^{u_{1},u_{2}}(t)^{\top}Q_{i}(t)X^{u_{1},u_{2}}(t)+X^{u_{1},u_{2}}(t-\delta)^{\top}\bar{Q}_{i}(t)X^{u_{1},u_{2}}(t-\delta)$
(2.2)
$\displaystyle\quad+u_{i}(t)^{\top}R_{i}(t)u_{i}(t)+u_{i}(t-\delta)^{\top}\bar{R}_{i}(t)u_{i}(t-\delta)\big{]}dt+X^{u_{1},u_{2}}(T)^{\top}G_{i}X^{u_{1},u_{2}}(T)\bigg{]},\quad
i=1,2.$

In the above, $X(\cdot)\in\textbf{R}^{n}$ is the state process,
$u_{1}(\cdot)\in\textbf{R}^{k_{1}}$ and $u_{2}(\cdot)\in\textbf{R}^{k_{2}}$
are the control processes of the follower and the leader, respectively.
$\varphi(\cdot)\in C([-\delta,0];\textbf{R}^{n})$ is the initial trajectory of
the state, $\eta_{i}(\cdot)\in L^{2}([-\delta,0];\textbf{R}^{k_{i}})$,
$i=1,2$, are the initial trajectories of the follower’s and the leader’s
control, respectively. $A(\cdot),\bar{A}(\cdot),C(\cdot),\bar{C}(\cdot)\in
L^{\infty}([0,T];\textbf{R}^{n\times n})$,
$\bar{B}_{i}(\cdot),\bar{D}_{i}(\cdot)\in L^{\infty}([0,T];\textbf{R}^{n\times
k_{i}})$, $Q_{i}(\cdot),\bar{Q}_{i}(\cdot)\in
L^{\infty}([0,T];\textbf{S}^{n})$, $R_{i}(\cdot),\bar{R}_{i}(\cdot)\in
L^{\infty}([0,T];\textbf{S}^{k_{i}})$, and $G_{i}\in\textbf{S}^{n}$.

For any $u_{1}(\cdot)\in L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{k_{1}})$,
$u_{2}(\cdot)\in L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{k_{2}})$, (2.1) admits
a unique solution $X^{u_{1},u_{2}}(\cdot)\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))$ (see [15]) and the cost
functionals are well-defined.

Our linear quadratic stochastic Stackelberg differential game with time delay
is the following. For each choice of the leader
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]\triangleq
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{k_{2}})$, the follower would like to
choose a strategy $\bar{u}_{1}(\cdot)\in\mathcal{U}_{1}[0,T]\triangleq
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{k_{1}})$ such that his cost functional
$J_{1}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$ is the minimum of
$J_{1}(u_{1}(\cdot),u_{2}(\cdot))$ over $u_{1}(\cdot)\in\mathcal{U}_{1}[0,T]$.
The leader has the ability to know the optimal strategy of the follower
$\bar{u}_{1}(\cdot)$, thus the leader would like to choose a strategy
$\bar{u}_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$ such that his cost functional
$J_{2}(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))$ is the minimum of
$J_{2}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$ over
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$. Strictly speaking, the follower wants
to find a map
$\bar{\alpha}_{1}[\cdot]:\mathcal{U}_{2}[0,T]\rightarrow\mathcal{U}_{1}[0,T]$
and the leader wants to find a control
$\bar{u}_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$, such that

$\displaystyle\left\\{\begin{aligned}
&J_{1}(\bar{\alpha}_{1}(u_{2}(\cdot))(\cdot),u_{2}(\cdot))=\underset{u_{1}(\cdot)\in\mathcal{U}_{1}[0,T]}{\inf}J_{1}(u_{1}(\cdot),u_{2}(\cdot)),\quad\forall
u_{2}(\cdot)\in\mathcal{U}_{2}[0,T],\\\
&J_{2}(\bar{\alpha}_{1}(\bar{u}_{2}(\cdot))(\cdot),\bar{u}_{2}(\cdot))=\underset{u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]}{\inf}J_{2}(\bar{\alpha}_{1}(u_{2}(\cdot)),u_{2}(\cdot)).\end{aligned}\right.$

Then $\bar{u}_{1}(\cdot)\triangleq\bar{\alpha}_{1}[\bar{u}_{2}(\cdot)]$, the
pair $(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))$ is called an open-loop
Stackelberg strategy to the above game, and the corresponding solution
$\bar{X}(\cdot)\equiv X^{\bar{u}_{1},\bar{u}_{2}}(\cdot)$ is called the
optimal state trajectory. The purpose of this paper is to find and
characterize the unique open-loop Stackelberg strategy.

##  3 Optimization problem of the follower

In this section, we deal with the optimization problem of the follower, which
is a linear quadratic stochastic optimal control problem with time delay, for
any choice $u_{2}(\cdot)$ of the leader.

We give the following detailed statement.

Problem (F-DLQ): For any $u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$, minimize the
cost functional $J_{1}(u_{1}(\cdot),u_{2}(\cdot))$ over
$u_{1}(\cdot)\in\mathcal{U}_{1}[0,T]$ such that (2.1) is satisfied.

To solve Problem (F-DLQ), we first introduce the adjoint equation:

$\left\\{\begin{aligned}
dp(t)&=\Big{\\{}-A(t)^{\top}p(t)-C(t)^{\top}q(t)-\big{[}Q_{1}(t)+\bar{Q}_{1}(t+\delta)\big{]}X^{\bar{u}_{1},u_{2}}(t)\\\
&\qquad-\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{A}(t+\delta)^{\top}p(t+\delta)+\bar{C}(t+\delta)^{\top}q(t+\delta)\big{]}\Big{\\}}dt+q(t)dW(t),\
t\in[0,T],\\\ p(T)&=G_{1}X^{\bar{u}_{1},u_{2}}(T),\ p(t)=q(t)=0,\
t\in(T,T+\delta].\end{aligned}\right.$ (3.1)

(3.1) is called an _anticipated backward stochastic differential equation_ (in
short, ABSDE), which has a natural adjoint relation with the SDDE (2.1) (see
[8]). By Peng and Yang [21], (3.1) admits a unique solution
$(p(\cdot),q(\cdot))\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))\times
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{n})$.

Then we can obtain the following result.

######  Theorem 3.1.

Assume $\bar{Q}_{1}(t)=\bar{R}_{1}(t)=0$ for $t\in[T,T+\delta]$. For any
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$, let $\bar{u}_{1}(\cdot)$ be an optimal
control of the follower and $X^{\bar{u}_{1},u_{2}}(\cdot)$ be the optimal
state trajectory for Problem (F-DLQ), then

$\displaystyle\big{[}R_{1}(t)+\bar{R}_{1}(t+\delta)\big{]}\bar{u}_{1}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{T}p(t+\delta)+\bar{D}_{1}(t+\delta)^{T}q(t+\delta)\big{]}=0,$
(3.2) $\displaystyle\hskip 256.0748pta.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

###### Proof.

For any $v_{1}(\cdot)\in\mathcal{U}_{1}[0,T]$ with $v_{1}(t)=\eta_{1}(t)$ for
$t\in[-\delta,0]$, define
$u_{1}^{\varepsilon}(\cdot)=\bar{u}_{1}(\cdot)+\varepsilon(v_{1}(\cdot)-\bar{u}_{1}(\cdot))$,
$\varepsilon\in[0,1]$. Suppose $X^{v_{1},u_{2}}(\cdot)$ and
$X^{u_{1}^{\varepsilon},u_{2}}(\cdot)$ are the state trajectories
corresponding to $v_{1}(\cdot)$ and $u_{1}^{\varepsilon}(\cdot)$,
respectively. Then we have

$\displaystyle
J_{1}(u_{1}^{\varepsilon}(\cdot),u_{2}(\cdot))-J_{1}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$
(3.3)
$\displaystyle=2\varepsilon\mathbb{E}\bigg{\\{}\int_{0}^{T}\Big{[}\big{\langle}Q_{1}(t)X^{\bar{u}_{1},u_{2}}(t),X^{v_{1},u_{2}}(t)-X^{\bar{u}_{1},u_{2}}(t)\big{\rangle}$
$\displaystyle\qquad\qquad\qquad+\big{\langle}\bar{Q}_{1}(t)X^{\bar{u}_{1},u_{2}}(t-\delta),X^{v_{1},u_{2}}(t-\delta)-X^{\bar{u}_{1},u_{2}}(t-\delta)\big{\rangle}$
$\displaystyle\qquad\qquad\qquad+\big{\langle}R_{1}(t)\bar{u}_{1}(t),v_{1}(t)-\bar{u}_{1}(t)\big{\rangle}+\big{\langle}\bar{R}_{1}(t)\bar{u}_{1}(t-\delta),v_{1}(t-\delta)-\bar{u}_{1}(t-\delta)\big{\rangle}\Big{]}dt$
$\displaystyle\qquad\qquad+\big{\langle}G_{1}X^{\bar{u}_{1},u_{2}}(T),X^{v_{1},u_{2}}(T)-X^{\bar{u}_{1},u_{2}}(T)\big{\rangle}\bigg{\\}}+\varepsilon^{2}\Big{\\{}\cdots\Big{\\}}.$

Applying Itô’s formula to
$\big{\langle}p(\cdot),X^{v_{1},u_{2}}(\cdot)-X^{\bar{u}_{1},u_{2}}(\cdot)\big{\rangle}$,
and substituting it into (LABEL:eq3.3), we deduce

$\displaystyle
J_{1}(u_{1}^{\varepsilon}(\cdot),u_{2}(\cdot))-J_{1}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$
(3.4)
$\displaystyle=2\varepsilon\mathbb{E}\int_{0}^{T}\Big{[}\big{\langle}\big{[}R_{1}(t)+\bar{R}_{1}(t+\delta)\big{]}\bar{u}_{1}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{T}p(t+\delta)$
$\displaystyle\qquad\qquad\quad+\bar{D}_{1}(t+\delta)^{T}q(t+\delta)\big{]},v_{1}(t)-\bar{u}_{1}(t)\big{\rangle}\Big{]}dt+\varepsilon^{2}\Big{\\{}\cdots\Big{\\}}.$

Dividing both sides of (LABEL:eq3.4) by $\varepsilon$, we derive

$\displaystyle 0$
$\displaystyle\geq\frac{1}{\varepsilon}\big{[}J_{1}(u_{1}^{\varepsilon}(\cdot),u_{2}(\cdot))-J_{1}(\bar{u}_{1}(\cdot),u_{2}(\cdot))\big{]}$
$\displaystyle=2\mathbb{E}\int_{0}^{T}\Big{[}\big{\langle}\big{[}R_{1}(t)+\bar{R}_{1}(t+\delta)\big{]}\bar{u}_{1}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{T}p(t+\delta)$
$\displaystyle\qquad\qquad\quad+\bar{D}_{1}(t+\delta)^{T}q(t+\delta)\big{]},v_{1}(t)-\bar{u}_{1}(t)\big{\rangle}\Big{]}dt+\varepsilon\Big{\\{}\cdots\Big{\\}}.$

Finally letting $\varepsilon\rightarrow 0$, then due to the arbitrariness
chosen of $v_{1}(\cdot)$, we complete the proof. ∎

Next we try to give the feedback expression of $\bar{u}_{1}(\cdot)$. For this
target, for some technical reason, we have to impose the following assumptions
on the coefficients of (2.1) and (LABEL:cost):

$\textbf{(A1)}\left\\{\begin{aligned}
&C(t)^{\top}P_{1}(t)\bar{D}_{1}(t)+P_{1}(t)\bar{B}_{1}(t)=0,\quad\
t\in[0,T],\\\ &C(t)^{\top}P_{1}(t)\bar{C}(t)+P_{1}(t)\bar{A}(t)=0,\quad\
t\in[0,T],\end{aligned}\right.$

where $P_{1}(\cdot)$ is the solution to the following Pseudo-Riccati equation:

$\left\\{\begin{aligned}
\dot{P}_{1}(t)&=-P_{1}(t)A(t)-A(t)^{\top}P_{1}(t)-C(t)^{\top}P_{1}(t)C(t)\\\
&\quad-\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta)-Q_{1}(t)-\bar{Q}_{1}(t+\delta)\\\
&\quad+\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{1}(t+\delta)\Omega_{1}^{-1}(t)\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta),\
t\in[0,T],\\\ P_{1}(T)&=G_{1},\ P_{1}(t)=0,\quad t\in(T,T+\delta],\\\
\Omega_{1}(t)&\triangleq
R_{1}(t)+\bar{R}_{1}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{1}(t+\delta)>0,\
\forall\ t\in[0,T].\end{aligned}\right.$ (3.5)

And we choose $(\zeta_{1}(\cdot),\bar{\zeta}_{1}(\cdot))$ satisfying the
following ABSDE:

$\left\\{\begin{aligned}
d\zeta_{1}(t)&=\Big{\\{}-A(t)^{\top}\zeta_{1}(t)-C(t)^{\top}\bar{\zeta}_{1}(t)-\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{A}(t+\delta)^{\top}\zeta_{1}(t+\delta)+\bar{C}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)\big{]}\\\
&\qquad+\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{1}(t+\delta)\Omega_{1}^{-1}(t)\big{\\{}\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)\\\
&\qquad\quad+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)\big{]}+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\big{\\}}\\\
&\qquad-\big{[}P_{1}(t)\bar{B}_{2}(t)+C(t)^{\top}P_{1}(t)\bar{D}_{2}(t)\big{]}u_{2}(t-\delta)\\\
&\qquad-\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\Big{\\}}dt+\bar{\zeta}_{1}(t)dW(t),\
t\in[0,T],\\\ \zeta_{1}(t)&=\bar{\zeta}_{1}(t)=0,\
t\in(T,T+\delta].\end{aligned}\right.$ (3.6)

######  Remark 3.1.

We point out that (3.5) is not a Riccati-type equation. In fact, (3.5) can be
solved step by step as a linear ordinary differential equation (in short,
ODE). Furthermore, its unique solvability can be guaranteed by the boundedness
of its coefficients. In details, for $t\in(T-\delta,T]$, (3.5) becomes

$\left\\{\begin{aligned}
\dot{P}_{1}(t)&=-P_{1}(t)A(t)-A(t)^{\top}P_{1}(t)-C(t)^{\top}P_{1}(t)C(t)-Q_{1}(t)-\bar{Q}_{1}(t+\delta),\
t\in(T-\delta,T],\\\ P_{1}(T)&=G_{1}.\end{aligned}\right.$

Obviously, it admits a unique solution $P_{1}(\cdot)\in
C((T-\delta,T];\textbf{S}^{n})$. Repeating the above steps on
$(T-2\delta,T-\delta]$, $(T-3\delta,T-2\delta]$, $\cdots$, we can derive the
solution to (3.5) on $[0,T]$. On the other hand, due to the boundedness of the
coefficients, ABSDE (3.6) has a unique solution
$(\zeta_{1}(\cdot),\bar{\zeta}_{1}(\cdot))\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))\times
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{n})$ for any
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$ (see [21]).

######  Remark 3.2.

From Remark 3.1, we know (3.5) is essentially a linear ODE and hence is easier
to solve than the Riccati equation (6) in [28].

Next we can derive the sufficient and necessary conditions of the solvability
for Problem (F-DLQ).

######  Theorem 3.2.

Let (A1) hold, and $\bar{Q}_{1}(t)=\bar{R}_{1}(t)=0$ for $t\in[T,T+\delta]$.
Let $P_{1}(\cdot)$ satisfy (3.5) and
$(\zeta_{1}(\cdot),\bar{\zeta}_{1}(\cdot))$ satisfy (3.6). Then for any
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$, Problem (F-DLQ) is solvable and the
optimal control $\bar{u}_{1}(\cdot)$ is of the following state feedback form:

$\displaystyle\bar{u}_{1}(t)=$
$\displaystyle-\Omega_{1}^{-1}(t)\Big{\\{}\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta)X^{\bar{u}_{1},u_{2}}(t)$
(3.7)
$\displaystyle\qquad\qquad+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)\big{]}$
$\displaystyle\qquad\qquad+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\Big{\\}},\
a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

Moreover, the optimal cost can be obtained as follows:

$\displaystyle\underset{u_{1}(\cdot)\in\mathcal{U}_{1}[0,T]}{\inf}J_{1}(u_{1}(\cdot),u_{2}(\cdot))=J_{1}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$
(3.8)
$\displaystyle=\mathbb{E}\bigg{\\{}\big{\langle}P_{1}(0)\varphi(0)+2\zeta_{1}(0),\varphi(0)\big{\rangle}+\int_{-\delta}^{0}I_{\delta}(t)dt+\int_{0}^{T}\Big{[}\big{\langle}P_{1}(t)\bar{D}_{2}(t)u_{2}(t-\delta),\bar{D}_{2}(t)u_{2}(t-\delta)\big{\rangle}$
$\displaystyle\qquad+2\langle\bar{B}_{2}(t)^{\top}\zeta_{1}(t)+\bar{D}_{2}(t)^{\top}\bar{\zeta}_{1}(t),u_{2}(t-\delta)\big{\rangle}-\big{|}\Omega_{1}^{-\frac{1}{2}}(t)\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)$
$\displaystyle\qquad+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\big{]}\big{|}^{2}\Big{]}dt\bigg{\\}},$

where

$\displaystyle I_{\delta}(t)$
$\displaystyle\triangleq\big{\langle}\varphi(t),\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta)\varphi(t)+\bar{Q}_{1}(t+\delta)\varphi(t)+\bar{C}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)$
(3.9)
$\displaystyle\qquad+\bar{A}(t+\delta)^{\top}\zeta_{1}(t+\delta)\big{\rangle}+\big{\langle}\eta_{1}(t),2\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)$
$\displaystyle\qquad+\bar{R}_{1}(t+\delta)\eta_{1}(t)+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{1}(t+\delta)\eta_{1}(t)\big{\rangle}$
$\displaystyle\quad+2\big{\langle}\eta_{1}(t),\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t)\varphi(t)\big{\rangle}+2\big{\langle}\eta_{1}(t),\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)\eta_{2}(t)\big{\rangle}$
$\displaystyle\quad+2\big{\langle}\eta_{2}(t),\bar{D}_{2}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta)\varphi(t)\big{\rangle}.$

###### Proof.

For any $u_{1}(\cdot)\in\mathcal{U}_{1}[0,T]$, applying Itô’s formula to
$\big{\langle}P_{1}(\cdot)X^{u_{1},u_{2}}(\cdot),X^{u_{1},u_{2}}(\cdot)\big{\rangle}$
and $2\big{\langle}\zeta_{1}(\cdot),\\\ X^{u_{1},u_{2}}(\cdot)\big{\rangle}$,
with some computation, by (A1) we deduce

$\displaystyle J_{1}(u_{1}(\cdot),u_{2}(\cdot))$
$\displaystyle=\mathbb{E}\bigg{\\{}\big{\langle}P_{1}(0)\varphi(0)+2\zeta_{1}(0),\varphi(0)\big{\rangle}+\int_{-\delta}^{0}I_{\delta}(t)dt+\int_{0}^{T}\Big{[}\big{\langle}P_{1}(t)\bar{D}_{2}(t)u_{2}(t-\delta),\bar{D}_{2}(t)u_{2}(t-\delta)\big{\rangle}$
$\displaystyle\qquad+2\big{\langle}\bar{B}_{2}(t)^{\top}\zeta_{1}(t)+\bar{D}_{2}(t)^{\top}\bar{\zeta}_{1}(t),u_{2}(t-\delta)\big{\rangle}-\big{|}\Omega_{1}^{-\frac{1}{2}}(t)\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)$
$\displaystyle\qquad\quad+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\big{]}\big{|}^{2}$
$\displaystyle\qquad+\Omega_{1}(t)\big{|}u_{1}(t)+\Omega_{1}^{-1}(t)\big{\\{}\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{C}(t+\delta)X^{u_{1},u_{2}}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\bar{B}_{1}(t+\delta)^{\top}\zeta_{1}(t+\delta)$
$\displaystyle\qquad\quad+\bar{D}_{1}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)\big{]}+\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\bar{D}_{2}(t+\delta)u_{2}(t)\big{\\}}\big{|}^{2}\Big{]}dt\bigg{\\}},$

which implies (3.7) is the optimal control and (LABEL:follower-optimal-cost)
holds. ∎

######  Remark 3.3.

In order to overcome the difficulty caused by the time delay, in this paper we
impose the assumption (A1). Otherwise, we will get the feedback in the form of

$\bar{u}_{1}(t)=\\{\cdots\\}X^{\bar{u}_{1},u_{2}}(t)+\\{\cdots\\}\mathbb{E}^{\mathcal{F}_{t}}[X^{\bar{u}_{1},u_{2}}(t+\delta)],$

which is more complex than (3.7) (in which the terms like
$\mathbb{E}^{\mathcal{F}_{t}}[X^{\bar{u}_{1},u_{2}}(t+\delta)]$ disappears),
and leads to our inability to deal with the optimization problem for the
leader. In the future we will consider some weaker assumptions than (A1) to
deal with this problem.

##  4 Optimization problem of the leader

In this section, we will address the optimization problem of the leader, which
is a linear quadratic stochastic optimal control problem with a state equation
formed by an SDDE and an ABSDE.

Now the follower takes the optimal control $\bar{u}_{1}(\cdot)$ of form
$(\ref{u1-feedback})$, consequently the leader has the following state
equation:

$\left\\{\begin{aligned}
dX^{\bar{u}_{1},u_{2}}(t)&=\Big{\\{}\hat{A}_{1}(t)X^{\bar{u}_{1},u_{2}}(t)+\hat{A}_{2}(t)X^{\bar{u}_{1},u_{2}}(t-\delta)+\hat{B}(t)u_{2}(t-\delta)+\hat{F}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\zeta_{1}(t)]\\\
&\qquad+\hat{H}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\zeta}_{1}(t)]+\bar{B}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\Big{\\}}dt\\\
&\quad+\Big{\\{}\hat{C}_{1}(t)X^{\bar{u}_{1},u_{2}}(t)+\hat{C}_{2}(t)X^{\bar{u}_{1},u_{2}}(t-\delta)+\hat{D}(t)u_{2}(t-\delta)+\hat{K}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\zeta_{1}(t)]\\\
&\qquad+\hat{M}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\zeta}_{1}(t)]+\bar{D}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\Big{\\}}dW(t),\\\
d\zeta_{1}(t)&=\Big{\\{}-\hat{A}_{1}(t)^{\top}\zeta_{1}(t)-\hat{C}_{1}(t)^{\top}\bar{\zeta}_{1}(t)-\mathbb{E}^{\mathcal{F}_{t}}[\tilde{A}_{2}(t+\delta)^{\top}\zeta_{1}(t+\delta)+\tilde{C}_{2}(t+\delta)^{\top}\bar{\zeta}_{1}(t+\delta)]\\\
&\qquad+\hat{N}_{1}(t)u_{2}(t)+\hat{N}_{2}(t)u_{2}(t-\delta)\Big{\\}}dt+\bar{\zeta}_{1}(t)dW(t),\
t\in[0,T],\\\ X^{\bar{u}_{1},u_{2}}(t)&=\varphi(t),\ u_{2}(t)=\eta_{2}(t),\
t\in[-\delta,0],\\\ \zeta_{1}(t)&=\bar{\zeta}_{1}(t)=0,\
t\in[T,T+\delta],\end{aligned}\right.$ (4.1)

where

$\left\\{\begin{aligned} &\hat{A}_{1}(t)\triangleq
A(t),\qquad\hat{C}_{1}(t)\triangleq
C(t),\qquad\hat{A}_{2}(t)\triangleq\bar{A}(t)-\bar{B}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t),\\\
&\tilde{A}_{2}(t)\triangleq\bar{A}(t)-\bar{B}_{1}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t),\\\
&\hat{B}(t)\triangleq\bar{B}_{2}(t)-\bar{B}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t),\\\
&\hat{C}_{2}(t)\triangleq\bar{C}(t)-\bar{D}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t),\\\
&\tilde{C}_{2}(t)\triangleq\bar{C}(t)-\bar{D}_{1}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t),\\\
&\hat{D}(t)\triangleq\bar{D}_{2}(t)-\bar{D}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t),\\\
&\hat{F}(t)\triangleq-\bar{B}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{B}_{1}(t)^{\top},\quad\quad\hat{H}(t)\triangleq-\bar{B}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top},\\\
&\hat{K}(t)\triangleq-\bar{D}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{B}_{1}(t)^{\top},\quad\quad\hat{M}(t)\triangleq-\bar{D}_{1}(t)I_{[\delta,T]}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top},\\\
&\hat{N}_{1}(t)\triangleq-\bar{C}(t+\delta)^{\top}P_{1}(t+\delta)\big{[}I-\bar{D}_{1}(t+\delta)\Omega_{1}^{-1}(t)\bar{D}_{1}(t+\delta)^{\top}P_{1}(t+\delta)\big{]}\bar{D}_{2}(t+\delta),\\\
&\hat{N}_{2}(t)\triangleq-
P_{1}(t)\bar{B}_{2}(t)-C(t)^{\top}P_{1}(t)\bar{D}_{2}(t).\end{aligned}\right.$

Now we formulate the optimization problem of the leader as follows.

Problem (L-DLQ): Minimize the cost functional
$J_{2}(\bar{u}_{1}(\cdot),u_{2}(\cdot))$ over
$u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]$ such that (4.1) is satisfied.

First we introduce the adjoint equation as follows:

$\left\\{\begin{aligned}
d\xi(t)&=\Big{\\{}\hat{A}_{1}(t)\xi(t)+\tilde{A}_{2}(t)\xi(t-\delta)+\mathbb{E}^{\mathcal{F}_{t-\delta}}\big{[}\hat{F}(t)^{\top}p_{2}(t)+\hat{K}(t)^{\top}q_{2}(t)\big{]}\Big{\\}}dt\\\
&\quad+\Big{\\{}\hat{C}_{1}(t)\xi(t)+\tilde{C}_{2}(t)\xi(t-\delta)+\mathbb{E}^{\mathcal{F}_{t-\delta}}\big{[}\hat{H}(t)^{\top}p_{2}(t)+\hat{M}(t)^{\top}q_{2}(t)\big{]}\Big{\\}}dW(t),\\\
dp_{2}(t)&=\Big{\\{}-\hat{A}_{1}(t)^{\top}p_{2}(t)-\hat{C}_{1}(t)^{\top}q_{2}(t)-\big{[}Q_{2}(t)+\bar{Q}_{2}(t+\delta)\big{]}\bar{X}(t)\\\
&\qquad-\mathbb{E}^{\mathcal{F}_{t}}\big{[}\hat{A}_{2}(t+\delta)^{\top}p_{2}(t+\delta)+\hat{C}_{2}(t+\delta)^{\top}q_{2}(t+\delta)\big{]}\Big{\\}}dt+q_{2}(t)dW(t),\
t\in[0,T],\\\ \xi(t)&=0,\ t\in[-\delta,0],\\\ p_{2}(T)&=G_{2}\bar{X}(T),\
p_{2}(t)=0,\ t\in(T,T+\delta],\ q_{2}(t)=0,\
t\in[T,T+\delta],\end{aligned}\right.$ (4.2)

where $\bar{X}(\cdot)\equiv X^{\bar{u}_{1},\bar{u}_{2}}(\cdot)$. The adjoint
equation (4.2) is formed by an SDDE and an ABSDE, which is partial coupled. By
[21] the second equation in (4.2) admits a unique solution
$(p_{2}(\cdot),q_{2}(\cdot))\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))\times
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{n})$, then by [15] the first equation in
(4.2) admits a unique solution $\xi(\cdot)\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))$.

Then we can obtain the following result similar to Theorem 3.1.

######  Theorem 4.1.

Assume $\bar{Q}_{2}(t)=\bar{R}_{2}(t)=0$ for $t\in[T,T+\delta]$. Let
$\bar{u}_{2}(\cdot)$ be the optimal control of the leader and $\bar{X}(\cdot)$
be the optimal state strategy for the Problem (L-DLQ), then
$\bar{u}_{2}(\cdot)$ satisfies

$\displaystyle\big{[}R_{2}(t)+\bar{R}_{2}(t+\delta)\big{]}\bar{u}_{2}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\hat{B}(t+\delta)^{\top}p_{2}(t+\delta)+\hat{D}(t+\delta)^{\top}q_{2}(t+\delta)$
(4.3)
$\displaystyle-\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{]}-\hat{N}_{1}(t)^{\top}\xi(t)=0,\quad
a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

Next we would like to give the feedback expression of $\bar{u}_{2}(\cdot)$. In
addition (A1), now we have to impose some other assumptions to the
coefficients of (2.1) and (LABEL:cost):

$\textbf{(A2)}\left\\{\begin{aligned}
&C(t)^{\top}P_{2}(t)\bar{D}_{1}(t)+P_{2}(t)\bar{B}_{1}(t)=0,\quad\
t\in[0,T],\\\ &C(t)^{\top}P_{2}(t)\bar{C}(t)+P_{2}(t)\bar{A}(t)=0,\quad\
t\in[0,T],\\\
&C(t)^{\top}P_{2}(t)\bar{D}_{2}(t)+P_{2}(t)\bar{B}_{2}(t)=0,\quad\
t\in[0,T],\end{aligned}\right.$

where $P_{2}(\cdot)$ is the solution to the following Pseudo-Riccati equation:

$\left\\{\begin{aligned}
\dot{P}_{2}(t)&=-P_{2}(t)\hat{A}_{1}(t)-\hat{A}_{1}(t)^{\top}P_{2}(t)-\hat{C}_{1}(t)^{\top}P_{2}(t)\hat{C}_{1}(t)\\\
&\quad-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{C}_{2}(t+\delta)-Q_{2}(t)-\bar{Q}_{2}(t+\delta)\\\
&\quad+\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{C}_{2}(t+\delta),\
t\in[0,T],\\\ P_{2}(T)&=G_{2},P_{2}(t)=0,\ t\in(T,T+\delta],\\\
\Omega_{2}(t)&\triangleq
R_{2}(t)+\bar{R}_{2}(t+\delta)+\hat{D}(t+\delta)^{T}P_{2}(t+\delta)\hat{D}(t+\delta)>0,\quad\forall\
t\in[0,T].\end{aligned}\right.$ (4.4)

Let $(\zeta_{2}(\cdot),\bar{\zeta}_{2}(\cdot))$ satisfy the following ABSDE:

$\left\\{\begin{aligned}
d\zeta_{2}(t)&=\Big{\\{}-\hat{A}_{1}(t)^{\top}\zeta_{2}(t)-\hat{C}_{1}(t)^{\top}\bar{\zeta}_{2}(t)+\big{[}\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{B}(t+\delta)^{\top}\\\
&\qquad-\hat{A}_{2}(t+\delta)^{\top}\big{]}\mathbb{E}^{\mathcal{F}_{t}}[\zeta_{2}(t+\delta)]+\big{[}\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{D}(t+\delta)^{\top}\\\
&\qquad-\hat{C}_{2}(t+\delta)^{\top}\big{]}\mathbb{E}^{\mathcal{F}_{t}}[\bar{\zeta}_{2}(t+\delta)]-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\big{\\{}\hat{N}_{1}(t)^{\top}\xi(t)\\\
&\qquad+\mathbb{E}^{\mathcal{F}_{t}}[\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)]\big{\\}}+\big{[}-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{K}(t+\delta)\\\
&\qquad+\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{K}(t+\delta)\big{]}\mathbb{E}^{\mathcal{F}_{t}}[\zeta_{1}(t+\delta)]\\\
&\qquad+\big{[}\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{M}(t+\delta)\\\
&\qquad-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{M}(t+\delta)\big{]}\mathbb{E}^{\mathcal{F}_{t}}[\bar{\zeta}_{1}(t+\delta)]\Big{\\}}dt+\bar{\zeta}_{2}(t)dW(t),\
t\in[0,T],\\\ \zeta_{2}(t)&=\bar{\zeta}_{2}(t)=0,\
t\in[T,T+\delta].\end{aligned}\right.$ (4.5)

######  Remark 4.1.

Similar to (3.5), (4.4) is not a Riccati-type equation and can be solved step
by step as a linear ODE, like (19) in [28]. On the other hand, ABSDE (4.5) has
the unique solution $(\zeta_{2}(\cdot),\bar{\zeta}_{2}(\cdot))\in
L_{\mathcal{F}}^{2}(\Omega;C([0,T];\textbf{R}^{n}))\times
L_{\mathcal{F}}^{2}([0,T];\textbf{R}^{n})$ (see [21]).

It is easy to verify that

$\left\\{\begin{aligned} p_{2}(t)&=P_{2}(t)\bar{X}(t)+\zeta_{2}(t),\
t\in[0,T],\ P-a.s.\\\
q_{2}(t)&=P_{2}(t)\hat{C}_{1}(t)\bar{X}(t)+P_{2}(t)\hat{C}_{2}(t)\bar{X}(t-\delta)+P_{2}(t)\hat{D}(t)\bar{u}_{2}(t-\delta)\\\
&\quad+P_{2}(t)\hat{K}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\zeta_{1}(t)]+P_{2}(t)\hat{M}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\zeta}_{1}(t)]\\\
&\quad+P_{2}(t)\bar{D}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)+\bar{\zeta}_{2}(t),\
t\in[0,T],\ P-a.s..\end{aligned}\right.$ (4.6)

Furthermore, with some computations, (LABEL:u2-optimal) becomes

$\displaystyle\bar{u}_{2}(t)=$
$\displaystyle-\Omega_{2}^{-1}(t)\Big{\\{}\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{C}_{2}(t+\delta)\bar{X}(t)+\hat{B}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\zeta_{2}(t+\delta)]$
(4.7) $\displaystyle\
+\hat{D}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\bar{\zeta}_{2}(t+\delta)]+\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{K}(t+\delta)\mathbb{E}^{\mathcal{F}_{t}}[\zeta_{1}(t+\delta)]$
$\displaystyle\
+\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{M}(t+\delta)\mathbb{E}^{\mathcal{F}_{t}}[\bar{\zeta}_{1}(t+\delta)]-\hat{N}_{1}(t)^{\top}\xi(t)$
$\displaystyle\
-\mathbb{E}^{\mathcal{F}_{t}}\big{[}\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{]}\Big{\\}},\quad
a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

Noting the representation of optimal control $\bar{u}_{2}(\cdot)$ in (4.7) is
not satisfactory, since in order to determine $\xi(\cdot)$, we need to solve
the adjoint equation (4.2). Hence we have to know the information
$\bar{X}(T)$, this is not the desired result. We expect to obtain a kind of
feedback expression for the optimal control $u_{2}(\cdot)$ similar to (3.7).
For this target, we stack the forward variables and the backward variables
obtained in the optimization of the follower and the leader, respectively.

We denote

$\displaystyle\begin{aligned} &\phi(t)\triangleq\begin{bmatrix}\xi(t)\\\
\bar{X}(t)\end{bmatrix},\quad\psi(t)\triangleq\begin{bmatrix}\zeta_{1}(t)\\\
\zeta_{2}(t)\end{bmatrix},\quad\bar{\psi}(t)\triangleq\begin{bmatrix}\bar{\zeta}_{1}(t)\\\
\bar{\zeta}_{2}(t)\end{bmatrix},\quad\mathcal{A}_{1}(t)\triangleq\begin{bmatrix}\hat{A}_{1}(t)&0\\\
0&\hat{A}_{1}(t)\end{bmatrix},\\\
&\mathcal{A}_{2}(t)\triangleq\begin{bmatrix}0&\hat{F}(t)^{\top}P_{2}(t)+\hat{K}(t)^{\top}P_{2}(t)\hat{C}_{1}(t)\\\
0&0\end{bmatrix},\quad\mathcal{A}_{3}(t)\triangleq\begin{bmatrix}\tilde{A}_{2}(t)&\hat{K}(t)^{\top}P_{2}(t)\hat{C}_{2}(t)\\\
0&\hat{A}_{2}(t)\end{bmatrix},\\\
&\mathcal{B}(t)\triangleq\begin{bmatrix}\hat{K}(t)^{\top}P_{2}(t)\hat{K}(t)&\hat{F}(t)^{\top}\\\
\hat{F}(t)&0\end{bmatrix},\quad\mathcal{C}(t)\triangleq\begin{bmatrix}\hat{K}(t)^{\top}P_{2}(t)\hat{M}(t)&\hat{K}(t)^{\top}\\\
\hat{H}(t)&0\end{bmatrix},\\\
&\bar{\mathcal{A}}_{1}(t)\triangleq\begin{bmatrix}\hat{C}_{1}(t)&0\\\
0&\hat{C}_{1}(t)\end{bmatrix},\quad\bar{\mathcal{A}}_{2}(t)\triangleq\begin{bmatrix}0&\hat{H}(t)^{\top}P_{2}(t)+\hat{M}(t)^{\top}P_{2}(t)\hat{C}_{1}(t)\\\
0&0\end{bmatrix},\\\
&\bar{\mathcal{A}}_{3}(t)\triangleq\begin{bmatrix}\tilde{C}_{2}(t)&\hat{M}(t)^{\top}P_{2}(t)\hat{C}_{2}(t)\\\
0&\hat{C}_{2}(t)\end{bmatrix},\quad\bar{\mathcal{C}}(t)\triangleq\begin{bmatrix}\hat{M}(t)^{\top}P_{2}(t)\hat{M}(t)&\hat{M}(t)^{\top}\\\
\hat{M}(t)&0\end{bmatrix},\\\
&\mathcal{D}(t)\triangleq\begin{bmatrix}\hat{K}(t)^{\top}P_{2}(t)\hat{D}(t)\\\
\hat{B}(t)\end{bmatrix},\quad\bar{\mathcal{D}}(t)\triangleq\begin{bmatrix}\hat{M}(t)^{\top}P_{2}(t)\hat{D}(t)\\\
\hat{D}(t)\end{bmatrix},\\\
&\mathcal{G}_{1}(t)\triangleq\begin{bmatrix}\hat{N}_{1}(t)\\\
-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\end{bmatrix},\quad\mathcal{G}_{2}(t)\triangleq\begin{bmatrix}\hat{N}_{2}(t)\\\
0\end{bmatrix},\\\ &\mathcal{E}(t)\triangleq\begin{bmatrix}0&0\\\
0&-\hat{C}_{2}(t+\delta)^{\top}P_{2}(t+\delta)\hat{D}(t+\delta)\Omega_{2}^{-1}(t)\hat{D}(t+\delta)^{\top}P_{2}(t+\delta)\hat{C}_{2}(t+\delta)\end{bmatrix},\\\
&\mathcal{M}(t)\triangleq\begin{bmatrix}\hat{K}(t)^{\top}P_{2}(t)\bar{D}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\\\
\bar{B}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\end{bmatrix},\quad\bar{\mathcal{M}}(t)\triangleq\begin{bmatrix}\hat{M}(t)^{\top}P_{2}(t)\bar{D}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\\\
\bar{D}_{1}(t)\eta_{1}(t-\delta)I_{[0,\delta]}(t)\end{bmatrix},\\\
\end{aligned}$

then we have

$\left\\{\begin{aligned}
d\phi(t)&=\Big{\\{}\mathcal{A}_{1}(t)\phi(t)+\mathcal{A}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+\mathcal{A}_{3}(t)\phi(t-\delta)+\mathcal{B}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]\\\
&\qquad+\mathcal{C}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]+\mathcal{D}(t)\bar{u}_{2}(t-\delta)+\mathcal{M}(t)\Big{\\}}dt\\\
&\quad+\Big{\\{}\bar{\mathcal{A}}_{1}(t)\phi(t)+\bar{\mathcal{A}}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+\bar{\mathcal{A}}_{3}(t)\phi(t-\delta)+\mathcal{C}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]\\\
&\qquad+\bar{\mathcal{C}}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]+\bar{\mathcal{D}}(t)\bar{u}_{2}(t-\delta)+\bar{\mathcal{M}}(t)\Big{\\}}dW(t),\\\
d\psi(t)&=\Big{\\{}\mathcal{E}(t)\phi(t)-\mathcal{A}_{1}(t)^{\top}\psi(t)-\mathcal{A}_{2}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]-\mathcal{A}_{3}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\psi(t+\delta)]\\\
&\qquad-\bar{\mathcal{A}_{1}}(t)^{\top}\bar{\psi}(t)-\bar{\mathcal{A}_{2}}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]-\bar{\mathcal{A}_{3}}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\bar{\psi}(t+\delta)]\\\
&\qquad+\mathcal{G}_{1}(t)\bar{u}_{2}(t)+\mathcal{G}_{2}(t)\bar{u}_{2}(t-\delta)\Big{\\}}dt+\bar{\psi}(t)dW(t),\
t\in[0,T],\\\ \phi(t)&=(0,\varphi(t)^{\top})^{\top},\ t\in[-\delta,0],\\\
\psi(t)&=\bar{\psi}(t)=(0,0)^{\top},\ t\in[T,T+\delta],\end{aligned}\right.$
(4.8)

and (4.7) can be simplified as

$\displaystyle\bar{u}_{2}(t)$
$\displaystyle=-\Omega_{2}^{-1}(t)\Big{\\{}-\mathcal{G}_{1}(t)^{T}\phi(t)+\mathcal{D}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\psi(t+\delta)]+\bar{\mathcal{D}}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\bar{\psi}(t+\delta)]$
(4.9) $\displaystyle\qquad\qquad\quad\
-\mathcal{G}_{2}(t+\delta)^{\top}\mathbb{E}^{\mathcal{F}_{t}}[\phi(t+\delta)]\Big{\\}},\quad
a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

To derive the feedback expression of the optimal control $\bar{u}_{2}(\cdot)$,
we will establish a nonhomogeneous relationship between $\phi(\cdot)$ and
$\psi(\cdot)$ (see [28]). Next basing on (A1), (A2), we also impose the
following assumptions to the coefficients of (2.1) and (LABEL:cost):

$\textbf{(A3)}\left\\{\begin{aligned}
&I-\bar{D}_{1}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)=0,\quad\
t\in[0,T],\\\
&\bar{A}(t)-\bar{B}_{1}(t)\Omega_{1}^{-1}(t-\delta)\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t)=0,\quad\
t\in[0,T].\end{aligned}\right.$

Hence

$\left\\{\begin{aligned} &\hat{A}_{2}(t)=\bar{A}(t),\
\hat{C}_{2}(t)=\bar{C}(t),\ \hat{D}(t)=\bar{D}_{2}(t),\
\tilde{A}_{2}(t)=\tilde{C}_{2}(t)=\hat{N}_{1}(t)=0,\quad t\in[0,\delta),\\\
&\tilde{A}_{2}(t)=\tilde{C}_{2}(t)=\hat{A}_{2}(t)\equiv\hat{C}_{2}(t)\equiv\hat{D}(t)\equiv\hat{N}_{1}(t)\equiv
0,\quad t\in[\delta,T],\\\
&\mathcal{A}_{3}(t)\equiv\bar{\mathcal{A}_{3}}(t)\equiv\mathcal{E}(t)\equiv
0,\quad\bar{\mathcal{D}}(t)\equiv\mathcal{G}_{1}(t)\equiv 0,\quad
t\in[\delta,T],\\\
&\mathcal{A}_{3}(t)=\begin{bmatrix}0&\hat{K}(t)^{T}P_{2}(t)\bar{C}(t)\\\
0&\bar{A}(t)\end{bmatrix},\quad\bar{\mathcal{A}_{3}}(t)=\begin{bmatrix}0&\hat{M}(t)^{T}P_{2}(t)\bar{C}(t)\\\
0&\bar{C}(t)\end{bmatrix},\quad t\in[0,\delta),\\\ &\mathcal{E}(t)\equiv
0,\quad\bar{\mathcal{D}}(t)=\begin{bmatrix}\hat{M}(t)^{T}P_{2}(t)\bar{D}_{2}(t)\\\
\bar{D}_{2}(t)\end{bmatrix},\quad\mathcal{G}_{1}(t)\equiv 0,\quad
t\in[0,\delta).\\\ \end{aligned}\right.$

Denote further

$\left\\{\begin{aligned} \Omega_{3}(t)&\triangleq
R_{2}(t)+\bar{R}_{2}(t+\delta),\\\
\Xi_{1}(t)&\triangleq\mathcal{A}_{2}(t)+\mathcal{B}(t)L(t)+\mathcal{C}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)]\\\
&\quad+\mathcal{D}(t)\Omega_{3}^{-1}(t-\delta)\big{[}\mathcal{G}_{2}(t)^{\top}-\mathcal{D}(t)^{\top}L(t)\big{]},\\\
\Xi_{2}(t)&\triangleq-\mathcal{B}(t)-\mathcal{C}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{\top}+\mathcal{D}(t)\Omega_{3}^{-1}(t-\delta)\mathcal{D}(t)^{\top},\\\
\Xi_{3}(t)&\triangleq[\mathcal{G}_{2}(t)-L(t)\mathcal{D}(t)]\Omega_{3}^{-1}(t-\delta)[\mathcal{G}_{2}(t)^{\top}-\mathcal{D}(t)^{\top}L(t)]-\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)\\\
&\quad+\mathcal{C}(t)^{\top}L(t)\big{]}^{\top}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}\\\
&\quad-\mathcal{A}_{2}(t)^{\top}L(t)-L(t)\mathcal{A}_{2}(t)-L(t)\mathcal{B}(t)L(t)+\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}_{1}}(t),\end{aligned}\right.$

where $L(\cdot)$ is the solution to the following matrix equation:

$\left\\{\begin{aligned}
\dot{L}(t)&=-L(t)\mathcal{A}_{1}(t)-\mathcal{A}_{1}(t)^{\top}L(t)+\Pi(t,t+\delta)I_{[0,T-\delta]}(t)-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{1}(t),\
t\in[0,T],\\\ L(T)&=0,\\\ [I-L&(t)\bar{\mathcal{C}}(t)]^{-1}\ \mbox{exists},\
t\in[0,T],\end{aligned}\right.$ (4.10)

and $\Pi(\cdot,\cdot)$ satisfies

$\left\\{\begin{aligned} -\frac{\partial\Pi(t,\theta)}{\partial
t}&=\Pi(t,\theta)\mathcal{A}_{1}(t)+\mathcal{A}_{1}(t)^{\top}\Pi(t,\theta),\
\theta-\delta\leq t\leq\theta,\\\ \Pi(t,t)&=\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{1}(t)+\Xi_{1}(t)^{\top}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\\\ &+\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{2}(t)\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta+\Xi_{3}(t).\end{aligned}\right.$ (4.11)

Apparently $\Pi(\cdot,\cdot)$ is symmetric, hence so is $L(\cdot)$.

######  Remark 4.2.

Comparing with (25) in [28], the matrix equation (4.10) is very simple.

Then we have the nonhomogeneous relationship between $\psi(\cdot)$ and
$\phi(\cdot)$ which is shown below.

######  Lemma 4.1.

Provided that the matrix equation $(\ref{L-equation})$ and $(\ref{Pi-
equation})$ have the unique solutions, then it holds that

$\psi(t)=L(t)\phi(t)-\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)\hat{\phi}(t|\theta-\delta)d\theta,\quad t\in[\delta,T],$
(4.12)

where
$\hat{\phi}(t|\theta-\delta)\triangleq\mathbb{E}^{\mathcal{F}_{\theta-\delta}}[\phi(t)]$.

###### Proof.

Let

$\Theta(t)\triangleq\psi(t)-L(t)\phi(t),\quad t\in[\delta,T],$ (4.13)

and suppose $d\Theta(t)=\Theta_{1}(t)dt+\Theta_{2}(t)dW(t)$. Applying Itô’s
formula to (4.13), we have

$\displaystyle d\Theta(t)$
$\displaystyle=\Big{\\{}-\mathcal{A}_{1}(t)^{\top}\psi(t)-\mathcal{A}_{2}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]-\bar{\mathcal{A}_{1}}(t)^{\top}\bar{\psi}(t)-\bar{\mathcal{A}_{2}}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]$
(4.14)
$\displaystyle\qquad+\mathcal{G}_{2}(t)\bar{u}_{2}(t-\delta)+\mathcal{A}_{1}(t)^{\top}L(t)\phi(t)-\Pi(t,t+\delta)I_{[0,T-\delta]}(t)\phi(t)$
$\displaystyle\qquad+\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{1}(t)\phi(t)-L(t)\mathcal{A}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]-L(t)\mathcal{B}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]$
$\displaystyle\qquad-L(t)\mathcal{C}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]-L(t)\mathcal{D}(t)u_{2}(t-\delta)\Big{\\}}dt$
$\displaystyle\quad+\Big{\\{}\bar{\psi}(t)-L(t)\bar{\mathcal{A}}_{1}(t)\phi(t)-L(t)\bar{\mathcal{A}}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]$
$\displaystyle\qquad-L(t)\mathcal{C}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]-L(t)\bar{\mathcal{C}}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]\Big{\\}}dW(t),\quad
t\in[\delta,T].$

Then we obtain

$\displaystyle\bar{\psi}(t)$
$\displaystyle=\Theta_{2}(t)+L(t)\bar{\mathcal{A}}_{1}(t)\phi(t)+L(t)\bar{\mathcal{A}}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+L(t)\mathcal{C}(t)^{T}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]$
(4.15)
$\displaystyle\quad+L(t)\bar{\mathcal{C}}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)],\quad
t\in[\delta,T].$

It follows that

$\displaystyle\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]$
$\displaystyle=[I-L(t)\bar{\mathcal{C}}(t)]^{-1}\Big{\\{}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta_{2}(t)]+L(t)\bar{\mathcal{A}}_{1}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]$
(4.16)
$\displaystyle\quad+L(t)\bar{\mathcal{A}}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+L(t)\mathcal{C}(t)^{T}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]\Big{\\}},\quad
t\in[\delta,T].$

Substituting (4.13) into (4.9), we deduce

$\displaystyle\bar{u}_{2}(t)$
$\displaystyle=-\Omega_{3}^{-1}(t)\Big{\\{}\big{[}\mathcal{D}(t+\delta)^{\top}L(t+\delta)-\mathcal{G}_{2}(t+\delta)^{\top}\big{]}\mathbb{E}^{\mathcal{F}_{t}}[\phi(t+\delta)]$
(4.17)
$\displaystyle\quad\quad\quad\quad\quad\quad+\mathcal{D}(t+\delta)^{T}\mathbb{E}^{\mathcal{F}_{t}}[\Theta(t+\delta)]\Big{\\}},\quad
a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

Substituting (4.16) into (4.15), we derive

$\displaystyle\bar{\psi}(t)$
$\displaystyle=L(t)\bar{\mathcal{A}}_{1}(t)\phi(t)+\Big{\\{}L(t)\bar{\mathcal{C}}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}$
(4.18)
$\displaystyle\quad+L(t)\bar{\mathcal{A}}_{2}(t)+L(t)\mathcal{C}(t)^{\top}L(t)\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+\Big{\\{}L(t)\bar{\mathcal{C}}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{\top}$
$\displaystyle\quad+L(t)\mathcal{C}(t)^{\top}\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]+\Theta_{2}(t)+L(t)\bar{\mathcal{C}}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta_{2}(t)],\quad
t\in[\delta,T].$

Hence we have

$\displaystyle\Theta_{1}(t)$
$\displaystyle=\Big{\\{}-\big{[}\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}$
(4.19)
$\displaystyle\qquad-\mathcal{A}_{2}(t)^{\top}L(t)-L(t)\mathcal{A}_{2}(t)-L(t)\mathcal{B}(t)L(t)\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]-\bar{\mathcal{A}_{1}}(t)^{\top}\bar{\psi}(t)$
$\displaystyle\quad-\Pi(t,t+\delta)I_{[0,T-\delta]}(t)\phi(t)+\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{1}(t)\phi(t)-\Big{\\{}[\bar{\mathcal{A}}_{2}(t)^{\top}$
$\displaystyle\qquad+L(t)\mathcal{C}(t)][I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{\top}+\mathcal{A}_{2}(t)^{\top}+L(t)\mathcal{B}(t)\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]$
$\displaystyle\quad-\mathcal{A}_{1}(t)^{\top}\Theta(t)+\big{[}\mathcal{G}_{2}(t)-L(t)\mathcal{D}(t)\big{]}\bar{u}_{2}(t-\delta)$
$\displaystyle\quad-\big{[}\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta_{2}(t)],\quad
t\in[\delta,T].$

Plugging (4.17) and (4.18) into (4.19), it follows that

$\displaystyle\Theta_{1}(t)$
$\displaystyle=\Big{\\{}-\big{[}\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}$
(4.20)
$\displaystyle\qquad-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{C}}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}$
$\displaystyle\qquad-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{2}(t)-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\mathcal{C}(t)^{\top}L(t)-\mathcal{A}_{2}(t)^{\top}L(t)-L(t)\mathcal{A}_{2}(t)$
$\displaystyle\qquad-L(t)\mathcal{B}(t)L(t)+\big{[}\mathcal{G}_{2}(t)-L(t)\mathcal{D}(t)\big{]}\Omega_{3}^{-1}(t-\delta)\big{[}\mathcal{G}_{2}(t)^{\top}-\mathcal{D}(t)^{\top}L(t)\big{]}\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]$
$\displaystyle\quad+\Big{\\{}-\bar{\mathcal{A}_{1}}(t)^{\top}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{\top}-\big{[}\bar{\mathcal{A}}_{2}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{\top}$
$\displaystyle\qquad-\mathcal{A}_{2}(t)^{\top}-L(t)\mathcal{B}(t)-\big{[}\mathcal{G}_{2}(t)-L(t)\mathcal{D}(t)\big{]}\Omega_{3}^{-1}(t-\delta)\mathcal{D}(t)^{\top}\Big{\\}}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]$
$\displaystyle\quad-\Pi(t,t+\delta)I_{[0,T-\delta]}(t)\phi(t)-\mathcal{A}_{1}(t)^{\top}\Theta(t)-\bar{\mathcal{A}_{1}}(t)^{\top}\Theta_{2}(t)$
$\displaystyle\quad-\big{[}\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{C}}(t)+\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta_{2}(t)],\quad
t\in[\delta,T].$

With some computations, it yields

$\displaystyle-\big{[}\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}$
(4.21)
$\displaystyle-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{C}}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{2}(t)$
$\displaystyle-\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\mathcal{C}(t)^{\top}L(t)=-\big{[}\bar{\mathcal{A}_{1}}(t)^{\top}+\bar{\mathcal{A}_{2}}(t)^{\top}+L(t)\mathcal{C}(t)][I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)$
$\displaystyle+\bar{\mathcal{A}_{2}}(t)+\mathcal{C}(t)^{\top}L(t)\big{]}+\bar{\mathcal{A}_{1}}(t)^{\top}L(t)\bar{\mathcal{A}}_{1}(t).$

On the other hand, for $t\in[T-\delta,T]$, denote
$\tilde{\Theta}_{1}(t)=-\int_{t}^{T}\Pi(t,\theta)\hat{\phi}(t|\theta-\delta)d\theta$,
we can prove that $(\tilde{\Theta}_{1}(\cdot),0)$ is the solution to the BSDE

$\left\\{\begin{aligned} d\Theta(t)&=\Theta_{1}(t)dt+\Theta_{2}(t)dW(t),\quad
t\in[T-\delta,T],\\\ \Theta(T)&=0.\end{aligned}\right.$

Hence

$\Theta(T-\delta)=\tilde{\Theta}_{1}(T-\delta)=-\int_{T-\delta}^{T}\Pi(T-\delta,\theta)\hat{\phi}(T-\delta|\theta-\delta)d\theta.$

For $t\in[\delta,T-\delta]$, denote
$\tilde{\Theta}_{2}(t)=-\int_{t}^{t+\delta}\Pi(t,\theta)\hat{\phi}(t|\theta-\delta)d\theta$,
next we aim to prove $(\tilde{\Theta}_{2}(\cdot),0)$ is the solution to the
BSDE

$\left\\{\begin{aligned} d\Theta(t)&=\Theta_{1}(t)dt+\Theta_{2}(t)dW(t),\
t\in[\delta,T-\delta],\\\
\Theta(T-\delta)&=-\int_{T-\delta}^{T}\Pi(T-\delta,\theta)\hat{\phi}(T-\delta|\theta-\delta)d\theta.\end{aligned}\right.$
(4.22)

Differentiating on $\tilde{\Theta}_{2}(t)$ with respect to $t$, we get

$\displaystyle d\tilde{\Theta}_{2}(t)=$
$\displaystyle-\Pi(t,t+\delta)\phi(t)+\Pi(t,t)\hat{\phi}(t|t-\delta)-\int_{t}^{t+\delta}\dot{\Pi}(t,\theta)\hat{\phi}(t|\theta-\delta)d\theta$
(4.23)
$\displaystyle-\int_{t}^{t+\delta}\Pi(t,\theta)\Big{\\{}\mathcal{A}_{1}(t)\hat{\phi}(t|\theta-\delta)+\mathcal{A}_{2}(t)\hat{\phi}(t|t-\delta)+\mathcal{B}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]$
$\displaystyle\qquad+\mathcal{C}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]+\mathcal{D}(t)u_{2}(t-\delta)\Big{\\}}d\theta,\quad
t\in[\delta,T-\delta].$

Noting

$\displaystyle\mathbb{E}^{\mathcal{F}_{t-\delta}}[\tilde{\Theta}_{2}(t)]=-\int_{t}^{t+\delta}\Pi(t,\theta)d\theta\hat{\phi}(t|t-\delta),$
(4.24)

and recalling (4.16) and (4.17), we have

$\displaystyle\quad\mathcal{B}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]+\mathcal{C}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]+\mathcal{D}(t)u_{2}(t-\delta)$
(4.25)
$\displaystyle=\mathcal{B}(t)L(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+\mathcal{B}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]+\mathcal{C}(t)[I-L(t)\bar{\mathcal{C}}(t)]^{-1}\Big{\\{}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta_{2}(t)]$
$\displaystyle\quad+L(t)\bar{\mathcal{A}}_{1}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+L(t)\bar{\mathcal{A}}_{2}(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]+L(t)\mathcal{C}(t)^{\top}L(t)\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]$
$\displaystyle\quad+L(t)\mathcal{C}(t)^{\top}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]\Big{\\}}-\mathcal{D}(t)\Omega_{3}^{-1}(t-\delta)\Big{\\{}\big{[}\mathcal{D}(t)^{T}L(t)-\mathcal{G}_{2}(t)^{T}\big{]}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)]$
$\displaystyle\quad+\mathcal{D}(t)^{T}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\Theta(t)]\Big{\\}},\quad
t\in[\delta,T].$

Plugging (4.11), (4.24) and (LABEL:eq4.24) into (4.23), combing (4.20) and
(LABEL:eq4.21), it is easy to verify that $(\tilde{\Theta}_{2}(\cdot),0)$ is
the solution to BSDE (4.22). Thus we complete the proof. ∎

Now we give the necessary condition of the solvability for Problem (L-DLQ):

######  Theorem 4.2.

Let (A1)-(A3) hold, assume $\bar{Q}_{i}(t)=\bar{R}_{i}(t)=0$ for
$t\in[T,T+\delta]$, $i=1,2$. Suppose the matrix equation $(\ref{L-equation})$
and $(\ref{Pi-equation})$ have the unique solutions $L(\cdot)$ and
$\Pi(\cdot,\cdot)$ respectively. Let $\bar{u}_{2}(\cdot)$ be the optimal
control of the leader and $\bar{X}(\cdot)$ be the optimal state strategy for
the Problem (L-DLQ), then $\bar{u}_{2}(\cdot)$ is of the following state
feedback form:

$\displaystyle\bar{u}_{2}(t)=$
$\displaystyle-\Omega_{3}^{-1}(t)\Big{\\{}\big{[}\mathcal{D}(t+\delta)^{\top}L(t+\delta)-\mathcal{G}_{2}(t+\delta)^{\top}\big{]}$
(4.26)
$\displaystyle\qquad\quad-\mathcal{D}(t+\delta)^{\top}\int_{t+\delta}^{(t+2\delta)\wedge
T}\Pi(t+\delta,\theta)d\theta\Big{\\}}\hat{\phi}(t+\delta|t),\quad a.e.\
t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

Moreover, the optimal cost of the leader can be expressed as follows:

$\displaystyle\underset{u_{2}(\cdot)\in\mathcal{U}_{2}[0,T]}{\inf}J_{2}(\bar{u}_{1}(\cdot),u_{2}(\cdot))=J_{2}(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))=\mathbb{E}\bigg{\\{}\big{\langle}\varphi(0),P_{2}(0)\varphi(0)+\zeta_{2}(0)\big{\rangle}$
(4.27)
$\displaystyle\quad+\int_{-\delta}^{0}\Big{[}\big{\langle}\varphi(t),\bar{A}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{C}(t+\delta)^{\top}q_{2}(t+\delta)+\bar{Q}_{2}(t+\delta)\varphi(t)\big{\rangle}$
$\displaystyle\qquad\qquad+\big{\langle}\eta_{1}(t),\bar{B}_{1}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}q_{2}(t+\delta)\big{\rangle}+\big{\langle}\eta_{2}(t),\bar{R}_{2}(t+\delta)\eta_{2}(t)$
$\displaystyle\qquad\qquad+\bar{B}_{2}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{2}(t+\delta)^{\top}q_{2}(t+\delta)-\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{\rangle}\Big{]}dt\bigg{\\}},$

where $P_{2}(\cdot)$ satisfies (4.4), $(\xi(\cdot),p_{2}(\cdot),q_{2}(\cdot))$
satisfies (4.2) and $(\zeta_{2}(\cdot),\bar{\zeta}_{2}(\cdot))$ satisfies
(4.5).

###### Proof.

The feedback form (4.26) can be obtained by (4.12), (4.13) and (4.17). Thus we
only need to prove (LABEL:leaderer-optimal-cost). Recall (4.1) and (4.2),
applying Itô’s formula to
$\big{\langle}p_{2}(\cdot),\bar{X}(\cdot)\rangle-\langle\zeta_{1}(\cdot),\xi(\cdot)\big{\rangle}$,
we deduce

$\displaystyle
J_{2}(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))=\mathbb{E}\big{[}\langle
p_{2}(0),\varphi(0)\rangle\big{]}+\mathbb{E}\int_{-\delta}^{0}\Big{[}\big{\langle}\varphi(t),\bar{A}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{C}(t+\delta)^{\top}q_{2}(t+\delta)$
(4.28)
$\displaystyle+\bar{Q}_{2}(t+\delta)\varphi(t)\big{\rangle}+\big{\langle}\eta_{1}(t),\bar{B}_{1}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}q_{2}(t+\delta)\big{\rangle}+\big{\langle}\eta_{2}(t),\bar{R}_{2}(t+\delta)\eta_{2}(t)$
$\displaystyle+\bar{B}_{2}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{2}(t+\delta)^{\top}q_{2}(t+\delta)-\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{\rangle}\Big{]}dt$
$\displaystyle+\mathbb{E}\int_{0}^{T}\big{\langle}\bar{u}_{2}(t),\big{[}R_{2}(t)+\bar{R}_{2}(t+\delta)\big{]}\bar{u}_{2}(t)+\mathbb{E}^{\mathcal{F}_{t}}\big{[}\hat{B}(t+\delta)^{\top}p_{2}(t+\delta)-\hat{N}_{2}(t+\delta)^{T}\xi(t+\delta)\big{]}\big{\rangle}dt.$

Since $(\ref{u2-optimal})$ holds, we obtain

$\displaystyle
J_{2}(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))=\mathbb{E}\big{[}\langle
p_{2}(0),\varphi(0)\rangle\big{]}+\mathbb{E}\int_{-\delta}^{0}\Big{[}\big{\langle}\varphi(t),\bar{A}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{C}(t+\delta)^{\top}q_{2}(t+\delta)$
$\displaystyle\quad+\bar{Q}_{2}(t+\delta)\varphi(t)\big{\rangle}+\big{\langle}\eta_{1}(t),\bar{B}_{1}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}q_{2}(t+\delta)\big{\rangle}+\big{\langle}\eta_{2}(t),\bar{R}_{2}(t+\delta)\eta_{2}(t)$
$\displaystyle\quad+\bar{B}_{2}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{2}(t+\delta)^{\top}q_{2}(t+\delta)-\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{\rangle}\Big{]}dt$
$\displaystyle=\mathbb{E}\Big{\\{}\big{\langle}\varphi(0),P_{2}(0)\varphi(0)+\zeta_{2}(0)\big{\rangle}+\int_{-\delta}^{0}\Big{[}\big{\langle}\varphi(t),\bar{A}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{C}(t+\delta)^{\top}q_{2}(t+\delta)$
$\displaystyle\qquad+\bar{Q}_{2}(t+\delta)\varphi(t)\big{\rangle}+\big{\langle}\eta_{1}(t),\bar{B}_{1}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{1}(t+\delta)^{\top}q_{2}(t+\delta)\big{\rangle}+\big{\langle}\eta_{2}(t),\bar{R}_{2}(t+\delta)$
$\displaystyle\qquad\eta_{2}(t)+\bar{B}_{2}(t+\delta)^{\top}p_{2}(t+\delta)+\bar{D}_{2}(t+\delta)^{\top}q_{2}(t+\delta)-\hat{N}_{2}(t+\delta)^{\top}\xi(t+\delta)\big{\rangle}\Big{]}dt\Big{\\}},$

which completes the proof. ∎

Finally we summarize the above contents and state the main results for the
linear quadratic Stackelberg differential game with time delay.

######  Theorem 4.3.

Let (A1)-(A3) hold, assume $\bar{Q}_{i}(t)=\bar{R}_{i}(t)=0$ for
$t\in[T,T+\delta]$, $i=1,2$. Suppose $(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))$
is the optimal open-loop strategy and the matrix equation $(\ref{L-equation})$
and $(\ref{Pi-equation})$ have the unique solutions $L(\cdot)$ and
$\Pi(\cdot,\cdot)$ respectively, then the optimal open-loop strategy is given
by

$\displaystyle\bar{u}_{1}(t-\delta)$
$\displaystyle=K_{1}^{\bar{u}_{1}}(t-\delta)\phi(t-\delta)+K_{2}^{\bar{u}_{1}}(t-\delta)\hat{\phi}(t|t-\delta),\quad
a.e.\ t\in[\delta,T+\delta],\ \mathbb{P}\mbox{-}a.s.,$ (4.29)
$\displaystyle\bar{u}_{2}(t-\delta)$
$\displaystyle=K^{\bar{u}_{2}}(t-\delta)\hat{\phi}(t|t-\delta),\quad a.e.\
t\in[\delta,T+\delta],\ \mathbb{P}\mbox{-}a.s.,$ (4.30)

where

$\displaystyle\hat{\phi}(t|t-\delta)$
$\displaystyle\triangleq\mathbb{E}^{\mathcal{F}_{t-\delta}}[\phi(t)],$
$\displaystyle K^{\bar{u}_{2}}(t-\delta)$
$\displaystyle\triangleq-\Omega_{3}^{-1}(t-\delta)\bigg{\\{}\mathcal{D}(t)^{\top}L(t)-\mathcal{G}_{2}(t)^{\top}-\mathcal{D}(t)^{\top}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\bigg{\\}},$ $\displaystyle
K_{1}^{\bar{u}_{1}}(t-\delta)$
$\displaystyle\triangleq-\Omega_{1}^{-1}(t-\delta)\big{[}0,\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t)\big{]},$
$\displaystyle K_{2}^{\bar{u}_{1}}(t-\delta)$
$\displaystyle\triangleq\Omega_{1}^{-1}(t-\delta)\bigg{\\{}\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t)\Omega_{3}^{-1}(t-\delta)\big{[}\mathcal{D}(t)^{\top}L(t)-\mathcal{G}_{2}(t)^{\top}\big{]}$
(4.31)
$\displaystyle\qquad-\big{[}\bar{B}_{1}(t)^{\top},0\big{]}L(t)-\big{[}\bar{D}_{1}(t)^{\top},0\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\big{[}\bar{\mathcal{A}_{1}}(t)+\bar{\mathcal{A}_{2}}(t)$
$\displaystyle\qquad+\mathcal{C}(t)^{\top}L(t)\big{]}-\Big{[}\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t)\Omega_{3}^{-1}(t-\delta)\mathcal{D}(t)^{\top}-\big{[}\bar{B}_{1}(t)^{\top},0\big{]}$
$\displaystyle\qquad-\big{[}\bar{D}_{1}(t)^{\top},0\big{]}[I-L(t)\bar{\mathcal{C}}(t)]^{-1}L(t)\mathcal{C}(t)^{T}\Big{]}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\bigg{\\}}.$

###### Proof.

(4.30) is the same as (4.26). Thus we only need to prove (4.29). Substituting
(4.30) into (3.7), it follows that

$\displaystyle\bar{u}_{1}(t-\delta)$
$\displaystyle=-\Omega_{1}^{-1}(t-\delta)\Big{\\{}\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t)\bar{X}(t-\delta)+\mathbb{E}^{\mathcal{F}_{t-\delta}}\big{[}\bar{B}_{1}(t)^{T}\zeta_{1}(t)$
(4.32)
$\displaystyle\qquad+\bar{D}_{1}(t)^{\top}\bar{\zeta}_{1}(t)]+\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t)\bar{u}_{2}(t-\delta)\Big{\\}}$
$\displaystyle=-\Omega_{1}^{-1}(t-\delta)\bigg{\\{}\big{[}0,\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{C}(t)\big{]}\phi(t-\delta)+\big{[}\bar{B}_{1}(t)^{\top},0\big{]}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\psi(t)]$
$\displaystyle\qquad+\big{[}\bar{D}_{1}(t)^{\top},0\big{]}\mathbb{E}^{\mathcal{F}_{t-\delta}}[\bar{\psi}(t)]-\bar{D}_{1}(t)^{\top}P_{1}(t)\bar{D}_{2}(t)\Omega_{3}^{-1}(t-\delta)\bigg{\\{}\mathcal{D}(t)^{\top}L(t)$
$\displaystyle\qquad-\mathcal{G}_{2}(t)^{\top}-\mathcal{D}(t)^{\top}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\bigg{\\}}\hat{\phi}(t|t-\delta)\bigg{\\}}.$

Plugging (4.12) and (4.16) into (4.32), we can deduce (4.29). Hence we
complete the proof. ∎

######  Remark 4.3.

From Theorem 4.3, to derive the state feedback expression of the open-loop
Stackelberg strategy $(\bar{u}_{1}(\cdot),\bar{u}_{2}(\cdot))$, where we
impose the assumptions (A1)-(A3) to the coefficients of (2.1) and
(LABEL:cost). Especially, in the one-dimensional case, we can obtain the
explicit solutions to the Pseudo-Riccati equation (3.5) and (4.4) as follows:

$\displaystyle
P_{1}(t)=G_{1}e^{\int_{t}^{T}(2A(s)+C^{2}(s))}ds+\int_{t}^{T}e^{\int_{t}^{s}(2A(r)+C^{2}(r))dr}\big{[}Q_{1}(s)+\bar{Q}_{1}(s+\delta)\big{]}ds,$
$\displaystyle
P_{2}(t)=G_{2}e^{\int_{t}^{T}(2A(s)+C^{2}(s))ds}+\int_{t}^{T}e^{\int_{t}^{s}(2A(r)+C^{2}(r))dr}\big{[}Q_{2}(s)+\bar{Q}_{2}(s+\delta)\big{]}ds.$

Therefore (A1)-(A3) can be simplified in the following two cases:

  1. 1.

$P_{1}>0,\quad P_{2}\neq 0.$

$\displaystyle\bar{B}_{1}(t)+C(t)\bar{D}_{1}(t)=0,\quad\bar{A}(t)+C(t)\bar{C}(t)=0,\quad\bar{B}_{2}(t)+C(t)\bar{D}_{2}(t)=0,\quad\forall\
t\in[0,T],$ $\displaystyle\bar{D}_{1}\neq
0,\quad\bar{R}_{1}(t)+R_{1}(t-\delta)=0,\quad
R_{2}(t-\delta)+\bar{R}_{2}(t)>0,\quad\forall\ t\in[0,T].$

  2. 2.

$P_{1}>0,\quad P_{2}=0.$

$\displaystyle\bar{B}_{1}(t)+C(t)\bar{D}_{1}(t)=0,\quad\bar{A}(t)+C(t)\bar{C}(t)=0,\quad\forall\
t\in[0,T].$ $\displaystyle\bar{D}_{1}\neq
0,\quad\bar{R}_{1}(t)+R_{1}(t-\delta)=0,\quad
R_{2}(t-\delta)+\bar{R}_{2}(t)>0,\quad\forall\ t\in[0,T].$

##  5 Applications

In this section, we will display two examples, one example is a simple one-
dimensional case and can be solved completely. Another example is related to a
class of resource allocation problems, although the explicit solution can not
be obtained, the numerical algorithm is given and some simulations are done.

###  5.1 One-dimensional case

Suppose

$\displaystyle A=\bar{A}=\bar{C}=0,\quad
C=-1,\quad\bar{B}_{1}(\cdot)=\bar{D}_{1}(\cdot)\neq
0,\quad\bar{B}_{2}(\cdot)=\bar{D}_{2}(\cdot)\neq 0,$ $\displaystyle
R_{1}(\cdot)+\bar{R}_{1}(\cdot+\delta)=0,\quad G_{1}=G_{2}=G>0,$
$\displaystyle
Q_{1}(\cdot)+\bar{Q}_{1}(\cdot+\delta)=Q_{2}(\cdot)+\bar{Q}_{2}(\cdot+\delta)>0.$

Thu the linear controlled SDDE (2.1) is reduced to the following:

$\left\\{\begin{aligned}
dX(t)&=\big{[}\bar{B}_{1}(t)u_{1}(t-\delta)+\bar{B}_{2}(t)u_{2}(t-\delta)\big{]}dt\\\
&\quad+\big{[}-X(t)+\bar{D}_{1}(t)u_{1}(t-\delta)+\bar{D}_{2}(t)u_{2}(t-\delta)\big{]}dW(t),\
t\in[0,T],\\\ u_{1}(t)&=\eta_{1}(t),\ u_{2}(t)=\eta_{2}(t),\
t\in[-\delta,0],\end{aligned}\right.$ (5.1)

and the cost functionals of the leader and follower are as follows: For
$i=1,2$,

$\displaystyle J_{i}(u_{1}(\cdot),u_{2}(\cdot))$
$\displaystyle=\mathbb{E}\bigg{[}\int_{0}^{T}\Big{[}X(t)^{\top}Q_{i}(t)X(t)+X(t-\delta)^{\top}\bar{Q}_{i}(t)X(t-\delta)+u_{i}(t)^{\top}R_{i}(t)u_{i}(t)$
(5.2)
$\displaystyle\qquad\qquad\quad+u_{i}(t-\delta)^{\top}\bar{R}_{i}(t)u_{i}(t-\delta)]dt+X(T)^{\top}GX(T)\bigg{]}.$

It is obvious that the coefficients satisfies (A1)-(A3), thus

$\Omega_{1}(t)\equiv
P_{1}(t+\delta)\bar{D}_{1}^{2}(t+\delta),\quad\Omega_{2}(t)=\Omega_{3}(t)\equiv
R_{2}(t)+\bar{R}_{2}(t+\delta),$

where $P_{1}(\cdot)$ is the solution to the following linear ODE:

$\left\\{\begin{aligned}
\dot{P}_{1}(t)&=-P_{1}(t)-Q_{1}(t)-\bar{Q}_{1}(t+\delta),\ t\in[0,T],\\\
P_{1}(T)&=G.\end{aligned}\right.$

It is easy to verify that
$P_{1}(t)=Ge^{T-t}+\int_{t}^{T}e^{s-t}[Q_{1}(s)+\bar{Q}_{1}(s+\delta)]ds>0$,
hence $\Omega_{1}>0$ and
$\Omega_{1}^{-1}(t)=P_{1}^{-1}(t+\delta)\bar{D}_{1}^{-2}(t+\delta)$. Now (4.4)
reads

$\left\\{\begin{aligned}
\dot{P}_{2}(t)&=-P_{2}(t)-Q_{2}(t)-\bar{Q}_{2}(t+\delta),\ t\in[0,T],\\\
P_{2}(T)&=G,\end{aligned}\right.$

and it follows that
$P_{2}(t)=Ge^{T-t}+\int_{t}^{T}e^{s-t}[Q_{2}(s)+\bar{Q}_{2}(s+\delta)]ds=P_{1}(t)>0$.
For simplicity, assume
$\bar{B}_{1}(t)=\bar{B}_{2}(t)=\bar{D}_{1}(t)=\bar{D}_{2}(t)=0$ for
$t\in[0,\delta]$. In this case,

$\displaystyle\begin{aligned}
&\mathcal{A}_{1}\equiv\mathcal{A}_{2}\equiv\mathcal{A}_{3}\equiv\bar{\mathcal{A}}_{2}\equiv\bar{\mathcal{A}}_{3}\equiv\mathcal{E}\equiv
0,\quad\quad\quad\quad\mathcal{D}\equiv\bar{\mathcal{D}}\equiv\mathcal{G}_{1}\equiv\mathcal{G}_{2}\equiv\mathcal{M}\equiv\bar{\mathcal{M}}\equiv
0,\\\
&\mathcal{B}(t)=\mathcal{C}(t)=\bar{\mathcal{C}}(t)=P_{1}^{-1}(t)I_{[\delta,T]}(t)\begin{bmatrix}1&-1\\\
-1&0\end{bmatrix},\quad\bar{\mathcal{A}}_{1}(t)=\begin{bmatrix}-1&0\\\
0&-1\end{bmatrix}.\end{aligned}$

Denote

$\displaystyle L(t)\triangleq\begin{bmatrix}L_{1}(t)&L_{2}(t)\\\
L_{2}(t)&L_{3}(t)\end{bmatrix},\quad\quad\quad\Pi(t,\theta)\triangleq\begin{bmatrix}\Pi_{1}(t,\theta)&\Pi_{2}(t,\theta)\\\
\Pi_{2}(t,\theta)&\Pi_{3}(t,\theta)\end{bmatrix},$

and note that

$\displaystyle[I-L(t)\bar{\mathcal{C}}(t)]^{-1}=\begin{bmatrix}1-L_{1}(t)P_{1}^{-1}(t)+L_{2}(t)P_{1}^{-1}(t)&L_{1}(t)P_{1}^{-1}(t)\\\
-L_{2}(t)P_{1}^{-1}(t)+L_{3}(t)P_{1}^{-1}(t)&1+L_{2}(t)P_{1}^{-1}(t)\end{bmatrix}^{-1}$
$\displaystyle=\frac{P_{1}(t)}{(P_{1}(t)+L_{2}(t))^{2}-L_{1}(t)(P_{1}(t)+L_{3}(t))}\begin{bmatrix}P_{1}(t)+L_{2}(t)&-L_{1}(t)\\\
L_{2}(t)-L_{3}(t)&P_{1}(t)+L_{2}(t)-L_{1}(t)\end{bmatrix},\ t\in[\delta,T].$

Hence we have (omitting $t$)

$\displaystyle[I-L\bar{\mathcal{C}}]^{-1}L\big{[}\bar{\mathcal{A}_{1}}+\bar{\mathcal{A}_{2}}+\mathcal{C}^{\top}L\big{]}$
$\displaystyle=\frac{P_{1}}{(P_{1}+L_{2})^{2}-L_{1}(P_{1}+L_{3})}\begin{bmatrix}P_{1}L_{1}&P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}\\\
P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}&P_{1}L_{3}+L^{2}_{2}-L_{1}L_{3}\end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}-1+P_{1}^{-1}(L_{1}-L_{2})&P_{1}^{-1}(L_{2}-L_{3})\\\
-P_{1}^{-1}L_{1}&-1-P_{1}^{-1}L_{2}\end{bmatrix},\ t\in[\delta,T].$

It follows that for $t\in[\delta,T]$,

$\displaystyle\Xi_{1}(t)$
$\displaystyle=P_{1}^{-1}\begin{bmatrix}L_{1}-L_{2}&L_{2}-L_{3}\\\
-L_{1}&-L_{2}\end{bmatrix}+\frac{1}{(P_{1}+L_{2})^{2}-L_{1}(P_{1}+L_{3})}\begin{bmatrix}1&-1\\\
-1&0\end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}P_{1}L_{1}&P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}\\\
P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}&P_{1}L_{3}+L^{2}_{2}-L_{1}L_{3}\end{bmatrix}\begin{bmatrix}-1+P_{1}^{-1}(L_{1}-L_{2})&P_{1}^{-1}(L_{2}-L_{3})\\\
-P_{1}^{-1}L_{1}&-1-P_{1}^{-1}L_{2}\end{bmatrix},$ $\displaystyle\Xi_{2}(t)$
$\displaystyle=P_{1}^{-1}\begin{bmatrix}-1&1\\\
1&0\end{bmatrix}-\frac{P_{1}^{-1}}{(P_{1}+L_{2})^{2}-L_{1}(P_{1}+L_{3})}\begin{bmatrix}1&-1\\\
-1&0\\\ \end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}P_{1}+L_{2}&-L_{1}\\\
L_{2}-L_{3}&P_{1}+L_{2}-L_{1}\end{bmatrix}\begin{bmatrix}L_{1}-L_{2}&-L_{1}\\\
L_{2}-L_{3}&-L_{2}\end{bmatrix},$ $\displaystyle\Xi_{3}(t)$
$\displaystyle=-P_{1}^{-1}\begin{bmatrix}L^{2}_{1}-2L_{1}L_{2}&L_{1}L_{2}-L^{2}_{2}-L_{1}L_{3}\\\
L_{1}L_{2}-L^{2}_{2}-L_{1}L_{3}&L^{2}_{2}-2L_{2}L_{3}\end{bmatrix}+\begin{bmatrix}L_{1}&L_{2}\\\
L_{2}&L_{3}\end{bmatrix}$
$\displaystyle\quad-\frac{P_{1}}{(P_{1}+L_{2})^{2}-L_{1}(P_{1}+L_{3})}\begin{bmatrix}-1+P_{1}^{-1}(L_{1}-L_{2})&-P_{1}^{-1}L_{1}\\\
P_{1}^{-1}(L_{2}-L_{3})&-1-P_{1}^{-1}L_{2}\end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}P_{1}L_{1}&P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}\\\
P_{1}L_{2}+L^{2}_{2}-L_{1}L_{3}&P_{1}L_{3}+L^{2}_{2}-L_{1}L_{3}\end{bmatrix}\begin{bmatrix}-1+P_{1}^{-1}(L_{1}-L_{2})&P_{1}^{-1}(L_{2}-L_{3})\\\
-P_{1}^{-1}L_{1}&-1-P_{1}^{-1}L_{2}\end{bmatrix}.$

Then (4.10) takes the following form:

$\left\\{\begin{aligned} \dot{L}(t)&=\Pi(t,t+\delta)I_{[0,T-\delta]}(t)-L(t),\
t\in[0,T],\\\ L(T)&=0,\\\ [I-L&(t)\bar{\mathcal{C}}(t)]^{-1}\ \mbox{exists},\
t\in[0,T].\end{aligned}\right.$ (5.3)

and $\Pi(\cdot,\cdot)$ satisfies

$\left\\{\begin{aligned} -\frac{\partial\Pi(t,\theta)}{\partial t}&=0,\\\
\Pi(t,t)&=\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{1}(t)+\Xi_{1}(t)^{\top}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\\\ &\quad+\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{2}(t)\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta+\Xi_{3}(t).\end{aligned}\right.$ (5.4)

From the above, we can easily see that even though in the simple one-
dimensional case, the components $L_{1}(\cdot),L_{2}(\cdot)$ and
$L_{3}(\cdot)$ in the matrix equation (5.3) are heavily cross-coupled. Hence
it is more complex to derive the analytical solution for the general case of
(4.10) and (4.11). However, Our example is very special, coincidentally we
find that $L(\cdot)=0$ and $\Pi(\cdot,\cdot)=0$ are the solutions to (5.3) and
(5.4), respectively. Hence the optimal open-loop strategy is

$\bar{u}_{1}(t)=\bar{u}_{2}(t)=0,\ a.e.\ t\in[0,T],\ \mathbb{P}\mbox{-}a.s.$

###  5.2 Resource allocation problem

In this section a modified model for a dynamic research and development
resource allocation problem under rivalry is considered (see Scherer [22]). It
is assumed that two firms, labeled as firm A and firm B, are competing with
each other for a share of the market for a specific consumer goods. The size
of the market is assumed to be fixed for the interval of consideration and it
is assumed that it will not change with time nor will it be affected by the
research and development effort of either firm. However, each firm’s share of
the market does depend on the quality of their product and in turn depends on
the research and development effort of each firm. It is further assumed that
the product is of such a nature that all other costs, such as the cost of
modifying plants, are negligible compared with the cost of research and
development. For highly technical products, this assumption may be fairly
reasonable.

Let $V_{A}(t)$ and $V_{B}(t)$ be the amounts of money invested in the research
and development by firm A and firm B at time $t$, respectively. Let $x(\cdot)$
be a certain kind of measurement of technical gap between these two firms. The
evolution of this gap is modeled by

$\dot{x}(t)=V_{B}^{\frac{1}{2}}(t-\delta)-\alpha
V_{A}^{\frac{1}{2}}(t-\delta),$ (5.5)

where $\delta$ is the time delay, which represents the time required for the
scientific study, i.e. if the firm invests in research at time $t-\delta$, he
will get a return at time $t$. The multiplying factor $\alpha\geq 1$ accounts
for the fact that it is easier for a developing firm, firm A, to catch up than
for firm B, which is technically advanced, to innovate.

It is assumed that the shares of the market are equal when there is no
difference in technical levels. When there is a gap between technical levels
of these two firms, a portion of the market of the firm lagging technically is
temporarily taken over by its rival. This portion is assumed to be
proportional to the square of the difference in technical levels. It is
assumed that there is no permanent or lasting effect of this take-over of the
market. In other words, the share of the market at each instant is determined
by the technical levels at that instant only. Thus, the revenues of these two
firms for finite horizon $[0,T]$ are

$\displaystyle
J_{A}=\int_{0}^{T}\bigg{[}\frac{V}{2x_{0}^{2}}(x^{2}(t)-x^{2}(t-\delta))+(V_{A}(t)-e^{\gamma\delta}V_{A}(t-\delta))\bigg{]}e^{-\gamma
t}dt,$ (5.6) $\displaystyle
J_{B}=\int_{0}^{T}\bigg{[}-\frac{V}{2x_{0}^{2}}(x^{2}(t)-e^{\gamma\delta}x^{2}(t-\delta))+(V_{B}(t)-e^{\frac{1}{2}\gamma\delta}V_{B}(t-\delta))\bigg{]}e^{-\gamma
t}dt,$

where $x_{0}$ is some constant such that, when $x$ reaches $x_{0}$, the market
is completely taken over by firm B, $V$ is the quasi-rent that is assumed to
be a constant and $\gamma$ is the discount rate.
$V_{A}(t)-e^{\gamma\delta}V_{A}(t-\delta)$,
$V_{B}(t)-e^{\frac{1}{2}\gamma\delta}V_{B}(t-\delta)$ represent the difference
of the scientific research for firm A, B in the time interval $[t-\delta,t]$,
respectively. $e^{\gamma\delta}$, $e^{\frac{1}{2}\gamma\delta}$ are the
investment weights, since the firm A lags technically, he should invest more
money in the scientific research and hence the weight is more high.
$\frac{x^{2}(t)-x^{2}(t-\delta)}{x_{0}^{2}}$ represents the technology gap
improvement ratio at time $t$ comparing with time $t-\delta$, since the firm B
has stronger technology strength, the weight $e^{\gamma\delta}$ is imposed on
the measurement of technology gap improvement ratio for firm B. In other word,
the firm B has to improve more technology strength to achieve the same
technology gap improvement ratio as the firm A. There is no doubt that both
companies want to minimize scientific research investment in order to obtain
more technological improvements, i.e. the firm A wants to minimize
$\frac{x^{2}(t)-x^{2}(t-\delta)}{x_{0}^{2}}$ while minimize
$V_{A}(t)-e^{\gamma\delta}V_{A}(t-\delta)$, and the firm B wants to maximize
$\frac{x^{2}(t)-e^{\gamma\delta}x^{2}(t-\delta)}{x_{0}^{2}}$ while minimize
$V_{B}(t)-e^{\frac{1}{2}\gamma\delta}V_{B}(t-\delta)$.

The above problem can be converted to a standard linear quadratic Stackelberg
differential game with time delay by the following change of variables:

$X(t)=e^{-\frac{\gamma t}{2}}x(t),\quad
u_{1}(t)=V_{A}^{\frac{1}{2}}(t)e^{-\frac{\gamma}{2}t},\quad
u_{2}(t)=V_{B}^{\frac{1}{2}}(t)e^{-\frac{\gamma}{2}t},\quad
Q=\frac{V}{2x_{0}^{2}},\quad\bar{Q}=\frac{e^{-\gamma\delta}V}{2x_{0}^{2}},$

then (5.5)-(LABEL:cost-1) become

$\left\\{\begin{aligned} \dot{X}(t)=&-\frac{\gamma}{2}X(t)-\alpha
e^{-\frac{\gamma\delta}{2}}u_{1}(t-\delta)+e^{-\frac{\gamma\delta}{2}}u_{2}(t-\delta),\
t\in[0,T],\\\
J_{A}=&\int_{0}^{T}\bigg{[}QX^{2}(t)-\bar{Q}X^{2}(t-\delta)+u_{1}^{2}(t)-u_{1}^{2}(t-\delta)\bigg{]}dt,\\\
J_{B}=&\int_{0}^{T}\bigg{[}-QX^{2}(t)+QX^{2}(t-\delta)+u_{2}^{2}(t)-e^{-\frac{\gamma\delta}{2}}u_{2}^{2}(t-\delta)\bigg{]}dt.\end{aligned}\right.$
(5.7)

There must be some uncertain factors affecting the scientific research output,
so we extend the above model to the stochastic case:

$\left\\{\begin{aligned} dX(t)&=\Big{[}-\frac{\gamma}{2}X(t)-\alpha
e^{-\frac{\gamma\delta}{2}}u_{1}(t-\delta)+e^{-\frac{\gamma\delta}{2}}u_{2}(t-\delta)\Big{]}dt\\\
&\quad+\big{[}CX(t)+\bar{D}_{1}u_{1}(t-\delta)+\bar{D}_{2}u_{2}(t-\delta)\big{]}dW(t),\
t\in[0,T],\\\
J_{A}&=\mathbb{E}\int_{0}^{T}\bigg{[}QX^{2}(t)-\bar{Q}X^{2}(t-\delta)+u_{1}^{2}(t)-u_{1}^{2}(t-\delta)\bigg{]}dt,\\\
J_{B}&=\mathbb{E}\int_{0}^{T}\bigg{[}-QX^{2}(t)+QX^{2}(t-\delta)+u_{2}^{2}(t)-e^{-\frac{\gamma\delta}{2}}u_{2}^{2}(t-\delta)\bigg{]}dt,\end{aligned}\right.$
(5.8)

where $C,\bar{D}_{1},\bar{D}_{2}$ are all constants satisfying

$\bar{D}_{1}\neq
0,\quad\bar{B}_{1}+C\bar{D}_{1}=0,\quad\bar{B}_{2}+C\bar{D}_{2}\neq 0.$

Now we apply the theoretical results in the above two sections to seek the
optimal open-loop strategy. The coefficients take the following values:

$\displaystyle A(t)=-\frac{\gamma}{2},\quad\bar{B}_{1}(t)=-\alpha
e^{-\frac{\gamma\delta}{2}},\quad\bar{B}_{2}(t)=e^{-\frac{\gamma\delta}{2}},\quad\bar{A}(t)=\bar{C}(t)=0,$
$\displaystyle Q_{1}(t)=Q,\quad\bar{Q}_{1}(t)=-\bar{Q},\quad
R_{1}(t)=1,\quad\bar{R}_{1}(t)=-1,\quad G_{1}=0,$ $\displaystyle
Q_{2}(t)=-Q,\quad\bar{Q}_{2}(t)=Q,\quad
R_{2}(t)=1,\quad\bar{R}_{2}(t)=-e^{-\frac{\gamma\delta}{2}},\quad G_{2}=0.$

Apparently the coefficients satisfies (A1)-(A3), and now

$\Omega_{1}(t)=P_{1}(t+\delta)\bar{D}_{1}^{2}(t+\delta),\quad\Omega_{2}(t)=\Omega_{3}(t)=1-e^{-\frac{\gamma\delta}{2}},$

where $P_{1}(\cdot)$ is the solution the following linear ODE:

$\left\\{\begin{aligned} \dot{P}_{1}(t)&=(\gamma-C^{2})P_{1}(t)-Q+\bar{Q},\
t\in[0,T],\\\ P_{1}(T)&=0.\end{aligned}\right.$ (5.9)

It is easy to verify that
$P_{1}(t)=(Q-\bar{Q})\int_{t}^{T}e^{(C^{2}-\gamma)(s-t)}ds>0$, hence
$\Omega_{1}>0$ and
$\Omega_{1}^{-1}(t)=P_{1}^{-1}(t+\delta)\bar{D}_{1}^{-2}(t+\delta)$. Now (4.4)
reads

$\left\\{\begin{aligned} \dot{P}_{2}(t)&=(\gamma-C^{2})P_{2}(t),\
t\in[0,T],\\\ P_{2}(T)&=0,\end{aligned}\right.$ (5.10)

and it follows that $P_{2}(t)\equiv 0$. In this case,

$\displaystyle\begin{aligned}
&\mathcal{A}_{2}\equiv\mathcal{A}_{3}\equiv\bar{\mathcal{A}}_{2}\equiv\bar{\mathcal{A}}_{3}\equiv\mathcal{E}\equiv
0,\quad\quad\quad\quad\mathcal{G}_{1}\equiv
0,\quad\quad\quad\quad\mathcal{A}_{1}(t)=-\frac{\gamma}{2}\begin{bmatrix}1&0\\\
0&1\end{bmatrix},\\\
&\mathcal{B}(t)=-C^{2}P_{1}^{-1}(t)I_{[\delta,T]}(t)\begin{bmatrix}0&1\\\
1&0\end{bmatrix},\quad\quad\quad\quad\mathcal{C}(t)=CP_{1}^{-1}(t)I_{[\delta,T]}(t)\begin{bmatrix}0&1\\\
1&0\end{bmatrix},\\\ &\bar{\mathcal{A}}_{1}(t)=\begin{bmatrix}C&0\\\
0&C\end{bmatrix},\quad\quad\quad\quad\quad\quad\bar{\mathcal{C}}(t)=-P_{1}^{-1}(t)I_{[\delta,T]}(t)\begin{bmatrix}0&1\\\
1&0\end{bmatrix},\\\ &\mathcal{D}(t)=\begin{bmatrix}0\\\
\bar{B}_{2}+C\bar{D}_{2}I_{[\delta,T]}(t)\end{bmatrix},\quad\quad\quad\quad\quad\quad\mathcal{G}_{2}(t)=\begin{bmatrix}-P_{1}(t)(\bar{B}_{2}+C\bar{D}_{2})\\\
0\end{bmatrix}.\end{aligned}$

Denote

$\displaystyle L(t)\triangleq\begin{bmatrix}L_{1}(t)&L_{2}(t)\\\
L_{2}(t)&L_{3}(t)\end{bmatrix},\quad\quad\quad\Pi(t,\theta)\triangleq\begin{bmatrix}\Pi_{1}(t,\theta)&\Pi_{2}(t,\theta)\\\
\Pi_{2}(t,\theta)&\Pi_{3}(t,\theta)\end{bmatrix}.$

Similar to Section 5.1, with some computations, we can get for
$t\in[\delta,T]$,

$\displaystyle[I-L(t)\bar{\mathcal{C}}(t)]^{-1}$
$\displaystyle=\frac{P_{1}(t)}{L_{1}(t)L_{3}(t)-(P_{1}(t)+L_{2}(t))^{2}}\begin{bmatrix}-P_{1}(t)-L_{2}(t)&L_{1}(t)\\\
L_{3}(t)&-P_{1}(t)-L_{2}(t)\end{bmatrix}.$

It follows that for $t\in[\delta,T]$ (omitting $t$)

$\displaystyle\Xi_{1}(t)$
$\displaystyle=-C^{2}P_{1}^{-1}\begin{bmatrix}L_{2}&L_{3}\\\
L_{1}&L_{2}\end{bmatrix}+\frac{C^{2}P_{1}^{-1}}{L_{1}L_{3}-(P_{1}+L_{2})^{2}}\begin{bmatrix}L_{3}&-P_{1}-L_{2}\\\
-P_{1}-L_{2}&L_{1}\end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}L_{1}&L_{2}\\\
L_{2}&L_{3}\end{bmatrix}\begin{bmatrix}P_{1}+L_{2}&L_{3}\\\
L_{1}&P_{1}+L_{2}\end{bmatrix}-[1-e^{-\frac{\gamma\delta}{2}}]^{-1}(\bar{B}_{2}+C\bar{D}_{2})^{2}\begin{bmatrix}0&0\\\
P_{1}+L_{2}&L_{3}\end{bmatrix},$ $\displaystyle\Xi_{2}(t)$
$\displaystyle=C^{2}P_{1}^{-1}\begin{bmatrix}0&1\\\
1&0\end{bmatrix}-\frac{C^{2}P_{1}^{-1}}{L_{1}L_{3}-(P_{1}+L_{2})^{2}}\begin{bmatrix}L_{3}&-P_{1}-L_{2}\\\
-P_{1}-L_{2}&L_{1}\\\ \end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}L_{2}&L_{1}\\\
L_{3}&L_{2}\end{bmatrix}+[1-e^{-\frac{\gamma\delta}{2}}]^{-1}(\bar{B}_{2}+C\bar{D}_{2})^{2}\begin{bmatrix}0&0\\\
0&1\end{bmatrix},$ $\displaystyle\Xi_{3}(t)$
$\displaystyle=C^{2}P_{1}^{-1}\begin{bmatrix}2L_{1}L_{2}&L^{2}_{2}+L_{1}L_{3}\\\
L^{2}_{2}+L_{1}L_{3}&2L_{2}L_{3}\end{bmatrix}+[1-e^{-\frac{\gamma\delta}{2}}]^{-1}(\bar{B}_{2}+C\bar{D}_{2})^{2}$
$\displaystyle\quad\times\begin{bmatrix}(P_{1}+L_{2})^{2}&L_{3}(P_{1}+L_{2})\\\
L_{3}(P_{1}+L_{2})&L_{3}^{2}\end{bmatrix}+C^{2}\begin{bmatrix}L_{1}&L_{2}\\\
L_{2}&L_{3}\end{bmatrix}$
$\displaystyle\quad-\frac{C^{2}P_{1}^{-1}}{L_{1}L_{3}-(P_{1}+L_{2})^{2}}\begin{bmatrix}P_{1}+L_{2}&L_{1}\\\
L_{3}&P_{1}+L_{2}\end{bmatrix}$
$\displaystyle\quad\times\begin{bmatrix}-P_{1}-L_{2}&L_{1}\\\
L_{3}&-P_{1}-L_{2}\end{bmatrix}\begin{bmatrix}L_{1}&L_{2}\\\
L_{2}&L_{3}\end{bmatrix}\begin{bmatrix}P_{1}+L_{2}&L_{3}\\\
L_{1}&P_{1}+L_{2}\end{bmatrix}.$

Then (4.10) becomes:

$\left\\{\begin{aligned}
\dot{L}(t)&=(\gamma-C^{2})L(t)+\Pi(t,t+\delta)I_{[0,T-\delta]}(t),\
t\in[\delta,T],\\\ L(T)&=0,\\\ [I-L&(t)\bar{\mathcal{C}}(t)]^{-1}\
\mbox{exists},\ t\in[0,T].\end{aligned}\right.$ (5.11)

and $\Pi(\cdot,\cdot)$ satisfies

$\left\\{\begin{aligned} -\frac{\partial\Pi(t,\theta)}{\partial
t}=&-\gamma\Pi(t,\theta),\\\ \Pi(t,t)=&\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{1}(t)+\Xi_{1}(t)^{\top}\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\\\ &+\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\Xi_{2}(t)\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta+\Xi_{3}(t).\end{aligned}\right.$ (5.12)

Because the components $L_{1}(\cdot),L_{2}(\cdot)$ and $L_{3}(\cdot)$ in the
matrix equation are coupled, hence we can not give the analytical solution to
(5.11).

Moreover, the optimal open-loop strategy is as follows:

$\displaystyle\bar{u}_{1}(t-\delta)$
$\displaystyle=K_{1}(t-\delta)\hat{\phi}(t|t-\delta),\ a.e.\
t\in[\delta,T+\delta],\ \mathbb{P}\mbox{-}a.s.,$
$\displaystyle\bar{u}_{2}(t-\delta)$
$\displaystyle=K_{2}(t-\delta)\hat{\phi}(t|t-\delta),\ a.e.\
t\in[\delta,T+\delta],\ \mathbb{P}\mbox{-}a.s.,$

where

$\displaystyle K_{1}(t-\delta)$
$\displaystyle\triangleq\Omega_{1}^{-1}(t-\delta)\bigg{\\{}\bar{D}_{1}\bar{D}_{2}P_{1}(t)\Omega_{3}^{-1}(t-\delta)(\bar{B}_{2}+C\bar{D}_{2})\big{[}L_{2}(t)+P_{1}(t),L_{3}(t)\big{]}$
$\displaystyle\quad-\big{[}\bar{B}_{1}L_{1}(t),\bar{B}_{1}L_{2}(t)\big{]}-\frac{C\bar{D}_{1}}{L_{1}(t)L_{3}(t)-(P_{1}(t)+L_{2}(t))^{2}}\big{[}-P_{1}(t)-L_{2}(t),L_{1}(t)\big{]}$
$\displaystyle\qquad\times\begin{bmatrix}L_{1}(t)&L_{2}(t)\\\
L_{2}(t)&L_{3}(t)\end{bmatrix}\begin{bmatrix}P_{1}(t)+L_{2}(t)&L_{3}(t)\\\
L_{1}(t)&P_{1}(t)+L_{2}(t)\end{bmatrix}$
$\displaystyle\quad-\bigg{[}\bar{D}_{1}\bar{D}_{2}P_{1}(t)\Omega_{3}^{-1}(t-\delta)\big{[}0,\bar{B}_{2}+C\bar{D}_{2}\big{]}-\big{[}\bar{B}_{1},0\big{]}$
$\displaystyle\qquad-\frac{C\bar{D}_{1}}{L_{1}(t)L_{3}(t)-(P_{1}(t)+L_{2}(t))^{2}}\big{[}-P_{1}(t)-L_{2}(t),L_{1}(t)\big{]}\begin{bmatrix}L_{2}(t)&L_{1}(t)\\\
L_{3}(t)&L_{2}(t)\end{bmatrix}\bigg{]}$
$\displaystyle\qquad\times\int_{t}^{(t+\delta)\wedge
T}\Pi(t,\theta)d\theta\bigg{\\}},$ $\displaystyle K_{2}(t-\delta)$
$\displaystyle\triangleq-(\bar{B}_{2}+C\bar{D}_{2})\Omega_{3}^{-1}(t-\delta)\begin{bmatrix}L_{2}(t)+P_{1}(t)-\int_{t}^{(t+\delta)\wedge
T}\Pi_{2}(t,\theta)d\theta\\\ L_{3}(t)-\int_{t}^{(t+\delta)\wedge
T}\Pi_{3}(t,\theta)d\theta\end{bmatrix}^{\top}.$

Finally, we give some simulation results. The following algorithm schemes are
designed to simulate the numerical solution to (5.11):

Startinput $T$, $\delta$, $N$ $\Delta=\frac{T}{N+1}$ $d=\frac{\delta}{\Delta}$
$k=N+1$ $k>N-d+1$ $L(k)=0$ $\Pi(k,k+d)=0$ $k=k-1$ $k\geq 0$
$L(k)=L(k+1)+(C^{2}-\gamma)\Delta L(k+1)-\Delta\Pi(k+1,k+1+d)$ $j=k+2$
$mid\Pi=0$ $j\leq\min(N+1,k+1+d)$ $\Pi(k+1,k+1)=\Delta
mid\Pi\Xi_{1}(k+1)+\Delta\Xi_{1}(k+1)mid\Pi+\Delta^{2}mid\Pi\Xi_{2}(k+1)mid\Pi+\Xi_{3}(k+1)$
$i=1$ $mid\Pi=mid\Pi+\Pi(k+1,j)$ $j=j+1$ $i\leq\min(d,N+1-k)$
$\Pi(k,k+i)=(1-\gamma\Delta)\Pi(k+1,k+i)$ $i=i+1$ $k=k-1$
Stopyesnonoyesyesnonoyes Figure 1: Algorithm scheme of the solution to (5.11)

Let $\delta=1$, $\gamma=0.02$, $V=10$, $x_{0}=10$, $\alpha=2$,
$C=-\frac{\sqrt{\gamma}}{2}$, $\bar{D}_{1}=-\bar{B}_{1}C^{-1}$,
$\bar{D}_{2}=e^{\gamma\delta}$, $Q_{1}=\frac{V}{2x_{0}^{2}}$,
$\bar{Q}_{1}=-\frac{e^{-\gamma\delta}V}{2x_{0}^{2}}$, $t_{0}=0$, $T=10$,
$\varphi(0)=1$. Using the above algorithm scheme, we can plot the pictures of
$P_{1}(\cdot),L_{1}(\cdot),L_{2}(\cdot),L_{3}(\cdot)$ as follows:

(a) The solution to linear ODE (5.9)

(b) The solution to (5.11) — $L_{1}(t)$

(c) The solution to (5.11) — $L_{2}(t)$

(d) The solution to (5.11) — $L_{3}(t)$

The simulation results of the optimal strategies $u_{1}(\cdot)$,
$u_{2}(\cdot)$ are as follows:

(e) The optimal strategy $u_{1}(t)$

(f) The optimal strategy $u_{2}(t)$

In the above two figures, the left/right figure draws the optimal strategy of
the follower/leader. Judging from the trend of the two figures, the optimal
strategy of the follower follows the leader, which is consistent with the
essence of the leader-follower problem. Furthermore, the value of
$u_{2}(\cdot)$ seems to be greater than $u_{1}(\cdot)$, this is rational
because the leader wants to keep the technology ahead and so he should invest
more money than the follower.

##  6 Concluding remarks

This paper is an extension of [28]. Our model is quite general, in which the
state equations of leader and follower both contain state delay and control
delay, moreover, which both enter into the diffusion. First we deal with the
optimization problem of the follower, which is an linear quadratic stochastic
optimal control problem with time delay, for any choice of the leader. By
introducing a Pseudo-Riccati equation (see (3.5)), we give the sufficient and
necessary condition of the solvability for follower’s problem (see Theorem
3.2). Next we address the optimization problem of the leader, which is an
linear quadratic stochastic optimal control problem with a state equation
formed by an SDDE and an ABSDE. We stack the forward variables and the
backward variables obtained in the optimization of the follower and the
leader, then introduce a special matrix equation (see (4.10)), and finally we
derive the necessary condition of the solvability for the leader’s problem
(see Theorem 4.2). In summary, the open-loop stackelberg strategy is shown in
Theorem 4.3. At the end of this paper, we take two examples to illustrate the
applications of the above theoretical results.

The assumptions ${\bf(A1)\mbox{-}(A3)}$ could be weakened, and some more
general cases can be dealt with. Some reasonable relations among the adjoint
variables and the state variable (including its delayed and time-advanced
terms) should be proposed and investigated carefully. The solvability of
(Pseudo-)Riccati equations are interesting. However, these topics are
difficult and very challenging. We will consider them in the near future.

## References

  * [1] Y. F. Bai, Z. B. Zhou, H. L. Xiao and R. Gao, A Stackelberg reinsurance investment game with asymmetric information and delay. _Optimization_ , 2020. DOI: 10.1080/02331934.2020.1777125
  * [2] T. Başar, Stochastic stagewise Stackelberg strategies for linear quadratic systems. _In: Stochastic Control Theory and Stochastic Differential Systems._ Berlin: Springer, 1979.
  * [3] T. Başar, G. J. Olsder, _Dynamic Noncooperative Game Theory, 2nd Edition_. Philadelphia: SIAM, 1998.
  * [4] A. Bensoussan, M. H. M. Chau, Y. Lai and S. C. P. Yam, Linear-quadratic mean field Stackelberg games with state and control delays. _SIAM J. Control Optim._ , 2017, 55(4): 2748-2781.
  * [5] A. Bensoussan, M. H. M. Chau and S. C. P. Yam, Mean field Stackelberg games: aggregation of delayed instructions. _SIAM J. Control Optim._ , 2015, 53(4): 2237-2266.
  * [6] A. Bensoussan, S. K. Chen and S. P. Sethi, The maximum principle for global solutions of stochastic Stackelberg differential games. _SIAM J. Control Optim._ , 2015, 53(4): 1956-1981.
  * [7] D. Castanon, M. Athans, On stochastic dynamic Stackelberg strategies. _Automatica_ , 1976: 12(2), 177-183.
  * [8] L. Chen, Z. Wu, Maximum principle for the stochastic optimal control problem with delay and application. _Automatica_ , 2010, 46(6): 1074-1080.
  * [9] G. Freiling, G. Jank and S. R. Lee, Existence and uniqueness of open-loop Stackelberg equilibria in linear-quadratic differential games. _J. Optim. Theory Appl._ , 2001, 110(3): 515-544.
  * [10] J. Harband, Existence of monotonic solutions of a nonlinear car-follwing equation. _J. Math. Anal. Appl._ , 1977, 57(2): 257-72.
  * [11] J. H. Huang, X. Li and J. T. Shi, Forward-backward linear quadratic stochastic optimal control problem with delay, _Syst. $\&$ Control Lett._, 2012, 61(5): 623-630.
  * [12] T. Ishida, E. Shimemura, Sufficient conditons for the team-optimal closed-loop Stackelberg strategies in linear differential games with time-delay. _Int. J. Control_ , 1983, 37(3): 441-454.
  * [13] T. Ishida, E. Shimemura, Open-loop Stackelberg strategies in a linear-quadratic differential game with time delay. _Int. J. Control_ , 1987, 45(5): 1847-1855.
  * [14] X. Q. Li, W. Wang, J. J. Xu and H. S. Zhang, A Stackelberg strategy for continuous-time mixed $H_{2}/H_{\infty}$ control problem with time delay. _Control Theory Tech._ , 2018, 16(3): 191-202.
  * [15] S. E. A. Mohammed, _Stochastic Functional Differential Equations_. Pitman, 1984.
  * [16] S. E. A. Mohammed, _Stochastic differential equations with memory: theory, examples and applications_. Progress in Probability, Stochastic Analysis and Related Topics 6, The Geilo Workshop, Birkhauser, 1996.
  * [17] B. Øksendal, L. Sandal and J. Ubøe, Stackelberg equilibria in a continuous-time vertical contracting model with uncertain demand and delayed information. _J. App. Proba._ , 2014. 51(A): 213-226.
  * [18] B. Øksendal, A. Sulem, A maximum principle for optimal control of stochastic systems with delay, with applications to finance. In: _Optimal Control and Partial Differential Equations_ , J. M. Menaldi, E. Rofman, A. Sulem (Eds.), ISO Press, Amsterdam, 2000, 64-79.
  * [19] L. P. Pan, J. M. Yong, A differential game with multi-level of hierarchy. _J. Math. Anal. Appl._ , 1991, 161(2): 522-544.
  * [20] G. P. Papavassilopoulos, J. B. Cruz, Nonclassical control problems and Stackelberg games. _IEEE Trans. Autom. Control_ , 1979, 24(2): 155-166.
  * [21] S. G. Peng, Z. Yang, Anticipated backward stochastic differential equations. _Ann. Proba._ , 2009, 37(3): 877-902.
  * [22] F. M. Scherer, Research and development resource allocation under rivalry. _Quart. J. Econ._. 1967, 81:359-394.
  * [23] J. T. Shi, G. C. Wang and J. Xiong, Leader-follower stochastic differential game with asymmetric information and applications. _Automatica_ , 2016, 63: 60-73.
  * [24] J. T. Shi, G. C. Wang, and J. Xiong, Linear-quadratic stochastic Stackelberg differential game with asymmetric information. _Sci. China Infor. Sci._ , 2017, 60: 092202:1-092202:15.
  * [25] J. T. Shi, G. C. Wang, and J. Xiong, Stochastic linear quadratic Stackelberg differential game with overlapping information. _ESAIM: Control, Optim. Calcu. Varia._ , 2020, 26: Article Number 83.
  * [26] M. Simaan, J. B. Cruz, On the Stackelberg strategy in nonzero-sum games. _J. Optim. Theory Appl._ , 1973, 11(5): 533-555.
  * [27] H. von Stackelberg, _Marktform und Gleichgewicht_. Vienna: Springer, 1934.
  * [28] J. J. Xu, J. T. Shi and H. S. Zhang, A leader-follower stochastic linear quadratic differential game with time delay. _Sci. China Inf. Sci._ , 2018, 61: 112202:1-112202:13.
  * [29] J. J. Xu, H. S. Zhang, Sufficient and necessary open-loop Stackelberg strategy for two-player game with time delay. _IEEE Trans. Cyber._ , 2016, 46(2): 438-449.
  * [30] J. M. Yong, A leader-follower stochastic linear quadratic differential game. _SIAM J. Control Optim._ , 2002, 41(4): 1015-1041.
  * [31] H. S. Zhang, J. J. Xu, Control for Itô stochastic systems with input delay. _IEEE Trans. Autom. Control_ , 2017, 62(1): 350-365.