# An adapted solution of a fractional backward stochastic differential
equation

Arzu Ahmadova Corresponding author. Email<EMAIL_ADDRESS>Department
of Mathematics, Eastern Mediterranean University, Mersin 10, 99628, T.R.
Northern Cyprus  Nazim I. Mahmudov  Email<EMAIL_ADDRESS>Department of Mathematics, Eastern Mediterranean University, Mersin 10, 99628,
T.R. Northern Cyprus

###### Abstract

Our aim in this paper is to deal with a new type differential equation so-
called Caputo fractional backward stochastic differential equations (for short
Caputo fBSDEs) and study the global existence and uniqueness of an adapted
solution to Caputo fBSDEs of order $\alpha\in(\frac{1}{2},1)$ whose
coefficients satisfy Lipschitz condition by applying fundamental lemma which
plays a crucial role in the theory of Caputo fBSDEs. The interesting point
here is to use a new weighted norm in square-integrable measurable function
space for fundamental lemma and therefore for the whole paper. For this class
of systems, we then show the coincidence between the notion of stochastic
Volterra integral equation and mild solution.

Keywords: Fractional backward stochastic differential equations, backward
stochastic nonlinear Volterra integral equation, existence and uniqueness,
adapted process, weighted norm

##  1 Introduction

Fractional stochastic differential equations (FSDEs) which are a
generalization of differential equations by the use of fractional and
stochastic calculus are more popular due to their applications in mathematical
modelling and finance [1, 2, 3]. Recently, FSDEs are intensively applied to
model mathematical problems in finance [4, 5], dynamics of complex systems in
engineering [6] and other areas [7, 8]. Several results have been investigated
on the qualitative theory and applications of fractional stochastic
differential equations (FSDEs) [9]-[16].

Studying backward stochastic differential equations (BSDEs) has necessary
applications in stochastic optimal control, stochastic differential game,
probabilistic formula for the solutions of quasilinear partial differential
equations and financial markets. The adapted solution for a linear BSDE which
arises as the adjoint process for a stochastic control problem was first
investigated by Bismut [17] in 1973, then by Bensousssan [18], and while
Pardoux and Peng [19] first studied the result for the existence and
uniqueness of an adapted solution to a continuous general non-linear BSDE
which is a terminal value problem for an Itô type stochastic differential
equation under the uniform Lipschitz conditions of the following form:

$\displaystyle\begin{cases*}\mathrm{d}Y(t)=h(t,Y(t),Z(t))\mathrm{d}t+Z(t)\mathrm{d}W(t),t\in[0,T],\\\
Y(T)=\xi.\end{cases*}$

They established existence and uniqueness of an adapted solution using
Bihari’s inequality which is the most important generalization of the
Gronwall-Bellman inequality.

Since then, the theory of BSDE became a powerful tool in many fields such as
mathematical finance, optimal control, semi-linear and quasi-linear partial
differential equations [20, 21, 22, 23]. Later Peng and Pardoux developed the
theory and applications of continuous BSDEs in a series of papers [19, 24, 25,
26, 27] under the assumptions the coefficients satisfy Lipschitz conditions.
Tang and Li [28] then applied the idea of Peng [19] to get the first result on
the existence of an adapted solution to a BSDE with Poisson jumps for a fixed
terminal time and with Lipschitz conditions. Moreover, Mao [29] obtained more
general result than that of Pardoux and Peng [19] in which he proved existence
and uniqueness result under mild assumptions applying Bihari’s inequality
which was the key tool in the proof.

A few years later, Lin [30] considered the following backward stochastic
nonlinear Volterra integral equation

$X(t)+\int_{t}^{T}f(t,s,X(s),Z(t,s))\mathrm{d}s+\int_{t}^{T}\left[g(t,s,X(s))+Z(t,s)\right]\mathrm{d}W(s)=X.$

His aim in [30] is to look for a pair $\left\\{X(s),Z(t,s)\right\\}$ which
requires this pair to be $\left\\{\mathscr{F}_{t\vee s}\right\\}$-adapted and
$Z(t,s)$ is related to $t$. This is the intersection point of our results on
linear Caputo fBSDEs with [30]. The author also defines
$Z(t,s)=\tilde{Z}(t,s)-g(t,s)$,
$(t,s)\in\mathcal{D}=\left\\{(t,s)\in\mathbb{R}^{2}_{+};0\leq t\leq s\leq
T\right\\}$ which we have defined it for linear Caputo fBSDEs. Another main
point of intersection with [30, 31, 32, 33, 34] is to use the well-known
extended martingale representation theorem in which we consider extended
martingale representation to an adapted solution
$\left\\{x(t),y(t,s)\right\\}$, $(t,s)\in\mathcal{D}$ for linear Caputo fBSDEs
as well.

In 2005, Mahmudov and Mckibben [31] studied the existence and uniqueness of
the adapted solution to a backward stochastic evolution equation in Hilbert
space in the following form of

$\displaystyle\begin{cases*}\mathrm{d}y(t)=-[Ay(t)+F(t,y(t),z(t))]\mathrm{d}t-[G(t,y(t))+z(t)]\mathrm{d}w(t),\\\
y(T)=\xi,\end{cases*}$

where $A$ is a linear operator which generates a $C_{0}$-semigroup. The
authors also applied Bihari’s inequality to show existence and uniqueness
under a non-Lipschitz condition. The main idea on this work is to establish
fundamental lemma which plays an important role in the theory of BSDEs.
Furthermore, they discussed a stochastic maximum principle for optimal control
problems in Hilbert space.

After a year, Yong [33] introduced backward stochastic Volterra integral
equation (BSVIEs) which takes the following form:

$Y(t)=f(t)-\int_{t}^{T}h(t,s,Y(s),Z(t,s),Z(s,t))\mathrm{d}s-\int_{t}^{T}Z(t,s)\mathrm{d}W(s),\quad
t\in[0,T].$

The interesting features are $Z(t,s)$ depends on $t$ and the drift depends on
both $Z(t,s)$ and $Z(s,t)$ in general. As applications of BSVIEs, Yong proved
comparison theorem for one-dimensional BSVIEs, and a Pontryagin type maximum
principle for optimal control of stochastic differential equations.

Recently, Wang and Yong [34] considered BSVIEs. By taking a look at special
case of BSVIEs they reduced the given BSVIEs to the following form:

$\displaystyle
Y(t)=\xi+\int_{t}^{T}g(s,Y(s),Z(t,s))\mathrm{d}s-\int_{t}^{T}Z(t,s)\mathrm{d}W(s),\quad
t\in[0,T].$ (1.1)

The unknown pair that they are looking for is
$\left(Y(\cdot),Z(\cdot,\cdot)\right)$ of stochastic processes. Eq. (1.1) is
comparable with the integral form of BSDE which takes the following form:

$Y(t)=\xi+\int_{t}^{T}g(s,Y(s),Z(s))\mathrm{d}s-\int_{t}^{T}Z(s)\mathrm{d}W(s),\quad
t\in[0,T].$ (1.2)

If (1.2) admits a unique $\mathscr{F}_{t}$-adapted solution
$(Y(\cdot),Z(\cdot))$ , then this solution is also an adapted solution of
(1.1) with $Z(t,s)=Z(s)$ and it turns out that BSVIE can be regarded as an
extension of BSDE.

To the best of our knowledge, we study a new fractional analogue of BSDEs
which is an untreated topic in the present literature and so-called Caputo
fractional backward stochastic differential equations of order
$\alpha\in(\frac{1}{2},1)$ on $[0,T]$ as follows:

$\begin{cases}\left(\prescript{C}{T}{D^{\alpha}_{t}}x\right)(t)=Ax(t)-f(t,x(t),y(t,s))-\left[g(t,s,x(t))+y(t,s)\right]\frac{\mathrm{d}w(t)}{\mathrm{d}t},\\\
x(T)=\xi.\end{cases}$ (1.3)

where $A$ is a $n\times n$ constant matrix and $(w(t))_{t\geq 0}$ denote a
standard $m$-dimensional Brownian motion defined on a complete probability
space $(\Omega,\mathscr{F},\mathbb{F},\mathbb{P})$ with natural filtration
$\mathbb{F}\coloneqq\left\\{\mathscr{F}_{t}\right\\}_{t\geq 0}$ satisfying
usual conditions. Moreover,
$f:[0,T]\times\mathbb{R}^{n}\times\mathbb{R}^{n\times m}\to\mathbb{R}^{n}$ and
$g:\mathcal{D}\times\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$ are assumed to be
measurable mappings and terminal value
$\xi\in\mathbb{L}^{2}(\Omega,\mathscr{F}_{T},\mathbb{P};\mathbb{R}^{n})$ is an
$\mathscr{F}_{T}$-measurable $\mathbb{R}^{n}$-valued random variable such that
$\textbf{E}\|\xi\|^{2}<\infty$. Let $x(t,\omega)=x(t)$ be a stochastic process
on $[0,T]$ and $\omega\in\Omega$. The expectation operator E is denoted by

$\left(\textbf{E}x\right)(t)\coloneqq\int_{\Omega}x(t)\mathbb{P}(\mathrm{d}\omega),\quad
t\in[0,T].$

Our aim in this paper is to look for a pair of stochastic processes
$\left\\{x(t),y(t,s);(t,s)\in\mathcal{D}\right\\}$ with values
$\mathbb{R}^{n}\times\mathbb{R}^{n\times m}$ which we require to be
$\mathscr{F}_{t\vee s}$ -adapted and satisfies (1.3) in the usual Itô’s sense.
Such a pair is called an adapted solution of the equation (1.3). Our main
result will be an existence and uniqueness result for an adapted pair
$\left\\{x(t),y(t,s);(t,s)\in\mathcal{D}\right\\}$ which solves (1.3). We
first derive representations of the adapted solution and then we present that
the operator is well-defined before proving contraction mapping principle.
Next we study existence and uniqueness results to the adapted solution of Eq.
(1.3) under Lipschitz condition.

Therefore, the plan of this paper is organized as below: In Section 2 we
introduce main definitions from stochastic and fractional calculus, and some
necessary inequalities from stochastic calculus which play a key role in
certain estimations of the main results. Section 3 is devoted to proving
global existence and uniqueness of the adapted solution to Caputo fBSDE (1.3)
of order $\alpha\in(\frac{1}{2},1)$ using the Banach’s fixed point approach.
Section 4 is dedicated to presenting the coincidence between the notion of
integral equation and mild solution. Finally, Section 5 is for conclusion and
introducing some open problems.

To conclude introductory section, we introduce some notations which will be
used throughout this paper. Let $\mathbb{R}^{n}$ be endowed with the standard
Euclidean norm of $x\in\mathbb{R}^{n}$ and $\langle x,\bar{x}\rangle$ denote
the inner product of $x,\bar{x}\in\mathbb{R}^{n}$. An element
$y\in\mathbb{R}^{n\times m}$ will be considered as a $n\times m$ matrix and
its Euclidean norm is defined by $\|y\|=\sqrt{\text{tr}(yy^{T})}$ and
$(x,y)=\text{tr}(xy^{*})$. For any given $0\leq t\leq T$, we denote by
$\mathbb{L}^{2}_{\mathscr{F}}(0,T;\mathbb{R}^{n})$ (accord.
$\mathbb{L}^{2}_{\mathscr{F}}(\mathcal{D};\mathbb{R}^{n\times m})$) the family
of $\mathbb{R}^{n}$-valued (resp. $\mathbb{R}^{n\times m}$-valued )
$\mathscr{F}_{t}$-adapted processes (resp. $\mathscr{F}_{t\vee s}$) which are
measurable and square integrable on $\Omega\times[0,T]$ with respect to
$\mathbb{P}\times\lambda$ where $\lambda$ denotes the Lebesgue measure on
$[0,T]$.

Let us recall some spaces. We define:

$\mathbb{L}^{2}_{\mathscr{F}}(0,T;\mathbb{R}^{n})=\left\\{\xi(\cdot)\in\mathbb{L}^{2}(0,T;\mathbb{R}^{n})\mid\xi(t)\quad\text{is}\quad\mathbb{R}^{n}-\text{valued}\quad\mathscr{F}_{T}-\text{adapted}\right\\}$;

$\mathbb{L}^{2}_{\mathscr{F}}(\mathcal{D};\mathbb{R}^{n\times
m})=\left\\{\xi:\mathcal{D}\times\Omega\to\mathbb{R}^{n\times
m}\mid\xi(\cdot)\quad\text{is}\quad\mathcal{B}([0,T])\otimes\mathscr{F}_{T}-\text{measurable}\right\\}$
with $\mathcal{B}$ be the Borel $\sigma$-field of $[0,T]$.

######  Definition 1.1.

Let $\frac{1}{2}<\alpha<1$ and $1\leq p<\infty$ be fixed. We say that a
measurable function $h:\Omega\times[0,T]\to\mathbb{R}^{n}$ belongs to
$\mathbb{L}^{p,\alpha}([0,T],\mathbb{R}^{n})$ if and only if the quantity

$\|h\|_{p,\alpha,t}\coloneqq\sup_{t\leq\tau\leq
T}\textbf{E}\left(\int_{\tau}^{T}\frac{\|h(s)\|^{p}}{(s-\tau)^{1-\alpha}}\mathrm{d}s\right)^{\frac{1}{p}}<\infty.$

######  Lemma 1.1.

Let $\frac{1}{2}<\alpha<1$ and $1\leq p<\infty$, $\mathbb{L}^{p,\alpha}$ is a
Banach space.

###### Proof.

Obviously, the defined weighted norm satisfies the conditions of norm in
$\mathbb{L}^{p,\alpha}$ space. It is only left to show that is complete.
Hence, a proof of this part is similar to the proof of Theorem 3.4 in [35], so
we omit it here. ∎

In particular, we consider $p=2$ and $\frac{1}{2}<\alpha<1$ throughout this
paper such that the weighted norm is defined by

$\|h\|_{2,\alpha,t}\coloneqq\sup_{t\leq\tau\leq
T}\textbf{E}\left(\int_{\tau}^{T}\frac{\|h(s)\|^{2}}{(s-\tau)^{1-\alpha}}\mathrm{d}s\right)^{\frac{1}{2}}.$
(1.4)

To show existence and uniqueness, we consider the following norm:

$\|h\|_{2,\alpha,0}\coloneqq\sup_{0\leq\tau\leq
T}\textbf{E}\left(\int_{\tau}^{T}\frac{\|h(s)\|^{2}}{(s-\tau)^{1-\alpha}}\mathrm{d}s\right)^{\frac{1}{2}}.$
(1.5)

We define
$M[t,T]\coloneqq\mathbb{L}^{2}_{\mathscr{F}}(t,T;\mathbb{R}^{n})\times\mathbb{L}^{2}_{\mathscr{F}}(\mathcal{D};\mathbb{R}^{n\times
m})$ to be a Banach space endowed with the norm

$\|(x,y)\|^{2}_{t}=\|x\|^{2}_{2,\alpha,t}+|\|y\||^{2}_{2,\alpha,t},$

where

$\displaystyle|\|y\||^{2}_{2,\alpha,t}\coloneqq\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|y(s,u)\|^{2}\mathrm{d}u\mathrm{d}s,\quad
t\leq s\leq u\leq T.$ (1.6)

Then
$M[0,T]\coloneqq\mathbb{L}^{2}_{\mathscr{F}}(0,T;\mathbb{R}^{n})\times\mathbb{L}^{2}_{\mathscr{F}}([0,T]^{2};\mathbb{R}^{n\times
m})$ is also Banach space equipped with the norm as below:

$\|(x,y)\|^{2}_{0}=\|x\|^{2}_{2,\alpha,0}+|\|y\||^{2}_{2,\alpha,0}.$

##  2 Fractional and Stochastic Calculus

We embark on this section by briefly introducing the essential structure of
fractional calculus and fractional operators, as well as some important
inequalities of stochastic calculus. For the more salient details on these
matters, see the textbooks [36, 37, 38, 39, 40] and research paper [41, 42].
We introduce the classical fractional operators.

Since we study backward stochastic fractional differential equation throughout
this research paper, we consider the right stochastic Riemann-Liouville and
Caputo fractional operators of order $\alpha>0$ on $[t,T]$.

  * •

the right Riemann-Liouville stochastic fractional derivative of order $\alpha$
is as follows:

$\displaystyle\prescript{}{t}{D}^{\alpha}_{T}x(t)=\frac{(-1)^{n}}{\Gamma(n-\alpha)}\left(\frac{d}{dt}\right)^{n}\int_{t}^{T}(s-t)^{n-\alpha-1}x(s)\mathrm{d}s,\quad\text{where}\quad
n-1<\alpha\leq n,\quad t<T.$

  * •

The right Riemann-Liouville stochastic fractional integral of order $\alpha$
is given by

$\prescript{}{t}{I}^{\alpha}_{T}x(t)=\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}x(s)\,\mathrm{d}s\\\
,\quad\text{for}\quad t<T,$ (2.1)

where $\Gamma$ is the gamma function.

  * •

The right Caputo stochastic fractional derivative operator of order $\alpha$
is defined by:

$\prescript{C}{t}{D}^{\alpha}_{T}x(t)=\frac{(-1)^{n}}{\Gamma(n-\alpha)}\int_{t}^{T}(s-t)^{n-\alpha-1}x^{(n)}(s)\mathrm{d}s,\quad\text{where}\quad
n-1<\alpha\leq n,\quad t<T.$

In particular, for $\alpha\in(0,1)$

$\prescript{}{t}{I}^{\alpha}_{T}\prescript{C}{t}{D}^{\alpha}_{T}x(t)=x(t)-x(T).$

  * •

the relationship between the Riemann–Liouville and Caputo fractional
derivatives are as follows:

$\prescript{C}{t}{D}^{\alpha}_{T}x(t)=\prescript{}{t}{D}^{\alpha}_{T}x(t)-\sum_{k=0}^{n-1}\frac{(T-t)^{k-\alpha}x^{(k)}(T)}{\Gamma(k-\alpha+1)}.$
(2.2)

  * •

the property of the Riemann–Liouville fractional integral operator and the
Caputo fractional derivative of order $\alpha$ is given by:

$\prescript{}{t}{I}^{\alpha}_{T}(\prescript{C}{t}{D}^{\alpha}_{T}x(t))=x(t)-\sum_{k=0}^{n-1}\frac{(T-t)^{k}x^{(k)}(T)}{\Gamma(k+1)}.$
(2.3)

The following lemma plays a crucial role on proofs of main results in Section
3 and 4.

######  Lemma 2.1 (Hölder’s inequality).

Suppose that $\alpha_{1}$,$\alpha_{2}>1$ and
$\frac{1}{\alpha_{1}}+\frac{1}{\alpha_{2}}=1$. If $|f(t)|^{\alpha_{1}}$,
$|g(t)|^{\alpha_{2}}\in L^{1}(\Omega)$, then $|f(t)g(t)|\in L^{1}(\Omega)$ and

$\int_{\Omega}|f(t)g(t)|\mathrm{d}t\leq\left(\int_{\Omega}|f(t)|^{\alpha_{1}}\mathrm{d}t\right)^{\frac{1}{\alpha_{1}}}\left(\int_{\Omega}|g(t)|^{\alpha_{2}}\mathrm{d}t\right)^{\frac{1}{\alpha_{2}}},$

where $L^{1}(\Omega)$ represents the Banach space of all Lebesgue measurable
functions $f:\Omega\to\mathbb{R}$ with
$\int_{\Omega}|f(t)|\mathrm{d}t<\infty$. Especially, when
$\alpha_{1}=\alpha_{2}=2$, the Hölder’s inequality reduces to the Cauchy-
Schwartz inequality

$\left(\int_{\Omega}|f(t)g(t)|\mathrm{d}t\right)^{2}\leq\int_{\Omega}|f(t)|^{2}\mathrm{d}t\int_{\Omega}|g(t)|^{2}\mathrm{d}t.$
(2.4)

######  Lemma 2.2 (Jensen’s inequality).

Let $n\in\mathbb{N}$, $q>1$ and $x_{i}\in\mathbb{R}_{+}$, $i=1,2,\ldots,n$.
Then, the following inequality holds true:

$\|\sum_{i=1}^{n}x_{i}\|^{q}\leq n^{q-1}\sum_{i=1}^{n}\|x_{i}\|^{q}.$

In particular, we consider the following inequality with $q=2$ within the
estimations on this paper:

$\|\sum_{i=1}^{n}x_{i}\|^{2}\leq n\sum_{i=1}^{n}\|x_{i}\|^{2}.$ (2.5)

The following inequalities are fundamental inequalities of stochastic
calculus. First, we introduce Jensen’s inequality in probability setting plays
an important role in the stochastic process theory. It has become a standard
result which appears in almost every introductory text in this field.

######  Theorem 2.1 (Jensen’s inequality in probabilistic setting ).

Let $X$ be an integrable real-valued random variable and $\varphi$ be a convex
function. Then:

$\varphi\left(\textbf{E}[X]\right)\leq\textbf{E}\left[\varphi(X)\right].$

In this probability setting, $\varphi$ is intended as the integral with
respect to an expected value E.

######  Theorem 2.2 (The law of total expectation).

Let $(\Omega,\mathscr{F}_{t},\mathbb{P})$ be a probability space on which two
sub $\sigma$-algebras $G_{1}\subset G_{2}\subset\mathscr{F}$ are defined. For
a random variable $X$ on such a space, the smoothing law states that if
$\textbf{E}[X]$ is defined, then

$\textbf{E}[\textbf{E}[X\mid\mathcal{G}_{2}]\mid\mathcal{G}_{1}]=\textbf{E}[X\mid\mathcal{G}_{1}],\quad\text{a.s.}.$

In special case, when $\mathcal{G}_{1}=\\{\emptyset,\Omega\\}$ and
$\mathcal{G}_{2}=\sigma(Y)$, the smoothing law reduces to

$\textbf{E}[\textbf{E}[X\mid Y]]=\textbf{E}[X].$

##  3 Main results for fractional backward stochastic differential equations

In this section, we study existence and uniqueness results to a mild solution
of the equation (1.3). To state our main results, we resort the following
assumptions which will be considered in Section 3 and 4. More precisely, let
us propose the standing assumptions for the function $f$ and $g$ as follows.

######  Assumption 3.1.

For all $x,\bar{x}\in\mathbb{R}^{n}$, $y,\bar{y}\in\mathbb{R}^{n\times m}$ and
$0\leq t\leq T$, there exists $c>0$ such that

$\displaystyle\|f(t,x,y)-f(t,\bar{x},\bar{y})\|^{2}\leq
c\left(\|x-\bar{x}\|^{2}+\|y-\bar{y}\|^{2}\right),\quad\text{a.s},$
$\displaystyle\|g(t,s,x)-g(t,s,\bar{x})\|^{2}\leq
c\|x-\bar{x}\|^{2}\quad\text{a.s}.$

######  Assumption 3.2.

$f(\cdot,0,0)$ is $\mathbb{L}^{2}(0,T;\mathbb{R}^{n})$ integrable i.e.

$\int_{0}^{T}\|f(r,0,0)\|^{2}\mathrm{d}r<\infty,$

and $g(\cdot,\cdot,0)$ is essentially bounded i.e.

$ess\sup\limits_{r\in[0,T]}\|g(r,u,0)\|<\infty.$

######  Definition 3.1.

A stochastic process
$\left\\{\left(x(t),y(t,s)\right),(t,s)\in\mathcal{D}\right\\}$ is called a
mild solution of (1.3) if

  * •

$\left(x,y\right)$ is adapted to $\left\\{\mathscr{F}_{t}\right\\}_{t\geq 0}$
with $\int_{0}^{T}\|(x,y)\|^{2}\mathrm{d}t<\infty$ almost everywhere;

  * •

$(x,y)\in M[t,T]$ has continuous path on $t\in[0,T]$ a.s. and satisfies the
following Volterra integral equation of second kind on $t\in[0,T]$:

$\displaystyle x(t)$
$\displaystyle=\xi-\frac{1}{\Gamma(\alpha)}\int_{t}^{T}A(s-t)^{\alpha-1}x(s)\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}f(s,x(s),y(t,s))\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}\left[g(t,s,x(s))+y(t,s)\right]\mathrm{d}w(s).$
(3.1)

######  Definition 3.2.

A pair of adapted process $(x,y)\in M[t,T]$ is a mild solution of (1.3) for
all $t\in[0,T]$ if satisfies the following backward stochastic nonlinear
Volterra integral equation

$\displaystyle x(t)=E_{\alpha}(A(T-t)^{\alpha})\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s,x(s),y(t,s))\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s,x(s))+y(t,s)\right]\mathrm{d}w(s),\qquad\text{P-a.s},$
(3.2)

where $E_{\alpha}(\cdot)$ and $E_{\alpha,\alpha}(\cdot)$ are one and two-
parameter Mittag-Leffler functions (see [43]).

We then introduce a fundamental lemma which plays an important role to state
and prove existence and uniqueness results.

######  Lemma 3.1.

For any $(x(\cdot),y(\cdot,\cdot))\in M[t,T]$, the linear Caputo fBSDE
equation

$\displaystyle x(t)=E_{\alpha}(A(T-t)^{\alpha})\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s)\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s)+y(t,s)\right]\mathrm{d}w(s),\quad\text{P-a.s.}$
(3.3)

admits a unique solution in $M[0,T]$ and moreover

$\displaystyle\|x\|^{2}_{2,\alpha,t}+|\|y\||^{2}_{2,\alpha,t}$
$\displaystyle\leq
2M^{2}_{\alpha}\textbf{E}\|\left\\{\xi\mid\mathscr{F}_{\cdot}\right\\}\|^{2}_{2,\alpha,t}+16M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}$
$\displaystyle+8M^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\|f\|^{2}_{2,\alpha,t}+2|\|g\||^{2}_{2,\alpha,t},$
(3.4)

where $M_{\alpha}\coloneqq\sup\left\\{\|E_{\alpha}(t^{\alpha}A)\|,0\leq t\leq
T\right\\}$ and
$M_{\alpha,\alpha}\coloneqq\sup\left\\{\|E_{\alpha,\alpha}(t^{\alpha}A)\|,0\leq
t\leq T\right\\}$.

###### Proof.

Uniqueness: Let $(x_{1},y_{1})$ and $(x_{2},y_{2})$ be two solutions of (3.1).

$\displaystyle
x_{1}(t)-x_{2}(t)=\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[y_{1}(t,s)-y_{2}(t,s)\right]\mathrm{d}w(s),$

Taking $\textbf{E}\left\\{\cdot\mid\mathscr{F}_{t}\right\\}$ from above, we
can deduce that

$\displaystyle\textbf{E}\left\\{x_{1}(t)-x_{2}(t)\mid\mathscr{F}_{t}\right\\}=0,\quad\forall
t\in[0,T],$

It is obvious that $x_{1}(t)=x_{2}(t)$ and this follows that
$y_{1}(t,s)=y_{2}(t,s)$, $(t,s)\in\mathcal{D}$.

Existence: Taking a conditional expectation from (3.1), we have

$x(t)=E_{\alpha}(A(T-t)^{\alpha})\textbf{E}\left\\{\xi\mid\mathscr{F}_{t}\right\\}+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\textbf{E}\left\\{f(s)\mid\mathscr{F}_{t}\right\\}\mathrm{d}s.$

From extended martingale representation theorem, there exists
$L(\cdot)\in\mathbb{L}^{2}_{\mathscr{F}}(0,T;L^{2}(\mathbb{R}^{n}))$ and
uniquely
$K(t,\cdot)\in\mathbb{L}^{2}_{\mathscr{F}}(\mathcal{D};L^{2}(\mathbb{R}^{n\times
m}))$ which satisfy the following relations:

$\textbf{E}\left\\{\xi\mid\mathscr{F}_{t}\right\\}=\textbf{E}\xi+\int_{0}^{t}L(u)\mathrm{d}w(u),$
(3.5)
$\textbf{E}\left\\{f(s)\mid\mathscr{F}_{t}\right\\}=\textbf{E}f(s)+\int_{0}^{t}K(s,u)\mathrm{d}w(u).$
(3.6)

Note also from (3.6), we can easily deduce that $\forall s\in[0,T]$

$K(s,u)=0,\quad\text{a.e.},\quad u\in[s,T],\quad\text{a.s.}$

and that

$\textbf{E}\int_{0}^{T}\int_{0}^{s}|K(s,u)|^{2}\mathrm{d}u\mathrm{d}s\leq
4\textbf{E}\int_{0}^{T}|f(s)|^{2}\mathrm{d}s.$ (3.7)

Since $t\in[0.T]$, it is obvious that

$\displaystyle\xi$
$\displaystyle=\textbf{E}\xi+\int_{0}^{T}L(u)\mathrm{d}w(u)$
$\displaystyle=\textbf{E}\xi+\int_{0}^{t}L(u)\mathrm{d}w(u)+\int_{t}^{T}L(u)\mathrm{d}w(u)$
$\displaystyle=\textbf{E}\left\\{\xi\mid\mathscr{F}_{t}\right\\}+\int_{t}^{T}L(u)\mathrm{d}w(u),$

and since $s\geq t$, we have

$\displaystyle f(s)$
$\displaystyle=\textbf{E}f(s)+\int_{0}^{s}K(s,u)\mathrm{d}w(u)$
$\displaystyle=\textbf{E}f(s)+\int_{0}^{t}K(s,u)\mathrm{d}w(u)+\int_{t}^{s}K(s,u)\mathrm{d}w(u)$
$\displaystyle=\textbf{E}\left\\{f(s)\mid\mathscr{F}_{t}\right\\}+\int_{t}^{s}K(s,u)\mathrm{d}w(u).$

Therefore, we obtain

$\textbf{E}\left\\{\xi\mid\mathscr{F}_{t}\right\\}=\xi-\int_{t}^{T}L(u)\mathrm{d}w(u),$
(3.8)

and

$\textbf{E}\left\\{f(s)\mid\mathscr{F}_{t}\right\\}=f(s)-\int_{t}^{s}K(s,u)\mathrm{d}w(u).$
(3.9)

Substituting (3.8) and (3.9) into (3.10) and using stochastic Fubini’s
theorem, we have

$\displaystyle x(t)$
$\displaystyle=E_{\alpha}(A(T-t)^{\alpha})\left(\xi-\int_{t}^{T}L(u)\mathrm{d}w(u)\right)+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left(f(s)-\int_{t}^{s}K(s,u)\mathrm{d}w(u)\right)\mathrm{d}s$
$\displaystyle=E_{\alpha}(A(T-t)^{\alpha})\xi+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s)\mathrm{d}s$
$\displaystyle-
E_{\alpha}(A(T-t)^{\alpha})\int_{t}^{T}L(u)\mathrm{d}w(u)-\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\int_{t}^{s}K(s,u)\mathrm{d}w(u)\mathrm{d}s$
$\displaystyle=E_{\alpha}(A(T-t)^{\alpha})\xi+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s)\mathrm{d}s$
$\displaystyle-
E_{\alpha}(A(T-t)^{\alpha})\int_{t}^{T}L(u)\mathrm{d}w(u)-\int_{t}^{T}\int_{u}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})K(s,u)\mathrm{d}s\mathrm{d}w(u).$

Thus, we get

$\displaystyle
x(t)=E_{\alpha}(A(T-t)^{\alpha})\xi+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s)\mathrm{d}s+\int_{t}^{T}\tilde{y}(t,u)\mathrm{d}w(u).$

Then there exists a mild solution $(x,y)\in M[0,T]$ of (3.1) given by

$x(t)=E_{\alpha}(A(T-t)^{\alpha})\textbf{E}\left\\{\xi\mid\mathscr{F}_{t}\right\\}+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\textbf{E}\left\\{f(s)\mid\mathscr{F}_{t}\right\\}\mathrm{d}s,$
(3.10)

and

$\tilde{y}(t,u)=-E_{\alpha}(A(T-t)^{\alpha})L(u)-\int_{u}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})K(s,u)\mathrm{d}s.$
(3.11)

We finally define $y(t,u)=\tilde{y}(t,u)-g(t,u)$,
$(t,u)\in\mathcal{D}=\left\\{(t,u)\in\mathbb{R}^{2}_{+};0\leq t\leq u\leq
T\right\\}$. It is easily seen that the pair $(x,y)$ solves (3.1). Therefore,
the existence is proved.

From (3.8) and (3.9), we invoke the following inequalities for $0\leq t\leq
s\leq T$:

$\textbf{E}\int_{t}^{T}\|L(u)\|^{2}\mathrm{d}u\leq 4\textbf{E}\|\xi\|^{2},$

and

$\textbf{E}\int_{t}^{s}\|K(s,u)\|^{2}\mathrm{d}u\leq 4\textbf{E}\|f(s)\|^{2}.$

Now we estimate the solution $(x,y)$ given by (3.10) and (3.11) in $[0,T]$.
From (3.10) it follows that

$\displaystyle\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\|x(s)\|^{2}\mathrm{d}s$
$\displaystyle\leq 2M^{2}_{\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\|\textbf{E}\left\\{\xi|\mathscr{F}_{s}\right\\}\|^{2}\mathrm{d}s$
$\displaystyle+2M^{2}_{\alpha,\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\Big{(}\int_{s}^{T}(r-s)^{\alpha-1}\textbf{E}\left\\{\|f(r)\|\mid\mathscr{F}_{s}\right\\}\mathrm{d}r\Big{)}^{2}\mathrm{d}s$
$\displaystyle\coloneqq\mathcal{I}_{1}+\mathcal{I}_{2}.$

From law of total expectation, it follows that

$\displaystyle\mathcal{I}_{1}$ $\displaystyle\coloneqq
2M^{2}_{\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\|\textbf{E}\left\\{\xi|\mathscr{F}_{s}\right\\}\|^{2}\mathrm{d}s=2M^{2}_{\alpha}\textbf{E}\|\left\\{\xi\mid\mathscr{F}_{\cdot}\right\\}\|^{2}_{2,\alpha,t}$

and Jensen’s inequality in probabilistic setting imply that

$\displaystyle\mathcal{I}_{2}$ $\displaystyle\coloneqq
2M^{2}_{\alpha,\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\left(\int_{s}^{T}(r-s)^{\alpha-1}\textbf{E}\left\\{\|f(r)\|\mid\mathscr{F}_{s}\right\\}\mathrm{d}r\right)^{2}\mathrm{d}s$
$\displaystyle=2M^{2}_{\alpha,\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\left(\textbf{E}\Biggl{\\{}\int_{s}^{T}(r-s)^{\alpha-1}\|f(r)\|\mathrm{d}r\mid\mathscr{F}_{s}\Biggr{\\}}\right)^{2}\mathrm{d}s$
$\displaystyle\leq 2M^{2}_{\alpha,\alpha}\sup_{t\leq s\leq
T}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\textbf{E}\left(\int_{s}^{T}(r-s)^{\alpha-1}\|f(r)\|\mathrm{d}r\right)^{2}\mathrm{d}s$
$\displaystyle\leq 2M^{2}_{\alpha,\alpha}\sup_{t\leq\tau\leq
T}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\textbf{E}\left(\int_{s}^{T}(r-s)^{\alpha-1}\mathrm{d}r\int_{s}^{T}(r-s)^{\alpha-1}\|f(r)\|^{2}\mathrm{d}r\right)\mathrm{d}s$
$\displaystyle\leq
2M^{2}_{\alpha,\alpha}\frac{(T-t)^{\alpha}}{\alpha}\sup_{t\leq\tau\leq
T}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\textbf{E}\int_{s}^{T}(r-s)^{\alpha-1}\|f(r)\|^{2}\mathrm{d}r\mathrm{d}s$
$\displaystyle=2M^{2}_{\alpha,\alpha}\frac{(T-t)^{\alpha}}{\alpha}\sup_{t\leq\tau\leq
T}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\mathrm{d}s\|f\|^{2}_{2,\alpha,t}$
$\displaystyle=2M^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha}}{\alpha^{2}}\|f\|^{2}_{2,\alpha,t}.$

Thus, considering (1.4), we can write

$\displaystyle\|x\|^{2}_{2,\alpha,t}\leq
2M^{2}_{\alpha}\textbf{E}\|\left\\{\xi\mid\mathscr{F}_{\cdot}\right\\}\|^{2}_{2,\alpha,t}+2M^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha}}{\alpha^{2}}\|f\|^{2}_{2,\alpha,t}.$
(3.12)

Next we estimate $\tilde{y}$ using Hölder’s inequality. We attain,

$\displaystyle\|\tilde{y}(s,u)\|^{2}$ $\displaystyle\leq
2\|E_{\alpha}(A(T-s)^{\alpha}L(u))\|^{2}+2\left\lVert\int_{u}^{T}(r-s)^{\alpha-1}E_{\alpha,\alpha}(A(r-s)^{\alpha})K(r,u)\mathrm{d}r\right\rVert^{2}$
$\displaystyle\leq
2M^{2}_{\alpha}\|L(u)\|^{2}+2M^{2}_{\alpha,\alpha}\left(\frac{(T-s)^{2\alpha-1}}{2\alpha-1}-\frac{(u-s)^{2\alpha-1}}{2\alpha-1}\right)\int_{u}^{T}\|K(r,u)\|^{2}\mathrm{d}r$
$\displaystyle\leq
2M^{2}_{\alpha}\|L(u)\|^{2}+2M^{2}_{\alpha,\alpha}\frac{(T+u)^{2\alpha-1}}{2\alpha-1}\int_{u}^{T}\|K(r,u)\|^{2}\mathrm{d}r$
$\displaystyle\leq
2M^{2}_{\alpha}\|L(u)\|^{2}+2M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\int_{u}^{T}\|K(r,u)\|^{2}\mathrm{d}r.$

Taking double integral of above inequality and applying stochastic Fubini’s
theorem twice yield that

$\displaystyle\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|\tilde{y}(s,u)\|^{2}\mathrm{d}u\mathrm{d}s\leq
2M^{2}_{\alpha}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|L(u)\|^{2}\mathrm{d}u\mathrm{d}s$
$\displaystyle+2M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\int_{u}^{T}\|K(r,u)\|^{2}\mathrm{d}r\mathrm{d}u\mathrm{d}s$
$\displaystyle\leq 8M^{2}_{\alpha}\sup_{t\leq\tau\leq
T}\int_{\tau}^{T}(s-\tau)^{2}\textbf{E}\|\xi\|^{2}\mathrm{d}s$
$\displaystyle+2M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\int_{s}^{r}\|K(r,u)\|^{2}\mathrm{d}u\mathrm{d}r\mathrm{d}s$
$\displaystyle\leq
8M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}+8M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|f(r)\|^{2}\mathrm{d}r\mathrm{d}s$
$\displaystyle\leq
8M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}+8M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}\int_{\tau}^{r}(s-\tau)^{\alpha-1}\|f(r)\|^{2}\mathrm{d}s\mathrm{d}r$
$\displaystyle\leq
8M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}+8M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}}{2\alpha-1}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}\int_{\tau}^{r}(s-\tau)^{\alpha-1}\mathrm{d}s\|f(r)\|^{2}\mathrm{d}r$
$\displaystyle\leq
8M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}+8M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}(T-t)}{\alpha(2\alpha-1)}\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(r-\tau)^{\alpha-1}\|f(r)\|^{2}\mathrm{d}r.$

Thus, we get

$\displaystyle\sup_{t\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|\tilde{y}(s,u)\|^{2}\mathrm{d}u\mathrm{d}s\leq
8M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}+8M^{2}_{\alpha,\alpha}\frac{(2T)^{2\alpha-1}(T-t)}{\alpha(2\alpha-1)}\|f\|^{2}_{2,\alpha,t}.$
(3.13)

Since $\frac{1}{2}<\alpha\leq 1$ and $0<2\alpha-1\leq 1$

$\frac{(T-t)^{2\alpha}}{\alpha^{2}}+2\frac{(2T)^{2\alpha-1}(T-t)}{\alpha(2\alpha-1)}\leq\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)},$

and taking into account (1.6), then the inequalities (3.12) and (3.13) imply
that

$\displaystyle\|x\|^{2}_{2,\alpha,t}+2|\|y\||^{2}_{2,\alpha,t}$
$\displaystyle\leq
2M^{2}_{\alpha}\textbf{E}\|\left\\{\xi\mid\mathscr{F}_{\cdot}\right\\}\|^{2}_{2,\alpha,t}+16M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}$
$\displaystyle+8M^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\|f\|^{2}_{2,\alpha,t}.$

Then for any $(x,y)$ the associated mild solution of (3.1) satisfies the
following estimate:

$\displaystyle\|x\|^{2}_{2,\alpha,t}+|\|y\||^{2}_{2,\alpha,t}$
$\displaystyle\leq
2M^{2}_{\alpha}\textbf{E}\|\left\\{\xi\mid\mathscr{F}_{\cdot}\right\\}\|^{2}_{2,\alpha,t}+16M^{2}_{\alpha}\frac{(T-t)^{\alpha}}{\alpha}\textbf{E}\|\xi\|^{2}$
$\displaystyle+8M^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\|f\|^{2}_{2,\alpha,t}+2|\|g\||^{2}_{2,\alpha,t}.$

∎

######  Theorem 3.1.

If

$8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)T<1,$
(3.14)

then Caputo fBSDE (1.3) admits a unique solution $(x,y)\in M[0,T]$ under
Assumptions 3.1 and 3.2.

###### Proof.

For any fixed $(\bar{x},\bar{y})\in M[0,T]$, it follows from Assumption 3.2
that

$\displaystyle
f(\cdot)=f(\cdot,\bar{x}(\cdot),\bar{y}(\cdot,\cdot))\in\mathbb{L}^{2}_{\mathscr{F}}([0,T],\mathbb{R}^{n}).$

By Lemma 3.1, the equation

$\displaystyle x(t)=E_{\alpha}(A(T-t)^{\alpha})\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s,\bar{x}(s),\bar{y}(t,s))\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s)+y(t,s)\right]\mathrm{d}w(s),\qquad\text{P-a.s}$
(3.15)

has a unique solution in $M[0,T]$.

Thus, the operator $\Psi:M[0,T]\to M[0,T]$ defined by

$\Psi(\bar{x},\bar{y})=(x,y)$

where $(x,y)$ is a solution of (3), is well -defined. Now we prove the
contractivity of the operator $\Psi$. To do so, applying Lemma 3.1 implies
that

$\displaystyle\|\Psi(\bar{x},\bar{y})-\Psi(\tilde{x},\tilde{y})\|^{2}_{0}=\|\Psi(\bar{x}-\tilde{x},\bar{y}-\tilde{y})\|^{2}_{0}$
$\displaystyle\leq
8M^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\sup_{0\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|f(s,\bar{x}(s),\bar{y}(s,u))-f(s,\tilde{x}(s),\tilde{y}(s,u))\|^{2}\mathrm{d}u\mathrm{d}s$
$\displaystyle\leq
8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\sup_{0\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\left(\|\bar{x}(s)-\tilde{x}(s)\|^{2}+\|\bar{y}(s,u)-\tilde{y}(s,u)\|^{2}\right)\mathrm{d}u\mathrm{d}s$
$\displaystyle\leq
8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\sup_{0\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|\bar{x}(s)-\tilde{x}(s)\|^{2}\mathrm{d}u\mathrm{d}s$
$\displaystyle+8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\sup_{0\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\|\bar{y}(s,u)-\tilde{y}(s,u)\|^{2}\mathrm{d}u\mathrm{d}s$
$\displaystyle\leq
8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)T\sup_{0\leq\tau\leq
T}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\|\bar{x}(s)-\tilde{x}(s)\|^{2}\mathrm{d}s$
$\displaystyle+8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)\sup_{0\leq\tau\leq
T}\textbf{E}\int_{\tau}^{T}(s-\tau)^{\alpha-1}\int_{s}^{T}\textbf{E}\|\bar{y}(s,u)-\tilde{y}(s,u)\|^{2}\mathrm{d}u\mathrm{d}s$
$\displaystyle=8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)T\left(\|\bar{x}-\tilde{x}\|^{2}_{2,\alpha,0}+|\|\bar{y}-\tilde{y}\||_{2,\alpha,0}^{2}\right)$
$\displaystyle=8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)T\|\left(\bar{x}-\tilde{x},\bar{y}-\tilde{y}\right)\|_{0}^{2}$
$\displaystyle=8cM^{2}_{\alpha,\alpha}\left(\frac{T^{2\alpha}}{\alpha^{2}}+\frac{(2T)^{2\alpha}}{\alpha(2\alpha-1)}\right)T\|(\bar{x},\bar{y})-(\tilde{x},\tilde{y})\|^{2}_{0}.$

which together with (3.14), we get contractivity of operator $\Psi$ on
$M[0,T]$, which in turn implies the existence and uniqueness of the solution
to (3.1). ∎

##  4 A coincidence between the notion of integral equation and mild solution

In this section, we are going to prove the coincidence between the notion of
stochastic Volterra integral equation and mild solution by following theorem.

######  Theorem 4.1.

The unique mild solution of(1.3) with terminal value $\xi$ on $[0,T]$ is given
by

$\displaystyle\psi(t,\xi)=E_{\alpha}(A(T-t)^{\alpha})\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s,\psi(s,\xi),y(t,s))\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s,\psi(s,\xi))+y(t,s)\right]\mathrm{d}w(s).$

###### Proof.

Before providing the proof of above theorem, we first need to show preparatory
lemma and remark. In doing so, we present martingale representation theorem
for any function $h\in\mathbb{L}^{2}(\Omega,\mathscr{F}_{T},\mathbb{R}^{m})$,
there exists a unique adapted process
$\Theta\in\mathbb{L}^{2}(\Omega,\mathscr{F}_{T},\mathbb{R}^{m})$ such that

$h=\textbf{E}h+\int_{0}^{T}\Theta(s)\mathrm{d}w(s).$

It is clear that

$h=\sum_{i=1}^{m}h_{i}e_{i}=\sum_{i=1}^{m}\Big{(}\textbf{E}h_{i}+\int_{0}^{T}\theta_{i}(s)\mathrm{d}w(s)\Big{)}e_{i},$

where

$h_{i}=\textbf{E}h_{i}+\int_{0}^{T}\theta_{i}(s)\mathrm{d}w(s),\quad
h_{i}\in\mathbb{L}^{2}(\Omega,\mathscr{F}_{T},\mathbb{R}^{m}).$

It is sufficient to show that

$\psi(t,\xi)=\bar{\psi}(t,\xi).$ (4.1)

To show (4.1), it is enough to prove that for any
$h\in\mathbb{L}^{2}(\Omega,\mathbb{F}_{T},\mathbb{R}^{m})$,

$\textbf{E}\langle\psi(t,\xi),h\rangle=\textbf{E}\langle\bar{\psi}(t,\xi),h\rangle.$

In other words,

$\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle=\sum_{i=1}^{m}\textbf{E}\langle(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i},e_{i}\rangle.$

It follows that

$\displaystyle|\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle|^{2}$
$\displaystyle\leq\|\sum_{i=1}^{m}\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq\sum_{i=1}^{m}1^{2}\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}.$
$\displaystyle\leq
m\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}.$

Before estimating $|\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle|$,
define the following functions:

$\displaystyle\chi_{i}(t)=\textbf{E}\psi(t,\xi)h_{i},\quad\kappa_{i}(t)=\textbf{E}f(t,\psi(t,\xi),y(t,s))h_{i},$
$\displaystyle\tilde{\chi}_{i}(t)=\textbf{E}\bar{\psi}(t,\xi)h_{i},\quad\tilde{\kappa}_{i}(t)=\textbf{E}f(t,\bar{\psi}(t,\xi),y(t,s))h_{i}.$

######  Remark 4.1.

Since $\psi(t,\xi),\bar{\psi}(t,\xi)\in M[0,T]$ , the functions
$\chi_{i},\kappa_{i},\tilde{\chi}_{i},\tilde{\kappa}_{i}$ are measurable and
bounded on $[t,T]$.

######  Lemma 4.1.

For all $t\in[0,T]$ and $c\in\mathbb{R}^{m}$, the following statements hold:

$\displaystyle\chi_{i}(t)=cE_{\alpha}(A(T-t)^{\alpha})\textbf{E}\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\kappa_{i}(s)\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\textbf{E}\theta_{i}(s)\left[g(s,u,\psi(s,\xi))+y(t,s)\right]\mathrm{d}w(s)$
(4.2)

and

$\displaystyle\tilde{\chi}_{i}(t)=cE_{\alpha}(A(T-t)^{\alpha})\textbf{E}\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\tilde{\kappa}_{i}(s)\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\textbf{E}\theta_{i}(s)\left[g(s,u,\bar{\psi}(s,\xi))+y(t,s)\right]\mathrm{d}w(s),\quad\text{P-a.s}.$
(4.3)

###### Proof.

Since $\psi(t,\xi)$ is a solution of (1.3), it follows that

$\displaystyle\psi(t,\xi)$
$\displaystyle=\xi-\frac{1}{\Gamma(\alpha)}\int_{t}^{T}A(s-t)^{\alpha-1}\psi(s,\xi)\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}f(s,\psi(s,\xi),y(t,s))\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}\left[g(t,s,\psi(s,\xi))+y(t,s)\right]\mathrm{d}w(s).$
(4.4)

Taking product of both sides of (4) with $h_{i}$ and then taking expectation
of both sides yield that

$\displaystyle\chi_{i}(t)$
$\displaystyle=c\textbf{E}\xi-\frac{1}{\Gamma(\alpha)}\int_{t}^{T}A(s-t)^{\alpha-1}\chi_{i}(s)\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}\kappa_{i}(s)\mathrm{d}s$
$\displaystyle+\frac{1}{\Gamma(\alpha)}\langle\int_{t}^{T}(s-t)^{\alpha-1}\left[g(t,s,\psi(s,\xi))+y(t,s)\right]\mathrm{d}w(s),\int_{0}^{T}\theta_{i}(s)\mathrm{d}w(s)\rangle.$

Using Itô’s representation theorem, we attain

$\displaystyle\chi_{i}(t)=c\textbf{E}\xi-\frac{1}{\Gamma(\alpha)}\int_{t}^{T}(s-t)^{\alpha-1}\Big{(}A\chi_{i}(s)-\kappa_{i}(s)-\textbf{E}\theta_{i}(s)\left[g(t,s,\psi(s,\xi))+y(t,s)\right]\Big{)}\mathrm{d}s$

such that $\chi_{i}(t)$ is a solution of the following fractional stochastic
backward differential equation:

$\prescript{C}{T}{D^{\alpha}_{t}}x(t)=Ax(t)-\kappa_{i}(t)-\textbf{E}\theta_{i}(t)\left[g(t,s,\psi(t,\xi))+y(t,s)\right],\quad
x(T)=c\textbf{E}\xi.$ (4.5)

Then, by means of Remark 4.1, (4.1) is proved.

Next, we define

$\displaystyle\bar{\psi}(t,\xi)=E_{\alpha}(A(T-t)^{\alpha})\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})f(s,\bar{\psi}(s,\xi),y(t,s))\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s,\bar{\psi}(s,\xi))+y(t,s)\right]\mathrm{d}w(s),\qquad
P-a.s.$ (4.6)

Similarly, by taking product of both sides of (4) with $h_{i}$ and then taking
expectation of both sides follow that

$\displaystyle\tilde{\chi}_{i}(t)=cE_{\alpha}(A(T-t)^{\alpha})\textbf{E}\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\tilde{\kappa}_{i}(s)\mathrm{d}s$
$\displaystyle+\langle\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\left[g(t,s,\bar{\psi}(s,\xi))+y(t,s)\right]\mathrm{d}w(s),\int_{0}^{T}\theta_{i}(s)\mathrm{d}w(s)\rangle.$

Applying Itô’s isometry theorem, (4.1) is proved:

$\displaystyle\tilde{\chi}_{i}(t)=cE_{\alpha}(A(T-t)^{\alpha})\textbf{E}\xi$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\tilde{\kappa}_{i}(s)\mathrm{d}s$
$\displaystyle+\int_{t}^{T}(s-t)^{\alpha-1}E_{\alpha,\alpha}(A(s-t)^{\alpha})\textbf{E}\theta_{i}(s)\left[g(t,s,\bar{\psi}(s,\xi))+y(t,s)\right]\mathrm{d}w(s),\qquad\text{P-a.s}.$

Therefore, $\tilde{\chi}_{i}(t)$ is a solution of the following fractional
stochastic backward differential equations:

$\prescript{C}{T}{D^{\alpha}_{t}}x(t)=Ax(t)-\tilde{\kappa}_{i}(t)-\textbf{E}\theta_{i}(t)\left[g(t,s,\bar{\psi}(t,\xi))+y(t,s)\right],\quad
x(T)=cE_{\alpha}(A(T-t)^{\alpha})\textbf{E}\xi.$

∎

######  Remark 4.2.

For any $h\in\mathbb{L}^{2}(\Omega,\mathscr{F}_{T},\mathbb{R}^{m})$, we have

$\displaystyle|\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle|^{2}$
$\displaystyle=m\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq
4mcM^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha-1}}{2\alpha-1}\int_{t}^{T}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\textbf{E}\|h\|^{2}$
$\displaystyle+4mcM^{2}_{\alpha,\alpha}\int_{t}^{T}(s-t)^{2\alpha-2}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\textbf{E}\|h\|^{2}.$
(4.7)

###### Proof.

To proof Remark 4.2, we start with the following inequality:

$\displaystyle|\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle|^{2}$
$\displaystyle=|\sum_{i=1}^{m}\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h_{i}\rangle|^{2}$
(4.8)
$\displaystyle\leq\sum_{i=1}^{m}1^{2}\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq
m\sum_{i=1}^{m}|\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h_{i}\rangle|^{2}$
$\displaystyle\leq
m\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq m\sum_{i=1}^{m}\|\chi_{i}(t)-\tilde{\chi}_{i}(t)\|^{2},$

First, we estimate $\|\chi_{i}(t)-\tilde{\chi}_{i}(t)\|$ as below:

$\displaystyle\|\chi_{i}(t)-\tilde{\chi}_{i}(t)\|$ $\displaystyle\leq
M_{\alpha,\alpha}\int_{t}^{T}(s-t)^{\alpha-1}\|\kappa_{i}(s)-\tilde{\kappa}_{i}(s)\|\mathrm{d}s$
$\displaystyle+mM_{\alpha,\alpha}\int_{t}^{T}(s-t)^{\alpha-1}\textbf{E}(\|\theta_{i}(s)\|\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|)\mathrm{d}s.$

Applying Cauchy-Schwarz inequality yields that

$\displaystyle\|\chi_{i}(t)-\tilde{\chi}_{i}(t)\|$ $\displaystyle\leq
M_{\alpha,\alpha}\sqrt{\frac{(T-t)^{2\alpha-1}}{2\alpha-1}}\Big{(}\int_{t}^{T}\|\kappa_{i}(s)-\tilde{\kappa}_{i}(s)\|^{2}\mathrm{d}s\Big{)}^{\frac{1}{2}}$
$\displaystyle+mM_{\alpha,\alpha}\Big{(}\int_{t}^{T}\textbf{E}\|\theta_{i}(s)\|^{2}ds\Big{)}^{\frac{1}{2}}\Big{(}\int_{t}^{T}(s-t)^{2\alpha-2}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\Big{)}^{\frac{1}{2}}.$
(4.9)

By definition of $\kappa$ and $\tilde{\kappa}$ for all $s\in[t,T]$,

$\displaystyle\|\kappa_{i}(s)-\tilde{\kappa}_{i}(s)\|^{2}$
$\displaystyle=\|\textbf{E}(f(s,\psi(s,\xi),y(t,s))-f(s,\bar{\psi}(s,\xi),y(t,s)))h_{i}\|^{2}$
$\displaystyle=\sum_{i=1}^{m}|\textbf{E}\langle
f_{i}(s,\psi(s,\xi),y(t,s))-f_{i}(s,\bar{\psi}(s,\xi),y(t,s)),h_{i}\rangle|^{2}$
$\displaystyle\leq\sum_{i=1}^{m}\textbf{E}\|f_{i}(s,\psi(s,\xi),y(t,s))-f_{i}(s,\bar{\psi}(s,\xi),y(t,s))\|^{2}\textbf{E}\|h_{i}\|^{2}$
$\displaystyle=\textbf{E}\|f(s,\psi(s,\xi),y(t,s))-f(s,\bar{\psi}(s,\xi),y(t,s))\|^{2}\textbf{E}\|h_{i}\|^{2}$
$\displaystyle\leq
c\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\textbf{E}\|h_{i}\|^{2}.$

Then, we make use of above estimation in (4), we obtain

$\displaystyle\|\chi_{i}(t)-\tilde{\chi}_{i}(t)\|$ $\displaystyle\leq
mM_{\alpha,\alpha}\sqrt{\frac{(T-t)^{2\alpha-1}}{2\alpha-1}}\Big{(}\int_{t}^{T}\left(\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\right)^{\frac{1}{2}}\left(\textbf{E}\|h_{i}\|^{2}\right)^{\frac{1}{2}}$
$\displaystyle+mM_{\alpha,\alpha}\Big{(}\int_{t}^{T}\textbf{E}\|\theta_{i}(s)\|^{2}ds\Big{)}^{\frac{1}{2}}\Big{(}\int_{t}^{T}(s-t)^{2\alpha-2}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\Big{)}^{\frac{1}{2}}.$
(4.10)

Now we take expectation of (4.8) and plug (4) into (4.8) , we attain desired
result as follows:

$\displaystyle|\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h\rangle|^{2}$
$\displaystyle=|\sum_{i=1}^{m}\textbf{E}\langle\psi(t,\xi)-\bar{\psi}(t,\xi),h_{i}\rangle|^{2}$
$\displaystyle\leq
m\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq
m\sum_{i=1}^{m}\|\textbf{E}(\psi(t,\xi)-\bar{\psi}(t,\xi))h_{i}\|^{2}$
$\displaystyle\leq
4mcM^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha-1}}{2\alpha-1}\int_{t}^{T}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\textbf{E}\|h\|^{2}$
$\displaystyle+4cmM^{2}_{\alpha,\alpha}\int_{t}^{T}(s-t)^{2\alpha-2}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s\textbf{E}\|h\|^{2},$

which completes the proof. ∎

Proof of Theorem 4.1. Let
$T^{*}=\inf\left\\{t\in\left[0,T\right];\psi(t,\xi)\neq\bar{\psi}(t,\xi)\right\\}$.
Then it is sufficient to show that $T^{*}=T$.
Suppose the contrary : $T^{*}<T$. Choose and fix an arbitrary $\delta>0$
satisfying the following expression:

$4mcM^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha-1}}{2\alpha-1}\delta+4mcM^{2}_{\alpha,\alpha}\frac{\delta^{2\alpha-1}}{2\alpha-1}<1.$
(4.11)

To lead contradiction, we show that $\psi(t,\xi)=\bar{\psi}(t,\xi)$ for all
$t\in[T^{*}-\delta,T^{*}]$. Using Ito’s representation, there exists a unique
$h\in\mathbb{L}^{2}$ such that $\psi(t,\xi)-\bar{\psi}(t,\xi)=h$. Therefore,
we have

$\textbf{E}\|\psi(t,\xi)-\bar{\psi}(t,\xi)\|^{2}=\textbf{E}\|h\|^{2}.$

Using Remark (4.2), we attain

$\displaystyle\textbf{E}\|\psi(t,\xi)-\bar{\psi}(t,\xi)\|^{2}$
$\displaystyle\leq
4mcM^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha-1}}{2\alpha-1}\int_{t}^{T^{*}}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s$
$\displaystyle+4mcM^{2}_{\alpha,\alpha}\int_{t}^{T^{*}}(s-t)^{2\alpha-2}\textbf{E}\|\psi(s,\xi)-\bar{\psi}(s,\xi)\|^{2}\mathrm{d}s.$

As a consequence,

$\displaystyle\sup_{t\in[T^{*}-\delta,T^{*}]}\textbf{E}\|\psi(t,\xi)-\bar{\psi}(t,\xi)\|^{2}$
$\displaystyle\leq\Big{[}4mcM^{2}_{\alpha,\alpha}\frac{(T-t)^{2\alpha-1}}{2\alpha-1}\delta+4mcM^{2}_{\alpha,\alpha}\frac{\delta^{2\alpha-1}}{2\alpha-1}\Big{]}$
$\displaystyle\times\sup_{t\in[T^{*}-\delta,T^{*}]}\textbf{E}\|\psi(t,\xi)-\bar{\psi}(t,\xi)\|^{2}.$

By selecting $\delta$ as in (4.11), we have
$\sup_{t\in[T^{*}-\delta,T^{*}]}\textbf{E}\|\psi(t,\xi)-\bar{\psi}(t,\xi)\|^{2}=0$.
This leads to a contradiction and the proof is complete. ∎

##  5 Conclusions and future works

In this paper, we first formulated new problem in BSDE theory so-called Caputo
fBSDE which is an untreated topic in recent literature. We recalled some
definitions from stochastic and fractional calculus and present a new weighted
maximum norm in square-integrable measurable space. To derive adapted pair of
stochastic processes, we first established fundamental lemma which plays a
crucial role in the theory of Caputo fBSDE. The main results in our paper were
to show global existence and uniqueness of adapted solution to (1.3) in finite
dimensional setting with the help of aforementioned fundamental lemma. The key
point in the proof of this lemma was to apply extended martingale
representation theorem and some inequalities from stochastic calculus. Last
but not least, we proved the coincidence between the notion of integral
equation and mild solution using martingale representation theorem.

Since our results are sufficiently new in the theory of BSDEs, there are still
open problems to discuss related to fractional stochastic control theory and
risk sensitive control problems. If we consider the same statement of our
results shown above, an interesting problem in Caputo fBSDE is for multi-
dimensional case: $m\geq 2$. The problem of existence with coefficient $f$
only continuous in $(x,y)$ becomes very hard. Similar problem appears in
situation of the quadratic growth condition. Successfully applied techniques
in one-dimensional case fails here due to the lack of comparison theorems. In
general, multi-dimensional case has many interesting applications. For
instance, one can apply Caputo fBSDE to optimal control theory using
stochastic maximum principle.

## References

  * [1] B. Oksendal, Stochastic Differential Equations: An Introduction with Applications, Springer-Verlag, Heidelberg, 2000.
  * [2] K. Itô, Stochastic Differential Equations, Memoirs of the American Mathematical Society, 4(1951) 1-51.
  * [3] G.D. Prato, J. Zabczyk, Stochastic equations in infinite dimensions, Cambridge University Press, Cambridge, 1992.
  * [4] D.N. Tien, Fractional stochastic differential equations with applications to finance. J. Math. Anal. Appl. 397(2013) 334–348.
  * [5] A. Farhadi, G.H. Erjaee, M. Salehi, Derivation of a new Merton’s optimal problem presented by fractional stochastic stock price and its applications. Comput. Math. Appl. 73(2017) 2066–2075.
  * [6] Y. Wang, S. Zheng, W. Zhang, J. Wang, Complex and entropy of fluctuations of agent-based interacting financial dynamics with random jump. Entropy. 19(2017) 512. https://doi.org/10.3390/e19100512.
  * [7] P. C. Jean, L.S. Gangaram, Stochastic fractional differential equations: Modeling, method and analysis, Chaos, Solitons & Fract. 45(2012) 279-293.
  * [8] B. Mandelbrot, J. Van Ness, Fractional Brownian motions, fractional noises and applications. SIAM Rev. 10(1968) 422–437.
  * [9] A. Ahmadova, N.I. Mahmudov, Existence and uniqueness results for a class of fractional stochastic neutral differential equations, Chaos Solitons&Fract. 139 (2020) https://doi.org/10.1016/j.chaos.2020.110253.
  * [10] D. Barbu, Local and global existence for mild solutions of stochastic differential equations, Port, Math 55(1998) 411-424.
  * [11] F. Jiang, Y. Shen, A note on the existence and uniqueness of mild solutions to neutral stochastic partial differential equations with non-Lipschitz coefficients, Computers and Mathematics with Applications, 61 (2011) 1590-1594.
  * [12] G. Shen, R. Sakthivel, Y. Ren, M. Li, Controllability and stability of fractional stochastic functional systems driven by Rosenblatt process, Collect. Math. 71 (2020) 63–82; https://doi.org/10.1007/s13348-019-00248-3.
  * [13] A. Ahmadova, I.T. Huseynov, N.I.Mahmudov, Controllability of fractional stochastic delay dynamical systems, arXiv:2009.10654.
  * [14] X. Mao, Stability of Stochastic Differential Equations with Respect to Semimartingales, Longman Scientific and Technical, New York, 1991.
  * [15] X. Mao, Exponential Stability of Stochastic Differential Equations, Marcel Dekker, New York, 1994.
  * [16] A. Ahmadova, N.I. Mahmudov, Ulam-Hyers stability of Caputo type stochastic neutral differential equations, Stat. Probab. Lett. 108949. https://doi.org/10.1016/j.spl.2020.108949.
  * [17] J.M. Bismut, Theorie probabiliste du controle des diffusions, Mem Amer Math. Soc. 176 (1973) 1-30.
  * [18] A. Bensoussan, Lectures on stochastic control, in: S.K. Mittler and A. Moro, eds., Nonlinear Filtering and Stochastic Control, Lecture Notes in Mathematics, 972. (1982) 1-62.
  * [19] E. Pardoux, S. Peng, Adapted solution of a backward stochastic differential equation, Systems Control Lett. 14 (1990) 55–61.
  * [20] Y. Hu, S. Peng, Adapted solutions of a backward semilinear stochastic evolution equation, Stochastic Anal. Appl. 9 (4) (1991) 445–459.
  * [21] S. Rong, On solutions of backward stochastic differential equations with jumps and applications, Stoch. Process. Appl. 66 (1997) 209-236.
  * [22] S. Rong, On solutions of backward stochastic differential equations with jumps and with non-Lipschitzian coefficients in Hilbert spaces and stochastic control, Statist. Probab. Lett. 60 (3) (2002) 279–288.
  * [23] G. Tessitore, Existence, uniqueness, and space regularity of the adapted solutions of a backward SPDE, Stochastic Anal. Appl. 14 (4) (1996) 461–486.
  * [24] E. Pardoux, A. Rascanu, Backward stochastic variational inequalities, Stoch. Stoch. Rep. 67 (3–4) (1999) 159–167.
  * [25] S. Peng, Backward stochastic differential equations and applications to optimal control, Appl. Math. Optim. 27 (1993) 125–144.
  * [26] S. Peng, A generalized dynamic programming principle and Hamilton-Jacobi-Bellman equation, Stochastics and Stochastics Reports 38 (1992) 119-134.
  * [27] S. Peng, Probabilistic interpletation for systems of quasilinear parabolic partial differential equations, Stochastics and Stochastics Reports 32 (1991) 61-74.
  * [28] S. Tang and X. Li, Necessary conditions for optimal control of stochastic systems with random jumps, SIAM J. Control Optim. 32 (1994) 1447-1475.
  * [29] X. Mao, Adapted solutions of backward stochastic differential equations with non-Lipschitz coefficients, Stoch. Process. Appl. 58 (1995) 281-292.
  * [30] J. Lin, Adapted solution of a backward stochastic nonlinear Volterra integral equation, Stochastic Anal. Appl. 20(1) (2002) 165-183.
  * [31] N.I. Mahmudov, M.A. McKibben, On backward stochastic evolution equations in Hilbert spaces and optimal control, Nonlinear Analysis. 67 (2007) 1260–1274.
  * [32] J.P. Dauer, N.I. Mahmudov, M.M. Matar, Approximate controllability of backward stochastic evolution equations in Hilbert spaces, J. Math. Anal, Appl. 323(1) (2006) 42-56.
  * [33] J. Yong, Backward stochastic Volterra integral equations and some related problems, Stoch. Process. Appl. 116 (2006) 779–795.
  * [34] T. Wang, J. Yong, Backward stochastic Volterra integral equations—Representation of adapted solutions, Stoch. Process. Appl. 129 (2019) 4926–4964.
  * [35] O. Baghani, On fractional Langevin equation involving two fractional orders, Commun Nonlinear Sci Numer Simulat, 42(2017), 675-681. https://doi.org/10.1016/j.cnsns.2016.05.023
  * [36] I. Podlubny, Fractional Differential Equations, Academic Press, New York, 1999.
  * [37] K.S. Miller, B. Ross, An Introduction to the Fractional Calculus and Fractional Differential Equations, Wiley, New York, 1993.
  * [38] A.A. Kilbas, H.M. Srivastava, J.J. Trujillo, Theory and applications of fractional differential equations, Elsevier Sceince B.V. 2006.
  * [39] S. G. Samko, A.A. Kilbas, O.I. Marichev, Fractional Integrals and Derivatives: Theory and Applications, Gordon and Breach, New York, 1993.
  * [40] K.B. Oldham, J. Spanier, The Fractional Calculus, Academic Press, San Diego, 1974.
  * [41] G. Anastassiou, Trigonometric Fractional Approximation of Stochastic Processes, Bulletin of TICMI, 24(1) 2020, 27–58.
  * [42] H. Zine, D.F. Torres, A Stochastic Fractional Calculus with Applications to Variational Principles, Fractal and Fract. 4(38) 2020, doi:10.3390/fractalfract4030038
  * [43] K. Diethelm, The analysis of fractional differential equations. An application-oriented exposition using differential operators of Caputo type, Lecture notes in Mathematics, Springer-Verlag, Berlin, 2010.