# Almost paracontact almost paracomplex Riemannian manifolds as extensions
of 2-dimensional space-forms

Mancho Manev  and  Veselina Tavkova  Department of Algebra and Geometry,
Faculty of Mathematics and Informatics, University of Plovdiv Paisii
Hilendarski, 24, Tzar Asen St, 4000 Plovdiv, Bulgaria<EMAIL_ADDRESS><EMAIL_ADDRESS>Department of Medical Informatics, Biostatistics and
E-Learning, Faculty of Public Health, Medical University of Plovdiv, 15A,
Vasil Aprilov Blvd, 4002 Plovdiv, Bulgaria

###### Abstract.

Almost paracontact Riemannian manifolds of the lowest dimension are studied,
whose paracontact distributions are equipped with an almost paracomplex
structure. These manifolds are constructed as a product of a real line and a
2-dimensional Riemannian space-form. Their metric is obtained in two ways: as
a cone metric and as a hyperbolic extension of the metric of the underlying
paracomplex 2-manifold. The resulting manifolds are studied and characterised
in terms of the classification used and their curvature properties.

###### Key words and phrases:

Almost paracontact structure, almost paracomplex structure, Riemannian metric,
cone, hyperbolic extension

###### 2010 Mathematics Subject Classification:

Primary 53C15, 53C17; Secondary 53C25, 53C42

## Introduction

In 1976, I. Sato [9] introduced the concept of an almost paracontact structure
compatible with a Riemannian metric as an analogue of an almost contact
Riemannian manifold. The study of the differential geometry of these manifolds
began with [1], [10], [11] by I. Sato, T. Adati and T. Miyazawa. After that,
in [8], S. Sasaki defined the notion of an almost paracontact Riemannian
manifold of type $(p,q)$, where $p$ and $q$ are the multiples of the
eigenvalues $+1$ and $-1$ of the paracontact endomorphism, respectively. It
also has a simple eigenvalue of 0.

In [4], M. Manev and M. Staikova gave a classification of almost paracontact
Riemannian manifolds $(\mathcal{M},\phi,\xi,\eta,g)$ of type $(n,n)$. The
dimension of $\mathcal{M}$ is $2n+1$ and the induced almost product structure
$P$ of $\phi$ on the paracontact distribution $\ker(\eta)$ is traceless, i.e.
$P$ is an almost paracomplex structure. Because of this, the present authors
called them _almost paracontact almost paracomplex Riemannian manifolds_ in
[5] and continued their study together with S. Ivanov and H. Manev (e.g. [3,
5, 6]).

In the present paper, we study the geometry of almost paracontact almost
paracomplex Riemannian manifolds of the lowest dimension 3. In Section 1, we
recall some necessary facts about the studied manifolds. In Section 2 and
Section 3, we use two different approaches to construct a manifold of the
studied type as a product of a real line and a 2-dimensional manifold. In the
first case, $g$ is the cone metric, and in a second one, $g$ is the so-called
hyperbolic extension, introduced in [3], where the underlying manifold is
paraholomorphic paracomplex Riemannian. Our goal in the present work is to
study the basic curvature properties of the resulting manifolds.

##  1\. Preliminaries

###  1.1. Almost paracontact almost paracomplex Riemannian manifolds

Let $(\mathcal{M},\phi,\allowbreak{}\xi,\eta,g)$ be an _almost paracontact
almost paracomplex Riemannian manifold_ (abbr. _apapR manifold_). This means
that $\mathcal{M}$ is a $(2n+1)$-dimensional differentiable manifold, $\phi$
is a paracontact endomorphism of the tangent bundle $T\mathcal{M}$, $\xi$ is a
characteristic vector field and $\eta$ is its dual 1-form, and $g$ is a
compatible Riemannian metric, such that:

(1.1)
$\begin{array}[]{c}\phi^{2}=\mathcal{I}-\eta\otimes\xi,\quad\eta(\xi)=1,\quad\eta\circ\phi=0,\quad\phi\xi=0,\quad{\rm
tr}\phi=0,\\\\[4.0pt] g(\phi x,\phi y)=g(x,y)-\eta(x)\eta(y),\end{array}$

where $\mathcal{I}$ denotes the identity on $T\mathcal{M}$ [9], [4].

Here and further $x$, $y$, $z$, $w$ will stand for arbitrary elements of the
Lie algebra $\mathfrak{X}(\mathcal{M})$ of tangent vector fields on
$\mathcal{M}$ or vectors in the tangent space $T_{p}\mathcal{M}$ at
$p\in\mathcal{M}$.

The associated metric $\tilde{g}$ of $g$ on
$(\mathcal{M},\allowbreak{}\phi,\allowbreak{}\xi,\allowbreak{}\eta,g)$ is
determined by $\tilde{g}(x,y)=g(x,\phi y)+\eta(x)\eta(y)$. In [4], it is shown
that $\tilde{g}$ is a compatible metric with $(\mathcal{M},\phi,\xi,\eta)$ as
$g$, but $\tilde{g}$ is a pseudo-Riemannian metric of signature $(n+1,n)$.

The fundamental tensor $F$ of type $(0,3)$ on $(\mathcal{M},\phi,\xi,\eta,g)$
is defined by

$F(x,y,z)=g\bigl{(}\left(\nabla_{x}\phi\right)y,z\bigr{)},$

where $\nabla$ is the Levi-Civita connection of $g$. The basic properties of
$F$ with respect to the structure are the following:

$\begin{array}[]{l}F(x,y,z)=F(x,z,y)\\\\[4.0pt] \phantom{F(x,y,z)}=-F(x,\phi
y,\phi z)+\eta(y)F(x,\xi,z)+\eta(z)F(x,y,\xi).\end{array}$

The Lee forms of $(\mathcal{M},\phi,\xi,\eta,g)$ are the following 1-forms
associated with $F$:

(1.2) $\theta(z)=g^{ij}F(e_{i},e_{j},z),\quad\theta^{*}(z)=g^{ij}F(e_{i},\phi
e_{j},z),\quad\omega(z)=F(\xi,\xi,z),$

where $g^{ij}$ are the components of the inverse matrix of $g$ with respect to
a basis $\left\\{\xi;e_{i}\right\\}$ $(i=1,2,\dots,2n)$ of $T_{p}\mathcal{M}$
at an arbitrary point $p\in\mathcal{M}$.

A classification of apapR manifolds is made in [4]. It consists of 11 basic
classes $\mathcal{F}_{1}$, $\mathcal{F}_{2}$, $\dots$, $\mathcal{F}_{11}$ and
it is made with respect to $F$. The components $F_{i}$ of $F$ corresponding to
$\mathcal{F}_{i}$ are determined in [5]. Namely,
$(\mathcal{M},\allowbreak{}\phi,\allowbreak{}\xi,\allowbreak{}\eta,g)$ belongs
to $\mathcal{F}_{i}$ $(i\in\\{1,2,\dots,11\\})$ if and only if the equality
$F=F_{i}$ is valid. Moreover,
$(\mathcal{M},\allowbreak{}\phi,\allowbreak{}\xi,\allowbreak{}\eta,g)\in\mathcal{F}_{i}\oplus\mathcal{F}_{j}\oplus\cdots$
if and only if the following condition is satisfied $F=F_{i}+F_{j}+\cdots$.

In the present work, we consider the case of the lowest dimension of
$(\mathcal{M},\phi,\xi,\eta,g)$, i.e. $\dim{\mathcal{M}}=3$. Let
$\\{e_{0},e_{1},e_{2}\\}$ be a $\phi$-basis of $T_{p}\mathcal{M}$, which
satisfies the following conditions:

(1.3) $\begin{array}[]{ll}\phi e_{0}=0,\quad\phi e_{1}=e_{2},\quad\phi
e_{2}=e_{1},&\xi=e_{0},\\\\[4.0pt]
\eta(e_{0})=1,\quad\eta(e_{1})=\eta(e_{2})=0,&g(e_{i},e_{j})=\delta_{ij},\;i,j\in\\{0,1,2\\}.\end{array}$

The components of $F$, $\theta$, $\theta^{*}$, $\omega$ with respect to
$\left\\{e_{0},e_{1},e_{2}\right\\}$ are denoted by
${F_{ijk}=F(e_{i},e_{j},e_{k})}$, ${\theta_{k}=\theta(e_{k})}$,
${\theta^{*}_{k}=\theta^{*}(e_{k})}$ and ${\omega_{k}=\omega(e_{k})}$,
respectively. According to [5], we have the following:

(1.4)
$\begin{array}[]{c}\begin{array}[]{ll}\theta_{0}=F_{110}+F_{220},&\theta_{1}=F_{111}=-F_{122}=-\theta^{*}_{2},\\\\[4.0pt]
\theta^{*}_{0}=F_{120}+F_{210},&\theta_{2}=F_{222}=-F_{211}=-\theta^{*}_{1},\\\\[4.0pt]
\end{array}\\\
\begin{array}[]{lll}\omega_{0}=0,&\omega_{1}=F_{001},&\omega_{2}=F_{002}.\end{array}\end{array}$

Then, if $F^{s}$, $s\in\\{1,2,\dots,11\\}$, are the components of $F$ in the
corresponding basic classes $\mathcal{F}_{s}$ and $x=x^{i}e_{i}$,
$y=y^{i}e_{i}$, $z=z^{i}e_{i}$ are arbitrary vectors in $T_{p}\mathcal{M}$, we
have the following: [5]

(1.5)
$\begin{array}[]{l}F^{1}(x,y,z)=\left(x^{1}\theta_{1}-x^{2}\theta_{2}\right)\left(y^{1}z^{1}-y^{2}z^{2}\right);\\\\[4.0pt]
F^{2}(x,y,z)=F^{3}(x,y,z)=0;\\\
F^{4}(x,y,z)=\frac{1}{2}\theta_{0}\bigl{\\{}x^{1}\left(y^{0}z^{1}+y^{1}z^{0}\right)+x^{2}\left(y^{0}z^{2}+y^{2}z^{0}\right)\bigr{\\}};\\\\[4.0pt]
F^{5}(x,y,z)=\frac{1}{2}\theta^{*}_{0}\bigl{\\{}x^{1}\left(y^{0}z^{2}+y^{2}z^{0}\right)+x^{2}\left(y^{0}z^{1}+y^{1}z^{0}\right)\bigr{\\}};\\\\[4.0pt]
F^{6}(x,y,z)=F^{7}(x,y,z)=0;\\\\[4.0pt]
F^{8}(x,y,z)=\lambda\bigl{\\{}x^{1}\left(y^{0}z^{1}+y^{1}z^{0}\right)-x^{2}\left(y^{0}z^{2}+y^{2}z^{0}\right)\bigr{\\}},\\\\[4.0pt]
\hskip 38.0pt\lambda=F_{110}=-F_{220};\\\\[4.0pt]
F^{9}(x,y,z)=\mu\bigl{\\{}x^{1}\left(y^{0}z^{2}+y^{2}z^{0}\right)-x^{2}\left(y^{0}z^{1}+y^{1}z^{0}\right)\bigr{\\}},\\\\[4.0pt]
\hskip 38.0pt\mu=F_{120}=-F_{210};\\\\[4.0pt] F^{10}(x,y,z)=\nu
x^{0}\left(y^{1}z^{1}-y^{2}z^{2}\right),\quad\nu=F_{011}=-F_{022};\\\\[4.0pt]
F^{11}(x,y,z)=x^{0}\bigl{\\{}\omega_{1}\left(y^{0}z^{1}+y^{1}z^{0}\right)+\omega_{2}\left(y^{0}z^{2}+y^{2}z^{0}\right)\bigr{\\}}.\end{array}$

By virtue of the latter equations, the 3-dimensional manifolds of the
considered type can belong only to the basic classes $\mathcal{F}_{1}$,
$\mathcal{F}_{4}$, $\mathcal{F}_{5}$, $\mathcal{F}_{8}$, $\mathcal{F}_{9}$,
$\mathcal{F}_{10}$, $\mathcal{F}_{11}$ and their direct sums [5].

The curvature $(1,3)$-tensor $R$ of $\nabla$ is defined as usually by
$R=\left[\nabla,\nabla\right]-\nabla_{[\,\ ]}$. The corresponding
$(0,4)$-tensor is denoted by the same letter and it is determined by
$R(x,y,z,w)=g(R(x,y)z,w)$.

The Ricci tensor $\rho$ and the scalar curvature $\tau$ for $R$ as well as
their associated quantities are defined respectively by:

(1.6)
$\begin{array}[]{ll}\rho(y,z)=g^{ij}R(e_{i},y,z,e_{j}),&\tau=g^{ij}\rho(e_{i},e_{j}),\\\\[4.0pt]
\rho^{*}(y,z)=g^{ij}R(e_{i},y,z,\phi
e_{j}),&\tau^{*}=g^{ij}\rho^{*}(e_{i},e_{j}).\end{array}$

The following tensors are essential curvature-like tensors of type $(1,3)$:

(1.7) $\begin{array}[]{l}\pi_{1}(x,y)z=g(y,z)x-g(x,z)y,\\\\[4.0pt]
\pi_{2}(x,y)z=g(y,\phi z)\phi x-g(x,\phi z)\phi y.\end{array}$

Their corresponding curvature-like $(0,4)$-tensors are determined by $g$ as
usually, $\pi_{i}(x,y,z,w)=g\left(\pi_{i}(x,y)z,w\right)$, $i=1,2$.

Let $\alpha$ be a non-degenerate 2-plane in $T_{p}\mathcal{M}$,
$p\in\mathcal{M}$, having a basis $\\{x,y\\}$. The sectional curvature
$k(\alpha;p)$ is determined by

(1.8) $k(\alpha;p)=\frac{R(x,y,y,x)}{\pi_{1}(x,y,y,x)}.$

###  1.2. Almost paracomplex Riemannian manifold

Consider a differentiable manifold $\mathcal{N}$ of arbitrary dimension. Let
us recall that an _almost product structure_ $P$ on $\mathcal{N}$ is an
endomorphism in the tangent bundle $T\mathcal{N}$ of $\mathcal{N}$ such that
$P^{2}$ is the identity $I$ in $T\mathcal{N}$, but $P$ does not coincide with
$I$. Then, such a manifold $(\mathcal{N},P)$ is called an _almost product
manifold_. In the particular case when the eigenvalues $+1$ and $-1$ of $P$
have the same multiplicity $n$, the structure $P$ is called an _almost
paracomplex structure_ and $(\mathcal{N},P)$ is known as an _almost
paracomplex manifold_ of dimension $2n$ [2]. In this case $P$ is traceless,
i.e. ${\rm tr}P=0$. This kind of manifolds are also known as almost product
Riemannian manifolds with ${\rm tr}P=0$ in [12].

As it is known, the $2n$-dimensional paracontact distribution
$\mathcal{H}=\ker(\eta)$ of $(\mathcal{M},\phi,\xi,\eta,g)$ can be considered
as an almost paracomplex manifold $\mathcal{N}$ equipped with an almost
paracomplex structure $P=\phi|_{\mathcal{H}}$ and a metric
$h=g|_{\mathcal{H}}$, where $\phi|_{\mathcal{H}}$ and $g|_{\mathcal{H}}$ are
the restrictions of $\phi$ and $g$ on $\mathcal{H}$, respectively.

Let $x^{\prime}$, $y^{\prime}$, $z^{\prime}$, $w^{\prime}$ denote arbitrary
vector fields or vectors on $\mathcal{H}$ of $\mathcal{M}$.

Since $g$ is a Riemannian metric of $(\mathcal{M},\phi,\xi,\eta)$, then $h$ is
the corresponding Riemannian metric on $\mathcal{H}$ and due to (1.1) it is
compatible with $P$ as follows

(1.9) $h(Px^{\prime},Py^{\prime})=h(x^{\prime},y^{\prime}).$

The associated Riemannian metric $\widetilde{h}$ of $h$ is determined by

$\widetilde{h}(x^{\prime},y^{\prime})=h(x^{\prime},Py^{\prime})$

and it is a pseudo-Riemannian of signature $(n,n)$.

Let us note that an $2n$-dimensional manifold $\mathcal{N}$, which is equipped
with an almost paracomplex structure $P$ and a Riemannian metric $h$
satisfying (1.9), is known as an _almost paracomplex Riemannian manifold_
$(\mathcal{N},P,h)$.

In [7], A. M. Naveira gave a classification of almost product Riemannian
manifolds $(\mathcal{N},P,h)$ with respect to the covariant derivative
$\nabla^{\prime}P$ for the Levi-Civita connection $\nabla^{\prime}$ of $h$.

Furthermore, using this classification, M. Staikova and K. Gribachev present a
classification of almost paracomplex Riemannian manifolds $(\mathcal{N},P,h)$
in [12]. The basic classes of the Staikova-Gribachev classification are three,
$\mathcal{W}_{1}$, $\mathcal{W}_{2}$ and $\mathcal{W}_{3}$. Their intersection
is the class $\mathcal{W}_{0}$ determined by the condition
$\nabla^{\prime}P=0$. The manifolds of the latter class are known as _locally
product Riemannian manifolds_ [7], _Riemannian $P$-manifolds_ [12] or
_paraholomorphic paracomplex Riemannian manifolds_ [3]. In the present paper,
we call such manifolds briefly $\mathcal{W}_{0}$-manifolds.

Let us remark that $\mathcal{W}_{1}$ contains the manifolds which are locally
conformal equivalent to $\mathcal{W}_{0}$-manifolds.

In the present paper we deal with the case of almost paracomplex Riemannian
manifolds $(\mathcal{N},P,h)$ of the lowest dimension, i.e.
$\dim\mathcal{N}=2$. As it is known such a manifold is a space-form, i.e. the
manifold has a curvature tensor of the form
$R^{\prime}=k^{\prime}\,\pi^{\prime}_{1}$, where $k^{\prime}$ is its pointwise
constant sectional curvature, and $\pi^{\prime}_{1}$ is the essential
curvature-like tensor, such as $\pi_{1}$ in (1.7), but with respect to $h$.
Moreover, each $(\mathcal{N},P,h)$ is a $\mathcal{W}_{1}$-manifold determined
by

(1.10)
$\begin{array}[]{l}F^{\prime}(x^{\prime},y^{\prime},z^{\prime})=\frac{1}{2}\\{h(x^{\prime},y^{\prime})\theta^{\prime}(z^{\prime})+h(x^{\prime},z^{\prime})\theta^{\prime}(y^{\prime})\\\\[4.0pt]
\phantom{F^{\prime}(x^{\prime},y^{\prime},z^{\prime})=\frac{1}{2}\\{}+h(x^{\prime},Py^{\prime})\theta^{\prime*}(z^{\prime})+h(x^{\prime},Pz^{\prime})\theta^{\prime*}(y^{\prime})\\},\end{array}$

where $F^{\prime}$ is the fundamental tensor of $(\mathcal{N},P,h)$ defined by
the following equality
$F^{\prime}(x^{\prime},y^{\prime},z^{\prime})=h\bigl{(}\left(\nabla^{\prime}_{x^{\prime}}P\right)y^{\prime},z^{\prime}\bigr{)}$,
$\theta^{\prime}$ is the Lee form determined analogously as in (1.2) and
$\theta^{\prime*}=-\theta^{\prime}\circ P$ is valid [12].

##  2\. Cone over a 2-dimensional paracomplex Riemannian space-form

Let us consider $\mathcal{C}(\mathcal{N})=\mathbb{R}^{+}\times\mathcal{N}$,
the cone over a 2-dimensional paracomplex space-form $(\mathcal{N},P,h)$,
where $\mathbb{R}^{+}$ is the set of positive reals. We introduce a Riemannian
metric $g$ on $\mathcal{C}(\mathcal{N})$ defined by

(2.1)
$g\left(\left(x^{\prime},a\tfrac{\mathrm{d}}{\mathrm{d}t}\right),\left(y^{\prime},b\tfrac{\mathrm{d}}{\mathrm{d}t}\right)\right)=t^{2}\,h(x^{\prime},y^{\prime})+ab,$

where $t$ is the coordinate on $\mathbb{R}^{+}$ and $a$, $b$ are
differentiable functions on $\mathcal{C}(\mathcal{N})$.

We equip $\mathcal{C}(\mathcal{N})$ with an almost paracontact almost
paracomplex structure $(\phi,\xi,\eta)$ by the following way

(2.2)
$\phi|_{\mathcal{H}}=P,\quad\xi=\tfrac{\mathrm{d}}{\mathrm{d}t},\quad\eta=\mathrm{d}t,\quad\phi\xi=0,\quad\eta\circ\phi=0.$

Obviously, we establish the truthfulness of the following

######  Proposition 2.1.

The manifold $(\mathcal{C}(\mathcal{N}),\phi,\xi,\eta,g)$ is a 3-dimensional
apapR manifold.

According to (2.1) and (2.2) and the well known Koszul equality

(2.3) $\begin{array}[]{l}2g(\nabla_{x}y,z)=xg(y,z)+yg(z,x)-zg(x,y)\\\\[4.0pt]
\phantom{2g(\nabla_{x}y,z)=}+g([x,y],z)+g([z,x],y)+g([z,y],x),\end{array}$

we get the following equalities for the Levi-Civita connection $\nabla$ of
$g$:

$\begin{array}[]{ll}g\left(\nabla_{x^{\prime}}y^{\prime},z^{\prime}\right)=t^{2}\,h\left(\nabla^{\prime}_{x^{\prime}}y^{\prime},z^{\prime}\right),&g\left(\nabla_{\xi}y^{\prime},z^{\prime}\right)=t\,h\left(y^{\prime},z^{\prime}\right),\\\\[4.0pt]
g\left(\nabla_{x^{\prime}}y^{\prime},\xi\right)=-t\,h\left(x^{\prime},y^{\prime}\right),&g\left(\nabla_{x^{\prime}}\xi,z^{\prime}\right)=t\,h\left(x^{\prime},z^{\prime}\right).\end{array}$

Bearing in mind the latter equalities, we obtain the covariant derivatives
with respect to $\nabla$ as follows

(2.4)
$\nabla_{x^{\prime}}y^{\prime}=\nabla^{\prime}_{x^{\prime}}y^{\prime}-\frac{1}{t}g\left(x^{\prime},y^{\prime}\right)\xi,\quad\nabla_{\xi}y^{\prime}=\frac{1}{t}y^{\prime},\quad\nabla_{x^{\prime}}\xi=\frac{1}{t}x^{\prime}.$

By virtue of (2.4), we get

$\begin{array}[]{l}R(x^{\prime},y^{\prime})z^{\prime}=\frac{1}{t^{2}}(k^{\prime}-1)\pi_{1}(x^{\prime},y^{\prime})z^{\prime},\\\\[4.0pt]
R(x^{\prime},y^{\prime})\xi=R(x^{\prime},\xi)y^{\prime}=R(\xi,x^{\prime})y^{\prime}=R(x^{\prime},\xi)\xi=R(\xi,y^{\prime})\xi=0.\end{array}$

Thus, by direct computations, we establish the following

(2.5)
$\begin{array}[]{l}R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=\frac{1}{t^{2}}(k^{\prime}-1)\pi_{1}(x^{\prime},y^{\prime},z^{\prime},w^{\prime}),\\\\[4.0pt]
R(\xi,x^{\prime},y^{\prime},z^{\prime})=R(x^{\prime},\xi,y^{\prime},z^{\prime})=R(x^{\prime},y^{\prime},\xi,z^{\prime})=R(x^{\prime},y^{\prime},z^{\prime},\xi)\\\\[4.0pt]
\phantom{R(\xi,x^{\prime},y^{\prime},z^{\prime})}=R(y^{\prime},\xi,\xi,z^{\prime})=R(\xi,y^{\prime},z^{\prime},\xi)=0.\end{array}$

Using (1.3) and (2.1), we determine the components $h_{ij}=h(e_{i},e_{j})$ and
$g_{ij}=g(e_{i},e_{j})$ with respect to the basis. The non-zero ones of them
are the following:

(2.6) $h_{11}=h_{22}=\frac{1}{t^{2}},\quad g_{00}=g_{11}=g_{22}=1.$

Taking into account (2.3), (2.4) and (2.6), we determine the components of the
covariant derivatives of $e_{i}$ with respect to $\nabla$:

(2.7)
$\begin{array}[]{lll}\nabla_{e_{1}}e_{1}=\nabla^{\prime}_{e_{1}}e_{1}-\frac{1}{t}e_{0},&\nabla_{e_{1}}e_{2}=\nabla^{\prime}_{e_{1}}e_{2},&\nabla_{e_{1}}e_{0}=\frac{1}{t}e_{1},\\\\[4.0pt]
\nabla_{e_{2}}e_{1}=\nabla^{\prime}_{e_{2}}e_{1},&\nabla_{e_{2}}e_{2}=\nabla^{\prime}_{e_{2}}e_{2}-\frac{1}{t}e_{0},&\nabla_{e_{2}}e_{0}=\frac{1}{t}e_{2},\\\\[4.0pt]
\nabla_{e_{0}}e_{1}=\frac{1}{t}e_{1},&\nabla_{e_{0}}e_{2}=\frac{1}{t}e_{2},&\nabla_{e_{0}}e_{0}=0.\end{array}$

Bearing in mind (2.2), (2.6) and (2.7), we obtain the components $F_{ijk}$ of
$F$ with respect to the basis $\\{e_{0},e_{1},e_{2}\\}$. The non-zero ones of
them are:

(2.8) $\begin{array}[]{l}F_{111}=-F_{122}=\theta^{\prime}_{1},\qquad
F_{222}=-F_{211}=\theta^{\prime}_{2},\\\\[4.0pt]
F_{120}=F_{102}=F_{210}=F_{201}=-\frac{1}{t},\end{array}$

where $\theta^{\prime}_{i}=\theta^{\prime}(e_{i})$ for $i=1,2$.

According to (1.4), we compute the components of the Lee forms of
$(\mathcal{C}(\mathcal{N}),\phi,\xi,\allowbreak{}\eta,g)$ and the non-zero
ones of them are:

(2.9)
$\theta_{1}=\theta^{\prime}_{1},\quad\theta_{2}=\theta^{\prime}_{2},\quad\theta^{*}_{1}=-\theta^{\prime}_{2},\quad\theta^{*}_{2}=-\theta^{\prime}_{1},\quad\theta^{*}_{0}=-\frac{2}{t}.$

By virtue of (1.5) and (2.8), we establish the following form of the tensor
$F$:

$F(x,y,z)=(F^{1}+F^{5})(x,y,z).$

According to (1.5) and (1.10), the non-zero components of $F^{1}$ and $F^{5}$
are the following:

(2.10) $\begin{array}[]{l}F^{1}_{111}=-F^{1}_{122}=\theta_{1},\quad
F^{1}_{222}=-F^{1}_{211}=\theta_{2},\\\\[4.0pt]
F^{5}_{120}=F^{5}_{102}=F^{5}_{210}=F^{5}_{201}=\frac{1}{2}\theta^{*}_{0}.\end{array}$

Thus, we obtain the following

######  Theorem 2.2.

The 3-dimensional apapR manifold $(\mathcal{C}(\mathcal{N}),\phi,\xi,\eta,g)$:

  1. (1)

belongs to $\mathcal{F}_{1}\oplus\mathcal{F}_{5}$,

  2. (2)

belongs to $\mathcal{F}_{5}$ if and only if $(\mathcal{N},P,h)$ is a
$\mathcal{W}_{0}$-manifold,

  3. (3)

cannot belong to $\mathcal{F}_{1}$.

###### Proof.

We compare (2.8), (2.9) and (2.10) to accomplish the proof. ∎

Next, bearing in mind (2.5), (2.6) and (2.7), we determine the basic
components $R_{ijk\ell}=R(e_{i},e_{j},e_{k},e_{\ell})$ of $R$. The non-zero
ones of them are obtained by the basic symmetries of $R$ and the following

(2.11) $R_{1212}=-\frac{1}{t^{2}}(k^{\prime}-1).$

Thus, it is valid the following

######  Theorem 2.3.

The 3-dimensional apapR manifold $(\mathcal{C}(\mathcal{N}),\phi,\xi,\eta,g)$
is flat if and only if $k^{\prime}=1$.

Moreover, using (1.8), (2.6) and (2.11), we compute the basic sectional
curvatures $k_{ij}=k(e_{i},e_{j})$ determined by the basis $\\{e_{i},e_{j}\\}$
of the corresponding 2-plane as follows

(2.12) $k_{12}=\frac{1}{t^{2}}(k^{\prime}-1),\qquad k_{01}=k_{02}=0.$

By virtue of (1.6), (2.6) and (2.11), we get the basic components
$\rho_{jk}=\rho(e_{j},e_{k})$ and $\rho^{*}_{jk}=\rho^{*}(e_{j},e_{k})$ of
$\rho$ and $\rho^{*}$, respectively, as well as the values of $\tau$ and
$\tau^{*}$. The non-zero ones of them are the following:

(2.13)
$\begin{array}[]{lll}\rho_{11}=\rho_{22}=-\rho^{*}_{12}=-\rho^{*}_{21}=\frac{1}{2}\tau=\frac{1}{t^{2}}(k^{\prime}-1).\end{array}$

Taking into account (2.12) and (2.13), we get the following

######  Theorem 2.4.

The following properties of $(\mathcal{C}(\mathcal{N}),\phi,\xi,\eta,g)$ are
valid:

  1. (1)

its sectional curvatures of $\xi$-sections vanish;

  2. (2)

it is $*$-scalar flat, i.e. $\tau^{*}=0$;

  3. (3)

$\tau<0$ if and only if $k^{\prime}<1$;

  4. (4)

$\tau>0$ if and only if $k^{\prime}>1$.

##  3\. Hyperbolic extension of a 2-dimensional paracomplex Riemannian space-
form

In this section, we construct a special type of a 3-dimensional warped product
manifold $\mathcal{S}(\mathcal{N})$ of $\mathbb{R}^{+}$ and a paracomplex
space-form $(\mathcal{N},P,h)$ from the class $\mathcal{W}_{1}$. Then,
$(\mathcal{N},P,h)$ is defined by (1.10) for some 1-form $\theta^{\prime}$.

Let $\mathrm{d}t$ be the coordinate 1-form on $\mathbb{R}^{+}$ and let us
introduce an almost paracontact almost paracomplex structure and a Riemannian
metric on $\mathcal{S}(\mathcal{N})$ as follows

(3.1)
$\phi|_{\mathcal{H}}=P,\quad\xi=\tfrac{\mathrm{d}}{\mathrm{d}t},\quad\eta=\mathrm{d}t,\quad\eta\circ\phi=0,\quad
g=\mathrm{d}t^{2}+\cosh{2t}\,h+\sinh{2t}\,\widetilde{h}.$

Then, it is easy to check the following

######  Proposition 3.1.

The manifold $(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$ is a 3-dimensional
apapR manifold.

In the partial case, when $(\mathcal{N},P,h)$ is paraholomorphic, i.e.
$\nabla^{\prime}P=0$, then the Riemannian manifold
$(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$ is para-Sasaki-like [3].

Taking into account (1.10), (2.3) and (3.1), we get the following formulae for
the Levi-Civita connections $\nabla$ and $\nabla^{\prime}$ of $g$ and $h$,
respectively:

(3.2)
$\begin{array}[]{c}\nabla_{x^{\prime}}y^{\prime}=\nabla^{\prime}_{x^{\prime}}y^{\prime}+\frac{1}{2}\sinh{2t}\left\\{g\left(x^{\prime},y^{\prime}\right)\theta^{\prime\sharp}-g\left(x^{\prime},Py^{\prime}\right)P\theta^{\prime\sharp}\right\\}-g\left(x^{\prime},Py^{\prime}\right)\xi,\\\\[4.0pt]
\nabla_{\xi}y^{\prime}=Py^{\prime},\qquad\nabla_{x^{\prime}}\xi=Px^{\prime},\qquad\nabla_{\xi}\xi=0.\end{array}$

In the latter equalities and further, $\theta^{\prime\sharp}$ denotes the dual
vector of $\theta^{\prime}$ with respect to $h$ on $(\mathcal{N},P,h)$.
Analogously, $\theta^{\sharp}$ stands for the dual vector of $\theta$ with
respect to $g$ on $(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$. Bearing in
mind (3.1), we have the following relations

$\theta^{\sharp}|_{\mathcal{H}}=\cosh 2t\,\theta^{\prime\sharp}-\sinh
2t\,P\theta^{\prime\sharp},\qquad\theta(z^{\prime})=\theta^{\prime}(z^{\prime}).$

Taking into account (3.2), we compute the following

(3.3)
$\begin{array}[]{l}R(x^{\prime},y^{\prime})z^{\prime}=k^{\prime}\left\\{\cosh
2t\,\pi_{1}(x^{\prime},y^{\prime})z^{\prime}-\sinh
2t\,P\pi_{2}(x^{\prime},y^{\prime})z^{\prime}\right\\}-\pi_{2}(x^{\prime},y^{\prime})z^{\prime}\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime})z^{\prime}=}+\frac{1}{2}\sinh
2t\left[g\left(y^{\prime},z^{\prime}\right)\nabla_{x^{\prime}}\theta^{\prime\sharp}-g\left(x^{\prime},z^{\prime}\right)\nabla_{y^{\prime}}\theta^{\prime\sharp}\right.\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime})z^{\prime}=+\frac{1}{4}\sinh
2t\left[\right.}\left.-g\left(y^{\prime},Pz^{\prime}\right)\nabla_{x^{\prime}}P\theta^{\prime\sharp}+g\left(x^{\prime},Pz^{\prime}\right)\nabla_{y^{\prime}}P\theta^{\prime\sharp}\right]\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime})z^{\prime}=}+\left.\frac{1}{4}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},y^{\prime},z^{\prime},\theta^{\prime\sharp})\left\\{\sinh
2t\,P\theta^{\sharp}|_{\mathcal{H}}+2\cosh 2t\,\xi\right\\}\right.\\\\[4.0pt]
R(x^{\prime},y^{\prime})\xi=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},y^{\prime})\theta^{\sharp}|_{\mathcal{H}},\\\\[4.0pt]
R(\xi,y^{\prime})z^{\prime}=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(\theta^{\sharp}|_{\mathcal{H}},y^{\prime})z^{\prime}-g\left(y^{\prime},z^{\prime}\right)\xi,\\\\[4.0pt]
R(x^{\prime},\xi)z^{\prime}=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},\theta^{\sharp}|_{\mathcal{H}})z^{\prime}+g\left(x^{\prime},z^{\prime}\right)\xi.\end{array}$

Bearing in mind (3.1) and (3.3), we obtain the following

(3.4)
$\begin{array}[]{l}R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=k^{\prime}\\{\cosh
2t\,\pi_{1}(x^{\prime},y^{\prime},z^{\prime},w^{\prime})-\sinh
2t\,\pi_{2}(x^{\prime},y^{\prime},z^{\prime},Pw^{\prime})\\}\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=}-\pi_{2}(x^{\prime},y^{\prime},z^{\prime},w^{\prime})\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=}+\frac{1}{2}\sinh
2t\left\\{g\left(y^{\prime},z^{\prime}\right)g\left(\nabla_{x^{\prime}}\theta^{\prime\sharp},w^{\prime}\right)\right.\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=+\frac{1}{2}\sinh
2t\left\\{\right.}-g\left(x^{\prime},z^{\prime}\right)g\left(\nabla_{y^{\prime}}\theta^{\prime\sharp},w^{\prime}\right)\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=+\frac{1}{2}\sinh
2t\left\\{\right.}-g\left(y^{\prime},Pz^{\prime}\right)g\left(\nabla_{x^{\prime}}P\theta^{\prime\sharp},w^{\prime}\right)\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=+\frac{1}{2}\sinh
2t\left\\{\right.}\left.+g\left(x^{\prime},Pz^{\prime}\right)g\left(\nabla_{y^{\prime}}P\theta^{\prime\sharp},w^{\prime}\right)\right\\}\\\\[4.0pt]
\phantom{R(x^{\prime},y^{\prime},z^{\prime},w^{\prime})=}+\left.\frac{1}{4}\sinh
2t\,\\{\pi_{1}-\pi_{2}\\}(x^{\prime},y^{\prime},z^{\prime},\theta^{\prime\sharp})\theta^{\prime}(Pw^{\prime}),\right.\\\\[4.0pt]
R(x^{\prime},y^{\prime},z^{\prime},\xi)=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},y^{\prime},z^{\prime},\theta^{\sharp}|_{\mathcal{H}}),\\\\[4.0pt]
R(x^{\prime},y^{\prime},\xi,w^{\prime})=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},y^{\prime},\theta^{\sharp}|_{\mathcal{H}},w^{\prime}),\\\\[4.0pt]
R(x^{\prime},\xi,z^{\prime},w^{\prime})=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(x^{\prime},\theta^{\sharp}|_{\mathcal{H}},z^{\prime},w^{\prime}),\\\\[4.0pt]
R(\xi,y^{\prime},z^{\prime},w^{\prime})=\frac{1}{2}\\{\pi_{1}-\pi_{2}\\}(\theta^{\sharp}|_{\mathcal{H}},y^{\prime},z^{\prime},w^{\prime}),\\\\[4.0pt]
R(\xi,y^{\prime},z^{\prime},\xi)=-g\left(y^{\prime},z^{\prime}\right).\end{array}$

According to (1.3) and (3.1), we determine the components $g_{ij}$ and
$h_{ij}$ as follows

(3.5) $\begin{array}[]{ll}g_{00}=g_{11}=g_{22}=1,&g_{12}=g_{21}=0,\\\\[4.0pt]
h_{11}=h_{22}=\cosh 2t,&h_{12}=h_{21}=-\sinh 2t.\end{array}$

Bearing in mind (2.3), (3.2) and (3.5), we obtain

(3.6)
$\begin{array}[]{ll}\nabla_{e_{1}}e_{1}=\nabla^{\prime}_{e_{1}}e_{1}+\frac{1}{2}\sinh
2t\,\theta^{\prime\sharp},\qquad\nabla_{e_{1}}e_{2}=\nabla^{\prime}_{e_{1}}e_{2}-\frac{1}{2}\sinh
2t\,P\theta^{\prime\sharp}-\xi,\\\\[4.0pt]
\nabla_{e_{2}}e_{2}=\nabla^{\prime}_{e_{2}}e_{2}+\frac{1}{2}\sinh
2t\,\theta^{\prime\sharp},\qquad\nabla_{e_{2}}e_{1}=\nabla^{\prime}_{e_{2}}e_{1}-\frac{1}{2}\sinh
2t\,P\theta^{\prime\sharp}-\xi,\\\\[4.0pt]
\nabla_{e_{0}}e_{1}=\nabla_{e_{1}}e_{0}=e_{2},\qquad\nabla_{e_{0}}e_{2}=\nabla_{e_{2}}e_{0}=e_{1},\qquad\nabla_{e_{0}}e_{0}=0.\end{array}$

Then, applying (3.1), (3.5) and (3.6), we determine the components $F_{ijk}$
of $F$. The non-zero of them are the following

(3.7) $\begin{array}[]{l}F_{111}=-F_{122}=\theta^{\prime}_{1},\qquad
F_{222}=-F_{211}=\theta^{\prime}_{2},\\\\[4.0pt]
F_{101}=F_{110}=F_{202}=F_{220}=-1.\end{array}$

According to (1.4), we get the components of the Lee forms of
$(\mathcal{S}(\mathcal{N}),\phi,\xi,\allowbreak{}\eta,g)$ with respect to the
basis. The non-zero of them are:

(3.8)
$\theta_{0}=-2,\quad\theta_{1}=-\theta^{*}_{2}=\theta^{\prime}_{1},\quad\theta_{2}=-\theta^{*}_{1}=\theta^{\prime}_{2}.$

By virtue of (1.5) and (3.7), we establish the truthfulness of the following
equality

$F(x,y,z)=(F^{1}+F^{4})(x,y,z).$

By virtue of (1.5) and (1.10), we have the following non-zero components of
$F^{1}$ and $F^{4}$ with respect to the basis (1.3):

(3.9) $\begin{array}[]{l}F^{1}_{111}=-F^{1}_{122}=\theta_{1},\quad
F^{1}_{222}=-F^{1}_{211}=\theta_{2},\\\\[4.0pt]
F^{4}_{101}=F^{4}_{110}=F^{4}_{202}=F^{4}_{212}=\frac{1}{2}\theta_{0}.\end{array}$

Thus, we get the following

######  Theorem 3.2.

The 3-dimensional apapR manifold $(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$:

  1. (1)

belongs to $\mathcal{F}_{1}\oplus\mathcal{F}_{4}$,

  2. (2)

belongs to $\mathcal{F}_{4}$ if and only if $(\mathcal{N},P,h)$ is a
$\mathcal{W}_{0}$-manifold,

  3. (3)

cannot belong to $\mathcal{F}_{1}$.

###### Proof.

It follows from the above by comparing (3.7), (3.8) and (3.9). ∎

Taking into account (3.4), (3.5) and (3.6), we obtain the components
$R_{ijk\ell}$ of $R$ with respect to the basis. The non-zero of them are
determined by the basic symmetries of $R$ and the following equalities

(3.10) $\begin{array}[]{l}R_{1221}=k^{\prime}\cosh 2t+1\\\\[4.0pt]
\phantom{R_{1221}=}+\frac{1}{2}\sinh
2t\,\left\\{g\left(\nabla_{e_{1}}\theta^{\prime\sharp},e_{1}\right)+g\left(\nabla_{e_{2}}P\theta^{\prime\sharp},e_{1}\right)\right\\}+\frac{1}{2}\theta^{\prime}_{1}\theta^{\prime}_{2},\\\\[4.0pt]
R_{1210}=-\theta^{\prime}_{2},\quad R_{1220}=\theta^{\prime}_{1},\quad
R_{0110}=R_{0220}=-1.\end{array}$

Bu virtue of (1.8), (3.5) and (3.10), we calculate the basic sectional
curvatures $k_{ij}$ as follows

(3.11) $k_{12}=R_{1221},\qquad k_{01}=k_{02}=-1.$

Furthermore, from (1.6), (3.5) and (3.10), we obtain the basic components
$\rho_{jk}$ and $\rho^{*}_{jk}$ as well as the values $\tau$ and $\tau^{*}$ as
follows:

(3.12)
$\begin{array}[]{lll}\rho_{11}=\rho_{22}=R_{1221}-1,&\rho_{00}=-2,&\rho^{*}_{00}=\rho^{*}_{11}=\rho^{*}_{22}=0,\\\\[4.0pt]
\rho_{12}=\rho_{21}=0,&\rho_{01}=\rho_{10}=\theta^{\prime}_{1},&\rho_{02}=\rho_{20}=\theta^{\prime}_{2}\\\\[4.0pt]
\rho^{*}_{12}=\rho^{*}_{21}=-R_{1221},&\rho^{*}_{01}=\rho^{*}_{10}=-\theta^{\prime}_{2},&\rho^{*}_{02}=\rho^{*}_{20}=-\theta^{\prime}_{1},\\\\[4.0pt]
\tau=2R_{1221}-4,&\tau^{*}=0.\end{array}$

Using (3.11) and (3.12), we conclude the following

######  Proposition 3.3.

The manifold $(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$ has the following
properties:

  1. (1)

It has constant negative $\xi$-sectional curvatures;

  2. (2)

It is $*$-scalar flat.

Bearing in mind (3.12) and (3.5) for $g_{ij}$, we obtain the following

######  Theorem 3.4.

The following properties of $(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$ are
equivalent:

  1. (1)

$(\mathcal{N},P,h)$ is a $\mathcal{W}_{0}$-manifold;

  2. (2)

$\rho=k^{\prime}\cosh 2t\,g-(2+k^{\prime}\cosh 2t)\eta\otimes\eta$;

  3. (3)

$\rho^{*}=-(1+k^{\prime}\cosh 2t)(\widetilde{g}-\eta\otimes\eta)$.

Let us remark that (1) of the latter theorem is equivalent to the fact that
$(\mathcal{S}(\mathcal{N}),\phi,\xi,\eta,g)$ is an $\mathcal{F}_{4}$-manifold,
according to (2) of Theorem 3.2.

The expression of the Ricci tensor in (2) of Theorem 3.4 means that, in this
special case, $(\mathcal{S}(\mathcal{N}),\phi,\xi,\allowbreak{}\eta,g)$ is an
para-$\eta$-Einstein manifold following [6].

Then, we obtain immediately the following

######  Corollary 3.5.

If $(\mathcal{N},P,h)$ is a $\mathcal{W}_{0}$-manifold, then
$(\mathcal{S}(\mathcal{N}),\allowbreak{}\phi,\xi,\eta,g)$ has the properties:

  1. (1)

$\tau=2(k^{\prime}\cosh 2t-1)$;

  2. (2)

$k^{\prime}<0$ if and only if $\tau\leq 2(k^{\prime}-1)<-2$;

  3. (3)

$k^{\prime}=0$ if and only if $\tau=-2$;

  4. (4)

$k^{\prime}>0$ if and only if $-2<2(k^{\prime}-1)\leq\tau$.

### Acknowledgment

The authors were supported by MU21-FMI-008 and the first author was partially
supported by FP21-FMI-002, both projects of the Scientific Research Fund,
University of Plovdiv Paisii Hilendarski, Bulgaria.

## References

  * [1] Adati, T., Miyazawa, T., On paracontact Riemannian manifolds, TRU Math. 13 (1977), 27–39.
  * [2] Cruceanu, V., Fortuny, P., Gadea, P. M., A survey on paracomplex geometry, Rocky Mountain J. Math. 26 (1996), 83–115.
  * [3] Ivanov, S., Manev, H., Manev, M., Para-Sasaki-like Riemannian manifolds and new Einstein metrics, arXiv:2008.04163.
  * [4] Manev, M., Staikova, M., On almost paracontact Riemannian manifolds of type $(n,n)$, J. Geom. 72 (2001), 108–114. https://doi.org/10.1007/s00022-001-8572-2
  * [5] Manev, M., Tavkova, V., On the almost paracontact almost paracomplex Riemannian manifolds, Facta Univ. Ser. Math. Inform. 33 (2018), 637–657.
  * [6] Manev, M., Tavkova, V., Lie groups as 3-dimensional almost paracontact almost paracomplex Riemannian manifolds, J. Geom. 110 (2019), 43. https://doi.org/10.1007/s00022-019-0499-6
  * [7] Naveira, A.M., A classification of Riemannian almost product manifolds, Rend. Mat. (7) 3 (1983), 577–592.
  * [8] S. Sasaki, On paracontact Riemannian manifolds. TRU Math. 16 (1980), 75–86.
  * [9] I. Satō, On a structure similar to the almost contact structure. Tensor (N.S.) 30 (1976), 219–224.
  * [10] I. Satō, On a structure similar to almost contact structure II, Tensor (N.S.) 31 (1977), 199–205.
  * [11] I. Satō, On a Riemannian manifold admitting a certain vector field, Kodai Math. Sem. Rep. 29 (1978), 250–260.
  * [12] Staikova, M., Gribachev, K., Canonical connections and their conformal invariants on Riemannian P-manifolds, Serdica Math. P. 18 (1992), 150–161.


# Integration of Process Design and Dynamic Optimization (or Optimal Control)
using Reinforcement Learning

Steven Sachio  Dongda Zhang  Maria Papathanasiou  a.del-rio-
<EMAIL_ADDRESS><EMAIL_ADDRESS>*  *  *

(RECEIVED DATE; REVISED DATE; ACCEPTED DATE)

###### Abstract

The performance of a chemical plant is highly dependent on its design and
control. A design cannot be accurately evaluated without its controls and vice
versa. To optimally address design and control simultaneously, one could
formulate a bi-level mixed-integer nonlinear program with control as the inner
problem (dynamic optimization) and design as the outer problem (MINLP). This
was, however, a previously intractable problem. Now, by computing an optimal
policy using reinforcement learning, it is possible to design and embed a
controller with a closed-form expression into the mathematical program. The
approach is tested in two simple design case studies (tank and CSTR) and the
performance of the controller is evaluated. It is shown that the proposed
approach outperforms current state-of-the-art control strategies. This opens a
new avenue to address the simultaneous design and control of engineering
systems.

###### keywords:

Bi-level optimization; Reinforcement Learning; Policy Gradient; Process
Control; Optimal Design

††articletype: RESEARCH ARTICLE††journal: Data-Centric Engineering

2020 10.1017/dce.2020.7

0000-0003-3826-0548 Include Include

[1]Antonio E. del-Rio ChanonaInclude [3]Panagiotis PetsagkourakisInclude

Steven Sachio et al.

[1]Department of Chemical Engineering, Imperial College London, Imperial
College Rd, Kensington, London, SW7 2AZ, United Kingdom

[2]Department of Chemical Engineering and Analytical Science, The University
of Manchester, Manchester, M1 3AL, United Kingdom

[3]Department of Chemical Engineering, University College London, Bloomsbury,
London, WC1E 7JE, United Kingdom

[Impact Statement] To create efficient-high performing processes, one would
need to find an optimal design of the process and a controller that can ensure
operation at the optimum conditions in the presence of uncertainty. A design
cannot be accurately evaluated without its controls and vice versa.
Simultaneous process design and control aims to address this problem by
providing the framework to create an optimal high-performing process. However,
traditional approaches to this involves the formulation of a bi-level mixed-
integer nonlinear program with control as the inner problem (dynamic
optimization) and design as the outer problem (MINLP) which is intractable. In
this work, to address the intractable bi-level problem, an optimal policy was
computed using reinforcement learning to design and embed a controller with a
closed-form expression into the mathematical program. Compared to the
traditional approaches, the RL approach was able to solve the bi-level
problems and show great control performance. Furthermore, this method can be
applied to both renewable and non-renewable processes and from a methods point
of view it could also be applied to other bi-level optimization problem.

##  1 Introduction

With the extensive progress that the Process Systems Engineering (PSE)
community has made over the years, it has been long understood that to improve
operability, flexibility, reliability and economic aspect of a process, both
the design problem and the operational time-variant decisions must be
considered simultaneously in the initial design phase of a process plant.
Operational time-variant decision includes short-term process regulatory
decisions (process control) and long-term economical decisions (scheduling).
Recent studies are focusing on complete integration of design, control and
scheduling to guarantee operability and profitability of operation under all
foreseeable conditions (burnak2019).

The scheduling problem addresses the allocation of resources, time, process
unit utilization, and also maintenance based on some economic criteria over
long periods of time, while the control problem is about maintaining optimal
operating conditions under operational uncertainty while satisfying any
physical limitations such as environmental and safety constraints. Due to the
difference in objective and time scales, integration of design, scheduling and
control together remains to be a great challenge left to be unsolved.
Integration between design and scheduling, and between design and control on
the other hand has seen some progress (pistikopoulos2016).

In this work, we will be focusing on the integration of design and control.
The design problem of a chemical plant answers questions such as whether we
would need a certain process unit, how big the process unit should be and what
are my optimal operating conditions. It is common to formulate this problem as
a mixed-integer non-linear programming (MINLP) problem. The objective function
of the design problem is usually based on an economic index which accounts
various factors such as operating and capital costs.

The control problem on the other hand, most commonly known as the optimal
control problem (OCP), are usually formulated as a dynamic optimization (DO)
problem. It answers the question of how do I adjust the manipulated process
variables in order to maintain operation at the optimal operating condition.
Therefore, in order to fully evaluate a design we would need to ensure that a
control strategy exist such that the design can be maintained at the optimal
operating condition. Similarly, to evaluate a control strategy, we would need
to apply it to a design first.

Hence, to fully evaluate a process plant we would need to consider both the
design and control problem simultaneously, meaning that it will be necessary
to couple both optimization problems together. Traditional approach to solve
the problem is by simply nesting the control problem inside the design problem
and solving the bi-level optimization problem directly (narraway1994). This
approach becomes intractable for larger systems and therefore other approaches
which relies on the offline solution of the control problem using multi-
parametric model predictive control (mp-MPC) were proposed (diangelakis2017).
However, the mp-MPC method required reduction of the high-fidelity model to
solve the control problem which may result in loss of information. In this
work, to address the intractable bi-level problem, a new approach is proposed
using an optimal policy computed using reinforcement learning to design and
embed a controller with a closed-form expression into the mathematical
program.

###  1.1 Past Approaches

One of the most significant contributions to simultaneous design and control,
in the beginning, was the “back-off approach” presented by narraway1991. This
method can handle the design of linear and mildly non-linear processes, and it
can be broken down to three essential steps: (i) do steady-state non-linear
process optimization, (ii) linearize the problem at the optimum point and,
(iii) “back-off” from the optimal solution a distance to ensure that the
process is feasible under disturbances. narraway1994 used this approach
forming an MINLP problem using PI control to solve the double effect
evaporator problem and bahri1995 also employed this approach on optimal
control to handle process uncertainties. It was shown that the approach was
effective to select between different control schemes and alternative flow
sheets.

In 1992, the International Federation of Automatic Control (IFAC) organised
the first workshop on simultaneous design and control at Imperial College
London which paved the way for the emergence of many different approaches to
simultaneous control and design. To name a few, luyben1992 formulated a multi-
objective optimization algorithm based on cutting planes involving open-loop
controllability measures and economics, and shah1992 implemented the State-
Task Network (STN) scheme taking scheduling into consideration for
multipurpose bath plants. At the time, literature about PI controllers was
already abundant and well-established, coupled with the fact that PI
controllers are relatively easy to integrate due to its explicit form it
became the most popular option for numerous research. The use of PI
controllers was acceptable when dealing with less complex SISO (single input
single output) systems. When dealing with more complex MIMO (multiple input
multiple output) systems, PI controller could not be used reliably without any
advanced modifications.

Compared to the PI controller, the MPC uses an explicit model of the process
and postulates a constrained dynamic optimization problem on it. One of the
first significant contributions to implementing the MPC for simultaneous
design and control was done by brengel1992. In the formulation, the authors
implemented a bi-level optimization problem, where the outer level is the
design problem formulated as an MINLP with an economic related objective
function while the inner level is the control problem formulated as an MPC
formulation. This bi-level optimization problem is extremely expensive to
solve online and this is why there was significantly less work done in MPC
compared to PI controllers for simultaneous process design and control at the
time.

###  1.2 Contribution

Several contributions has been made to address this problem, chu2014 have
proposed a novel method utilizing a decomposition algorithm to solve bi-level
integrated scheduling and optimization for sequential batch processes, while
beykal2020 have proposed a data-driven framework to tackle bi-level
optimization problems. An approach to solve the bi-level problem was presented
by bemporad2002; sakizlis2004a where the authors replaced the inner problem
with a multi-parametric model predictive controller (mp-MPC) reducing it to a
simple look-up table algorithm. In diangelakis2017 the authors worked further
upon this approach and formulated a "design-dependent offline controller",
where it only requires solving the mp-MPC problem once offline. The biggest
drawback of these mp-MPC approach is that they rely in linearizations which do
not represent well nonlinear dynamics.

To solve this intractable bi-level optimization problem, we propose to take
advantage of the closed form (explicit) nature of reinforcement learning
controllers. Reinforcement learning (RL) controllers (sutton2018) are a
natural choice, as they can address disturbances in highly nonlinear and
complex processes without the need for linearizations which may result in loss
of information (petsagkourakis2020).

###  1.3 Structure

In the next section, Section 2, an introduction to the formulation of the
simultaneous design and control problem is shown. Afterwards, in Section 3,
background knowledge for bi-level optimization, neural networks and policy
gradient are presented. For readers knowledgeable in these topics it is
recommended to skip Section 3. In Section 4, the methodology of the new
approach is discussed. The results and discussion of the two case studies
carried out follows after in Section 5 and finally the conclusions in Section
6.

##  2 Problem Statement

Given a simultaneous process design and control problem, first a bi-level
optimization problem is formulated and then split into two, the design problem
and the control problem. The design problem is formulated as a mixed integer
dynamic optimization (MIDO) problem as shown in (1).

$\begin{split}\min_{\bm{Y},\bm{p}}~{}&J_{SDC}=\int_{0}^{T_{F}}\mathcal{P}(\bm{x},\bm{y},\bm{u},\bm{Y},\bm{d},\bm{p})d\tau\\\
s.t.~{}&\frac{dx}{d\tau}=f(\bm{x},\bm{y},\bm{u},\bm{Y},\bm{d},\bm{p})\\\
&\bm{y}_{min}\leq\bm{y}=g(\bm{x},\bm{u},\bm{Y},\bm{d},\bm{p})\leq\bm{y}_{max}\\\
&\bm{u}_{t}=\pi_{\theta}(\bm{x}_{t}^{obs},\bm{x}_{t-1}^{obs},\bm{y}_{t}^{obs},\bm{y}_{t-1}^{obs},\bm{p}^{obs})~{}\forall
t\\\ &\bm{Y}\in\\{0,1\\},~{}\bm{x}_{t}^{obs}\in\bm{x}_{t},\\\
&\bm{y}_{t}^{obs}\in\bm{y}_{t},~{}\bm{p}^{obs}\in\bm{p}\\\
&[\bm{x}_{min},\bm{d}_{min}]\leq[\bm{x}_{\tau},\bm{d}_{\tau}]\leq[\bm{x}_{max},\bm{d}_{max}]~{}\forall\tau\\\
&\bm{p}_{min}\leq\bm{p}\leq\bm{p}_{max}\\\
&\tau\in[0,T_{F}],~{}t\in\\{1,...n_{T}-1\\}\\\ \end{split}$ (1)

where $\tau$ is the continuous variable time $\tau\in[0,T_{F}]$, $T_{F}$ is
the time horizon discretized in $n_{T}$ time steps of size
$\Delta\tau=T_{F}/n_{T}$. The subscript $t$ is the time step number, it
represents the value of a variable at time $\tau=t\Delta\tau$ where
$t\in\\{0,~{}1,...,~{}n_{T}-1,~{}n_{T}\\}$. While $\bm{x}\in R^{n_{x}}$ is the
vector of states of the system, $\bm{y}\in R^{n_{y}}$ is the vector of system
outputs, $\bm{u}\in R^{n_{u}}$ is the control action represented discretely as
a piece-wise constant variable, $\bm{d}$ is the disturbance vector, $\bm{p}$
is the design variable vector including the steady-state manipulated
variables, $\mathcal{P}$ is the integral function of the objective function
$J_{SDC}$ (usually an economic/efficiency related cost function), $f$ are the
differential equations describing the process dynamics and $g$ are the
constraints. The vectors with superscript $obs$ are the values of the vector
which is observable by the controller, given that not the whole state of the
process is necessarily observable.

The controller in (1) is represented by a policy $\pi$ parameterized by a set
of parameters $\theta$. The policy could be given as a linear controller, e.g.
(chu2014; diangelakis2017). However, this approach has a limitation; there is
no close form expression for nonlinear dynamic systems. As a result an
appropriate approximation via linearization is implemented (diangelakis2017)
and subsequently a multi-parametric programming technique is applied to
compute the explicit solution of the optimal control problem. In this paper,
we avoid the use of approximations in the dynamic systems and reinforcement
learning is utilized to construct closed form expression for the general
stochastic closed-loop optimal control problem (see (2) and substitute in
(1)). Specifically policy gradient technique is proposed to construct an
optimal policy $\pi_{\theta}$ as an artificial neural network (ANN).

The goal of the design problem in (1) is to find combinations of the design
and binary variables (e.g. reactor type, process layout), given an optimized
controller to minimize an objective function $J_{SDC}$ over a time horizon
$T_{F}$ discretized in $n_{T}$ number of time steps of size
$\Delta\tau=T_{F}/n_{T}$ while satisfying the constraints $f$ and $g$. To get
the optimized controller, an optimal control problem (OCP) is formulated in a
way that the controller would also take in the observable design variable as
its input. The formulation is shown in (2).

$\begin{split}\max_{\pi_{\theta}}~{}&\mathbb{E}[J_{OCP}(\bm{x}_{t}^{obs},\bm{y}_{t}^{obs},\bm{u}_{t},\bm{p})]\\\
s.t.~{}&\bm{x}(0)=\bm{x}_{0}\\\
&\bm{x}_{t+1}=f(\bm{x}_{t},\bm{u}_{t},\bm{d}_{t},\bm{p})~{}\forall t\\\
&\bm{u}_{t}=\pi(\bm{x}_{t}^{obs},\bm{x}_{t-1}^{obs},\bm{y}_{t}^{obs},\bm{y}_{t-1}^{obs},\bm{p}^{obs})\\\
&\bm{x}_{t}^{obs}\in\bm{x}_{t},~{}\bm{y}_{t}^{obs}\in\bm{y}_{t},~{}\bm{p}^{obs}\in\bm{p}\\\
&[\bm{x}_{min},\bm{u}_{min}]\leq[\bm{x}_{t},\bm{u}_{t}]\leq[\bm{x}_{max},\bm{u}_{max}]~{}\forall
t\\\ &\bm{y}_{min}\leq\bm{y}_{t}\leq\bm{y}_{max}~{}\forall t\\\
&t\in\\{1,...,n_{T}-1\\}\\\ \end{split}$ (2)

where $J_{OCP}$ is the cost function of the OCP problem e.g. setpoint error
(not to be confused with the objective in (1)). The goal of this optimal
control problem is to find the optimum parameters of the policy that maximize
the expectation of the objective function. Notice that this problem is
intractable as a closed-loop policy $\pi$ needs to be found. To solve this
problem the policy is parameterized by an ANN ($\pi_{\theta}$) and then a
policy gradient method is used (petsagkourakis2020; sutton2018) to solve the
optimal control problem.

This is a bi-level optimization problem, with the outer level being a MINLP
and the inner level an OCP. By using RL this problem can be addressed. The bi-
level optimization problem can be separated and formulated as presented in
(2). Notice that the OCP takes into account the design variables to calculate
the control output. When this is solved by RL, the final form of the policy is
an explicit function, that takes states and design variables as input, and
outputs an optimal control action, this can be embedded into the MIDO problem
formulated in (1). Remark: If ReLu activation functions are used, these result
in a mixed integer linear function, which can be easily appended to the MINLP
problem.

##  3 Preliminaries

The methodology used in this work will involve solving a bi-level optimization
problem using a Policy Gradient controller with Artificial Neural Networks
(ANNs). In this section, a brief background of each topic is presented.

###  3.1 Bi-level Problem

A bi-level optimization problem is an optimization problem where another
problem is nested within. Let us consider a simple bi-level problem shown in
(3).

$\begin{split}\min_{x}~{}&OBJ_{1}(x,u)\\\ s.t.~{}&f_{1}(x)=0\\\ &g_{1}(x)\leq
0\\\ &\begin{split}\min_{u}~{}&OBJ_{2}(x,u)\\\ s.t.~{}&f_{2}(x,u)=0\\\
&g_{2}(x,u)\leq 0\\\ \end{split}\end{split}$ (3)

where $x$ is the design variable, $u$ is control variable, $OBJ_{1}$ is the
design objective (e.g. operating cost), $f_{1}$ and $g_{1}$ are the equality
and inequality constraints of the design problem respectively (e.g. maximum
column length), $OBJ_{2}$ is the control objective (e.g. error for setpoint
tracking), and $f_{2}$ and $g_{2}$ are the equality and inequality constraints
of the control problem respectively (e.g. maximum temperature).

The outer level problem is the design problem which goal is to optimize an
economic related objective function, while the inner level problem is the
control problem which goal is to find the optimum control actions to maintain
optimal operating condition given the design variables. In this approach, to
get an optimum value of $OBJ_{1}$ we would need to explore different values of
design variable, $x$, and at each different design we would need to solve the
control problem to ensure/create a controller which can maintain optimum
operating condition for that design. For large problems with high number of
variable and constraints, this formulation quickly becomes intractable.
Furthermore, it is often important to look into the dynamic perspective of the
system which would introduce the time dimension and hence larger problem to
solve.

In this work, the problem tackled is similar like in (3) but with more
variables, variable types, equations and including the time dimension. Using
traditional approach to solve this problem is intractable, there in this work
we propose to take advantage of the closed form (explicit) nature of
reinforcement learning controllers.

###  3.2 Artificial Neural Networks (ANNs)

The form of our policy gradient controller is the simplest form of neural
networks, feed-forward neural network. In brief, it is essentially a
parameterized function taking in states of the system and giving out control
actions as its output. An ANN is composed of individual nodes/neurons in which
each neuron may have the same or different parameters and activation
functions. To make the concept more concrete, let us consider a simple neural
network shown in Fig. 1.

Figure 1: Simple feed-forward artificial neural network.

Each neuron has its own parameters called weights and biases and they are
denoted by $w^{[l](n)}$ and $b^{[l](n)}$ respectively. The superscript $[l]$
corresponds to the layer number while the superscript $(n)$ corresponds to the
node number in that layer. In each neuron, the weight is a vector with the
same size as the input to that node and the bias is a scalar.

There are two steps of calculations done in a neuron, first is using the
weights and bias to calculate a value $z^{[l](n)}$ and then using the
activation function of that layer to calculate $a^{[l](n)}$. Activation
functions are used to help with the non-linearity, without it the neural
network would just be the same as a linear regression. More details and
equations on how the calculations are done in the neural network can be found
in (bebis1994).

The policy in our formulation is in the form of an ANN just like the one shown
in Fig. 1 but different numbers of layers and nodes. The input and all of the
hidden layers use the tanh activation function while the output layer nodes
use the ReLU6 and leaky ReLU function for the mean and std of the control
action, respectively. Details on the ReLU6 and leaky ReLU functions can be
found in (agarap2019deep). To get the ANN/controller to give out meaningful
results, we calculate the parameters of the ANN by using the reinforcement
learning technique.

###  3.3 Reinforcement Learning

Reinforcement learning problems are generally formulated as a Markov Decision
Process (MDP). It is a discrete-time control process used to describe
decision-making processes in a stochastic system. Consider a time
discretization as such where $\tau$ is the continuous variable time
$\tau\in[0,T_{F}]$, $T_{F}$ is the time horizon discretized in $n_{T}$ time
steps of size $\Delta\tau=T_{F}/n_{T}$. The subscript $t$ is the time step
number, it represents the value of a variable at time $\tau=t\Delta\tau$ where
$t\in\\{0,~{}1,...,~{}n_{T}-1,~{}n_{T}\\}$.

At each time step, the environment is at a state $\bm{x}_{t}\in R^{n_{x}}$
which the agent (decision-making component) can sense and the agent follows a
probability distribution called the policy $\pi_{\theta}$ which tells the
agent which action $\bm{u}_{t}=\pi(\bm{x}_{t})$ to choose from a set of
available actions $u_{t}\in R^{n_{u}}$ in the current state $\bm{x}_{t}$ Then
the environment shifts to another state $\bm{x}_{t+1}$ based on the transition
probability function $p(\bm{x}_{t+1}|\bm{x}_{t},\bm{u}_{t})$ which defines the
dynamics of the process and gives the agent a reward $R_{t+1}$ (sutton2018).
The system is often approximated as a discrete-time stochastic non-linear
system presented as a state-space model as shown in (4).

$\bm{x}_{t+1}=f(\bm{x}_{t},\bm{u}_{t},\bm{d}_{t})$ (4)

where the vector $\bm{d}_{t}$ is the disturbance. Shown in Fig. 2 below is a
block diagram illustration of the interaction between the agent and the
environment.

Figure 2: Interaction between the agent and the environment in an MDP.

The goal of the agent is to find the optimal policy which gives the maximum
total number of rewards, therefore the agent has to consider rewards in the
long run. This is expressed by $G_{t}$, also called the return. In the
calculation of return, another parameter called the discount factor $\gamma$
(constant real number between 0 and 1) is used to give a weighting to the
rewards at different time steps, the closer it is to 1 the more far-sighted
the agent is. Equation (5) expresses how the return is generally calculated
(sutton2018).

$G_{t}=R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\gamma^{3}R_{t+4}+\dots$ (5)

Equation (5) above can also be expressed recursively as shown in (6).

$G_{t}=R_{t+1}+\gamma G_{t+1}$ (6)

####  3.3.1 Policy Gradient

The Policy Gradient (PG) method estimates the optimal policy by using a
parameterised policy (usually ANNs). The main idea behind this method is that
it uses the gradient of a scalar performance index, the return ($G_{t}$), with
respect to the policy parameters to update and learn the policy (sutton2018).
The policy, $\pi$, calculates the control action,
$\bm{u}_{t}=\pi_{\theta}(\bm{x}_{t})$, given the current state, $\bm{x}_{t}\in
R^{n_{x}}$, and parameters, $\theta$. For a stochastic policy (which can help
exploration), the output from the neural network would be the mean and
standard deviation of the control action. Then the actual control would be
sampled from a distribution with the output mean and std (petsagkourakis2020).

$\bm{u}\sim\pi_{\theta}(\bm{u}|\bm{x})=\pi(\bm{u}|\bm{x},\theta)=p(\bm{u}_{t}=\bm{u}|\bm{x}_{t}=\bm{x},\theta_{t}=\theta)$
(7)

where $\pi$ is the policy. In PG, the network parameters are updated using the
gradient ascent strategy. Let us consider the most basic Reinforce PG
algorithm, where it does the Monte Carlo runs to get the trajectory of states,
control and rewards. Let $\bm{\kappa}$ be a joint variable of state, control
and rewards trajectories with a time horizon of $T_{F}$.

$\bm{\kappa}=(\bm{x}_{0},\bm{u}_{0},R_{0},\dots,\bm{x}_{n_{T}-1},\bm{u}_{n_{T}-1},R_{n_{T}-1},\bm{x}_{n_{T}},R_{n_{T}})$
(8)

Then, the return from the initial state, $G_{0}(\bm{\kappa})$ or
$J(\bm{\kappa})$ (objective criterion) and the probability density of a
trajectory can be expressed as shown in 9 and 10 (sutton2018;
petsagkourakis2020).

$J(\bm{\kappa})=\sum^{n_{T}}_{t=0}\gamma^{t}R_{t}(\bm{x}_{t},\bm{u}_{t})$ (9)
$p(\bm{\kappa}|\theta)=\hat{\mu}(\bm{x}_{0})\prod^{n_{T}-1}_{t=0}[\pi(\bm{u}_{t}|\bm{x}_{t},\theta)p(\bm{x}_{t+1}|\bm{x}_{t},\bm{u}_{t})]$
(10)

where $\hat{\mu}(\bm{x}_{0})$, is the probability density of the initial
state. The gradient update rule is shown in 11 (sutton2018;
petsagkourakis2020).

$\theta_{m+1}=\theta_{m}+\alpha_{m}\hat{J}(\theta)$ (11)
$\hat{J}(\theta)=\nabla_{\theta}\mathbb{E}_{\bm{\kappa}\sim
p(\bm{\kappa}|\theta)}[J(\bm{\kappa})]$ (12)

where $\alpha$ is the learning rate/step-size, $m$ is the current interation
at which the paramters are updated (epoch). However, computing the gradient of
the expectation, 12, directly is hard. Hence, the _Policy Gradient Theorem_
(sutton2018) is used. The derivation is shown in 13 (petsagkourakis2020).

$\begin{split}\hat{J}(\theta)~{}&=\nabla_{\theta}\mathbb{E}_{\bm{\kappa}\sim
p(\bm{\kappa}|\theta)}[J(\bm{\kappa})]=\nabla_{\theta}\int
p(\bm{\kappa}|\theta)J(\bm{\kappa})d\bm{\kappa}\\\
~{}&=\int\nabla_{\theta}p(\bm{\kappa}|\theta)J(\bm{\kappa})d\bm{\kappa}\\\
~{}&=\int
p(\bm{\kappa}|\theta^{\pi})\frac{\nabla_{\theta}p(\bm{\kappa}|\theta)}{p(\bm{\kappa}|\theta)}J(\bm{\kappa})d\bm{\kappa}\\\
~{}&=\int
p(\bm{\kappa}|\theta^{\pi})\nabla_{\theta}\log(p(\bm{\kappa}|\theta))J(\bm{\kappa})d\bm{\kappa}\\\
~{}&=\mathbb{E}_{\bm{\kappa}}[J(\bm{\kappa})\nabla_{\theta}\log(p(\bm{\kappa}|\theta))]\end{split}$
(13)

Substituting $p(\bm{\kappa}|\theta)$ from 10 to the log expression in the last
line of 13, 14 is formed.

$\nabla_{\theta}\log(p(\bm{\kappa}|\theta))=\nabla_{\theta}\sum^{n_{T}-1}_{t=0}log(\pi(\bm{x}_{t},\bm{u}_{t}))$
(14)

Since $p(\bm{x}_{t+1}|\bm{x}_{t},\bm{u}_{t})$ and $\hat{\mu}(\bm{x}_{0})$ does
not depend on $\theta$, they both disappear from 14. Finally, substituting the
derived results from 13, and 14 to 11 the following update rule is formed.

$\theta_{m+1}=\theta_{m}+\alpha_{m}\mathbb{E}_{\bm{\kappa}}[J(\bm{\kappa})\nabla_{\theta}\log(p(\bm{\kappa}|\theta))]$
(15)

where $m$ is the variable at epoch $m$. To estimate the value of the
expectation, the Monte Carlo method as previously discussed is used. Several
episodes using the current policy are obtained and then the data collected are
used to estimate the expectation as an average of $K$ samples. Therefore, the
final update rule for this Reinforce Monte Carlo algorithm is shown in 16.

$\theta_{m+1}=\theta_{m}+\alpha_{m}\frac{1}{K}\sum^{K}_{k=1}[J(\bm{\kappa}^{k})\nabla_{\theta}\sum^{n_{T}-1}_{t=0}\log(\pi(\bm{u}_{t}^{k}|\hat{\bm{x}}_{t}^{k},\theta))]$
(16)

where the superscript $k$ corresponds to sample number $k$ in $K$ total number
of samples. The full algorithm is presented in Algorithm 1.

Algorithm 1 Reinforce policy gradient algorithm (sutton2018;
petsagkourakis2020)

Input: Initialize policy parameter $\theta=\theta_{0}$, with
$\theta_{0}\in\Theta_{0}$, learning rate $\alpha$, $m:=0$, the number of
episodes $K$ and the number of epochs $N$.
Output: policy $\pi(\cdot|\cdot,\theta)$ and $\Theta$

for m = 1,…, N do:

  1. 1.

Monte Carlo episodes to collect $\textbf{u}_{t}^{k},\textbf{x}_{t}^{k}$ for
$T$ time steps for $K$ trajectories along with $J(\textbf{x}_{T}^{k})$, also
for $K$ trajectories.

  2. 2.

Update the policy, using the update rule shown in 16.

  3. 3.

$m:=m+1$

Past work on using PG method in process control has seen success in dealing
with non-linear highly complex bio-processes (petsagkourakis2020). In their
study, the approach were able to perform better than a non-linear MPC with
only 100 trainined batches. The PG method, compared to other value-based
reinforcement learning techniques, has a big advantage in the form of the end
result. In value based methods, such as Q-learning, the end results is a
table/function of action-value pairs (more details in (sutton2018)) which
needs another optimization to find out the best control action given a state
of the system. In PG, on the other hand, the end result is a function taking
in states of the system and giving out the optimal control action without the
need for another optimization step. This helps in solving the bi-level
optimization problem as the control problem reduces to a function-lookup
problem.

##  4 Method and Integration

###  4.1 Constructing Controller

The reinforcement learning policy is computed based on the OCP formulated in
Problem (2). The method used in this work to optimize the policy is a policy
gradient type method and the structure of the controller a feed-forward neural
network. To give the controller a head start, a pre-training (supervised
learning) scheme is used (for example, see (torabi2018behavioral)). The main
idea is that labelled data ($X,Y$) are fed into the algorithm and the
algorithm will try to find a relationship between them and then predict the
output ($Y$) given the state ($X$).

In a process control context, the data could be a trajectory of states
($\bm{x}_{PT}$) and the control action trajectory ($\bm{u}_{PT}$) taken given
the state trajectory. Then, the neural network is trained so that if it sees a
similar state $\bm{x}_{i}$, it would return a contro action which is close to
the control $\bm{u}_{i}$. The algorithm is shown in Algorithm 2.

Algorithm 2 Pre-training (supervised learning).

Input: Initialize policy with parameters $\theta=\theta_{0}$, with
$\theta_{0}\in\Theta_{0}$, learning rate $\alpha$, number of iterations
$n_{iter}$ and labelled data $\bm{x}_{PT}$ and $\bm{u}_{PT}$
Output: Pre-trained policy.

for m = 1,…, $n_{iter}$ do:

  1. 1.

Generate control actions, $\hat{\bm{u}}$, using the current policy with
parameters $\theta$ from the labelled state trajectory $\bm{x}_{PT}$.

  2. 2.

Improve $\pi_{\theta}$ by AdamOptimizer$(\hat{\bm{u}},\bm{u}_{PT})$
(kingma2017).

  3. 3.

$m:=m+1$

After the pre-training, the controller is deployed into the environment to
begin the reinforcement learning. The Reinforce (williams1992) (Monte Carlo)
algorithm was used with the addition of baseline and decaying learning rate.
The algorithm is shown in Algorithm 3.

Algorithm 3 Policy Gradient Algorithm.

Input: Initialize policy parameter $\theta=\theta_{0}$, with
$\theta_{0}\in\Theta_{0}$, learning rate $\alpha$, $m:=0$, the number of
episodes $K$ and the number of epochs $N$.
Output: policy $\pi(\cdot|\cdot,\theta)$ and $\Theta$

for m = 1,…, N do:

  1. 1.

Collect $\textbf{u}_{t}^{k},\textbf{x}_{t}^{k}$ for $T$ time steps for $K$
trajectories along with $J(\textbf{x}_{T}^{k})$, also for $K$ trajectories.

  2. 2.

Update the policy, using a policy gradient estimate
$\theta_{m+1}=\theta_{m}+\alpha_{m}\frac{1}{K}\sum_{k=1}^{K}\left[(J(\boldsymbol{\kappa}^{k})-b)\nabla_{\theta}\sum_{t=0}^{T-1}\text{log}\left(\pi(\textbf{u}_{t}^{k}|\hat{\textbf{x}}_{t}^{k},\theta)\right)\right]$

  3. 3.

$m:=m+1$

The update rule used in the algorithm (step 2 of Algorithm 3) is based on the
one used by petsagkourakis2020 derived from the policy gradient theorem
(sutton2018). In this update rule, $b$, is the baseline. Due to the
stochasticity of the policy coupled with Monte Carlo sampling, the
approximation of the expectation can have high variance, however, a baseline
can be used to reduce this variance without a bias (sutton2018). The baseline
used in this work is simply the mean expectation of rewards under the current
policy, this is the most commonly used baseline in policy gradient projects
and it has been proven to be effective (petsagkourakis2020).

While the baseline helps reducing the variance, the decaying learning rate on
the other hand helps the training to be faster during the early epochs, and
more stable during the later epochs. In this work we use Reinforce for
simplicity of presentation, however, other policy gradient-based algorithms
(e.g. TRPO (schulman2017trust), PPO (schulman2017proximal)) can be used.

###  4.2 Integration

Figure 3: Simultaneous Design and Control Optimization using RL methodology.

To summarize the methodology, a flow sheet of the full approach is presented
in (3). First, a bi-level optimization problem is formulated and split into
two problems, the design problem and the control problem. The control problem
is solved via reinforcement leaning, such that an optimal policy is produced.
This optimal policy is a controller in the form of an ANN such that it takes
states as inputs and outputs optimal control actions. Second, the policy is
embedded into the design problem and solved using standard MIP methods.

The approach is shown in Algorithm 4. For training the controller, any policy
optimization algorithm can be used (e.g. TRPO (schulman2017trust), PPO
(schulman2017trust), DDPG (lillicrap2019continuous)) but for simplicity the
Reinforce algorithm with baseline (sutton2018) is used in this work.

Algorithm 4 Full approach for simultaneous design and control optimization
using RL.

Input: Simultaneous process design and control problem.
Output: Optimal design and controller.

  1. 1.

State design-control simultaneous optimization: formulate the design problem
as a mixed-integer dynamic optimization (MIDO) problem as presented in (1).

  2. 2.

Optimal Control problem: Pose the optimal control problem (OCP) based on the
design problem as in (2).

  3. 3.

OCP as RL: Formulate the OCP to be solved via a RL policy optimization
framework.

    1. (a)

Pre-training: pre-train from an initial policy either via simulation or plant
data, this can be done via supervised or apprenticeship learning, amongst
others.

    2. (b)

Policy learning via policy gradients: Begin full- training using a policy
optimization algorithm.

  4. 4.

Embedded bi-level program: Embed the final policy obtained from step 3 into
the design (MIDO) problem and solve via a MIP algorithm.

It is important to highlight, that without a policy based method, the OCP is
an optimization problem, hence the integrated formulation in (1) is an
intractable bi-level optimization problem.

##  5 Results and Discussions

###  5.1 Tank Design

There are two case studies used to evaluate the approach, both were taken from
(diangelakis2017). In the first case study, it is desired to design a simple
tank with continuous inlet and outlet flows. There are no reactions involved,
the inlet flow is in the form of a sinusoidal signal and it has a nominal
inlet flow of $F_{nom}$ with a maximum deviation of $F_{dev}$. The outlet flow
is the manipulated variable of the PG (policy gradient) controller. A set-
point which depends on the values of $F_{nom}$ and $F_{dev}$ is inputted into
the controller. The design problem was formulated as a MIDO problem presented
in (17). In order to reduce the size of the problem while still maintaining a
reasonable sampling time to capture the dynamics of the sinusoidal signal, the
final time is set to allow only one period of the sinusoidal signal.

$\begin{split}&\max_{V_{tank},F_{dev},F_{nom},V(0)}~{}J_{SDC}=\int_{0}^{1}F_{dev}d\tau\\\
s.t.~{}&\frac{dV(\tau)}{d\tau}=F_{in}(\tau)-F_{out}(\tau)\\\
&F_{out}(\tau)=\alpha_{t}V(\tau)\\\
&F_{in}(\tau)=F_{nom}+F_{dev}\sin{(\tau/freq)}\\\ &freq=\frac{1}{2\pi}\\\
&V_{SP}=F_{nom}+F_{dev}\leq V_{tank}\\\
&err_{\pi_{\theta}}=\int_{0}^{1}\frac{||V(\tau)-V_{SP}||}{V_{SP}}d\tau\\\
&\text{End-point constraints:}\\\ &(1-\varepsilon/100)V(0)\leq
V(T_{F})\leq(1+\varepsilon/100)V(0)\\\
&err_{\pi_{\theta}}\leq\varepsilon/100\\\ &\text{Interior-point
constraints:}\\\ &a_{t}=\pi_{\theta}(F_{in,t},V_{SP},V_{t},V_{t-1})~{}\forall
t\in\\{0,...,n_{T}-1\\}\\\ \end{split}$ (17)

where $\tau$ is the continuous time variable, $V(t)$ is the volume of liquid
in the tank, $F_{in}(t)$ and $F_{out}(t)$ are the inlet and outlet flows in
$\mathrm{m}^{3}\,\mathrm{s}^{-1}$, respectively, $a$ is the valve position (1
is open, 0 is closed), and $freq$ is the frequency of the sinusoidal wave
disturbance in $\mathrm{s}^{-1}$. $V_{SP}$ is the volume set point, $V_{tank}$
is the volume of the tank, $err_{\pi_{\theta}}$ is the integral set point
error, $\varepsilon$ is the maximum allowable error and is equal to 1 %,
$a_{t}$ is the control action output from the policy applied to the system.

Constraints are enforced in the formulation to ensure the initial and final
states are the same, within some error threshold, to achieve cyclic operation
which allows for extrapolation of the operation to larger time horizons. The
same error threshold, is also used for the maximum setpoint deviation.

The goal of the design problem is to determine the maximum deviation for which
the controller is able to maintain the desired set-point. While the goal of
the OCP is set-point tracking. The OCP is presented in (18).

$\begin{split}&\max_{\pi_{\theta}}J_{OCP}=-10\sum_{t=0}^{n_{T}}(V_{t}-V_{SP})^{2}\\\
s.t.~{}&\frac{dV(\tau)}{d\tau}=F_{in}(\tau)-F_{out}(\tau)\\\
&F_{out}(\tau)=\alpha_{t}V(\tau)\\\
&F_{in}(\tau)=F_{nom}+F_{dev}\sin{(\tau/freq)}\\\ &freq=\frac{1}{2\pi}\\\
&V_{SP}=F_{nom}+F_{dev}\leq V_{tank}\\\
&a_{t}=\pi_{\theta}(F_{in,t},V_{SP},V_{t},V_{t-1})~{}\forall
t\in\\{0,...,n_{T}-1\\}\\\ \end{split}$ (18)

where $J_{OCP}$ is the objective function, it is the sum of negative squared
setpoint errors. Therefore, in theory, the maximum reward that is achievable
is 0. This objective function is also used for the training to calculate the
total reward earned at the end of each episode.

In this case study, a proportional derivative (PD) controller is used to
compare againts the approch proposed in this work. The final results showed
that the PG controller was able to handle deviation values of up to 3.84
$\mathrm{m}^{3}\,\mathrm{s}^{-1}$. The control performances are plotted in
Figure 4.

Figure 4: Tank case study control performances.

Figure 4 shows the performance of the controllers in the tank case study at
the MIDO solution averaged over 1000 simulations with 2 % measurement noise.
It can be observed that the PG controller outperforms the PD controller. The
PD control action shows a large variance on its control action indicating very
chaotic response. The mean set-point error of the PD controller is two times
larger than the PG controller mean error. Furthermore, the PG controller were
able to perform better than the mp-MPC used by diangelakis2017
($F_{dev}=2.69ms^{-1}$) as the PG controller managed to handle a higher
maximum deviation even with the presence of measurement noise.

###  5.2 CSTR Design

The second case study considers the design of a CSTR (continuous stirred tank
reactor). The reaction in the CSTR is a first-order endothermic reaction and a
heating jacket is equipped. The temperature of the jacket is controlled by the
PG controller. The feed flowrate and concentrations are disturbed by a
sinusoidal wave similar to the tank case study. The controller is not able to
see the value of inlet feed flowrate, however, in this case study, there is an
option to have a settling tank before the CSTR to normalize the feed mass
flowrate to its nominal value. The dynamic system can be found in
(diangelakis2017). The design problem formulation is shown in (19).

$\begin{split}&\min_{V,m_{dev},m_{nom},C_{A,dev},C_{A,nom},Y_{S}}J_{SDC}=Cost_{Total}\\\
s.t.~{}&\frac{dC_{A}}{d\tau}=\frac{m}{\rho
V}(C_{A_{0}}-C_{A})-k_{0}C_{A}e{{}^{-\frac{E_{A}}{RT}}}\\\ &\begin{split}V\rho
C_{P}\frac{dT}{d\tau}=mC_{P}(T_{0}-&T)+UA(T_{H}-T)\\\ &+V\Delta
H_{rxn}k_{0}C_{A}e^{-\frac{E_{A}}{RT}}\\\ \end{split}\\\
&C_{A_{0}}=C_{A_{0,nom}}+C_{A_{0,nom}}+C_{A_{0,dev}}\sin{(t/freq)}\\\
&m=m_{nom}+m_{dev}(1-Y_{S,t})\sin{(t/freq)}\\\ &freq=100/(2\pi)\\\
&T_{H,t}=\pi_{\theta}(C_{A,t},C_{A,t-1},T_{t},V,C_{A_{0},t},C_{A}^{SP})~{}\forall
t\\\ &err_{\pi_{\theta}}=\int_{0}^{T_{F}}C_{A}-C_{A}^{SP}d\tau\\\
&C_{A}^{SP}=0\\\ &\text{Objective function:}\\\
&Cost_{Total}=Cost_{Equip}+Cost_{Oper}\\\
&Cost_{Equip}=10((V-750)/\pi)+1000+400Y_{S,f}\\\
&\frac{dCost_{Oper}}{d\tau}=-m(C_{A_{0}}-C_{A})+4Y_{S}\\\ &\text{Endpoint
constraints:}\\\ &err_{\pi_{0}}\leq 100,~{}C_{A_{0,dev}}\leq
C_{A_{0,nom}},~{}m_{dev}\leq m_{nom}\\\ &\text{Interior point
constraints:}~{}\forall t\in\\{0,...,n_{T}\\}\\\ &Y_{S,t}-Y_{S,f}\leq
0,~{}Y_{S,t}\in\\{0,1\\},~{}Y_{S,f}\in\\{0,1\\}\\\ &T_{t}\leq 450\\\
\end{split}$ (19)

where $C_{A}$ and $C_{A_{0}}$ are the concentrations of A in the reactor and
in the feed, respectively, $m$ is the mass flowrate into and out of the
reactor, $\rho$ is the density of the liquid, $V$ is the volume of the
reactor, $k_{0}$ is the pre-exponential factor, $E_{A}$ is the activation
energy, $R$ is the ideal gas constant, $\Delta H_{rxn}$ is the heat of
reaction, $T$ is the temperature inside the reactor, $T_{H}$ is the heating
utility temperature, $C_{P}$ is the heat capacity, $UA$ is the overall heat
transfer coefficient multiplied by the heat transfer area, and $Y_{S,t}$ is a
piece-wise constant binary variable which determines the utilization of the
settling tank while $Y_{S,f}$ is a binary variable which denote the existence
of the settling tank.

The goal of the design problem is to minimize a cost related objective
function while satisfying the temperature constraint. While the goal of the
OCP is to minimize the concentration of A by manipulating the heating jacket
temperature while keeping the CSTR temperature below 450 $\mathrm{K}$. The OCP
is presented in (20)

$\begin{split}\max_{\pi_{\theta}}&J_{OCP}=-\sum_{t=0}^{n_{T}}(C_{A,t}-C_{A}^{SP})-100\sum_{t=0}^{n_{T}}\max(0,T_{t}-450)\\\
s.t.~{}&\frac{dC_{A}}{d\tau}=\frac{m}{\rho
V}(C_{A_{0}}-C_{A})-k_{0}C_{A}e{{}^{-\frac{E_{A}}{RT}}}\\\
~{}&\frac{dT}{d\tau}=\frac{mC_{P}(T_{0}-T)+UA(T_{H}-T)+V\Delta
H_{rxn}k_{0}C_{A}e^{-\frac{E_{A}}{RT}}}{V\rho C_{P}}\\\
&C_{A_{0}}=C_{A_{0,nom}}+C_{A_{0,nom}}+C_{A_{0,dev}}\sin{(t/freq)}\\\
&m=m_{nom}+m_{dev}\sin{(t/freq)}\\\ &freq=100/(2\pi)\\\
&T_{H,t}=\pi_{\theta}(C_{A,t},C_{A,t-1},T_{t},V,C_{A_{0},t},C_{A}^{SP})~{}\forall
t\\\ &C_{A}^{SP}=0,~{}T_{t}\leq 450\\\ \end{split}$ (20)

Figure 5: CSTR case study control performance.

A PID controller was not implemented in this case study as it would need
complex modifications for it to work with constraints. Again, the approach
were able to solve this otherwise intractable problem for traditional
approaches. The solution has final cost of 27,539 and the control performance
is presented in Figure 5. It shows that the settling tank was only needed for
a brief amount of time at the start of the process. At the start, the
concentration of A in the CSTR is higher than the feed concentration, making
it hard to satisfy the temperature constraint at initial time, therefore the
settling tank was used. Overall, the controller is performing very well. It
was able to minimize the concentration of A with very good performance (very
close to zero almost all of the time) and the temperature constraint is
satisfied at all times. This is also true for a wide range of design
variables.

##  6 Conclusions

In this work we proposed the use of RL to address a long standing challenge
for the process design and control community; the simultaneous design and
control of sustainable chemical processes. It is shown that using RL it is
possible to address the otherwise intractable nonlinear mixed integer dynamic
bi-level optimization problem. Through case studies, we show that the PG-based
controller was able to perform well in a wide range of design problems and
design variables enabling the optimization of the process. Policy gradient-
based controllers can handle measurement noise and stochastic systems
naturally and the final form is an explicit function, which can be directly
embedded into optimization problems. For future work, we will add the handling
of constraints with high probability such as in (petsagkourakis2020chance),
and a further hyperparameter tuning scheme (e.g snoek2012 Bayesian
optimization or other expensive black box optimization techniques). We aim to
address also harder and larger problems in the near future.

##### Acknowledgments

We are grateful for the technical assistance of A. Author.

##### Funding Statement

This research was supported by grants from the <funder-name><doi>(<award ID>);
<funder-name><doi>(<award ID>).

##### Competing Interests

A statement about any financial, professional, contractual or personal
relationships or situations that could be perceived to impact the presentation
of the work — or ‘None’ if none exist

##### Data Availability Statement

A statement about how to access data, code and other materials allowing users
to understand, verify and replicate findings — e.g. Replication data and code
can be found in Harvard Dataverse: `\url{https://doi.org/link}`.

##### Ethical Standards

The research meets all ethical guidelines, including adherence to the legal
requirements of the study country.

##### Author Contributions

Please provide an author contributions statement using the CRediT taxonomy
roles as a guide `\url{https://www.casrai.org/credit.html}`.
Conceptualization: A.RC.;P.P; Methodology: S.S.; A.RC.;P.P Data curation: S.S.
Data visualisation: S.S. Writing original draft: S.S.; A.RC.;P.P. All authors
approved the final submitted draft.

##### Supplementary Material

State whether any supplementary material intended for publication has been
provided with the submission.