# The Chow ring of a sequence of point blow-ups

Daniel Camazón 111The author was partially supported by PGC2018-096446-B-C21

###### Abstract

Given a sequence of point blow-ups of smooth $n-$dimensional projective
varieties $Z_{i}$ defined over an algebraically closed field $\mathit{k}$,
$Z_{s}\xrightarrow{\pi_{s}}Z_{s-1}\xrightarrow{\pi_{s-1}}\cdot\cdot\cdot\xrightarrow{\pi_{2}}Z_{1}\xrightarrow{\pi_{1}}Z_{0}$,
we give two presentations of the Chow ring of its sky $A^{\bullet}(Z_{s})$.
The first one using the classes of the total transforms of the exceptional
components as generators and the second one using the classes of the strict
transforms ones. We prove that the skies of two sequences of point blow-ups of
the same length have isomorphic Chow rings. Finally we give a characterization
of final divisor of a sequence of point blow-ups in terms of some relations
defined over the Chow group of zero-cycles of its sky $A_{0}(Z_{s})$.

##  1 Introduction

Sequences of blow-ups of smooth varieties along smooth centers are useful for
general algebraic geometric purposes, in particular, for resolution and
classification of singularities. This paper is devoted to explicitly compute
the Chow ring of the variety obtained after a sequence of point blow-ups. We
will assume, additionally, that the center of each blow up has normal
crossings with the already created exceptional divisors by the precedent blow
ups. In this case, the Chow ring is generated as a $\mathbb{Z}-$algebra by the
exceptional divisors and the generic hyperplane.

The composition of the successive blow ups of such sequences is a projective
(and therefore proper) birational morphisms $Z_{s}\rightarrow Z_{0}$, where
$Z_{s}$ and $Z_{0}$ are smooth algebraic varieties which are respectively
called sky and ground. In this paper, we restrict to the case of point blow-
ups and $Z_{0}\cong\mathbb{P}^{n}$, as it is involved in many geometric
contexts, in particular, the study of algebraic curves and surfaces.

Our main goal is to give an explicit presentation of the Chow ring of the sky
of a sequence of point blow-ups as a finite type $\mathbb{Z}-$algebra. In [4]
Keel gives a presentation of the Chow ring of a blow-up $\pi:Z_{1}\rightarrow
Z_{0}$ with center $C_{1}$ a regularly embedded subvariety under the
assumption that the restriction map $i_{1}^{*}:A^{\bullet}(Z_{0})\rightarrow
A^{\bullet}(C_{1})$ is surjective, and no more results have been found in the
literature.

We give two presentations of the Chow ring of the sky of a sequence of point
blow-ups $A^{\bullet}(Z_{s})$ by considering the total transforms and the
strict transforms of the irreducible components of the exceptional divisor as
generators respectively. We give polynomial generators of the ideal of
relations for the total (respectively strict) exceptional components. As a
corollary, we prove a surprising result: the Chow rings of the skies of two
sequences of point blow-ups of the same length are isomorphic.

An irreducible exceptional component $E$ is called final if there are not
blow-ups with center lying over the point producing $E$ for the first time,
see [1] and Definition 3.7. The presentation of the Chow ring in terms of the
strict transform of exceptional divisors, allow us to characterize final
components by polynomial conditions. This improves conditions given in [1].

##  2 Preliminaries

Fix an algebraically closed field $\mathit{k}$. Throughout this paper a
variety will mean a reduced projective scheme over $\mathit{k}$.

######  Definition 2.1.

A sequence of blow-ups over $\mathit{k}$ is defined as a sequence of morphisms

$Z_{s}\xrightarrow{\pi_{s}}Z_{s-1}\xrightarrow{\pi_{s-1}}\cdot\cdot\cdot\xrightarrow{\pi_{2}}Z_{1}\xrightarrow{\pi_{1}}Z_{0},$

where $Z_{0}$ is a smooth $n-$dimensional projective variety and

  1. i.

the morphism $\pi_{i}:Z_{i}\rightarrow Z_{i-1}$ is the blow-up at center
$C_{i}\subset Z_{i-1}$ for $i=1,...,s$. We denote by $E_{i}^{i}$ to be the
exceptional hypersurface of $\pi_{i}$, and for $j<i$ $E_{j}^{i}\subset Z_{i}$
the strcit transform of $E_{j}^{j}\subset Z_{j}$ in $Z_{i}$,

  2. ii.

$codim(C_{i+1},Z_{i})\geq 2$ for $i=0,...,s-1$,

  3. iii.

the center $C_{i}$ has simple normal crossings with
$\\{E_{1}^{i-1},E_{2}^{i},...,E_{i-1}^{i-1}\\}$, for $i=1,...,s$.

We will refer to $Z_{0}$ and $Z_{s}$ as the ground and the sky of the sequence
of blow-ups respectively.

Along the paper we fix a sequence of blow ups $(Z_{0},...,Z_{s},\pi)$ as in
Definition 2.1 and we set $\pi_{j,i}:Z_{j}\rightarrow Z_{i}$, for $j>i$, to be
the composition
$\pi_{j,i}=\pi_{i+1}\circ\pi_{i+2}\circ...\circ\pi_{j-1}\circ\pi_{j}$.

The centers $C_{i}$, in general, can have any dimension. We extend the well-
known notion of proximity for point blow-ups.

######  Definition 2.2.

We say that $C_{j}$ is proximate to $C_{i}$, and write $C_{j}\rightarrow
C_{i}$ if and only if $C_{j}\subset E_{i}^{j-1}$.

Note that, if $C_{j}$ is proximate to $C_{i}$ then $j>i$.

######  Remark 2.3.

For $j>i$ we denote by $E_{i}^{j*}$ the total transform of $E_{i}^{i}$ by the
morphism $\pi_{j,i}:Z_{j}\rightarrow Z_{i}$. By an abuse of notation
$E_{i}^{i*}=E_{i}^{i}$. Note that by definition of the total transform, one
has

$E_{i}^{k*}=E_{i}^{k}+\sum_{j>i}p_{ij}E_{j}^{k*}$

where $p_{ij}=1$ if $i<j\leq k$ and $C_{j}$ is proximate to $C_{i}$ and
$p_{ij}=0$ in any other case.
We can now define two free $\mathbb{Z}-$modules with basis
$\left\\{E_{i}^{k*}\right\\}_{i=1}^{k}$ and
$\left\\{E_{i}^{k}\right\\}_{i=1}^{k}$ respectively, and construct the change
of basis matrix $B_{k}$

$B_{k}=\begin{pmatrix}1&0&\cdots&0&0\\\ -p_{12}&1&\ddots&\vdots&\vdots\\\
\vdots&-p_{23}&\ddots&\vdots&\vdots\\\ \vdots&\vdots&\ddots&1&\vdots\\\
-p_{1k}&-p_{2k}&\cdots&-p_{k-1k}&1\end{pmatrix}$ (1)

Note that its inverse $B_{k}^{-1}$ has also integer entries.

Consider just one of the blow-ups conforming the sequence:

$\textstyle{E_{\alpha+1}^{\alpha+1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{j_{\alpha+1}}$$\scriptstyle{g_{\alpha+1}}$$\textstyle{Z_{\alpha+1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\pi_{\alpha+1}}$$\textstyle{C_{\alpha+1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{i_{\alpha+1}}$$\textstyle{Z_{\alpha}}$

for $\alpha\in\left\\{0,1,...,s-1\right\\}$.

The following result show us the multiplication rules of
$A^{\bullet}(Z_{\alpha+1})$ for any blow-up at some smooth center
$C_{\alpha+1}$.

######  Proposition 2.4.

[2, Proposition 13.12.] The Chow ring $A^{\bullet}(Z_{\alpha+1})$ is generated
by $\pi_{\alpha+1}^{*}A^{\bullet}(Z_{\alpha})$ and
$j_{\alpha+1*}A^{\bullet}(E_{\alpha+1}^{\alpha+1})$ as an algebra, that is, by
classes pulled back from $Z_{\alpha}$ and classes supported on
$E_{\alpha+1}^{\alpha+1}$. The rules for multiplication are the following:

$\displaystyle\pi$
${}_{\alpha+1}^{*}x\cdot\pi_{\alpha+1}^{*}y=\pi_{\alpha+1}^{*}(x\cdot y)$ for
$x,y\in A^{\bullet}(Z_{\alpha})$ (2) $\displaystyle\pi$
$\displaystyle\pi_{\alpha+1}^{*}x\cdot j_{\alpha+1*}t=j_{\alpha+1*}(t\cdot
g_{\alpha+1}^{*}i_{\alpha+1}^{*}x)$ for $x\in A^{\bullet}(Z_{\alpha}),t\in
A^{\bullet}(E_{\alpha+1}^{\alpha+1})$ (3) $\displaystyle\pi$ $\displaystyle
j_{\alpha+1*}t\cdot j_{\alpha+1*}u=-j_{\alpha+1*}(t\cdot u\cdot\varsigma)$ for
$t,u\in A^{\bullet}(E_{\alpha+1}^{\alpha+1})$ (4)

where $\varsigma=c_{1}(\mathcal{O}_{E_{\alpha+1}^{\alpha+1}}(1))$.
We identify $A^{\bullet}(Z_{\alpha})$ as a subring of
$A^{\bullet}(Z_{\alpha+1})$ by means of the ring homomorphism

$\pi_{\alpha+1}^{*}:A^{\bullet}(Z_{\alpha})\rightarrow
A^{\bullet}(Z_{\alpha+1}),$

######  Remark 2.5.

We will denote by $e_{i}^{\alpha*}$ (resp. $e_{i}^{\alpha}$) to be the class
of $\left[E_{i}^{\alpha*}\right]$ (resp. $\left[E_{i}^{\alpha}\right]$) in
$A^{1}(Z_{\alpha})$ for $i=1,...,\alpha$.

In the next section we will give a presentation of the Chow ring
$A^{\bullet}(Z_{\alpha})$ as a $\mathbb{Z}-$algebra of finite type.

##  3 Main results

Note that Proposition 2.4 does not give a presentation of
$A^{\bullet}(Z_{\alpha+1})$ as a $A^{\bullet}(Z_{\alpha})-$algebra, but only
states the rules of multiplication.
If we could find generators of $A^{\bullet}(E_{\alpha+1}^{\alpha+1})$ as a
$\mathbb{Z}-$algebra, $\left\\{\gamma_{1},...,\gamma_{r}\right\\}\in
A^{\bullet}(E_{\alpha+1}^{\alpha+1})$, then

$A^{\bullet}(Z_{\alpha+1})\cong
A^{\bullet}(Z_{\alpha})\left[j_{\alpha+1*}(\gamma_{1}),...,j_{\alpha+1*}(\gamma_{r})\right]$

would be a $A^{\bullet}(Z_{\alpha})-$algebra of finite type. One would like to
have a presentation

$A^{\bullet}(Z_{\alpha+1})\cong
A^{\bullet}(Z_{\alpha})\left[w_{1},...,w_{r}\right]/\mathcal{J}$

by sending $w_{i}$ to $j_{\alpha+1*}(\gamma_{i})$, with an explicit
description of the ideal $\mathcal{J}$. The ideal $\mathcal{J}$will be
computed in Theorems 3.3 and 3.6. We will restrict ourselves to the case of
sequences of point blow-ups, that is $C_{\alpha}=P_{\alpha}$, with the ground
variety $Z_{0}\cong\mathbb{P}^{n}$. By [2, Theorem 2.1.],
$A^{\bullet}(Z_{0})\cong\mathbb{Z}\left[u\right]/(u^{n+1})$, by sending $u$ to
$h$, where $h\in A^{1}(Z_{0})$ is the rational equivalence class of any
hyperplane $\left[H\right]$ in $\mathbb{P}^{n}$, and $\forall\alpha$
$A^{\bullet}(E_{\alpha}^{\alpha})\cong\mathbb{Z}\left[w\right]/(w^{n})$ by
sending $w$ to $\varsigma_{\alpha}$, with $\varsigma_{\alpha}\in
A^{1}(E_{\alpha}^{\alpha})$ is the rational class of any hyperplane.

In the case of sequences of point blow-ups, we are able to give generators of
the Chow ring $A^{\bullet}(Z_{s})$ as a $\mathbb{Z}-$algebra.

######  Lemma 3.1.

The Chow ring of the sky $A^{\bullet}(Z_{s})$ is generated by
$\left\\{h^{s*},\left\\{e_{i}^{s*}\right\\}_{i=1}^{s}\right\\}$ as a
$\mathbb{Z}-$algebra.

###### Proof.

This follows by induction on $\alpha$. It is clear that $A^{\bullet}(Z_{0})$
is generated by $\left\\{h\right\\}$. Let us suppose that
$A^{\bullet}(Z_{\alpha})$ is generated by
$\left\\{h^{\alpha*},\left\\{e_{i}^{\alpha*}\right\\}_{i=1}^{\alpha}\right\\}$.
Now by Proposition 2.4 and due to the fact that
$E_{\alpha+1}^{\alpha+1}\cong\mathbb{P}^{n-1}$, that is
$A^{\bullet}(E_{\alpha+1}^{\alpha+1})\cong\mathbb{Z}\left[t\right]/(t^{n})$,
by sending $t$ to $\varsigma_{\alpha+1}$, with $\varsigma_{\alpha+1}$ the
rational equivalence class of any hyperplane in $\mathbb{P}^{n-1}$, and
$e_{\alpha+1}^{\alpha+1*}\cdot
e_{\alpha+1}^{\alpha+1*}=-j_{\alpha+1*}(\varsigma_{\alpha+1})$ by equation (4)
then $A^{\bullet}(Z_{\alpha+1})$ is generated by
$\left\\{h^{\alpha+1*},\left\\{e_{i}^{\alpha+1*}\right\\}_{i=1}^{\alpha+1}\right\\}$
as a $\mathbb{Z}-$algebra. ∎

######  Remark 3.2.

It makes sense then to define the augmented free $\mathbb{Z}-$modules with
basis $\left\\{H^{k*},\left\\{E_{i}^{k*}\right\\}_{i=1}^{k}\right\\}$ and
$\left\\{\widetilde{H}^{k},\left\\{E_{i}^{k}\right\\}_{i=1}^{k}\right\\}$ and
the augmented change of basis matrix $B_{k}^{*}$

$B_{k}^{*}=\begin{pmatrix}1&0&\cdots&\cdots&0&0\\\
0&1&0&\cdots&\vdots&\vdots\\\ 0&-p_{12}&1&\ddots&\vdots&\vdots\\\
\vdots&-p_{13}&-p_{23}&\ddots&\vdots&\vdots\\\
\vdots&\vdots&\vdots&\ddots&1&\vdots\\\
0&-p_{1k}&-p_{2k}&\cdots&-p_{k-1k}&1\end{pmatrix}$ (5)

and its inverse $B_{k}^{*-1}$.

######  Theorem 3.3.

The Chow ring of the sky $A^{\bullet}(Z_{s})$ is isomorphic to

$A^{\bullet}(Z_{s})\cong\mathbb{Z}\left[x_{0},x_{1},...,x_{s}\right]/(\left\\{x_{i}\cdot
x_{j}\right\\}_{\begin{subarray}{c}i,j=0\\\ i\neq
j\end{subarray}}^{s},\left\\{(-1)^{n}(x_{i})^{n}+(x_{0})^{n}\right\\}_{i=1}^{s}),$
(6)

by sending $x_{0}$ to the class $h^{s*}$ and $x_{i}$ to the class $e_{i}^{s*}$
for $i=1,...,s$.

###### Proof.

By lemma 3.1 there exist a exists a surjective morphism

$\phi:\mathbb{Z}\left[x_{0},x_{1},...,x_{s}\right]\rightarrow
A^{\bullet}(Z_{s}),$

such that $\phi(x_{0})=h^{s*}$ and $\phi(x_{i})=e_{i}^{s*}$ for $i=1,...s$.
Firstly we will prove that
$\mathcal{J}:=\left\langle\left\\{x_{i}\cdot
x_{j}\right\\}_{\begin{subarray}{c}i,j=0\\\ i\neq
j\end{subarray}}^{s},\left\\{(-1)^{n}(x_{i})^{n}+(x_{0})^{n}\right\\}_{i=1}^{s}\right\rangle\subset
Ker(\phi)$. To begin with, let us express the classes of the basis
$\left\\{E_{i}^{\alpha+1*}\right\\}_{i=1}^{\alpha+1}$ in terms of the classes
of the basis $\left\\{E_{i}^{\alpha+1}\right\\}_{i=1}^{\alpha+1}$, that is,
since

$e_{i}^{\alpha*}=e_{i}^{\alpha}+\sum_{j=i+1}^{\alpha}b_{j,i}e_{j}^{\alpha},$

then

$e_{i}^{\alpha+1*}=e_{i}^{\alpha+1}+\sum_{j=i+1}^{\alpha}b_{j,i}e_{j}^{\alpha+1}+(\sum_{j=i}^{\alpha}p_{j\alpha+1}b_{j,i})e_{\alpha+1}^{\alpha+1}$

where $b_{j,i}$ denotes the coefficients of the augmented change of basis
matrix $B_{\alpha}^{*-1}$.
If we denote by $\varsigma_{\alpha+1}\in A^{1}(E_{\alpha+1}^{\alpha+1})$ the
class of any hyperplane in $E_{\alpha+1}^{\alpha+1}$ then we have the
following intersection products

$\displaystyle e$ ${}_{\alpha+1}^{\alpha+1}\cdot
e_{\alpha+1}^{\alpha+1}=-j_{\alpha+1*}(\varsigma_{\alpha+1})$ (7)
$\displaystyle e$ $\displaystyle e_{j}^{\alpha+1}\cdot
e_{\alpha+1}^{\alpha+1}=j_{\alpha+1*}(\varsigma_{\alpha+1})$ if
$P_{\alpha+1}\rightarrow P_{j}$ (8) $\displaystyle e$ $\displaystyle
e_{j}^{\alpha+1}\cdot e_{\alpha+1}^{\alpha+1}=0$ otherwise (9)

where equation (7) follows from equation (4) and equation (8) is a direct
consequence of [3, Corollary 6.7.1], that is
$\pi_{\alpha+1}^{*}(e_{j}^{\alpha})=e_{j}^{\alpha+1}+e_{\alpha+1}^{\alpha+1}$,
and equations (3) and (7). So the following intersection product is $0$

$(e_{j}^{\alpha+1}+p_{j\alpha+1}e_{\alpha+1}^{\alpha+1})\cdot
e_{\alpha+1}^{\alpha+1}=0,$ (10)

and we can conclude that

$e_{i}^{\alpha+1*}\cdot
e_{\alpha+1}^{\alpha+1*}=(e_{i}^{\alpha+1}+\sum_{j=i+1}^{\alpha}b_{j,i}e_{j}^{\alpha+1}+(\sum_{j=i}^{\alpha}p_{j\alpha+1}b_{j,i})e_{\alpha+1}^{\alpha+1})\cdot
e_{\alpha+1}^{\alpha+1}=0.$

On the other hand $h^{\alpha+1*}\cdot e_{\alpha+1}^{\alpha+1}=0$ is a
consequence of the moving lemma (see [3, 11.4 Moving lemma]). If we make the
pull back through $\pi_{s,\alpha+1}^{*}$ for all $\alpha$, then it follows
that $\left\langle\left\\{x_{i}\cdot
x_{j}\right\\}_{\begin{subarray}{c}i,j=0\\\ i\neq
j\end{subarray}}^{s}\right\rangle\subset ker(\phi)$. By [3, Example 16.1.11],
$A_{0}(Z_{0})$ is a birational invariant, that is
$A_{0}(Z_{i})\cong\mathbb{Z}(h^{i*})^{n}$ for $i=1,...,s$, so since
$(e_{\alpha+1}^{\alpha+1})^{n}=(-1)^{n-1}j_{\alpha+1*}(\varsigma_{\alpha+1}^{n})$
then $(e_{\alpha+1}^{\alpha+1})^{n}=(-1)^{n-1}(h^{\alpha+1*})^{n}$, and by
making the pull back through $\pi_{s,\alpha+1}^{*}$ we conclude that
$\left\langle\left\\{(-1)^{n}(x_{i})^{n}+(x_{0})^{n}\right\\}_{i=1}^{s}\right\rangle\subset
Ker(\phi)$.
Now we will prove that $Ker(\phi)\subset\mathcal{J}$. Note that
$\phi:\mathbb{Z}\left[x_{0},x_{1},...,x_{s}\right]\rightarrow
A^{\bullet}(Z_{s})$ is homogenous, so $ker(\phi)$ is an homogenous ideal, and
$\mathcal{J}$ is an homogenous ideal too by construction. Let us suppose that
$P\left[x\right]\in Ker(\phi)/\mathcal{J}$ with $deg(P)=\eta$. Then
$2\leq\eta\leq n$, since
$\left\\{x_{i}^{n+1}\right\\}_{i=0}^{s}\in\mathcal{J}$, and $P\left[x\right]$
must be of the form
$P\left[x\right]=\sum_{i=0}^{s}a_{i}x_{i}^{\eta}mod(\mathcal{J})$, since
$\left\\{x_{i}\cdot x_{j}\right\\}_{\begin{subarray}{c}i,j=0\\\ i\neq
j\end{subarray}}^{s}\in\mathcal{J}$. Now if $\eta<n$, then
$x_{i}^{n-\eta}P\left[x\right]$ will be also in $Ker(\phi)$, then
$\phi(x_{i}^{n-\eta}P\left[x\right])=a_{i}(e_{i}^{s*})^{n}=0$, since
$(e_{i}^{s*})^{n}\neq 0$ then $a_{i}=0$ for $i=0,1,...,s$. If $\eta=n$, since
$\left\\{(-1)^{n}(x_{i})^{n}+(x_{0})^{n}\right\\}_{i=1}^{s}\in Ker(\phi)$ then
it follows that $a_{0}+(-1)^{n+1}\sum_{i=1}^{s}a_{i}=0$, so
$P\left[x\right]=0mod(\mathcal{J})$. ∎

######  Remark 3.4.

Note that $\left\langle x_{0},x_{1},...,x_{s}\right\rangle
Ker(\phi)=\left\langle\left\\{x_{i}x_{j}x_{k}\right\\}_{\begin{subarray}{c}i,j,k=0\\\
i\neq j\\\ j\neq
k\end{subarray}}^{s},\left\\{x_{i}^{n+1}\right\\}_{i=0}^{s}\right\rangle$, so
we have that
$Ker(\phi)/\left\langle x_{0},x_{1},...,x_{s}\right\rangle Ker(\phi)$ is a
free $\mathbb{Z}-$module of finite rank $\binom{n+1}{2}+n$. Any set of
generators of the ideal $Ker(\phi)$ is a set of generators of
$Ker(\phi)/\left\langle x_{0},x_{1},...,x_{s}\right\rangle Ker(\phi)$ as
$\mathbb{Z}-$module, so
$\left\\{\left\\{x_{i}\cdot x_{j}\right\\}_{\begin{subarray}{c}i,j=0\\\ i\neq
j\end{subarray}}^{s},\left\\{(-1)^{n}(x_{i})^{n}+(x_{0})^{n}\right\\}_{i=1}^{s}\right\\}$
is a minimal set of generators for $Ker(\phi)$.

######  Corollary 3.5.

Given two sequences of point blow-ups $(Z_{0},...,Z_{s},\pi)$ and
$(Z_{0}^{{}^{\prime}},...,Z_{s^{{}^{\prime}}}^{{}^{\prime}},\pi^{{}^{\prime}})$,
if $s=s^{{}^{\prime}}$ then $A^{\bullet}(Z_{s})\cong
A^{\bullet}(Z_{s^{{}^{\prime}}}^{{}^{\prime}})$.

###### Proof.

It follows directly from equation $(\ref{EqCRTTPBU})$ in Theorem 3.3. ∎

We can use
$\left\\{\widetilde{h}^{s},\left\\{e_{i}^{s}\right\\}_{i=1}^{s}\right\\}$ as
generators of the Chow ring $A^{\bullet}(Z_{s})$ as $\mathbb{Z}-$algebra
instead.

######  Theorem 3.6.

A presentation of $A^{\bullet}(Z_{s})$ using
$\left\\{\widetilde{h}^{s},\left\\{e_{i}^{s}\right\\}_{i=1}^{s}\right\\}$ as
generators is the following one:

$A^{\bullet}(Z_{s})\cong\frac{\mathbb{Z}\left[y_{0},y_{1},...,y_{s}\right]}{\mathcal{A}},$
(11)

where

$\mathcal{A}=((\left\\{y_{0}\cdot
y_{i}\right\\}_{i=1}^{s},\left\\{(y_{i}+\sum_{k=i+1}^{s}b_{k,i}y_{k})\cdot(y_{j}+\sum_{l=j+1}^{s}b_{l,j}y_{l})\right\\}_{\begin{subarray}{c}i,j=1\\\
i\neq
j\end{subarray}}^{s},\left\\{(y_{i})^{n}+((-1)^{n}+\\#\left\\{j\right\\}_{j\rightarrow
i})(y_{0})^{n}\right\\}_{i=1}^{s}))$

by sending $y_{0}$ to $\widetilde{h}^{s}$ and $y_{i}$ to $e_{i}^{s}$ for
$i=1,...,s$.

###### Proof.

In this case there exists a surjective morphism

$\phi^{{}^{\prime}}:\mathbb{Z}\left[y_{0},y_{1},...,y_{s}\right]\rightarrow
A^{\bullet}(Z_{s})$

with $\phi^{{}^{\prime}}(y_{0})=h^{s*}$ and
$\phi^{{}^{\prime}}(y_{i})=e_{i}^{s}$ for $i=1,...,s$. Moreover we have the
following commutative diagram

$\textstyle{\mathbb{Z}\left[x_{0},...,x_{s}\right]\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\phi}$$\textstyle{\mathbb{Z}\left[y_{0},...,y_{s}\right]\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\rho}$$\scriptstyle{\phi^{{}^{\prime}}}$$\textstyle{A^{\bullet}(Z_{s})}$

where
$\rho:\mathbb{Z}\left[y_{0},...,y_{s}\right]\rightarrow\mathbb{Z}\left[x_{0},...,x_{s}\right]$
is the isomorphism induced by the augmentated change of basis matrix
$B_{s}^{*}$, that is $\rho(y_{0})=x_{0}$ and
$\rho(y_{i})=x_{i}-\sum_{j=i+1}^{s}p_{ij}x_{j}$. Now, by considering the
following images through $\rho$:

$\begin{cases}\rho((y_{i})^{n}+((-1)^{n}+\\#\left\\{j\right\\}_{j\rightarrow
i})(y_{0})^{n})&=(x_{i}-\sum_{k=i+1}^{s}p_{ik}x_{k})^{n}+((-1)^{n}+\\#\left\\{j\right\\}_{j\rightarrow
i})(x_{0})^{n}\\\
&=(x_{i})^{n}+(-1)^{n}\sum_{k=i+1}^{s}p_{ik}(x_{k})^{n}+((-1)^{n}+\\#\left\\{j\right\\}_{j\rightarrow
i})(x_{0})^{n}+\\\
&\sum\limits_{\begin{subarray}{c}n_{i}+n_{i+1}+...+n_{s}=n\\\
n_{i},...,n_{s}\neq
n\end{subarray}}(-1)^{n-n_{i}}\binom{n}{n_{i},n_{i+1}...,n_{s}}\prod_{\beta=i}^{s}(p_{i\beta}x_{\beta})^{n_{\beta}}\\\
&=(-1)^{n}((-1)^{n}(x_{i})^{n}+(x_{0})^{n})+\sum_{k=i+1}^{s}p_{ik}((-1)^{n}(x_{k})^{n}+(x_{0})^{n})+\\\
&\sum\limits_{\begin{subarray}{c}n_{i}+n_{i+1}+...+n_{s}=n\\\
n_{i},...,n_{s}\neq
n\end{subarray}}(-1)^{n-n_{i}}\binom{n}{n_{i},n_{i+1}...,n_{s}}\prod_{\beta=i}^{s}(p_{i\beta}x_{\beta})^{n_{\beta}}\\\
\rho(y_{0}\cdot y_{i})&=x_{0}\cdot(x_{i}-\sum_{k=i+1}^{s}p_{ik}x_{k})\\\
&=x_{0}\cdot x_{i}-\sum_{k=i+1}^{s}p_{i,k}x_{0}\cdot x_{k},\\\
\rho((y_{i}+\sum_{k=i+1}^{s}b_{k,i}y_{k})\cdot(y_{j}+\sum_{l=j+1}^{s}b_{l,j}y_{l}))&=x_{i}\cdot
x_{j}\end{cases}$

we can conclude that $\mathcal{A}\subset Ker(\phi^{{}^{\prime}})$. The
inclusion $Ker(\phi^{{}^{\prime}})\subset\mathcal{A}$ is straightforward by
Remark 3.4. ∎

######  Definition 3.7.

We define for a given a sequence of blow-ups $(Z_{0},...,Z_{s},\pi)$ that an
irreducible component $E_{i}$ is final if and only if $\nexists j$ such that
$P_{j}$ is proximate to $P_{i}$.

######  Remark 3.8.

In [1] the definition of final divisor is different, but in the case of
sequences of point blow-ups both definitions are equivalent.

Also, in [1], a characterization of final was expressed in terms of the
intersection of the irreducible components of the exceptional divisor. We can
now use Theorem 3.6 to refine the characterization.

######  Corollary 3.9.

$E_{i}$ is final if and only if its class in $A^{1}(Z_{s})$, that is
$e_{i}^{s}$, satisfies the following two conditions

$\displaystyle($ $\displaystyle
e_{i}^{s})^{n}=(-1)^{r}(e_{i}^{s})^{n-r}(e_{j}^{s})^{r}$ (12) $\displaystyle($
$\displaystyle(e_{j}^{s})^{n-1}e_{i}^{s}=(h^{s*})^{n}$ (13)

for every $j$ such that $e_{i}^{s}\cdot e_{j}^{s}\neq 0$.

###### Proof.

If $E_{i}$ is final then $\nexists k$ such that $P_{k}$ is proximate to
$P_{i}$. By equation (10) $(e_{j}^{i}+e_{i}^{i})\cdot e_{i}^{i}=0$ if $P_{i}$
is proximate to $P_{j}$ and $e_{i}^{i}\cdot e_{j}^{i}=0$ otherwise. Since
$E_{i}$ is final then it follows that

$\displaystyle($ $\displaystyle e_{j}^{s}+e_{i}^{s})\cdot e_{i}^{s}=0$ if
$P_{i}\rightarrow P_{j}$ (14) $\displaystyle($ $\displaystyle e_{i}^{s}\cdot
e_{j}^{s}=0$ otherwise (15)

From equation (14) we can deduce that
$(e_{i}^{s})^{n}=(-1)^{r}(e_{i}^{s})^{n-r}(e_{j}^{s})^{r}$. Moreover
$(h^{s*})^{n}=(-1)^{n+1}(e_{i}^{s*})^{n}$, so
$(h^{s*})^{n}=(-1)^{2n}e_{i}^{s}(e_{j}^{s})^{n-1}=e_{i}^{s}(e_{j}^{s})^{n-1}$.
Now we will prove that if $E_{i}$ is not final, then some of the above
conditions fails. Among all the index $\left\\{\beta\right\\}$ satisfying
$P_{\beta}\rightarrow P_{i}$ there must exist an index $j$ such that
$P_{j}\rightarrow P_{i}$ but that there not exists $k$ with $P_{k}\rightarrow
P_{i}$ and $P_{k}\rightarrow P_{j}$. Since $E_{j}^{j}$ is final for the
sequence $(Z_{0},...,Z_{j},\pi_{j,0})$, then
$(e_{j}^{j})\cdot(e_{i}^{j})^{n-1}=(h^{j*})^{n}$ and
$(e_{i}^{j})^{n-1-\beta}(e_{j}^{j})^{1+\beta}=(-1)^{\beta}(e_{i}^{j})^{n-1}e_{j}^{j}$.
Moreover, since $\nexists P_{k}$ with $P_{k}$ proximate to both $P_{i}$ and
$P_{j}$, then we can conclude that
$(e_{j}^{s})\cdot(e_{i}^{s})^{n-1}=(h^{s*})^{n}$ and
$(e_{i}^{s})^{n-1-\beta}(e_{j}^{s})^{1+\beta}=(-1)^{\beta}(e_{i}^{s})^{n-1}e_{j}^{s}$.
If $n$ is even, although $(e_{j}^{s})^{n-1}e_{i}^{s}=(h^{s*})^{n}$ since $n-2$
is even too, $(e_{i}^{s})^{n}\neq(-1)^{n-1}(e_{i}^{s})(e_{j}^{s})^{n-1}$ since
by Theorem 3.6 $(e_{i}^{s})^{n}=-(1+\\#\left\\{\beta\right\\})(h^{s*})^{n}$
with $\\#\left\\{\beta\right\\}\geq 1$ so condition (12) fails.
If $n$ is odd, $(e_{j}^{s})^{n-1}e_{i}^{s}=-(h^{j*})^{n}$, since $n-2$ is odd
too, so condition 13 fails. ∎

Some comments about the Chow ring of blow-ups at more general centers

The main result in the literature about the structure of the Chow ring of a
blow-up at a center of arbitrary dimension is the following one

######  Theorem 3.10.

[4, Appendix Theorem 1.] Suppose the map of bivariant rings

$i_{\alpha+1}^{*}:A^{\bullet}(Z_{\alpha})\rightarrow
A^{\bullet}(C_{\alpha+1})$

is surjective, then $A^{\bullet}(Z_{\alpha+1})$ is isomorphic to

$A^{\bullet}(Z_{\alpha})\left[T\right]/(P(T),(T\cdot Ker(i_{\alpha+1}^{*}))),$

where $P(T)\in A^{\bullet}(Z_{\alpha})\left[T\right]$ is any polynomial whose
constant term is $\left[C_{\alpha+1}\right]$ and whose restriction to
$A^{\bullet}(C_{\alpha+1})$is the Chern polynomial of the normal bundle
$\mathcal{N}_{C_{\alpha+1}/Z_{\alpha}}$ i.e.

$i_{\alpha+1}^{*}(P(T))=t^{d}+c_{1}(\mathcal{N}_{C_{\alpha+1}/Z_{\alpha}})T^{d-1}+\cdots+c_{d-1}(\mathcal{N}_{C_{\alpha+1}/Z_{\alpha}})T+c_{d}(\mathcal{N}_{C_{\alpha+1}/Z_{\alpha}}),$

(where $d=codim(C_{\alpha+1},Z_{\alpha})$). This isomorphism is induced by

$\pi_{\alpha+1}^{*}:A^{\bullet}(Z_{\alpha})\rightarrow
A^{\bullet}(Z_{\alpha+1})$

and by sending $-T$ to the class of the exceptional divisor.

However, the surjectivity hypothesis of the theorem is quite restrictive. For
example, let us consider the blow-up of a rational curve $C_{1}\subset
Z_{0}\cong\mathbb{P}^{3}$ of degree $\gamma>1$.

$\textstyle{E_{1}^{1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{j_{1}}$$\scriptstyle{g_{1}}$$\textstyle{Z_{1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\pi_{1}}$$\textstyle{C_{1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{i_{1}}$$\textstyle{Z_{0}}$

Note that in this case the restriction map
$i_{1}^{*}:A^{\bullet}(Z_{0})\rightarrow A^{\bullet}(C_{1})$ is not surjective
since $i_{1}^{*}(h)=\gamma\left[P\right]$, where $\left[P\right]$ denotes the
class of a point $P\in C_{1}$, so we can not apply Keel formula of theorem
3.10. $A^{\bullet}(Z_{1})$ is no longer generated by
$\left\\{h^{1*},e_{1}^{1*}\right\\}$ and we need to add an extra generator
$r_{1}=j_{1*}\left[g_{1}^{-1}(P)\right]$, which geometrically is the fiber of
a point in the blow-up.
One can prove that

$A^{\bullet}(Z_{1})\cong\mathbb{Z}\left[x_{0},x_{1},w_{1}\right]/\mathcal{I}$

where

$\mathcal{I}=((x_{0})^{2}\cdot x_{1},x_{0}\cdot x_{1}-\gamma
w_{1},(x_{1})^{2}-c_{1}(N)w_{1}+\gamma(x_{0})^{2},x_{0}\cdot
w_{1},(x_{0})^{3}+x_{1}\cdot w_{1})$

by sending $x_{0},x_{1}$ and $w_{1}$ to $h^{1*},e_{1}^{1*}$ and $r_{1}$
respectively.

## References

  * [1] Daniel Camazon and Santiago Encinas.  Sequences of point blow-ups from a combinatorial point of view, 2022.  arXiv:2207.04270.
  * [2] David Eisenbud and Joe Harris.  3264 and all that—a second course in algebraic geometry.  Cambridge University Press, Cambridge, 2016.
  * [3] William Fulton.  Intersection theory, volume 2 of Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics].  Springer-Verlag, Berlin, second edition, 1998.
  * [4] Sean Keel.  Intersection theory of moduli space of stable $n$-pointed curves of genus zero.  Trans. Amer. Math. Soc., 330(2):545–574, 1992.


# Real-time Forecast Models for TBM Load Parameters Based on Machine Learning
Methods

Xianjie Gao1, Xueguan Song2, Maolin Shi2, Chao Zhang3111Corresponding author,
Hongwei Zhang3
1Department of Basic Sciences,
Shanxi Agricultural University, Taigu, Shanxi 030801, P.R. China
<EMAIL_ADDRESS>
2School of Mechanical Engineering,
Dalian University of Technology, Dalian, Liaoning, 116024, P.R. China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
3School of Mathematical Sciences,
Dalian University of Technology, Dalian, Liaoning, 116024, P.R. China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Because of the fast advance rate and the improved personnel safety, tunnel
boring machines (TBMs) have been widely used in a variety of tunnel
construction projects. The dynamic modeling of TBM load parameters (including
torque, advance rate and thrust) plays an essential part in the design, safe
operation and fault prognostics of this complex engineering system. In this
paper, based on in-situ TBM operational data, we use the machine-learning (ML)
methods to build the real-time forecast models for TBM load parameters, which
can instantaneously provide the future values of the TBM load parameters as
long as the current data are collected. To decrease the model complexity and
improve the generalization, we also apply the least absolute shrinkage and
selection (Lasso) method to extract the essential features of the forecast
task. The experimental results show that the forecast models based on deep-
learning methods, e.g., recurrent neural network and its variants, outperform
the ones based on the shallow-learning methods, e.g., support vector
regression and random forest. Moreover, the Lasso-based feature extraction
significantly improves the performance of the resultant models.

Keywords: Tunnel boring machines; feature extraction; real-time forecast; load
parameters; machine learning

##  1 Introduction

With the increasing demand of long and deep tunnel construction, the
traditional tunneling techniques (e.g., drilling and blasting) have not
satisfied the construction requirement of high efficiency, small environmental
damage and low construction risk. Instead, tunnel boring machines (TBMs) have
become a primary method in various underground excavation projects (e.g.,
mountain tunnel [1], city subway [2] and coal mine [3]) with its predominance
of high advance rate, environmental friendliness and security. The widespread
use of TBMs benefits from the experiences gained from different conditions and
the new technological advances. However, there are still many challenges in
the machinery design and the project application of TBMs. For example, under
complicated construction conditions, TBMs’ excavation process will be faced
with high risk caused by an unanticipated presence of water leakage, rock
spalling, geological faulting, weak regions, break outs of the tunnel wall and
subsidence of work area. To improve TBMs’ reliability and reduce unscheduled
downtime, some research works have paid attention to TBMs’ risk evaluation
[4], design and R$\&$D [5], and safety decision [6].

One main research concern of TBM is to develop a health monitoring and
prognostics system that can automatically find out the anomalies and the
provides the fault prognostic during the TBM operation [7]. An essential part
of such a system is the real-time forecast of TBM load parameters (including
torque, advance rate and thrust), which are important indicators to evaluate
TBMs’ machine health condition by observing whether their forecast values
exceed the pre-defined threshold or not [8, 9]. However, due to the complex
geological condition, it is still challenging to achieve the accurate real-
time forecast of TBM load parameters (cf. [10, 11, 12, 13, 14, 15, 16]).

In general, the existing methods for forecasting TBM load parameters can be
categorized into three kinds:

  1. (i)

Empirical methods aim to developing the empirical equations, which describe
the relationship between the geological information and some important
operation parameters, e.g., cutter force, thrust and cutter power, to forecast
TBM load parameters [17, 18, 19, 20]. In order to improve the forecast
accuracy, there have also been some attempts to combine the empirical methods
with the cutting experiments (cf. [21, 22, 23]).

  2. (ii)

Under the assumption that one stratum or different stratums are distributed on
the excavation face, rock-soil mechanics methods explore the relationship
between the rock-soil mechanics characteristics of the stratums and TBMs’
operation state and then provide the estimate of TBM load parameters [24, 25,
26].

  3. (iii)

The numerical simulation methods employ some differential equation models to
describe the dynamic variation of TBMs’ operation parameters, and then provide
the estimate of these parameters with the aid of computer simulation, e.g.,
the three-dimensional finite element simulation model [27] and the 3D FEM
model [28].

However, the practical applications of these methods have the following
limitations: 1) we need the considerable specialty knowledge or the
participation of relevant professional technicians; 2) the forecast results
are lack of generality and cannot be (at least cannot be directly) generalized
to other construction conditions; and 3) these methods are unsuitable (at
least cannot be directly applied) to the real-time forecast, which requires
that the future forecast results should be instantaneously provided as long as
the current operational data are collected.

In this paper, we adopt the machine-learning (ML) methods to develop the real-
time forecast models for TBM load parameters based on TBM operational data
collected from the sensors equipped on a TBM in the construction process. We
mainly concern with two kinds of ML models: the deep-learning models including
recurrent neural networks (RNN) and its variants (e.g. long-short term memory
network (LSTM) and gated recurrent unit network (GRU)); and the shallow-
learning models including support vector regression (SVR), random forest (RF)
and feed-forward neural network (FNN). To capture the temporal information
encoded in the original data, the input of these models is a length of time
series data containing the previous and the current data, and the output is
the estimate of the future values of the TBM load parameters including torque,
advance rate and thrust.

Although the existing work [29] has discussed the prediction for TBM
operational parameters, the models given in [29] can only provide the
prediction value of one single operational parameter once a time and thus has
a low efficiency in the prediction task of multiple operational parameters.
Moreover, the training difficulty of multiple-output models is relatively
higher than the single-output models because the former have more undetermined
weights. Therefore, we introduce the Lasso method to extract essential
features from the original TBM operational data to simplify the model
structure and then to improve the training efficiency accordingly. The
experimental results support the effectiveness of the ML-based models for the
real-time forecast task.

The rest of this paper is organized as follows. In Section 2, we give the
necessary preliminaries on the ML methods considered in this paper. In Section
3, we introduce the ML-based models of the real-time forecast of the TBM load
parameters. Section 4 presents experimental results and the last section
concludes the paper. In the appendix, we list some important features
appearing in the in-situ TBM operational data.

##  2 Preliminaries on ML Methods

In this section, we give a brief introduction of the main ML methods used in
this paper including FNN, RNN, LSTM, GRU and Lasso. We will show that, because
of the existence of loop structure, RNN and its variants can maintain the
previous input information for the future forecast, and thus are more suitable
to the real-time forecast. The details of SVR and RF are referred to [30].

###  2.1 Feed-forward Neural Networks

Artificial neural networks (ANNs) are the typical representatives of
connectionism models. By using the weighted directed lines to connect neurons
squashed by non-linear functions, ANNs can accurately approximate arbitrary
continuous functions [31]. Because of this strong non-linear mapping
capability, ANNs have been widely used in various kinds of AI-related problems
[32]. One of the most frequently used ANNs is the feed-forward neural network
(FNN) that stacks multiple-layer neurons and the adjacent layers are connected
by the unidirectional lines such that the information flow is unidirectional
from the input layer to the output layer (cf. Fig. 1).

Take FNN with one hidden layer as an example. Let ${\bf V}=[{\bf v}_{1},{\bf
v}_{2},\cdots,{\bf v}_{L}]\in\mathbb{R}^{J\times L}$ be the weight matrix
between the output layer and the hidden layer, where ${\bf v}_{l}$ ($1\leq
l\leq L$) is the weight vector connecting the hidden layer with the $l$-th
output node. Denote ${\bf W}=[{\bf w}_{1},{\bf w}_{2},\cdots,{\bf
w}_{J}]\in\mathbb{R}^{I\times J}$ as the weight matrix between the input layer
and the hidden layer, where ${\bf w}_{j}$ ($1\leq j\leq J$) is the weight
vector connecting the input layer with the $j$-th hidden node. Let
$\sigma:\mathbb{R}\rightarrow\mathbb{R}$ be a sigmoid function. Given an
$I$-dimensional input ${\bf x}=(x^{(1)},\cdots,x^{(I)})^{T}\in\mathbb{R}^{I}$,
the output ${\bf y}\in\mathbb{R}^{L}$ of the FNN can be computed by:

${\bf y}=\bm{\sigma}({\bf V}^{\prime}{\bf
h})=\left(\begin{array}[]{c}\sigma(\langle{\bf v}_{1},{\bf h})\rangle\\\
\sigma(\langle{\bf v}_{2},{\bf h})\rangle\\\ \vdots\\\ \sigma(\langle{\bf
v}_{L},{\bf
h})\rangle\end{array}\right)=\left(\begin{array}[]{c}\sigma\big{(}\langle{\bf
v}_{1},\bm{\sigma}({\bf W}^{\prime}{\bf x}\rangle\big{)}\\\
\sigma\big{(}\langle{\bf v}_{2},\bm{\sigma}({\bf W}^{\prime}{\bf
x})\rangle\big{)}\\\ \vdots\\\ \sigma\big{(}\langle{\bf
v}_{L},\bm{\sigma}({\bf W}^{\prime}{\bf x})\rangle\big{)}\end{array}\right),$
(1)

where $\bm{\sigma}(\cdot)$ is the vector-valued function that is component-
wise w.r.t. the scalar-valued function $\sigma(\cdot)$. One popular FNN
training algorithm is the stochastic gradient descent method which updates the
weights ${\bf W}$ and ${\bf V}$ in the following way: given a sample set
$\\{({\bf x}_{t},{\bf y}_{t})\\}_{t=1}^{T}$, randomly select $t$ from
$\\{1,2,\cdots,T\\}$ and then

$\displaystyle{\bf W}_{k+1}=$ $\displaystyle{\bf
W}_{k}-\eta\frac{\partial\mathcal{L}_{t}^{k}({\bf V}_{k},{\bf
W}_{k})}{\partial{\bf h}_{t}^{k}(\bf W)}\frac{\partial{\bf h}_{t}^{k}(\bf
W)}{\partial{\bf W}_{k}};$ $\displaystyle{\bf V}_{k+1}=$ $\displaystyle{\bf
V}_{k}-\eta\frac{\partial\mathcal{L}_{t}^{k}({\bf V}_{k},{\bf
W}_{k})}{\partial{\bf h}_{t}^{k}(\bf W)}$

with $\mathcal{L}_{t}^{k}({\bf V},{\bf W})=\|{\bf y}_{t}^{k}-\widehat{{\bf
y}}_{t}^{k}\|_{2}^{2}$, where $\eta>0$ is a learning rate and $\widehat{{\bf
y}}_{t}^{k}$ is the network output w.r.t. the input ${\bf x}_{t}$ at the
$k$-th iteration.

###  2.2 Recurrent Neural Networks

Recurrent neural networks (RNNs) are a class of neural networks that have the
loop structures to maintain the previous learning information and then combine
it with the current input to generate the future network outputs [33]. In this
manner, the output of a RNN contains the information of the current and the
previous inputs. This characteristics makes RNNs quite suitable to dynamically
modeling the sequential behavior of time series. As shown in Fig. 2, given a
current input ${\bf x}_{t+1}$, the current output of the hidden node ${\bf
h}_{t+1}$ is computed by

${\bf h}_{t+1}={\bf tanh}({\bf U}{\bf x}_{t+1}+{\bf W}{\bf x}_{t}),$

where ${\bf tanh}(\cdot)$ is the vector-valued function that is component-wise
w.r.t. the scalar-valued function $\tanh(\cdot)$; ${\bf U}$ is the weight
matrix between the current input ${\bf x}_{t+1}$ and the current hidden output
${\bf h}_{t+1}$; and ${\bf W}$ is the weight matrix between the current hidden
output ${\bf h}_{t+1}$ and the previous hidden output ${\bf h}_{t}$. Then, the
fully-connected node gives the following output:

${\bf q}_{t+1}={\bf tanh}({\bf V}{\bf h}_{t+1}),$

where ${\bf V}$ is the weight matrix between the current hidden output ${\bf
h}_{t+1}$ and the current output ${\bf q}_{t+1}$.

RNNs perform well in modeling short sequences, while the training for the long
sequences will fail because of vanishing gradients or explosive gradients. The
former means that the gradient signal is too small to effectively update
weights; and the latter refers to the phenomenon that the magnitude of
gradients becomes so large that the training process diverges [34]. To
overcome this limitation, some specific gate operations are imposed into RNN’s
hidden nodes, and then there arise some variants of RNN, e.g., long-short term
memory network (LSTM) and gated recurrent unit network (GRU).

###  2.3 Long-short Term Memory Networks

In RNNs, the hidden nodes implement the inner-product operation and then are
squashed by a nonlinear activation function. For a long time series data, this
simple structure cannot efficiently capture the useful sequential
characteristic or eliminate the reductant information. Instead of the inner-
product operation, the long-short term memory (LSTM) networks impose three
kinds of adaptive and multiplicative gate operations into RNN’s hidden nodes
including the input gate, the forget gate and the output gate [35]. The input
and the output gates respectively control the information flowing in and out;
and the forget gate is used to eliminate the redundancy from time series data
and meanwhile to maintain the useful information.

In Fig. 3, we display a typical structure of LSTMs’ hidden nodes, where $i$,
$f$ and $o$ stand for the input gate, the forget gate and the output gate
respectively. Denote $c$ and $g$ as the memory cell and the new memory cell
respectively. Given the previous hidden output ${\bf h}_{t}$ and the current
input ${\bf x}_{t+1}$, the forward propagation equations of LSTM are given as
follows:

$\displaystyle{\bf i}_{t+1}=$ $\displaystyle\bm{\sigma}({\bf U}^{[i]}{\bf
x}_{t+1}+{\bf W}^{[i]}{\bf h}_{t});$ $\displaystyle{\bf f}_{t+1}=$
$\displaystyle\bm{\sigma}({\bf U}^{[f]}{\bf x}_{t+1}+{\bf W}^{[f]}{\bf
h}_{t});$ $\displaystyle{\bf o}_{t+1}=$ $\displaystyle\bm{\sigma}({\bf
U}^{[o]}{\bf x}_{t+1}+{\bf W}^{[o]}{\bf h}_{t});$ $\displaystyle{\bf
g}_{t+1}=$ $\displaystyle{\bf tanh}({\bf U}^{[g]}{\bf x}_{t+1}+{\bf
W}^{[g]}{\bf h}_{t});$ $\displaystyle{\bf c}_{t+1}=$ $\displaystyle{\bf
c}_{t}\odot{\bf f}_{t+1}+{\bf g}_{t+1}\odot{\bf i}_{t+1};$ $\displaystyle{\bf
h}_{t+1}=$ $\displaystyle{\bf tanh}({\bf c}_{t+1})\odot{\bf o}_{t+1},$

where $\odot$ is the Hadamard product, and ${\bf U}^{[\cdot]}$, ${\bf
W}^{[\cdot]}$ stand for the weight matrices for the corresponding gates.

###  2.4 Gated Recurrent Unit Networks

In some problems, the three-gate structure of LSTM’s hidden node is likely to
cause the overfitting, and thus Cho et al. [36] proposed gated recurrent unit
networks (GRUs) , which integrate the input and the forget gates in LSTMs’
hidden nodes into one gate (cf. Fig. 4). Because of the relatively simpler
structure, GRUs have a higher training efficiency but also perform better than
RNNs and LSTMs in many learning tasks [36]. The forward propagation equations
of GRU are given as follows:

$\displaystyle{\bf z}_{t+1}=$ $\displaystyle\bm{\sigma}({\bf U}^{[z]}{\bf
x}_{t+1}+{\bf W}^{[z]}{\bf h}_{t});$ $\displaystyle{\bf r}_{t+1}=$
$\displaystyle\bm{\sigma}({\bf U}^{[r]}{\bf x}_{t+1}+{\bf W}^{[r]}{\bf
h}_{t});$ (2) $\displaystyle{\bf g}_{t+1}=$ $\displaystyle{\bf
tanh}\big{(}{\bf U}^{[g]}{\bf x}_{t+1}+{\bf W}^{[g]}({\bf h}_{t}\odot{\bf
r}_{t+1})\big{)};$ $\displaystyle{\bf h}_{t+1}=$ $\displaystyle({\bf 1}-{\bf
z}_{t+1})\odot{\bf g}_{t+1}+{\bf z}_{t+1}\odot{\bf h}_{t},$

where ${\bf x}_{t+1}$, ${\bf h}_{t+1}$, ${\bf z}_{t+1}$ and ${\bf r}_{t+1}$
are the input, the output, the update gate and the reset gate vectors
respectively; and ${\bf U}^{[\cdot]}$, ${\bf W}^{[\cdot]}$ stand for the
weight matrices for the corresponding gates.

###  2.5 Least Absolute Shrinkage and Selection Operator

Given a sample set $D=\\{({\bf x}_{1},y_{1}),({\bf x}_{1},y_{1}),\ldots,({\bf
x}_{N},y_{N})\\}$, where ${\bf x}\in{\mathbb{R}}^{I}$, $y\in{\mathbb{R}}$,
consider the following linear regression model:

$\min_{\bm{\beta}}\frac{1}{2}\sum_{n=1}^{N}\left(y_{i}-(\langle{\bf
x}_{n},\bm{\beta}\rangle+\beta_{0})\right)^{2},$

where $\langle\cdot,\cdot\rangle$ stands for the inner product,
$\bm{\beta}\in\mathbb{R}^{I}$ is the weight vector and
$\beta_{0}\in\mathbb{R}$ is the bias. This model easily leads to overfitting
and has a low tolerance for noisy information. To overcome these shortcomings,
one way is to add the regularizer into the objective function:

$\min_{\bm{\beta}}\frac{1}{2}\sum_{n=1}^{N}\left(y_{i}-(\langle{\bf
x}_{n},\bm{\beta}\rangle+\beta_{0})\right)^{2}+\lambda\Omega(\bm{\beta}),$

where $\lambda>0$ is the regularization coefficient and $\Omega(\bm{\beta})$
is the regularizer w.r.t. the weight vector $\bm{\beta}$. Especially, when
$\Omega(\bm{\beta})=\|\bm{\beta}\|_{2}^{2}$, the model is called as the ridge
regression (cf. [37]); and when $\Omega(\bm{\beta})=\|\bm{\beta}\|_{1}$, the
model is called as the least absolute shrinkage and selection operator (Lasso)
(cf. [38]).

Lasso has become one of the most frequently used feature-extraction methods
and has been widely used in many engineering problems (e.g. [39, 40]). We note
that, one common way to use Lasso method for the feature extraction is to
remove the features whose Lasso coefficients are close to zero and maintain
the rest as the input features for the subsequent learning process.

##  3 Real-time Forecast of TBM Load Parameters

In this section, we present the models of the real-time forecast of TBM load
parameters. First, we give the problem setup of the real-time forecast. Then,
we discuss two output-modes of the forecast models. Finally, we introduce the
Lasso-based feature extraction for TBM operational data.

###  3.1 Real-time Forecast

The real-time forecast requires that models should instantaneously generate
the future values of the TBM load parameters during the next period as long as
the current operational data are collected. Therefore, a desired forecast
model should be able to dynamically capture the dynamic nature of time series
data. One reasonable way to handle this issue is to take a time-continuous
sequence from the whole time series as the input of the models.

Given a time series $\\{({\bf x}_{t},{\bf y}_{t})\\}_{t=1}^{\infty}$ with
${\bf x}_{t}=(x_{t}^{(1)},\cdots,x_{t}^{(I)})^{T}\in\mathbb{R}^{I}$ and ${\bf
y}_{t}=(y_{t}^{(1)},\cdots,y_{t}^{(J)})^{T}\in\mathbb{R}^{J}$, the real-time
forecast task aims to find a function
$f:\underbrace{\mathbb{R}^{I}\times\cdots\times\mathbb{R}^{I}}_{\tau}\rightarrow\mathbb{R}^{J}$
such that its output

$\widehat{{\bf y}}_{t+1}:=f({\bf x}_{t-\tau+1},{\bf x}_{t-\tau+2},\cdots,{\bf
x}_{t-1},{\bf x}_{t})$

can approximate the actual output ${\bf y}_{t+1}$ accurately, where $\tau>0$
is the window width that controls the length of input sequences. In general,
the Euclidean distance $\|{\bf y}_{t}-\widehat{{\bf y}}_{t}\|_{2}^{2}$ is used
to measure the difference between ${\bf y}_{t}$ and $\widehat{{\bf y}}_{t}$.
Especially, if the output vector ${\bf y}_{t}$ is composed of some specific
components of ${\bf x}_{t}$, i.e., ${\bf
y}_{t}:=(x_{t}^{(i_{1})},\cdots,x_{t}^{(i_{J})})^{T}$ ($J<I$), it is the so-
called self-forecast problem.

In practical applications, given a $T$ length of time series data $\\{({\bf
x}_{t},{\bf y}_{t})\\}_{t=1}^{T}$, we can draw $(T-\tau)$ sub-sequence $({\bf
x}_{t-\tau+1},{\bf x}_{t-\tau+2},\cdots,{\bf x}_{t-1},{\bf x}_{t})$ from it
and the length of each sub-sequence is $\tau$. Then, the function $f$ is
obtained by minimizing the mean squared error (MSE)

$\min_{f\in\mathcal{F}}\frac{1}{T-\tau}\sum_{t=\tau+1}^{T}\|{\bf y}_{t}-f({\bf
x}_{t-\tau+1},{\bf x}_{t-\tau+2},\cdots,{\bf x}_{t-1},{\bf x}_{t})\|_{2}^{2},$
(3)

where $\mathcal{F}$ is a function class that can be set as a shallow-learning
model (e.g. SVR, RF or FNN) or a deep-learning model (e.g. RNN, LSTM or GRU).

###  3.2 Output Modes and Lasso-based Feature Extraction

As addressed above, TBM’s load refer to three kinds of parameters: torque,
advance rate and thrust. Thus, forecasting these parameters can be achieved in
two kinds of output modes: single-output and multiple-output. In the single-
output mode, the aforementioned $f$ is a scalar function that only generates
the estimate of one of the three parameters each time. Because of the
relatively simpler model structure, the single-output models have a higher
training efficiency than that of multiple-output models. However, three
individual models should be built for the three TBM parameters respectively.
In contrast, one multiple-output model is enough to simultaneously forecast
the three parameters, and thus it has a higher operating efficiency.

One way to improve the performance of multiple-output models is to decrease
the model complexity by extracting useful features from TBM operational data.
As addressed in Section 2.5, Lasso provides a powerful method for extracting
features that are strongly related to learning tasks by observing the
magnitude of their Lasso coefficients. In this manner, we select the essential
features of the forecast task. The subsequent experimental results show that
the Lasso-based feature extraction can significantly improve the performance
of the forecast models.

##  4 Numerical Experiments

In this section, we use the in-situ TBM operational data to conduct the
numerical experiments for verifying the validity of the proposed models. All
experiments were processed by using PyTorch in a computer with
Intel$\circledR$Core${}^{\text{TM}}$ i7 CPU at $4$ GHz, $64$ GB RAM and two
NVIDIA GTX 1080 graphics cards.

###  4.1 Data Description

The data set is collected from the sensors equipped on an earth pressure
balance shield TBM of the subway tunnel project in Shenzhen, a city of China.
The TBM has $500T$ total mass and $120$ knifes on its cutterhead. The tunnel
length is about $2000m$ and its diameter is $6.3m$. The elevation of ground
surface is between $0.2m\sim 5.8m$, and the depth of the tunnel floor from the
ground surface is between $11.8m\sim 25.4m$. There are $44$ features in the
TBM operational data (cf. Tab. 3).

We choose a $3000$ length of time-continuous TBM operational data $\\{{\bf
x}_{1},{\bf x}_{2},\cdots,{\bf x}_{3000}\\}$ and then normalize them in the
component-wise way:

$x_{t}^{(i)}=\frac{x_{t}^{(i)}-\min\limits_{1\leq k\leq
3000}\big{\\{}x^{(i)}_{k}\big{\\}}}{\max\limits_{1\leq j,k\leq
3000}\big{\\{}|x^{(i)}_{k}-{x^{(i)}_{j}|\big{\\}}}},\quad
t=1,2,\cdots,3000,\;\;i=1,2,\cdots,44.$ (4)

This sequence is split into two time-continuous sub-sequences: $\\{{\bf
x}_{1},{\bf x}_{2},\cdots,{\bf x}_{2500}\\}$ and $\\{{\bf x}_{2501},{\bf
x}_{2502},\cdots,{\bf x}_{3000}\\}$. The former is used to train models, and
the latter serves as the test set to examine whether the trained models can
accurately forecast the load parameters during next period.

Moreover, we consider two error criterions: the root of MSE (RMSE) (cf. (3))
and the mean absolute percentage error (MAPE):

${\rm
MAPE}=\frac{100}{T-\tau}\sum_{t=\tau+1}^{T}\frac{|\widehat{y}_{t}^{(j)}-y_{t}^{(j)}|}{y_{t}^{(j)}}\%,\quad
j=\\{1,2,3\\},$

where $\widehat{y}_{t}^{(j)}$ and $y_{t}^{(j)}$ stand for the prediction and
the actual values of the load parameters “Torque” ($j=1$), “Advance Rate”
($j=2$) and “Thrust” ($j=3$).

###  4.2 Experimental Settings

Here, we consider the real-time forecast of TBM load parameters including
torque, thrust and advance rate. Note that the three parameters belong to the
$44$ features of TBM operational data, and thus it is a self-forecast problem.
Set the window width $\tau=5$, i.e., generating the current output needs the
previous five inputs.

The following are the specific settings of the aforementioned models in the
experiments:

  * •

The kernel function of support vector regression (SVR) is selected as “RBF”
with the parameter $\epsilon=0.1$ and the constant $C$ equals to $1$.

  * •

Random forest (RF) is an ensemble of ten decision trees with the maximal depth
being $5$.

  * •

Feed-forward neural network (FNN) has two hidden layers both of which have
$10$ nodes, its input layer contains $220$ nodes and its output layer has one
node (resp. three nodes) corresponding to single-output mode (resp. multiple-
output mode) (cf. Fig. 1). The network is trained by using stochastic gradient
descent with learning rate $\eta=0.01$ and the maximal training epoch is
$100$.

  * •

The recurrent neural network (RNN) is composed of an input layer with $44$
nodes, one RNN layer with $10$ RNN units, a fully-connected layer composed of
two layers both of which have $10$ hidden nodes and an output layer with one
or three nodes. The LSTM (resp. GRU) has the same structure as that of RNN
except that the RNN units in the RNN layer are replaced with LSTM (resp. GRU)
units (cf. Fig. 5). These models are trained by using RMSprop [41] with the
learning rate $\eta=0.0005$ and the maximal training epoch is $30000$.

The comparative experiments are conducted in the four kinds of settings:
single-output mode with (reps. without) Lasso-based feature extraction
(briefly denoted as “s.w.l.” (resp. “s.w/o.l.”)) and multiple-output mode with
(reps. without) Lasso-based feature extraction (briefly denoted as “m.w.l.”
(resp. “m.w/o.l.”)), respectively. In the s.w/o.l. and m.w/o.l. settings, the
inputs of these models are the original operation data that have $44$
features. The Lasso-based feature extraction will select the essential
features according to their Lasso coefficients whose magnitudes reflects their
importance to the forest task. Thus, in the s.w.l. setting, for each one of
the three parameters, the input features are extracted from the $44$ features
by using the Lasso method. Combine its own feature and extracted features, we
use Lasso-based single-output SVR, RF, FNN amd RNN-based (RNNs, LSTM and GRU)
predictors to real-time forecast. We change the number of input nodes
according to the number of extracted features. The number of nodes in the
input layer of the predicted torque FNN and RNN-based models are $30$ and $6$,
respectively. The number of nodes in the input layer of the predicted advance
rate FNN and RNN-based models are $35$ and $7$, respectively. The number of
nodes in the input layer of the predicted thrust FNN and RNN-based models are
$35$ and $7$, respectively. Comprehensive the features to be predicted and all
extracted features. We use multi-output SVR, RF, FNN amd RNN-based (RNNs, LSTM
and GRU) predictors. We set the number of nodes in the input layer of FNN and
RNN-based models to $90$ and $18$, respectively. Note that, in the m.w.l.
setting, the input features are the integration of the features extracted by
the Lasso method implemented for each of the load parameters.

###  4.3 Experimental Results

To exam the validity of the proposed models, we mainly concern with the
following issues:

  * •

The performance of these models for the real-time forecast task.

  * •

The performance discrepancy between the single-output mode and the multiple-
output mode.

  * •

The effectiveness of the Lasso-based feature extraction.

As shown in Fig. 6 \- Fig. 8, the deep-learning models (RNN, LSTM and GRU)
outperform the shallow-learning models (SVR, RF and FNN) for the forecast task
in most cases, and these deep models can accurately capture the dynamic
variation of the three parameters “Torque”, “Advance Rate” and “Thrust” during
the next period. Especially, in most cases, GRU performs better than RNN and
LSTM, which is because the relatively simpler gate structure is enough to
afford the forecast task and meanwhile avoids the overfitting.

As listed in Tab. 2, the performance of single-output models is better than
that of multiple-output models in most cases because the former has a simpler
structure that leads to a higher training efficiency. However, to achieve the
forecast of the three parameters, we need to build three single-output models
respectively. In contrast, one multiple-output model can simultaneously
provide the forecast results of the three TBM parameters and thus has a higher
operating efficiency because of a smaller amount of model parameters.

Table 1: Results of Lasso-based Feature Extraction Features Parameters Coefficient | Torque | Advance Rate | Thrust
---|---|---|---
Rotation speed of cutter (r/min) | 29.912 | 4.178 |
Cutter power (kw) | 28.871 |  |
Pressure of chamber at top left (bar) | -9.126 |  |
Pressure of chamber at bottom right (bar) | 1.451 |  |
Temperature of oil tank (∘C) | 0.813 |  |
Pressure of shield tail seal at left front (bar) |  | 48.521 |
Rolling angle (∘) |  | -29.760 |
Propelling pressure (bar) |  | 0.075 |
Propelling pressure of C group (bar) |  | 0.041 | 30.267
Pressure of screw pump (bar) |  | 0.023 |
Pressure of shield tail seal at top left back (bar) |  |  | 36.896
Propelling pressure of A group (bar) |  |  | 29.833
Propelling pressure of B group (bar) |  |  | 26.638
Propelling pressure of D group (bar) |  |  | 5.219
Displacement of A group of thrust cylinders (mm) |  |  | -0.001

In general, compared with the single-output models, the performance loss of
the multiple-output models is due to their complex structures. To overcome
this limitation, we use the Lasso-based feature extraction to eliminate the
redundant features from the inputs so as to decrease the model complexity. The
effectiveness of the Lasso-based feature extraction can be measured by the
quantity called “Performance Gain” (or briefly denoted as “PERF Gain”):

$\mathop{\mbox{PERF Gain}}\limits_{\mbox{(s.w/o.l. vs.
s.w.l.)}}=\frac{\mathop{\mbox{RMSE}}_{\mbox{s.w/o.l.}}-\mathop{\mbox{RMSE}}_{\mbox{s.w.l.}}}{\mathop{\mbox{RMSE}}_{\mbox{s.w/o.l.}}};$

and

$\mathop{\mbox{PERF Gain}}\limits_{\mbox{(m.w/o.l. vs.
m.w.l.)}}=\frac{\mathop{\mbox{RMSE}}_{\mbox{m.w/o.l.}}-\mathop{\mbox{RMSE}}_{\mbox{m.w.l.}}}{\mathop{\mbox{RMSE}}_{\mbox{m.w/o.l.}}},$

where “RMSE” can also be changed to “MAPE” accordingly. In Tab. 1, we have
listed the three groups of features whose Lasso coefficients have the large
magnitude and deem that they are strongly related with the forecast task.
Meanwhile, we omit the features with the coefficients close to zero. We find
that the essential features of the three TBM operation parameters differ from
each other except for two individual features. This fact suggests that the
three parameters have difference inherent characteristics. The experimental
results given in Tab. 2 point out that the introduction of Lasso-based feature
extraction brings a significant performance gain in most cases, and thus
support the validity of this method.

Table 2: RMSE (MAPE) of Models for Real-time TBM Multi-parameters Forecast

|  Models Settings Results | s.w/o.l. | s.w.l. | $\mathop{\mbox{PERF Gain}}\limits_{\mbox{(s.w/o.l. vs. s.w.l.)}}$ | m.w/o.l. | m.w.l. | $\mathop{\mbox{PERF Gain}}\limits_{\mbox{(m.w/o.l. vs. m.w.l.)}}$
---|---|---|---|---|---|---|---
Torque | SVR | 98.204(13.541%) | 36.510(3.999%) | 62.882%(70.467%) | 98.204(13.541%) | 66.564(8.976%) | 32.219%(33.712%)
RF | 94.795(8.471%) | 30.499(3.221%) | 67.826%(61.976%) | 77.331(7.959%) | 29.224(2.972%) | 62.209%(62.659%)
FNN | 41.857(5.144%) | 31.265(2.919%) | 25.305%(43.254%) | 93.226(11.672%) | 49.240 (6.362%) | 47.182%(45.493%)
RNN | 40.719(4.555%) | 28.472(2.281%) | 30.077%(49.923%) | 61.797(8.229%) | 56.020(7.283%) | 9.348%(11.496%)
GRU |  35.476(3.974%) | 28.112(2.593%) | 20.758%(34.751%) | 64.407(8.596%) | 55.912(7.328%) | 13.190%(14.751%)
LSTM | 38.239(4.266%) | 29.143(2.821%) | 23.787%(33.872%) |  39.688(4.710%) | 54.931(7.517%) | -38.407%(-59.597%)
Advance Rate | SVR | 3.613(54.953%) | 2.512(38.643%) | 30.473%(29.680%) | 3.613(54.953%) | 2.687 (43.608%) | 25.630%(20.645%)
RF | 2.215(28.719%) | 2.149(27.627%) | 2.980%(3.802%) | 2.405(29.062%) | 2.404(29.299%) | 0.042%(-0.815%)
FNN | 2.586(34.065%) | 1.861(25.646%) | 28.036%(24.715%) | 2.424(29.629%) |  1.943(27.001%) | 19.843%(8.870%)
RNN | 2.367(31.959%) | 1.786(25.972%) | 24.546%(18.733%) |  2.121(36.967%) | 1.971(28.685%) | 7.072%(22.404%)
GRU |  2.124(29.405%) | 1.749(28.723%) | 17.655%(2.319%) | 2.173(31.787%) | 2.137(29.780%) | 1.657%(6.314%)
LSTM | 2.338(31.997%) | 1.775(28.111%) | 24.080%(12.145%) | 2.152 (30.102%) | 2.189(31.483%) | -1.719%(-4.588%)
Thrust | SVR | 221.442(4.956%) | 190.923(4.294%) | 13.782%(13.358%) | 221.442(4.956%) | 145.007(3.109%) | 34.517%(37.268%)
RF | 85.340(0.956%) | 99.354(1.434%) | -16.421%(-50.000%) | 110.099 (1.137%) | 125.190(1.185%) | -13.707%(-4.222%)
FNN | 132.889(2.854%) | 65.777(0.590%) | 50.502%(79.327%) | 83.898(1.262%) | 76.749(0.928%) | 8.521%(26.466%)
RNN | 78.607(1.187%) | 68.987(0.588%) | 12.238%(50.463%) |  83.702(1.232%) | 82.603(1.101%) | 1.313%(10.633%)
GRU |  69.903(0.844%) | 69.119(0.855%) | 1.122%(-1.303%) | 86.797(1.428%) | 67.414 (0.621%) | 22.331%(56.513%)
LSTM | 80.748(1.302%) | 67.223(0.546%) | 16.750%(58.065%) | 86.724(1.458%) |  66.159(0.599%) | 23.713%(58.916%)

##  5 Conclusion

Although TBMs have been widely used in various tunnel construction projects,
there still remain some challenges in their machinery design and construction
safety. Many research interests are inspired to explore the TBMs’ risk
evaluation, fault diagnostics and prognostics. One key to these problems is
the real-time forecast of the TBM load parameters (including torque, advance
rate and thrust), which requires that the forecast models should
instantaneously provide the accurate future values of these parameters during
the next period as long as the current operational data are collected. There
have been many works on the forecast of TBM operational parameters, while it
is still challenging to realize a true sense of “real-time” forecast.

In this paper, we use the ML methods to develop the models for the real-time
forecast of the TBM load parameters. In particular, we adopt the shallow-
learning models (including SVR, RF and FNN) and the deep-learning models (RNN,
LSTM, GRU) as the forecast models, respectively. Different from the non-
sequential learning problems which require that models should provide the
estimate of the current real value when the current data are collected, the
inputs of these real-time forecast models are a length of time series data
containing the previous and the current data, and their outputs are the
estimate of the future real values of the load parameters. Because of the loop
structure, we find that these deep models outperform the shallow models in
most case.

Moreover, we consider the forecast task in two kinds of modes: single-output,
which allows the models generate the forecast result of one parameter each a
time, and multiple-output, which requires that the models should
simultaneously provide the forecast results of the three parameters. Although
the experimental results show that the single-output models perform better
than multiple-output models, three individual single-output models should be
develop for the three TBM load parameters. In contrast, one multiple-output
model can simultaneously provide the forecast of the three parameters, and
thus it has a higher operating efficiency.

In addition, we also use the least absolute shrinkage and selection (Lasso)
method to extract essential features of the three parameters according to the
magnitude of their Lasso coefficients. Based on the extracted features, we
then develop the real-time forecast models for the TBM load parameters. The
experimental results show that this manner brings a significant performance
gain in most cases. In our future works, we will consider the application of
the proposed ML-based models in TBM health monitoring and fault prognostics.

##  Appendix A Important Features in TBM Operational Data

In the appendix, we list some important features appearing in the in-situ TBM
operational data.

Table 3: Some Features in TBM Operation Data [29]

Parameter (Unit) Parameter (Unit) Temperature of oil tank (∘C) Temperature of
gear oil (∘C) Rotation speed of cutter(r/min) Cutter power (kw) Propelling
pressure (bar) Propelling pressure of A group (bar) Propelling pressure of B
group (bar) Propelling pressure of C group (bar) Propelling pressure of D
group (bar) Pressure of equipment bridge (bar) Pressure of articulation system
(bar) Pressure of Shield tail seal at top right front (bar) Pressure of Shield
tail seal at right front (bar) Pressure of Shield tail seal at bottom left
front (bar) Pressure of Shield tail seal at top right back (bar) Pressure of
Shield tail seal at right back (bar) Pressure of Shield tail seal at bottom
left back (bar) Pressure of Shield tail seal at left front (bar) Pressure of
Shield tail seal at top left front (bar) Pressure of Shield tail seal at left
back (bar) Pressure of Shield tail seal at top left back (bar) Pressure of
Shield tail seal at bottom right back (bar) Rolling angle (∘) Pressure of
screw pump at back (bar) Pressure of chamber at top left (bar) Pressure of
chamber at bottom left (bar) Pressure of chamber at bottom right (bar)
Bentonite pressure (bar) Temperature of screw conveyor (∘C) Pitch angle (∘)
Thrust of cutterhead (kN) Advance rate (mm/min) Torque of cutterhead (kNm)
Displacement of A group of thrust cylinders (mm) Displacement of B group of
thrust cylinders (mm) Displacement of C group of thrust cylinders (mm)
Displacement of D group of thrust cylinders (mm) Displacement of articulated
system at top right (mm) Displacement of articulated system at bottom left
(mm) Displacement of articulated system at top left (mm) Displacement of
articulated system at bottom right (mm) Bentonite pressure of shield shell
(bar) Pressure of screw conveyor at front (bar) Pressure of screw pump (bar)

## Acknowledgment

This work is supported by the National Key R&D Program of China (Grant No.
2018YFB1700704 & 2018YFB1702502) and the National Natural Science Foundation
of China (Grant No. 11401076 & 61473328).

## References

  * [1] Miura, K., 2003. Design and construction of mountain tunnels in japan. Tunn. Undergr. Space Technol. 18 (2-3), 115–126.  doi:http://dx.doi.org/10.1016/S0886-7798(03)00038-5.
  * [2] Edalat, K., Vahdatirad, M. J., Ghodrat, H., Firouzian, S., Barari, A., 2010. Choosing tbm for tabriz subway using multi criteria method. J. Civ. Eng. Manag. 16 (4), 531–539.  doi:https://doi.org/10.3846/jcem.2010.59.
  * [3] M. Cigla, S. Yagiz, L. Ozdemir., 2001. Application of tunnel boring machines in underground mine development. in: E. Ünal, B. Ünver, E. Tercan (Eds.), Proceeding of the 17th International Mining Congress and Exhibition of Turkey, Ankara, Turkey, pp. 155–164, ISBN:97S-395-417-4.
URL http://www.maden.org.tr/resimler/ekler/43857f4a06c8521_ek.pdf

  * [4] Hong, E. S., Lee, I. M., Shin, H. S., Nam, S. W., Kong, J. S., 2009. Quantitative risk evaluation based on event tree analysis technique: application to the design of shield tbm. Tunn. Undergr. Space Technol. 24 (3), 269–277.  doi:http://dx.doi.org/10.1016/j.tust.2008.09.004.
  * [5] Cho, S., Ahn, M. Y., Lee, D. W., Park, Y. H., Lee, E. H., Jin, H. G., Lee, C. W., Kim, T. K., Chun, Y. B., Kim, S. K., et al., 2014. Design and r&d progress of korean hccr tbm. Fusion Eng. Des. 89 (7-8), 1137–1143.  doi:http://dx.doi.org/10.1016/j.fusengdes.2014.01.032.
  * [6] Wu, X., Liu, H., Zhang, L., Skibniewski, M. J., Deng, Q., Teng, J., 2015. A dynamic bayesian network based approach to safety decision support in tunnel construction. Reliab. Eng. Syst. Saf. 134, 157–168.  doi:https://doi.org/10.1016/j.ress.2014.10.021.
  * [7] Kothamasu, R., Huang, S. H., VerDuin, W. H., 2006. System health monitoring and prognostics – a review of current paradigms and practices. Int. J. Adv. Manuf. Technol. 28 (9-10), 1012–1024.  doi:http://dx.doi.org/10.1007/s00t70-004-2131-6.
  * [8] Wang, L., Kang, Y., Zhao, X., Zhang, Q., 2015. Disc cutter wear prediction for a hard rock tbm cutterhead based on energy analysis. Tunn. Undergr. Space Technol. 50, 324–333.  doi:https://dx.doi.org/10.1016/j.tust.2015.08.003.
  * [9] Talebi, K., Memarian, H., Rostami, J., Gharahbagh, E. A., 2015. Modeling of soil movement in the screw conveyor of the earth pressure balance machines (epbm) using computational fluid dynamics. Tunn. Undergr. Space Technol. 47, 136–142.  doi:https://doi.org/10.1016/j.tust.2014.12.008.
  * [10] Javad, G., Narges, T., 2010. Application of artificial neural networks to the prediction of tunnel boring machine penetration rate. Min. Sci. Tech. 20 (5), 727–733.  doi:http://dx.doi.org/10.1016/S1674-5264(09)60271-4.
  * [11] Delisio, A., Zhao, J., Einstein, H., 2013. Analysis and prediction of tbm performance in blocky rock conditions at the lötschberg base tunnel. Tunn. Undergr. Space Technol. 33, 131–142.  doi:https://doi.org/10.1016/j.tust.2012.06.015.
  * [12] Gong, Q., Zhao, J., 2009. Development of a rock mass characteristics model for tbm penetration rate prediction. Int. J. Rock Mech. Min. Sci. 46 (1), 8–18.  doi:https://doi.org/10.1016/j.ijrmms.2008.03.003.
  * [13] Ates, U., Bilgin, N., Copur, H., 2014. Estimating torque, thrust and other design parameters of different type tbms with some criticism to tbms used in turkish tunneling projects. Tunn. Undergr. Space Technol. 40, 46–63.  doi:https://doi.org/10.1016/j.tust.2013.09.004.
  * [14] Wang, X., Yuan, Y., Mu, X., Sun, W., Song, X., 2019. Sensitivity of tbm¡¯s performance to structural, control and geological parameters under different prediction models. IEEE Access 7, 19738–19751.
  * [15] Shi, M., Sun, W., Zhang, T., Liu, Y., Wang, S., Song, X., 2019. Geology prediction based on operation data of tbm: comparison between deep neural network and soft computing methods. in: The 1st International Conference on Industrial Artificial Intelligence (IAI), pp. 1–5.  doi:10.1109/ICIAI.2019.8850794.
  * [16] Zhao, J., Shi, M., Hu, G., Song, X., Zhang, C., Tao, D., Wu, W., 2019. A data-driven framework for tunnel geological-type prediction based on tbm operating data. IEEE Access 7, 66703–66713.
  * [17] Krause, H., 1976. Geologische erfahrungen beim einsatz von tunnelvortriebsmaschinen in baden-wiirttemberg. Neue Erkenntnisse im Hohlraumbau$-$Fundierungen im Fels/Latest Findings in the Construction of Underground Excavations$-$Rock Foundations, Springer, Vienna, 49–60.  doi:https://doi.org/10.1007/978-3-7091-8452-3_3.
  * [18] Yagiz, S., 2017. New equations for predicting the field penetration index of tunnel boring machines in fractured rock mass. Arab. J. Geosci. 10 (2), 33.  doi:https://doi.org/10.1007/s12517-016-2811-1.
  * [19] Hamidi, J. K., Shahriar, K., Rezai, B., Rostami, J., 2010. Performance prediction of hard rock tbm using rock mass rating (rmr) system. Tunn. Undergr. Space Technol. 25 (4), 333–345.  doi:https://doi.org/10.1016/j.tust.2010.01.008.
  * [20] Hassanpour, J., Rostami, J., Khamehchiyan, M., Bruland, A., Tavakoli, H., 2010. Tbm performance analysis in pyroclastic rocks: a case history of karaj water conveyance tunnel. Rock Mech. Rock Eng. 43 (4), 427–445.  doi:https://doi.org/10.1007/s00603-009-0060-2.
  * [21] Gertsch, R., Gertsch, L., Rostami, J., 2007. Disc cutting tests in colorado red granite: Implications for tbm performance prediction. Int. J. Rock Mech. Min. Sci. 44 (2), 238–246.  doi:https://doi.org/10.1016/j.ijrmms.2006.07.007.
  * [22] Entacher, M., Lorenz, S., Galler, R., 2014. Tunnel boring machine performance prediction with scaled rock cutting tests. Int. J. Rock Mech. Min. Sci. 70, 450–459.  doi:https://doi.org/10.1016/j.ijrmms.2014.04.021.
  * [23] Xue, J., Xia, Y., Ji, Z., Zhou, X., 2009. Soft rock cutting mechanics model of tbm cutter and experimental research. International Conference on Intelligent Robotics and Applications. Springer, Berlin, Heidelberg, 383–391.  doi:https://doi.org/10.1007/978-3-642-10817-4_38.
  * [24] Shi, H., Yang, H., Gong, G., Wang, L., 2011. Determination of the cutterhead torque for epb shield tunneling machine. Autom. Constr. 20 (8), 1087–1095.  doi:https://doi.org/10.1016/j.autcon.2011.04.010.
  * [25] Wang, L., Gong, G., Shi, H., Yang, H., 2012. Modeling and analysis of thrust force for epb shield tunneling machine. Autom. Constr. 27, 138–146.  doi:https://doi.org/10.1016/j.autcon.2012.02.004.
  * [26] Zhang, Q., Su, C., Qin, Q., Cai, Z., Hou, Z., Kang, Y., 2016. Modeling and prediction for the thrust on epb tbms under different geological conditions by considering mechanical decoupling. Sci. China-Technol. Sci. 59 (9), 1428–1434.  doi:https://doi.org/10.1007/s11431-016-6096-0.
  * [27] Kasper, T., Meschke, G., 2006. On the influence of face pressure, grouting pressure and tbm design in soft ground tunnelling. Tunn. Undergr. Space Technol. 21 (2), 160–171.  doi:https://doi.org/10.1016/j.tust.2005.06.006.
  * [28] Su, C., Wang, Y., Zhao, H., Su, P., Qu, C., Kang, Y., Huang, T., Cai, Z., Wang, L., 2011. Analysis of mechanical properties of two typical kinds of cutterheads of shield machine. Adv. Sci. Lett. 4 (6-7), 2049–2053.  doi:https://doi.org/10.1166/asl.2011.1545.
  * [29] Gao, X., Shi, M., Song, X., Zhang, C., Zhang, H., 2019. Recurrent neural networks for real-time prediction of tbm operating parameters. Autom. Constr. 98, 225–235.  doi:https://doi.org/10.1016/j.autcon.2018.11.013.
  * [30] Bishop, C. M., 2006. Pattern recognition and machine learning. Springer.  doi:ISBM:978-0-387-31073-2.
  * [31] Cybenko, G., 1989. Approximation by superpositions of a sigmoidal function. Math. Control Signal Syst. 2 (4), 303–314.  doi:https://doi.org/10.1007/BF02551274.
  * [32] Jain, A. K., Mao, J., Mohiuddin, K., 1996. Artificial neural networks: A tutorial. Computer 29 (3), 31–44.  doi:http://dx.doi.org/10.1109/2.485891.
  * [33] Williams, D., Hinton, G., 1986. Learning representations by back-propagating errors. Nature 323 (6088), 533–538.  doi:https://doi.org/10.1038/323533a0.
  * [34] Pascanu, R., Mikolov, T., Bengio, Y., 2013. On the difficulty of training recurrent neural networks. in: S. Dasgupta, D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA , pp. 1310–1318.
URL https://arxiv.org/abs/1211.5063

  * [35] Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Comput. 9 (8), 1735–1780.  doi:https://doi.org/10.1162/neco.1997.9.8.1735.
  * [36] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
URL https://arxiv.org/abs/1406.1078

  * [37] Tikhonov, A., Arsenin, V., 1977. Solutions of ill-posed problems. Scripta series in mathematics. V. H. Winston & Sons, Washington, D.C.: John Wiley & Sons, New York.  doi:ISBN:9780470991244.
  * [38] Tibshirani, R., 1996. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B-Methodol. 58 (1), 267–288.  doi:http://dx.doi.org/10.1111/j.2517-6161.1996.tb02080.x.
  * [39] Lai, Z., Nagarajaiah, S., 2019. Sparse structural system identification method for nonlinear dynamic systems with hysteresis/inelastic behavior. Mech. Syst. Signal Proc. 117, 813–842.  doi:https://doi.org/10.1016/j.ymssp.2018.08.033.
  * [40] Bartkowiak, A., Zimroz, R., 2014. Dimensionality reduction via variables selection–linear and nonlinear approaches with application to vibration-based condition monitoring of planetary gearbox. Appl. Acoust. 77, 169–177.  doi:http://dx.doi.org/10.1016/j.apacoust.2013.06.017.
  * [41] Tieleman,T., Hinton, G., 2017. Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning, Technical report, Available online: https://zh. coursera. org/learn/neuralnetworks/lecture/YQHki/rmsprop-divide-the-gradient-by-a-running-average-of-its-recent-magnitude.

##

Figure 1: Structure of FNN with Two Hidden Layers Figure 2: Structure of
Recurrent Neural Networks Figure 3: Structure of LSTM’s Hidden Node Figure 4:
Structure of GRU’s Hidden Node

Figure 5: Structure of RNN

(a) SVR (s.w/o.l.)

(b) SVR (s.w.l.)

(c) SVR (m.w/o.l.)

(d) SVR (m.w.l.)

(e) RF (s.w/o.l.)

(f) RF (s.w.l.)

(g) RF (m.w/o.l.)

(h) RF (m.w.l.)

(i) FNN (s.w/o.l.)

(j) FNN (s.w.l.)

(k) FNN (m.w/o.l.)

(l) FNN (m.w.l.)

(m) RNN (s.w/o.l.)

(n) RNN (s.w.l.)

(o) RNN (m.w/o.l.)

(p) RNN (m.w.l.)

(q) GRU (s.w/o.l.)

(r) GRU (s.w.l.)

(s) GRU (m.w/o.l.)

(t) GRU (m.w.l.)

(u) LSTM (s.w/o.l.)

(v) LSTM (s.w.l.)

(w) LSTM (m.w/o.l.)

(x) LSTM (m.w.l.)

Figure 6: Forecast Results of “Torque”

(a) SVR (s.w/o.l.)

(b) SVR (s.w.l.)

(c) SVR (m.w/o.l.)

(d) SVR (m.w.l.)

(e) RF (s.w/o.l.)

(f) RF (s.w.l.)

(g) RF (m.w/o.l.)

(h) RF (m.w.l.)

(i) FNN (s.w/o.l.)

(j) FNN (s.w.l.)

(k) FNN (m.w/o.l.)

(l) FNN (m.w.l.)

(m) RNN (s.w/o.l.)

(n) RNN (s.w.l.)

(o) RNN (m.w/o.l.)

(p) RNN (m.w.l.)

(q) GRU (s.w/o.l.)

(r) GRU (s.w.l.)

(s) GRU (m.w/o.l.)

(t) GRU (m.w.l.)

(u) LSTM (s.w/o.l.)

(v) LSTM (s.w.l.)

(w) LSTM (m.w/o.l.)

(x) LSTM (m.w.l.)

Figure 7: Forecast Results of “Advance Rate”

(a) SVR (s.w/o.l.)

(b) SVR (s.w.l.)

(c) SVR (m.w/o.l.)

(d) SVR (m.w.l.)

(e) RF (s.w/o.l.)

(f) RF (s.w.l.)

(g) RF (m.w/o.l.)

(h) RF (m.w.l.)

(i) FNN (s.w/o.l.)

(j) FNN (s.w.l.)

(k) FNN (m.w/o.l.)

(l) FNN (m.w.l.)

(m) RNN (s.w/o.l.)

(n) RNN (s.w.l.)

(o) RNN (m.w/o.l.)

(p) RNN (m.w.l.)

(q) GRU (s.w/o.l.)

(r) GRU (s.w.l.)

(s) GRU (m.w/o.l.)

(t) GRU (m.w.l.)

(u) LSTM (s.w/o.l.)

(v) LSTM (s.w.l.)

(w) LSTM (m.w/o.l.)

(x) LSTM (m.w.l.)

Figure 8: Forecast Results of “Thrust”


# Deep Deterministic Path Following

Georg Hess  Chalmers University of Technology
Gothenburg, Sweden
<EMAIL_ADDRESS>William Ljungbergh  Chalmers University of
Technology
Gothenburg, Sweden
<EMAIL_ADDRESS>
###### Abstract

This paper deploys the Deep Deterministic Policy Gradient (DDPG) algorithm for
longitudinal and lateral control of a simulated car to solve a path following
task. The DDPG agent was implemented using PyTorch and trained and evaluated
on a custom kinematic bicycle environment created in Python. The performance
was evaluated by measuring cross-track error and velocity error, relative to a
reference path. Results show how the agent can learn a policy allowing for
small cross-track error, as well as adapting the acceleration to minimize the
velocity error.

###### Index Terms:

Reinforcement Learning, Deep Deterministic Policy Gradient, Vehicle Control,
Autonomous Vehicles

##  I Introduction

Autonomous vehicles have evolved quickly in the last years, mainly thanks to
recent development in the field of deep learning. Creating a self-driving
vehicle can generally be divided into three subtasks, perception and
localization, planning, and control and actuation. This paper is concerned
with the latter, namely following a given path, ensuring correct control and
actuation of the vehicle. More specifically, a Reinforcement Learning (RL)
approach was used for controlling the steering angle $\delta$ and acceleration
$a$ of a simulated land vehicle, trying to keep the cross-track error (CTE)
and velocity error small. While traditional controlling algorithms exist for
this coupled control task [1] and they provide promising performance, their
main drawback is that precise modeling of the vehicle at hand is required.
Many parameters must be measured or estimated accurately for the control
algorithm to perform adequately. This motivates the use of deep learning where
suitable parameters instead are learned by the algorithm itself. Although
traditional supervised deep learning can be applied to solve this problem [2],
this will not be considered here. Instead, Deep Deterministic Policy Gradient
will be applied, which is an off-policy learning algorithm suitable to use in
a continuous domain where continuous control actions are desirable [3]. The
algorithm and its application to the control problem will be explained closer
in the coming section.

##  II Related work

Reinforcement learning has been successfully applied to various scenarios,
such as playing Atari video games [4] and basic control tasks [3]. The basic
idea of reinforcement learning consists of having an agent interacting with an
environment by each time-step $t$ receiving an observation $x_{t}$, based on
this taking an action $a_{t}$ and receiving a reward $r_{t}$. The agent then
tries to select actions to maximize the cumulative reward. One of the more
commonly known algorithms for finding action policies is called Q-learning,
which is a model-free off-policy RL algorithm, introduced already in 1989 [5].
Model-free refers to that no model of the environment is needed to learn
suitable actions, while off-policy means that it can learn from previously
collected observations. The Q-learning algorithm is value-based, i.e.
essentially it uses observations to learn the optimal action for any giving
state by keeping track of all possible state-action pairs and continuously
evaluating and updating their quality $Q^{*}(s_{t},a_{t})$ (hence the name
Q-learning). After learning $Q^{*}(s_{t},a_{t})$, actions are selected such
that $Q^{*}(s_{t},a_{t})$ is maximized for each state. However, as the state-
space grows larger, the discrete approach used in Q-learning becomes
infeasible due to the curse of dimensionality. Furthermore, for a continuous
state-space, each state has to be discretized, where there is a trade-off
between small discretization steps and keeping low dimensionality.

The problem of continuous observation spaces can be solved using Deep
Q-learning (DQN)[6]. Instead of using a table to store the Q-values, a
function approximation based on a neural network is used to estimate the
values. In the algorithm a continuous state-vector is fed into a neural
network where the output is an estimation of the current Q-value. However,
this still leaves one with having to maximize the Q-value over all possible
actions, i.e. for each state, one has to search through the actions, finding
the one with the highest Q-value. For a large action space or large function
approximation network, this becomes computationally costly and time-consuming.

For handling continuous action spaces, one can use Deep Deterministic Policy
Gradient (DDPG) [3] which is an extension of the actor-critic approach in
Deterministic Policy Gradient [7] in combination with findings done by Deep
Q-learning [6]. The algorithm is based on two different neural networks,
called the actor $\mu(s|\theta^{\mu})$ and critic $Q(s,a|\theta^{Q})$
networks, where $\theta^{\mu}$ and $\theta^{Q}$ denote their weights
respectively. The actor network maps vectors $s$ in the state-space into a
vector $a$ in the action space, i.e. it uses the knowledge it has about its
current state and tries to determine the best possible action to take. The
critic network uses both the current state $s$ and the action $a$ taken by the
actor-network to decide how good that particular action is to take in this
particular state, i.e. trying to approximate the Q-value. The parameters of
the critic are updated by using gradient stochastic descent for minimizing the
Mean-Squared Bellman Error, similar as in traditional Q-learning. For the
actor however, [7] showed that there exist a policy gradient, i.e. gradient of
the policy’s performance, along which one updates the parameters of the
network

$\nabla_{\theta^{\mu}}J\approx\mathrm{E}_{s\sim\rho^{\beta}}[\nabla_{a}Q(s,a|\theta^{Q})|_{a=\mu(s|\theta^{\mu})}\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})]$
(1)

Deep Q-learning introduced two major ideas to scale Q-learning which are also
used in DDPG, namely the use of replay buffers and separate target networks.
Replay buffers save a large number of experiences $(s,a,r,s^{\prime},d)$,
where $s^{\prime}$ denotes the next state and $d$ whether that state is
terminal, in a dataset $\mathcal{D}$. One later randomly samples mini-batches
from this dataset, rather than taking the observations directly from the
simulation when training the actor and critic networks. By employing a replay
buffer, one can suppress the issue that the networks overfit to the most
recent data and achieve a more stable training process. Furthermore, it also
handles the issues that the observations taken from the simulation are
sequentially dependent on each other.

When training the critic network, the Mean-Squared Bellman Error (MSBE) is to
be minimized with respect to the parameters $\theta$, i.e. we are trying to
minimize the loss function

$L=\Big{(}Q_{\theta}(s,a)-\underbrace{\left(r+\gamma(1-d)\max_{a^{\prime}}Q_{\theta}\left(s^{\prime},a^{\prime}\right)\right)}_{T}\Big{)}^{2}$
(2)

by making $Q_{\theta}$ more similar to the target $T$. Here $\gamma$ denotes
the discount factor $\gamma\in[0,1]$ determining how much future rewards are
valued and $a^{\prime}$ is the action taken in the next state $s^{\prime}$. As
can be seen in (2), the target is dependent on the parameters we are trying to
learn, namely $\theta$. This can make the training very unstable, leading to
the agent not learning anything. Here is where the authors of DQN proposed
target networks, which are yet another set of parameters, $\theta_{targ}^{Q}$
and $\theta^{\mu}_{targ}$, which lag behind the actual parameters $\theta^{Q}$
and $\theta^{\mu}$. In the original DQN paper[6], the target networks were
updated after a fixed number of episodes, however in the DDPG algorithm[3] the
authors used Polyak averaging after each training step. The update is
therefore described as

$\theta_{targ}\longleftarrow(1-\rho)\theta_{targ}+\rho\theta,\ \rho\ll 1$ (3)

By using te target networks instead of the actual networks to calculate the
target in (2) one can achieve much more stable training.

Finally, in RL one has to balance the trade-off between exploration and
exploitation. By adding a certain level of randomness in the agent one allows
it to explore and learn more about the state-action space before optimizing
its policy. In the original DDPG paper [3], the authors proposed an
exploration policy where Gaussian noise was added to the actor-networks
output, i.e. directly to the selected action before executing it in the
environment. However, as described in [8], using such independent exploration
noise can be very inefficient as a vehicle system acts as a low pass filter
for the high-frequency changes in acceleration and steering angle that this
type of noise provides. Instead, they proposed an exploration policy where the
noise added, $\epsilon_{\delta}$ and $\epsilon_{T}$, is sinusoidal with
randomly sampled mean and deviation according to

$\displaystyle\epsilon_{\delta},\epsilon_{a}\sim N\left(\mu,\sigma^{2}\right)$
(4) $\displaystyle\mu_{\delta}=A\sin\left(\omega t+\varphi_{\delta}\right)$
(5) $\displaystyle\mu_{a}=A\sin\left(\omega t+\varphi_{a}\right)$ (6)

where $\sigma^{2}$, $A$ and $\omega$ are sampled from a zero-mean Gaussian
distribution, while $\varphi_{\delta},\varphi_{a}$ are uniformly sampled in
the range $(-\pi,\pi)$. Note here that the values $\sigma^{2}$, $A$,
$\varphi_{\delta},\varphi_{a}$ and $\omega$ are kept constant over one full
episode.

Furthermore, in [8], the authors showed that it is advantageous to have an
initial explore time (e.g. the first 500 episodes) where the agent, rather
than using the output of the actor-network, samples actions uniformly in the
range $(-1,1)$. They showed that this drastically improved the performance of
the agent and enabled it to learn much better behaviour as it allows for vast
exploration in the initial training period.

##  III Method

Before training the RL agent, an environment must be created with which it can
interact. The environment in this project was produced by the authors and
consisted of a kinematic bicycle model which took requested steering angles
and accelerations as input, and calculated a global position $x$, $y$,
velocity $v$, steering angle $\delta$, and heading $\theta$ of the agent. The
choice of the kinematic bicycle model is based on the model’s simplicity while
capturing the overall movements of a car. However, the model was adjusted such
that the requested steering angle $\delta_{\text{req}}$ was limited by a
maximum steering rate $\Delta\delta_{\text{max}}=40$, i.e. the change in
steering angle could not exceed a certain threshold. Further, the requested
accelerations $a_{\text{req}}$ were also clipped to lie within adequate limits
$[-a_{\text{max}},a_{\text{max}}]$ with $a_{\text{max}}=5m/s^{2}$. These
changes were introduced to simulate how actuators of a car might behave in
real-world conditions. The model can be summarized as selecting a time step
$\Delta t$ and vehicle length $L$, then iterating

$\displaystyle\Delta\delta$
$\displaystyle=\text{clip}\left(\frac{\delta_{\text{req}}-\delta_{t}}{\Delta
t},-\Delta\delta_{\text{max}},\Delta\delta_{\text{max}}\right)$ (7)
$\displaystyle a_{t}$
$\displaystyle=\text{clip}\left(a_{\text{req}},-a_{\text{max}},a_{\text{max}}\right)$
(8) $\displaystyle x_{t+1}$ $\displaystyle=v_{t}\cos(\theta_{t})\Delta t$ (9)
$\displaystyle y_{t+1}$ $\displaystyle=v_{t}\sin(\theta_{t})\Delta t$ (10)
$\displaystyle v_{t+1}$ $\displaystyle=a_{t}\Delta t$ (11)
$\displaystyle\delta_{t+1}$ $\displaystyle=\Delta\delta\Delta t$ (12)
$\displaystyle\theta_{t+1}$
$\displaystyle=\frac{v_{t}\tan(\delta_{t})}{L}\Delta t$ (13)

The model was also used to generate reference paths for the agent to follow.
For this, an average velocity $v_{avg}\sim\mathcal{U}(3,20)$ was sampled once
before each path generation started and then $\delta_{\text{req}}$ and
$a_{\text{req}}$ where sampled randomly every time step until the path had a
length of $400$m. $\delta_{\text{req}}$ was sampled from an uniform
distribution $\mathcal{U}(-30,30)$ and $a_{\text{req}}$ was sampled from two
different uniform distributions, $\mathcal{U}(-2,2)$ if $v_{t}>v_{avg}$,
otherwise from $\mathcal{U}(0,2)$. The reference trajectory was then processed
to consist of waypoints with reference speed every meter. Due to the
randomness of the generation, the agent received a unique path to follow each
episode. The agent’s state observations $s$ consisted of the distance to the
25 nearest future waypoints, expressed in the agent’s coordinate frame, along
with the difference between current velocity $v_{t}$ and reference velocity
$v_{ref}$ for all 25 waypoints. Additionally, the observed state also included
the agent’s current velocity and steering angle $v_{t}$, $\delta_{t}$. An
illustration is shown in Figure 1, where $(x_{i},y_{i})$ denote global
coordinates and $(x^{\prime}_{i},y^{\prime}_{i})$ coordinates in the agents
frame.

Figure 1: Illustration of state observations for agent.

As the agent intends to control the steering angle $\delta$ and the
acceleration $a$, the action space is continuous and the actions have been
chosen to be in the range of $\delta,a\in[-1,1]$. Each command must therefore
be multiplied with the maximally allowed acceleration and steering angle, as
defined by the model described above before it is passed to the simulation.

The reward signal was designed with inspiration from [8] and is described by

$R=\begin{cases}-1&\text{if $\text{cte}>0.2$}\\\
1.5-P_{cte}-P_{\delta}-P_{v}-P_{acc}&\text{otherwise}\end{cases}$ (14)

where, cte is the cross-track error between the agent and the reference path,
i.e. perpendicular distance from reference path to agent location, and
$P_{cte}$, $P_{\delta}$, $P_{v}$ and $P_{acc}$ are penalties based on the
cross-track error, steering angle, deviation from reference velocity, and the
acceleration performed by the agent. The penalties were introduced to minimize
cross-track errors, steering angles, velocity errors and jerky maneuvers
respectively, and are defined by the following equations

$\displaystyle P_{cte}$ $\displaystyle=0.8\text{cte},\
P_{\delta}=0.1\delta_{t}/\delta_{\text{max}},$ $\displaystyle P_{v}$
$\displaystyle=0.8|v_{t}-v_{ref}|/v_{ref},\ P_{acc}=0.2|a_{\text{req}}|$

Additional negative reward was assigned when the normalized velocity error
became to large, i.e.

$R:=R-1\ \text{if $|v_{t}-v_{ref}|/v_{ref}>0.25$}$ (15)

Each simulation was terminated when either the cross-track error $\geq 2$
meters, the velocity $v_{t}\leq 0$ m/s or the agent reached the end of the
path. However, only the to first two cases were marked as terminal states in
the target $T$.

As mentioned previously the DDPG architecture consists of two different
networks, the actor and the critic. The network architecture for each of the
networks are shown in Figure 2. The actor-network begins with two shared fully
connected layer and then branches out into two branches, each producing their
own action, i.e. either the steering angle $\delta$ or the acceleration $a$.
The two outputs are then concatenated before given as the final output of the
actor-network. The weights and biases of the last fully connected layers of
network were initialized to $\mathcal{U}(-10^{-6},10^{-6})$ as this was found
to be crucial for the agent to learn anything at all.

Figure 2: Critic (top) and actor (bottom) network architectures.

Furthermore, the replay-buffer size was set to $100,000$ and the sinusoidal
exploration policy explained in Section II was used. To transition smoothly
from exploration to exploitation, a noise amplitude multiplier was initialize
to 1 and reduced by $0.9996$ every episode. The agent was trained in total for
$10$ epochs, each lasting for $500$ episodes. The division into epochs was
done mainly for learning rate scheduling.

##  IV Results

In Figure 3 several different metrics from the training are shown for the
first 2500 episodes. Starting from the top we see the average cross-track
error and velocity error normalized with the travelled path length, as well as
the average reward per time-step. After $\sim 1000$ episodes a major bump in
performance was obtained, which is caused by switching from the aggressive
exploration policy used in the initial training phase to the more lenient
sinusoidal noise explained in Section II. Simply from sampling random actions
in the action space, the agent was able to learn enough about the environment
to perform adequately. When trained without the initial training phase, the
agent was not able to perform any satisfactory behaviour after the same number
of training steps, thus validating the results in [8]. After the initial bump
in performance, the behavior of the agent only changed marginally. While it
improved somewhat in all aspects shown in Figure 3, the biggest change was in
consistency. Towards the end of the training the agent was able to maintain a
lower variance in all of the aspects discussed above, including the average
percent of completion w.r.t. the reference path.

Figure 3: Average cross-track error, reward and velocity error, all normalized
with the travelled path length, shown as a function of first 2500 episodes.

After the training, the agent was evaluated over a set of 10 test tracks
randomly generated using the environment described earlier, where the
performance over these is shown in Table I.

TABLE I: Performance over a set of 10 test-tracks

Track | Avg cte | Max cte | Avg $\Delta v$ | Max $\Delta v$ | % of path
---|---|---|---|---|---
1 | 0.103 | 0.264 | 0.473 | 1.466 | 100.0
2 | 0.141 | 0.32 | 0.465 | 1.813 | 100.0
3 | 0.101 | 0.299 | 0.608 | 2.523 | 100.0
4 | 0.119 | 0.335 | 0.658 | 2.519 | 100.0
5 | 0.113 | 0.641 | 0.62 | 2.34 | 100.0
6 | 0.114 | 0.464 | 0.752 | 2.623 | 100.0
7 | 0.141 | 0.54 | 0.65 | 2.28 | 100.0
8 | 0.107 | 0.31 | 0.609 | 1.937 | 100.0
9 | 0.111 | 0.385 | 0.496 | 1.84 | 100.0
10 | 0.105 | 0.336 | 0.626 | 2.076 | 100.0
Avg | 0.115 | 0.39 | 0.596 | 2.142 | 100.0

From the evaluation one can conclude that the agent is performing passably.
The average cross-track error is kept low ($\sim 0.12$ m) while also ensuring
that the velocity is kept close to its reference (on average $\sim 0.60$ m/s
deviation). For the worst case scenario, the table shows the maximal cross-
track error during the 10 evaluations and it never exceeded $0.65$ m. This is
important to guarantee that the path following algorithm never strays to far
from its reference.

Furthermore, in Figure 4 the evaluation on test track # 6 (as it is deemed to
be representative over the 10 evaluations) is shown in more detail. Here the
reference and the actual path travelled is shown together with cross-track
error and the velocity error throughout the simulation. As can be seen, the
cross-track error rarely exceeds 0.3 m

Figure 4: Test Track #6 with the cross-track error [m] and velocity [m/s]
deviation shown over time.

##  V Conclusions & Future Work

In this paper, we leveraged DDPG to solve the lateral and longitudinal control
tasks. While conventionally solved with algorithms such as Pure Pursuit or
Model Predictive Control our RL algorithm showed very promising results. The
agent used relative distance and velocity to the 25 closest way-points on our
reference path to determine the appropriate action. During testing, the agent
was able to steer with an average cross-track error of only 0.12m while
keeping the agents velocity within 0.6m/s deviation from its reference on
average.

However, it should be noted that the agent was never applied in a real-world
setting, but merely a rather simplistic simulator. In future work, it would be
interesting not only to apply and test this on an actual vehicle but also to
compare it to the performance of the more conventional controllers.

Furthermore, one could make the simulation more realistic by including e.g.
noise in the state observations fed to the agent, or simply use a more
sophisticated simulator.

Finally, the authors recognize that the black-box nature of our RL approach
raises the issue of how we can guarantee safe behavior at all times. While
conventional controllers performance can be analyzed and determined under
which circumstances it will perform as intended, our solution is inherently
hard to analyze begging the question; can we ever guarantee satisfactory
behavior at all times, and if not, under which circumstances can we assure
that the RL approach will act appropriately?

## References

  * [1] A. Chebly, R. Talj, and A. Charara, “Coupled longitudinal and lateral control for an autonomous vehicle dynamics modeled using a robotics formalism,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 12 526–12 532, 2017.
  * [2] G. Devineau, P. Polack, F. Altché, and F. Moutarde, “Coupled longitudinal and lateral control of a vehicle using deep learning,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 642–649.
  * [3] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [4] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [5] C. J. C. H. Watkins, “Learning from delayed rewards,” Ph.D. dissertation, King’s College, Cambridge, may 1989. [Online]. Available: http://www.academia.edu/download/50360235/Learning_from_delayed_rewards_20161116-28282-v2pwvq.pdf
  * [6] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015. [Online]. Available: http://dx.doi.org/10.1038/nature14236
  * [7] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” ser. Proceedings of Machine Learning Research, E. P. Xing and T. Jebara, Eds., vol. 32, no. 1. Bejing, China: PMLR, 22–24 Jun 2014, pp. 387–395. [Online]. Available: http://proceedings.mlr.press/v32/silver14.html
  * [8] D. Kamran, J. Zhu, and M. Lauer, “Learning path tracking for real car-like mobile robots from simulation,” in _2019 European Conference on Mobile Robots (ECMR)_. IEEE, 2019, pp. 1–6.