# Parisian ruin with power-asymmetric variance near the optimal point with
application to many-inputs proportional reinsurance

Pavel Ievlev  Pavel Ievlev, Department of Actuarial Science, University of
Lausanne
UNIL-Dorigny, 1015 Lausanne, Switzerland<EMAIL_ADDRESS>
(Date: August 28, 2024)

Abstract: This paper investigates the Parisian ruin probability for processes
with power-asymmetric behavior of the variance near the unique optimal point.
We derive the exact asymptotics as the ruin boundary tends to infinity and
extend the previous result [1] to the case when the length of Parisian
interval is of Pickands scale. As a primary application, we extend the recent
result [2] on the many inputs proportional reinsurance fractional Brownian
motion risk model to the Parisian ruin.

Key Words: Parisian ruin, ruin probability, fractional Brownian motion

AMS Classification: Primary 60G15; secondary 60G70

##  1\. Introduction

Consider the following reinsurance scheme: $d$ companies share premiums and
one claim process proportionally. Suppose that the risk process $\bm{R}(t)$ is
composed of a large number of i.i.d. sub-risk processes $\bm{R}^{(i)}(t)$
representing independent businesses, and let each $\bm{R}^{(i)}$ be driven by
a fractional Brownian motion $B_{H}(t)$. That is, let

$\bm{R}_{N}(t)=\sum_{i=1}^{N}\bm{R}^{(i)}(t),\quad\text{where}\quad\bm{R}^{(i)}(t)=\bm{\alpha}+\bm{\mu}t-\bm{\sigma}B_{H}^{(i)}(t),$

where $B_{H}^{(i)},i\geq 1$ are independent fractional Brownian motions and
$\bm{\alpha},\bm{\mu},\bm{\sigma}\in\mathbb{R}^{d}$. In a recent contribution
[2], the authors derived the exact asymptotics of the simultaneous ruin
probability

$\mathbb{P}\left\\{\exists t\in[0,T]\colon\bm{R}_{N}(t)<0\right\\},\quad
N\to\infty$

in the case of $d=2$. In the present work we shall concentrate on the
simultaneous Parisian ruin probability

(1)
$\Pi(N)=\mathbb{P}\left\\{\mathcal{P}_{[0,S],[0,T_{N}]}(\bm{R}_{N})<0\right\\},\quad\text{where}\quad\mathcal{P}_{E,F}(\bm{Z})=\max_{i=1,\dots,d}\sup_{t\in
E}\inf_{s\in F}Z_{i}(t+s).$

Parisian stopping times have been first introduced in relation to barrier
options in mathematical finance, see [3], and since then attracted substantial
interest. For the applications to actuarial risk theory, we refer to [4],
where risk process is treated as a surplus process of an insurance company
with initial capital $u$.

In opposition to the well-studied classical ruin, when the failure is
recognized at the moment of surplus hitting zero, the Parisian ruin is
recognized only if the surplus process has spent a sufficient, pre-specified
amount of time below zero. We refer to [5, 6, 7, 8, 9] and the references
therein for analysis of Parisian ruin in the one-dimensional Lévy surplus
model.

In the univariate Gaussian setup, Parisian ruin has been investigated in [10]
for self-similar Gaussian processes and in [1] for general Gaussian processes,
satisfying some standard assumptions (see [11]). Another interesting
univariate case is of Parisian ruin over discrete sets. In [12], the authors
have proved that for the Brownian motion and equidistant grid the asymptotics
differs from the continuous one by some constant factor.

There are many possible extensions of the notion of the Parisian ruin to
multivariate risk processes, such as simultaneous Parisian ruin, when all the
components of a multivariate process should plunge below zero at the same time
and remain there long enough for the ruin to be attested. This problem has
recently been studied in [13] for the case when the risk process consists of
two correlated Brownian motions. Another possible extension is the joint or
non-simultaneous Parisian ruin, when ruin is attested if all the components
experience Parisian ruin during some interval of time, but not necessarily at
the same time. This problem has been studied in [14], also for the bivariate
Brownian motion with $\rho\in(-1,1)$. A third possible extension would be the
notion of ‘‘at least one’’ ruin, suggested in the classical ruin context in
[2]: the ruin is declared if either of the processes has a Parisian ruin over
time.

In this paper, we derive the exact asymptotics as $u\to\infty$ of the one-
dimensional Parisian ruin probability

$\Pi(u)=\mathbb{P}\left\\{\mathcal{P}_{[-S,S],[0,T_{u}]}\left(Z\right)>u\right\\}$

for $T_{u}\to 0$ at some specified rate and a class of Gaussian processes with
correlation structure

$\operatorname{corr}(Z(t),Z(s))=1-D|t-s|^{\alpha}+o\left(|t-s|^{\alpha}\right)$

and a unique optimal point $t_{*}=0$ of the variance with asymmetric behaviour
near this point:

$\sigma(t)=1-A_{\pm}|t|^{\gamma_{\pm}}\Big{(}1+o(1)\Big{)}\quad\text{as}\
t\to\pm 0,$

from which we further derive the exact asymptotics of the many-inputs Parisian
ruin probability (1).

The asymptotic behaviour of $\Pi(u)$ for such class of processes is of
interest by itself. Similar problems have recently been studied in [10] and
[1]. Our findings account for the previously discarded type of the Talagrand
case with $T>0$ (see Section 2) and discover the new type of asymptotics
therein:

$\Pi(u)=e^{-\min\\{A_{-}T^{\gamma_{-}},A_{+}T^{\gamma_{+}}\\}}\Psi(u),$

which rather surprisingly happens only if $\gamma_{+}<\gamma_{-},\alpha$ and
does not happen if $\gamma_{-}<\gamma_{+},\alpha$.

The paper is organized as follows. In Section 2 we present our main findings.
Theorem 1 provides the exact asymptotics of $\Pi(u)$ for the general Gaussian
process with power-asymmetric behaviour of the variance near the optimal point
and under some assumption on the speed $T_{u}\to 0$ convergence. It covers the
previously unaccounted for case when the size of Parisian interval is
equivalent to the Pickands scale of the process. Corollary 1 contains the
exact asymptotics of the many-inputs Parisian ruin probability. The proof of
Theorem 1 is presented in a separate Section 3. All known results and
technical details are relegated to the Appendix.

##  2\. Main results

In this section, we first explain how to rewrite many-inputs ruin probability
in a form suitable for applying Theorem 1, then specify the assumptions under
which the general theorem works and conclude with deriving the exact
asymptotics of the many-inputs proportional reinsurance ruin probability 1.

Observe that by properties of Gaussian distribution

$\bm{R}_{N}\stackrel{{\scriptstyle
d}}{{=}}\bm{\alpha}N+\bm{\mu}Nt-\bm{\sigma}\sqrt{N}B_{H}(t),$

we can rewrite the ruin probability (1) as

$\Pi_{[0,S]}(N)=\mathbb{P}\left\\{\mathcal{P}_{[0,S],[0,T_{N}]}(Z)>\sqrt{N}\right\\},\quad\text{where}\quad
Z(t)=\frac{B_{H}(t)}{D(t)},\quad
D(t)=\max_{i=1,\dots,d}(\alpha_{i}+\mu_{i}t).$

Next, we state a result concerning Parisian ruin probabilities

$\Pi(u)=\mathbb{P}\left\\{\mathcal{P}_{E,[0,T_{u}]}(Z)>u\right\\}$

for some large class of Gaussian processes and then apply it to
$\Pi_{[0,S]}(N)$, rewritten in the latter form.

###  2.1. Assumptions

Let $E$ be a compact subset of $\mathbb{R}$, containing point $0$ in its
interior, and let $Z(t),t\in E$ be a centered Gaussian process with a.s.
continuous sample paths satisfying the following two assumptions:

Assumption A1:

The standard variance function $\sigma_{Z}$ of the Gaussian process $Z$
attains its maximum on $E$ at the unique point $\widehat{\tau}=0$. Further,
there exist positive constants $\gamma_{\pm}$ and $A_{\pm}$ such that

(2)
$\sigma_{Z}(t)=1-A_{\pm}|t|^{\gamma_{\pm}}+o(|t|^{\gamma_{\pm}})\quad\text{as}\quad
t\to\pm 0.$

Assumption A2:

There exists some positive constant $\alpha\in(0,2]$ such that

$\operatorname{corr}\left(Z(t),Z(s)\right)=1-D\left|t-s\right|^{\alpha}+o\left(|t-s|^{\alpha}\right)\quad\text{as
}t,s\to 0.$

######  Remark 1.

Note that it follows from A2 that there exists such $\delta>0$ that

$\mathbb{E}\left\\{\Big{(}\overline{Z}(t)-\overline{Z}(s)\Big{)}^{2}\right\\}<C|t-s|^{\alpha}$

for all $t,s<\delta$.

As it turns out, there are two numbers

$\nu=\min\\{\alpha,\gamma_{-},\gamma_{+}\\}\quad\text{and}\quad\gamma=\max\\{\gamma_{-},\gamma_{+}\\}$

which determine the type of the asymptotics, but before proceeding to that, we
also need the following assumption on the convergence rate of $T_{u}\to 0$:

Assumption B:

$T_{u}=Tu^{-2/\nu}$ for some $T\in[0,\infty)$.

Next, we introduce two well-known and important constants in the theory of
Gaussian extremes, see [15, 10, 1]. Define for $T\geq 0$ and $\alpha\in(0,2]$
the generalized Pickands and Piterbarg constants

$\mathcal{H}_{\alpha}^{\mathcal{P}}(T)=\lim_{\lambda\to\infty}\frac{\mathcal{H}_{\alpha,0}^{\mathcal{P}}(\lambda,T)}{\lambda}\quad\text{and}\quad\mathcal{H}_{\alpha,h}^{\mathcal{P}}(T)=\lim_{\lambda\to\infty}\mathcal{H}_{\alpha,h}^{\mathcal{P}}(T),$

where

$\mathcal{H}_{\alpha,h}^{\mathcal{P}}(T)=\mathbb{E}\exp\left(\sup_{t\in[-\lambda,\lambda]}\inf_{s\in[0,T]}\Big{(}\sqrt{2}B_{\alpha/2}(t+s)-|t+s|^{\alpha}-h(t+s)\Big{)}\right)$

for such continuous $h$ that the limit exists. We are in a position to
formulate our main theorem.

######  Theorem 1.

Let $(Z(t))_{t\geq 0}$ be a centered Gaussian process satisfying assumptions
A1 and A2, and let $T_{u}$ be a positive measurable function of $u$ satisfying
assumption (B). Then

In the Pickands case $\nu=\alpha\neq\gamma$ :

we have

$\Pi(u)=C_{S}\
\mathcal{H}_{\alpha}^{\mathcal{P}}(D^{1/\alpha}T)\,u^{2/\nu-2/\gamma}\,\Psi(u)\Big{(}1+o(1)\Big{)},$

with

$C_{S}=A_{+}^{-1/\gamma_{+}}D^{1/\alpha}\Gamma\left(\frac{1}{\gamma_{+}}+1\right)1_{\gamma=\gamma_{+}}+A_{-}^{-1/\gamma_{-}}D^{1/\alpha}\Gamma\left(\frac{1}{\gamma_{-}}+1\right)1_{\gamma=\gamma_{-}}.$

In the Piterbarg case $\nu=\alpha=\gamma$:

we have

$\Pi(u)=\mathcal{H}_{\alpha,h}^{\mathcal{P}}(D^{1/\alpha}T)\Psi(u)(1+o(1)),$

where $h(t)=A_{-}D^{-1/\alpha}|t|^{\gamma_{-}}1_{t\leq
0}+A_{+}D^{-1/\alpha}|t|^{\gamma_{+}}1_{t\geq 0}$.

In the Talagrand-1 case $\gamma=\nu\neq\alpha$:

$\Pi(u)=C\Psi(u)\Big{(}1+o(1)\Big{)},\quad
C=\begin{cases}1,&\gamma_{+}\geq\gamma_{-},\\\
\exp(-\min\\{A_{-}T^{\gamma_{-}},A_{+}T^{\gamma_{+}}\\}),&\gamma_{+}<\gamma_{-}.\end{cases}$

Now we proceed with our initial problem, to which end we first have to study
the behaviour of $\operatorname{var}B_{H}(t)/D(t)$. Note that the derivative
$\sigma^{\prime}$ of the variance function

$\sigma(t)=\frac{t^{H}}{D(t)},\quad
D(t)=\max_{i=1,\dots,d}(\alpha_{i}+\mu_{i}t)$

changes its sign exactly once, since

$\sigma^{\prime}(t)=\frac{t^{H-1}}{D^{2}(t)}G(t),\quad
G(t)=HD(t)-tD^{\prime}(t),$

where $G(t)$ is monotone and decreasing, $G(0)>0$ and $G(t)\to-\infty$ as
$t\to\infty$. Since $\sigma^{\prime}$ must change sign (possibly in a
discontinuous manner) at the optimal point $t_{*}$ of $\sigma$, we have thus
proved that such point is unique. Let us assume that $t_{*}\in(0,S)$ or
$S=\infty$, since to account for the boundary maxima case an approach slightly
different to ours is needed.

The maximum can either be caused by intersection of some two lines
$l_{\pm}(t)=\alpha_{\pm}+\mu_{\pm}t$ from $D(t)$, that is,

$t_{*}=\frac{\alpha_{+}-\alpha_{-}}{\mu_{-}-\mu_{+}}$

in which case $\sigma^{\prime}$ is discontinuous at $t_{*}$, or by a point
away from the lines’ intersections, satisfying $\sigma^{\prime}(t_{*})=0$,
that is,

$t_{*}=\frac{H\alpha}{\mu(1-H)}.$

Finally, these two types of maxima can coincide, giving rise to a power-
asymmetric behavior near $t_{*}$

$\frac{\sigma(t)}{\sigma(t_{*})}=1-A_{\pm}|t-t_{*}|^{\gamma_{\pm}}\Big{(}1+o(1)\Big{)}$

with $\gamma_{\pm}\in\\{1,2\\}$. Precisely, if

$-C_{\pm}^{(1)}=\frac{\sigma_{\pm}^{\prime}(t_{*})}{\sigma(t_{*})}=\frac{H}{t_{*}}-\frac{\mu_{\pm}}{\alpha_{\pm}+\mu_{\pm}t_{*}}<0,$

then $\gamma_{\pm}=1$ and $A_{\pm}=C_{\pm}^{(1)}$. If on the other hand
$C_{\pm}^{(1)}=0$, then under the following non-degeneracy assumption

$-C_{\pm}^{(2)}=\frac{\sigma_{\pm}^{\prime\prime}(t_{*})}{\sigma(t_{*})}=-\frac{H}{t_{*}^{2}}+\frac{\mu_{\pm}^{2}}{(\alpha_{\pm}+\mu_{\pm}t_{*})^{2}}<0$

we have $\gamma_{\pm}=2$ and $A_{\pm}=C_{\pm}^{(2)}$.

Now we may introduce the natural asymptotic parameter

$\widehat{N}=\frac{\sqrt{N}}{\sigma_{Z}(t_{*})}$

and formulate the corollary on the MIPR asymptotics.

######  Corollary 1.

Let $T_{N}$ satisfy the condition

$\lim_{N\to\infty}T_{N}\widehat{N}^{1/H}=T\in[0,\infty).$

  * •

If either $\gamma_{+}$ or $\gamma_{-}$ equals $2$, then

$\Pi(N)=\frac{\sqrt{\pi}}{2}\frac{1}{\sqrt{A}}\Big{(}1_{\gamma_{+}=2}+1_{\gamma_{-}=2}\Big{)}\frac{\mathcal{H}_{2H}^{\mathcal{P}}\left(T/2^{1/2H}t_{*}\right)}{2^{1/2H}t_{*}}\widehat{N}^{\zeta}\Psi(\widehat{N})\Big{(}1+o(1)\Big{)}.$

  * •

If both $\gamma_{\pm}=1>2H$, then

$\Pi(N)=\left(\frac{1}{A_{-}}+\frac{1}{A_{+}}\right)\frac{\mathcal{H}_{2H}^{\mathcal{P}}\left(T/2^{1/2H}t_{*}\right)}{2^{1/2H}t_{*}}\widehat{N}^{\zeta}\Psi(\widehat{N})\Big{(}1+o(1)\Big{)}.$

  * •

If both $\gamma_{\pm}=1=2H$, then

$\Pi(N)=\mathcal{H}_{2H,h}^{\mathcal{P}}(T/2^{1/2H}t_{*})\Psi(\widehat{N})\Big{(}1+o(1)\Big{)}.$

  * •

If $\gamma_{\pm}=1<2H$, then

$\Pi(N)=\Psi(\widehat{N})\Big{(}1+o(1)\Big{)}.$

##  3\. Proof of Theorem 1

This section is dedicated to the proof of Theorem 1.

###  3.1. Large vicinities.

Looking ahead, we shall prove that only a small vicinity of the optimal point
contributes to the first order asymptotics, and to evaluate its contribution
we shall divide this small vicinity into even smaller parts of some size
$q(u)$ (referred to as the Pickands scale of the process $Z$, determined only
by the covariance structure of $Z$), on which the uniform local Pickands lemma
may be applied. It follows directly from the Piterbarg inequality and the
following obvious but important property of the Parisian functional:

(3) $\mathcal{P}_{E,F}(f)=\sup_{t\in E}\inf_{s\in F}f(t+s)\leq\sup_{t\in
E}f(t)$

that

$\limsup_{u\to\infty}\frac{\Pi_{E\setminus[-\delta_{-}(u),\delta_{+}(u)]}(u)}{u^{\kappa}\Psi(u)}=0$

for $\delta_{\pm}(u)=u^{-2/\gamma_{\pm}}\ln^{2/\gamma_{\pm}}u$ and all
$\kappa>0$. Since we intend to prove that
$\Pi(u)\sim\mathcal{H}u^{\kappa}\Psi(u)$ for some $\mathcal{H},\kappa>0$, from
this inequality will follow that
$\Pi(u)\sim\Pi_{[-\delta_{-}(u),\delta_{+}(u)]}(u)$ as $u\to\infty$. We can
narrow the vicinity even further by once again using (3) and applying Lemma
4.3 (Lemma 5.4 from [16])

$\limsup_{u\to\infty}\frac{\Pi_{[-\delta_{-}(u),\delta_{+}(u)]\setminus[-\Lambda
u^{-2/\gamma_{-}},\Lambda u^{-2/\gamma_{+}}]}(u)}{u^{\kappa}\Psi(u)}\leq
Ce^{-c\Lambda^{\gamma}}$

where $\gamma=\max\\{\gamma_{-},\gamma_{+}\\}$. Due to this inequality, we may
concentrate on the exact asymptotics of

$\Pi_{\Delta(u,\Lambda)}(u)\quad\text{where}\quad\Delta(u,\Lambda)=\Delta^{+}(u,\Lambda)\cup\Delta^{-}(u,\Lambda),\quad\Delta^{\pm}(u,\Lambda)=\pm[0,\Lambda
u^{-2/\gamma_{\pm}}]$

and then let $\Lambda\to\infty$.

###  3.2. Pickands intervals.

Next, we introduce the left and right Pickands intervals
$\Delta^{\pm}_{k}(u,\lambda)$ with some additional parameter $\lambda>0$

$\Delta^{\pm}_{k}(u,\lambda)=\pm\lambda
q(u)[k,k+1],\quad\text{where}\quad\nu=\min\\{\alpha,\gamma_{+},\gamma_{-}\\},\
q(u)=u^{-2/\nu},$

and the number of those fitting into the large vicinity
$\Delta^{\pm}(u,\Lambda)$:

$N^{\pm}(u,\lambda,\Lambda)=\left\lfloor\frac{|\Delta^{\pm}(u)|}{\lambda
u^{-2/\nu}}\right\rfloor=\left\lfloor\frac{\Lambda
u^{\zeta_{\pm}}}{\lambda}\right\rfloor,\quad\zeta_{\pm}=\frac{2}{\nu}-\frac{2}{\gamma_{\pm}}=\max\left\\{\frac{2}{\alpha}-\frac{2}{\gamma_{\pm}},\frac{2}{\gamma_{\mp}}-\frac{2}{\gamma_{\pm}},0\right\\}\geq
0.$

We shall split the proof in four cases:

Pickands case:

$\gamma\neq\nu=\alpha$

Piterbarg case:

$\gamma=\nu=\alpha$

Talagrand-1 case:

$\gamma=\nu\neq\alpha$

Talagrand-2 case:

$\gamma\neq\nu\neq\alpha$

In the Pickands case at least one of $\zeta_{\pm}$ is nonzero, therefore
$N^{\pm}$ grows as $u^{\zeta_{\pm}}$. In both Piterbarg and Talagrand-1 cases
$\zeta_{+}=\zeta_{-}=0$, hence $u\mapsto N^{\pm}$ is constant and we can set
$\lambda=\Lambda$, in which case $N^{\pm}=1$ – the zeroth Pickands interval
coincides with the informative vicinity. The Talagrand-2 case is to be treated
separately.

###  3.3. Pickands case.

To deal with the Pickands case, we employ the so-called double sum method,
which is based on the Bonferroni inequality

$\bm{\Sigma}_{1}(u,\lambda,\Lambda)-\bm{\Sigma}_{2}(u,\lambda,\Lambda)\leq\Pi_{\Delta(u,\Lambda)}(u)\leq\bm{\Sigma}_{1}^{\prime}(u,\lambda,\Lambda),$

where

$\bm{\Sigma}_{1}(u,\lambda,\Lambda)=\underbrace{\sum_{k=1}^{N^{+}(u,\lambda,\Lambda)}\Pi_{\Delta_{k}^{+}(u,\lambda)}(u)}_{=:\bm{\Sigma}_{1}^{+}(u,\lambda,\Lambda)}+\underbrace{\sum_{k=1}^{N^{-}(u,\lambda,\Lambda)}\Pi_{\Delta_{k}^{-}(u,\lambda,\Lambda)}(u)}_{=:\bm{\Sigma}_{1}^{-}(u,\lambda,\Lambda)}+\underbrace{\Pi_{\Delta_{0}^{+}(u,\lambda)\cup\Delta_{0}^{-}(u,\lambda)}(u)}_{=:\bm{\Sigma}_{0}(u,\lambda)},$

and $\bm{\Sigma}_{1}^{\prime\pm}$ and
$\bm{\Sigma}_{1}^{\prime}=\bm{\Sigma}_{1}^{\prime+}+\bm{\Sigma}_{1}^{\prime-}+\bm{\Sigma}_{0}$
denote the same $\bm{\Sigma}_{1}$ but with $N^{\pm}+1$ instead of $N^{\pm}$
(so that the collection of Pickands intervals indeed cover
$\Delta(u,\Lambda)$), and finally

$\bm{\Sigma}_{2}(u,\lambda,\Lambda)=\sum_{\begin{subarray}{c}\kappa,\kappa^{\prime}\in\\{+,-\\},\\\
1\leq k\leq N^{\kappa}(u,\lambda,\Lambda),\\\ 1\leq k^{\prime}\leq
N^{\kappa^{\prime}}(u,\lambda,\Lambda),\\\
(\kappa,k)\neq(\kappa^{\prime},k^{\prime})\end{subarray}}\mathbb{P}\left\\{\mathcal{P}_{\Delta^{\kappa}_{k}(u,\lambda,\Lambda),[0,T_{u}]}(Z)>u,\mathcal{P}_{\Delta^{\kappa^{\prime}}_{k^{\prime}}(u,\lambda,\Lambda),[0,T_{u}]}(Z)>u\right\\}.$

Since the Parisian functional is bounded by $\sup$ functional, we can reduce
the double sum estimate to the classical ($\sup$) case

$\bm{\Sigma}_{2}(u,\lambda,\Lambda)\leq\sum_{\begin{subarray}{c}\kappa,\kappa^{\prime}\in\\{+,-\\},\\\
1\leq k\leq N^{\kappa}(u,\lambda,\Lambda),\\\ 1\leq k^{\prime}\leq
N^{\kappa^{\prime}}(u,\lambda,\Lambda),\\\
(\kappa,k)\neq(\kappa^{\prime},k^{\prime})\end{subarray}}\mathbb{P}\left\\{\sup_{\Delta^{\kappa}_{k}(u,\lambda,\Lambda),}Z(t)>u,\sup_{\Delta^{\kappa^{\prime}}_{k^{\prime}}(u,\lambda,\Lambda),}Z(t)>u\right\\},$

and using similar arguments to those in [17] obtain

(4)
$\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\limsup_{u\to\infty}\frac{\bm{\Sigma}_{2}(u,\lambda,\Lambda)}{u^{k}\Psi(u)}=0\quad\text{for
all }k>0.$

We shall prove that if there exist two constants $C_{\pm}>0$ such that

$\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{1}^{\pm}(u,\lambda,\Lambda)}{u^{\zeta_{\pm}}\Psi(u)}=C_{\pm},$

which together with the double sum estimate above and

(5)
$\lim_{\lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{0}(u,\lambda)}{\Psi(u)}=\mathcal{H}_{0}\in(0,\infty),$

yields

$\lim_{\Lambda\to\infty}\lim_{u\to\infty}\frac{\Pi_{\Delta(u,\Lambda)}(u)}{u^{\zeta}\Psi(u)}=C,$

where $\zeta=\max\\{\zeta_{+},\zeta_{-}\\}>0$ and
$C=C_{+}1_{\zeta=\zeta_{+}}+C_{-}1_{\zeta=\zeta_{-}}$. Finally, we obtain

$\Pi(u)\sim Cu^{\zeta}\Psi(u).$

###  3.4. Piterbarg and Talagrand-1 cases.

As noted before, in both Piterbarg and Talagrand-1 cases $u\mapsto
N^{\pm}(u,\lambda,\Lambda)$ is constant. We may set $\lambda=\Lambda$, which
makes this constant equal one, and therefore

$\Pi_{\Delta(u,\Lambda)}=\bm{\Sigma}_{0}(u,\Lambda).$

By (5), we have

$\lim_{\Lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{0}(u,\Lambda)}{\Psi(u)}=\mathcal{H}_{0}$

for some $\mathcal{H}_{0}>0$, which together with Lemma 4.3 yeilds
$\Pi(u)\sim\mathcal{H}_{0}\Psi(u)$. In the Talagrand case $\nu\neq\alpha$ we
shall see that $\mathcal{H}_{0}=1$, and therefore $\Pi(u)\sim\Psi(u)$.

###  3.5. Talagrand-2 case.

In the Talagrand-2 case we shall directly (that is, without appealing to
Pickands intervals) prove that there exists positive and finite limit

$\lim_{u\to\infty}\frac{\Pi_{\Delta(\lambda,u)}(u)}{\Psi(u)}=\begin{cases}1,&\gamma_{-}<\gamma_{+},\\\
\exp(-\min\\{A_{-}T^{\gamma_{-}},A_{+}T^{\gamma_{+}}\\}),&\gamma_{+}<\gamma_{-},\end{cases}$

which ceases to depend on $\lambda$ as long as $\lambda>T$.

###  3.6. Asymptotics of $\Sigma_{0}(u,\lambda)$ and the Piterbarg and
Talagrand-1 cases

Denote

$q(u)=u^{-2/\nu},\quad\nu=\min\\{\alpha,\gamma_{+},\gamma_{-}\\}.$

Note that

$u^{2/\alpha}q(u)\xrightarrow[u\to\infty]{}1_{\nu=\alpha}\quad\text{and}\quad
u^{2/\gamma_{\pm}}q(u)\xrightarrow[u\to\infty]{}1_{\nu=\gamma_{\pm}}.$

To apply the uniform local Pickands Lemma 2, let us rewrite the probability
$\bm{\Sigma}_{0}(u,\lambda)$ in terms of a standardized process as follows:

$\bm{\Sigma}_{0}(u,\lambda)=\mathbb{P}\left\\{\mathcal{P}_{\Delta_{0}^{+}(u,\lambda)\cup\Delta_{0}^{-}(u,\lambda)}(Z)>u\right\\}=\mathbb{P}\left\\{\mathcal{P}^{\prime}(\xi_{u,0})>u\right\\},$

where we have defined
$\mathcal{P}^{\prime}=\mathcal{P}_{[-\lambda,\lambda],[0,1]}$ and the family
$\\{\xi_{u,0}\colon u>0\\}$ of centered Gaussian processes by

$\xi_{u,0}(t,s)=\frac{\overline{Z}(q(u)t+T_{u}s)}{1+g(q(u)t+T_{u}s)},\quad
1+g(t)=\frac{1}{\sigma_{Z}(t)},$

Note that in contrast to $\xi_{u,k}$, $k>0$ (see below), the process
$\xi_{u,0}$ is defined for $t\in[-\lambda,\lambda]$, not $[0,\lambda]$. This
is neither a coincidence, nor a technical decision: the two adjacent intervals
near the optimal point cannot be treated separately as it will be evident from
the result.

By (2) and the definition of $g$ we have

$u^{2}g(q(u)t+T_{u}s)\to
h(t+Ts)=h^{+}(t+Ts)1_{\nu=\gamma_{+}}+h^{-}(t+Ts)1_{\nu=\gamma_{-}}.$

where

$h^{\pm}(\mu)=A_{\pm}|\mu|^{\gamma_{\pm}}1_{\pm\mu>0}.$

By assumption A2 we have

$u^{2}\mathbb{E}\left\\{\left|Z(q(u)t+T_{u}s)-Z(q(u)t^{\prime}+T_{u}s^{\prime})\right|^{2}\right\\}\to
2D1_{\nu=\alpha}\Big{|}(t-t^{\prime})+T(s-s^{\prime})\Big{|}^{\alpha},$

which means that the condition C2

$\eta(t,s)=B_{\alpha/2}(t+s)1_{\nu=\alpha},\quad(t,s)\in[-D^{1/\alpha}\lambda,D^{1/\alpha}\lambda]\times[0,D^{1/\alpha}T]$

for $T\geq 0$. By the uniform local Pickands Lemma 2 (condition (C3) is
obviously satisfied) we have

$\lim_{u\to\infty}\frac{\bm{\Sigma}_{0}(u,\lambda)}{\Psi(u)}=\mathcal{H}_{\eta,h}^{\mathcal{P}_{0}}\Big{(}[-D^{1/\alpha}\lambda,D^{1/\alpha}\lambda]\times[0,D^{1/\alpha}T]\Big{)},$

where
$\mathcal{H}_{\eta,h}^{\mathcal{P}_{0}}(E)=\mathbb{E}\left\\{e^{\mathcal{P}(\eta^{h})}\right\\}$,
$\mathcal{P}_{0}=\mathcal{P}_{[-\lambda,\lambda],[0,T]}$ and

(6)
$\eta^{h}(t,s)=\Big{(}\sqrt{2}B_{\alpha/2}(t+s)-|t+s|^{\alpha}\Big{)}1_{\nu=\alpha}-\\\\[7.0pt]
-A_{-}D^{-\gamma_{-}/\alpha}|t+s|^{\gamma_{-}}1_{t+s\leq
0,\nu=\gamma_{-}}-A_{+}D^{-\gamma_{+}/\alpha}|t+s|^{\gamma_{+}}1_{t+s\geq
0,\nu=\gamma_{+}}.$

####  3.6.1. Piterbarg case.

To prove the main theorem in the Piterbarg case $\gamma=\alpha=\nu$, that is
when $\eta^{h}$ has all the terms

$\eta^{h}(t,s)=\Big{(}\sqrt{2}B_{\alpha/2}(t+s)-|t+s|^{\alpha}\Big{)}-A_{-}D^{-\gamma_{-}/\alpha}|t+s|^{\gamma_{-}}1_{t+s\leq
0}-A_{+}D^{-\gamma_{+}/\alpha}|t+s|^{\gamma_{+}}1_{t+s\geq 0},$

it remains to apply the standard result on the existence of Piterbarg
constants to see that

$\lim_{\lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{0}(u,\lambda)}{\Psi(u)}=\lim_{\lambda\to\infty}\mathcal{H}_{\eta,h}^{\mathcal{P}_{0}}\Big{(}[-D^{1/\alpha}\lambda,D^{1/\alpha}\lambda]\times[0,D^{1/\alpha}T]\Big{)}=\mathcal{H}_{\eta,h}^{\mathcal{P}_{0}}(D^{1/\alpha}T)$

exists and is finite. This ends the proof of the main theorem in the Piterbarg
case.

####  3.6.2. Talagrand-1 case.

In the Talagrand-1 case $\gamma=\nu\neq\alpha$ and, therefore, the random part
disappears from (6), whereas all non-random terms are present:

$\eta^{h}(t,s)=-A_{+}D^{-\gamma_{+}/\alpha}|t+s|^{\gamma_{+}}1_{t+s\geq
0}-A_{-}D^{-\gamma_{-}/\alpha}|t+s|^{\gamma_{-}}1_{t+s\leq 0}$

It remains to calculate $\mathcal{P}_{0}(\eta^{h})$ explicitly: if
$\lambda>T$, we have

$\displaystyle\mathcal{P}_{0}\left(\eta^{h}\right)$
$\displaystyle=\sup_{t\in[-\lambda,\lambda]}\inf_{s\in[0,T]}\Big{(}-A_{+}|t+s|^{\gamma_{+}}1_{t+s\geq
0}-A_{-}|t+s|^{\gamma_{-}}1_{t+s\leq 0}\Big{)}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,-T]}\inf_{s\in[0,T]}\left(-A_{-}|t+s|^{\gamma_{-}}\right),\sup_{t\in[-T,\lambda]}\inf_{s\in[0,T]}\left(-A_{+}|t+s|^{\gamma_{+}}\right)\right\\}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,-T]}(-A_{-}|t|^{\gamma_{-}}),\sup_{t\in[-T,\lambda]}\left(-A_{+}|t+T|^{\gamma_{+}}\right)\right\\}=$
$\displaystyle=\max\left\\{-A_{-}T^{\gamma_{-}},0\right\\}=0.$

Therefore, by lemma above in the Talagrand case

$\mathcal{H}_{\eta,h}^{\mathcal{P}_{0}}\Big{(}[-D^{1/\alpha}\lambda,D^{1/\alpha}\lambda]\times[0,D^{1/\alpha}T]\Big{)}=1$

for all $\lambda>T$. Thus, we have proved that

$\lim_{\lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{0}(u,\lambda)}{\Psi(u)}=1.$

This ends the proof of the main theorem in the Talagrand case.

###  3.7. Talagrand-2 case.

To apply the uniform local Pickands Lemma 2, let us rewrite the probability
$\Pi_{\Delta(u,\Lambda)}(u)$ in terms of a standardized process as follows.
First, observe that the trivial equality

$t+s=(t+s)1_{t+s\geq 0}+(t+s)1_{t+s\leq 0},\quad(t,s)\in[-\lambda
u^{-2/\gamma_{-}},\lambda u^{-2/\gamma_{+}}]\times[0,T_{u}]$

may be rewritten as

$t+s=q(u,t^{\prime},s^{\prime})=q_{+}(u,t^{\prime},s^{\prime})+q_{-}(u,t^{\prime},s^{\prime}),\quad(t^{\prime},s^{\prime})\in[-\lambda,\lambda]\times[0,1].$

where

$q_{\pm}(u,t^{\prime},s^{\prime})=\Big{(}u^{-2/\gamma_{\pm}}t^{\prime}+T_{u}s^{\prime}\Big{)}1_{\pm(u^{-2/\gamma_{\pm}}t^{\prime}+T_{u}s^{\prime})\geq
0}.$

Using this reparametrization, we rewrite

$\Pi_{\Delta(u,\lambda)}(u)=\mathbb{P}\left\\{\mathcal{P}_{[-\lambda
u^{-2/\gamma_{-}},\lambda
u^{-2/\gamma_{+}}]}(Z)>u\right\\}=\mathbb{P}\left\\{\mathcal{P}^{\prime}(\xi_{u,0})>u\right\\},$

where we have defined
$\mathcal{P}^{\prime}=\mathcal{P}_{[-\lambda,\lambda],[0,1]}$ and the family
$\\{\xi_{u,0}\colon u>0\\}$ of centered Gaussian processes by

$\xi_{u,0}(t^{\prime},s^{\prime})=\frac{\overline{Z}(q(u,t^{\prime},s^{\prime}))}{1+g(q(u,t^{\prime},s^{\prime}))},\quad
1+g(t)=\frac{1}{\sigma_{Z}(t)}.$

Note that

$u^{2/\gamma_{\pm}}q_{\pm}(u,t^{\prime},s^{\prime})\to\Big{(}t^{\prime}+1_{\nu=\gamma_{\pm}}Ts^{\prime}\Big{)}1_{\pm(t^{\prime}+1_{\nu=\gamma_{\pm}}Ts^{\prime})\geq
0}$

uniformly in $(t^{\prime},s^{\prime})$. By (2) and the definition of $g$ we
have

$u^{2}g(q(u,t^{\prime},s^{\prime}))\sim
A_{+}|u^{2/\gamma_{+}}q_{+}(u,t^{\prime},s^{\prime})|^{\gamma_{+}}+A_{-}|u^{2/\gamma_{-}}q_{-}(u,t^{\prime},s^{\prime})|^{\gamma_{-}}\sim\\\\[7.0pt]
\sim
A_{+}|t^{\prime}+1_{\nu=\gamma_{+}}Ts^{\prime}|^{\gamma_{+}}1_{t^{\prime}+1_{\nu=\gamma_{+}}Ts^{\prime}\geq
0}+A_{-}|t^{\prime}+1_{\nu=\gamma_{-}}Ts^{\prime}|^{\gamma_{-}}1_{t^{\prime}+1_{\nu=\gamma_{-}}Ts^{\prime}\geq
0}.$

By assumption A2 we have

$u^{2}\mathbb{E}\left\\{\left|Z(q(u,t^{\prime}_{1},s^{\prime}_{1}))-Z(q(u,t^{\prime}_{2},s^{\prime}_{2}))\right|^{2}\right\\}\to
0,$

which means that the condition C2 $\eta(t,s)=0$.

By the uniform local Pickands Lemma 2 (condition C3 is obviously satisfied) we
have

$\lim_{u\to\infty}\frac{\Pi_{\Delta(u,\lambda)}(u)}{\Psi(u)}=\mathcal{H}_{0,h}^{\mathcal{P}_{0}}([-\lambda,\lambda]\times[0,T]),$

where $\mathcal{H}_{0,h}^{\mathcal{P}_{0}}(E)=e^{\mathcal{P}_{0}(-h)}$,
$\mathcal{P}_{0}=\mathcal{P}_{[-\lambda,\lambda],[0,T]}$ and

$h(t,s)=A_{-}|t+1_{\nu=\gamma_{-}}s|^{\gamma_{-}}1_{t+1_{\nu=\gamma_{-}}s\leq
0}+A_{+}|t+1_{\nu=\gamma_{+}}s|^{\gamma_{+}}1_{t+1_{\nu=\gamma_{+}}s\geq 0}.$

Since we are looking at a case where $\gamma\neq\nu$, either
$1_{\nu=\gamma_{-}}$ or $1_{\nu=\gamma_{+}}$ is zero. Suppose,
$\nu=\gamma_{-}\neq\gamma_{+}$. Then

$h(t,s)=-A_{-}|t+s|^{\gamma_{-}}1_{t+s\leq 0}-A_{+}|t|^{\gamma_{+}}1_{t\geq
0}.$

therefore, we have

$\mathcal{P}(h)=\max\left\\{\sup_{t\in[-\lambda,0]}\inf_{s\in[0,t]}\big{(}-a_{-}|t+s|^{\gamma_{-}}1_{t+s\leq
0}\big{)},\sup_{t\in[0,\lambda]}\inf_{s\in[0,t]}\big{(}-a_{+}|t|^{\gamma_{+}}\big{)}\right\\}=\\\\[7.0pt]
=\max\left\\{\sup_{t\in[-\lambda,0]}\inf_{\mu\in[t,t+t]}\big{(}-a_{-}|\mu|^{\gamma_{-}}1_{\mu\leq
0}\big{)},0\right\\}=0$

since the first term is non-positive.

If, on the other hand, $\nu=\gamma_{+}\neq\gamma_{-}$, we have

$h(t,s)=-a_{-}|t|^{\gamma_{-}}1_{t\leq 0}-a_{+}|t+s|^{\gamma_{+}}1_{t+s\geq
0}$

and therefore for $\lambda>T$ we have

$\displaystyle\mathcal{P}(h)$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,0]}\inf_{s\in[0,T]}\Big{(}-A_{-}|t|^{\gamma_{-}}-A_{+}|t+s|^{\gamma_{+}}1_{t+s\geq
0}\Big{)},\sup_{t\in[0,\lambda]}\inf_{s\in[0,T]}\Big{(}-A_{+}|t+s|^{\gamma_{+}}\Big{)}\right\\}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,0]}\inf_{\mu\in[t,t+T]}\Big{(}-A_{-}|t|^{\gamma_{-}}-A_{+}|\mu|^{\gamma_{+}}1_{\mu\geq
0}\Big{)},-A_{+}|T|^{\gamma_{+}}\right\\}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,0]}\min\left\\{-A_{-}|t|^{\gamma_{-}},\inf_{\mu\in[0,\max(t+T,0)]}\Big{(}-A_{-}|t|^{\gamma_{-}}-A_{+}|\mu|^{\gamma_{+}}\Big{)}\right\\},-A_{+}|T|^{\gamma_{+}}\right\\}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,0]}\Big{(}-A_{-}|t|^{\gamma_{-}}-A_{+}|\max(t+T,0)|^{\gamma_{+}}\Big{)},-A_{+}|T|^{\gamma_{+}}\right\\}=$
$\displaystyle=\max\left\\{\sup_{t\in[-\lambda,-T]}\Big{(}-A_{-}|t|^{\gamma_{-}}\Big{)},\sup_{t\in[-T,0]}\Big{(}-A_{-}|t|^{\gamma_{-}}-A_{+}|t+T|^{\gamma_{+}}\Big{)},-A_{+}|T|^{\gamma_{+}}\right\\}=$
$\displaystyle=\max\left\\{-A_{-}|T|^{\gamma_{-}},\sup_{\mu\in[0,T]}\Big{(}-A_{-}|\mu|^{\gamma_{-}}-A_{+}|T-\mu|^{\gamma_{+}}\Big{)},-A_{+}|T|^{\gamma_{+}}\right\\}=$
$\displaystyle=-\min\left\\{A_{-}T^{\gamma_{-}},A_{+}T^{\gamma_{+}}\right\\}$

We have thus proved that

$\mathcal{H}_{0,h}^{\mathcal{P}}([-\lambda,\lambda]\times[0,T])=e^{\mathcal{P}(-h)}=\begin{cases}1,&\gamma_{-}<\gamma_{+},\\\
\exp(-\min\\{A_{-}T^{\gamma_{-}},A_{+}T^{\gamma_{+}}\\}),&\gamma_{+}<\gamma_{-}\end{cases}$

for all $\lambda>T$.

###  3.8. Pickands case.

Now we proceed to the Pickands case.

To find the aforementioned asymptotics of
$\bm{\Sigma}_{1}^{\pm}(u,\lambda,\Lambda)$ we shall first find the uniform in
$k\in\\{1,\dots,N^{\pm}(u,\lambda,\Lambda)\\}$ asymptotics of each summand
$\Pi_{\Delta^{\pm}(u,\lambda,\Lambda)}(u)$ and then sum them up. To this end,
let us rewrite the probability in the form required for the uniform local
Pickands Lemma 2

$\Pi_{\Delta_{k}^{\pm}(u,\lambda,\Lambda)}(u)=\mathbb{P}\left\\{\mathcal{P}_{\Delta_{k}^{\pm}(u,\lambda,\Lambda),[0,T_{u}]}(Z)>u\right\\}=\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\xi_{u,k}^{\pm})>u\right\\},$

where we have defined
$\mathcal{P}^{\prime\prime}=\mathcal{P}_{[0,\lambda],[0,1]}$ and the family

$\\{\xi_{u,k}^{\kappa}\colon\kappa\in\\{+,-\\},1\leq k\leq
N^{\pm}(u,\lambda,\Lambda),u>0\\}$

of centered Gaussian processes by

$\xi_{u,k}^{\pm}(t,s)=\frac{\widetilde{Z}_{u,k}^{\pm}(t,s)}{1+h_{u,k}^{\pm}(t,s)},\quad
1+h_{u,k}^{\pm}(t,s)=\frac{1}{\sigma_{Z}(\pm q(u)(\lambda k+t)+T_{u}s)}$

and

$\widetilde{Z}_{u,k}^{\pm}(t,s)=\frac{Z(\pm q(u)(\lambda
k+t)+T_{u}s)}{\sigma_{Z}(\pm q(u)(\lambda k+t)+T_{u}s)},$

$t\in[0,\lambda],s\in[0,1]$. Note that $\widetilde{Z}_{u,k}^{\pm}$ is a
centered Gaussian random field with unit variance and continuous paths.
Besides, $h_{u,k}^{\pm}\in C_{0}([0,\lambda]\times[0,T])$, that is, it is a
continuous function on $[0,\lambda]\times[0,T]$, such that
$h_{u,k}^{\pm}(0,0)=0$.

There is, however, a pitfall in trying to apply uLP directly to
$\xi_{u,k}^{\pm}$. Even though $u^{2}h_{u,k}$ may have a limit $h$ for each
$k$, it is never uniform. In other words, the condition (C1)

$\lim_{u\to\infty}\sup_{k\in
K_{u},(t,s)\in[0,\lambda]\times[0,T]}\left|u^{2}h_{u,k}(t,s)-h(t,s)\right|=0$

is not satisfied. To get around this inconvenience, we shall coarsen the
inequality describing the event by taking $1+h_{u,k}$ out of
$\mathcal{P}^{\prime\prime}$

$\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\widetilde{Z}_{u,k})/U(1+h_{u,k}^{\pm})>u\right\\}\leq\Pi_{\Delta_{k}^{\pm}(u,\lambda)}(u)=\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\xi_{u,k})>u\right\\}\leq\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\widetilde{Z}_{u,k})/L(1+h_{u,k}^{\pm})>u\right\\},$

where

$U(f)=\sup_{t\in[0,\lambda]}\sup_{s\in[0,T]}f(t+s)\quad\text{and}\quad
L(f)=\inf_{t\in[0,\lambda]}\inf_{s\in[0,T]}f(t+s).$

Let us rewrite it in a handier fashion as

$\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\widetilde{Z}_{u,k})>u\Big{(}1+U(h_{u,k}^{\pm})\Big{)}\right\\}\leq\Pi_{\Delta_{k}^{\pm}(u,\lambda)}=\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\xi_{u,k})>u\right\\}\leq\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\widetilde{Z}_{u,k})>u\Big{(}1+L(h_{u,k}^{\pm})\Big{)}\right\\}.$

Using the assumption that $A_{\pm}>0$, we get

$L(h_{u,k}^{+})=A_{+}\Big{|}\lambda q(u)k\Big{|}^{\gamma_{+}},\quad
U(h_{u,k}^{+})=A_{+}\Big{|}\lambda q(u)(k+1)+T_{u}\Big{|}^{\gamma_{+}}$

and

$L(h_{u,k}^{-})=A_{-}\Big{|}-\lambda q(u)k+T_{u}\Big{|}^{\gamma_{-}},\quad
U(h_{u,k}^{-})=A_{-}\Big{|}\lambda q(u)(k+1)\Big{|}^{\gamma_{-}}.$

All four bounds can be rewritten as follows:

$L(h_{u,k}^{\pm})=A_{\pm}\Big{|}\lambda
q(u)k-q(u)m_{\pm}(u)\Big{|}^{\gamma_{\pm}},\quad
U(h_{u,k}^{\pm})=A_{\pm}\Big{|}\lambda
q(u)k+q(u)(\lambda+m_{\mp}(u))\Big{|}^{\gamma_{\pm}},$

where $m_{+}(u)=0$ and $m_{-}(u)=T_{u}/q(u)\to T$ as $u\to\infty$. It is
important for us that $m_{\pm}$ do not depend on $k$ and have finite limits as
$u\to\infty$.

In order to apply the uniform local Pickands Lemma 2 to the upper bound, we
set

$g_{u,k}(\lambda)=u\Big{(}1+A_{\pm}\Big{|}\lambda
q(u)k-q(u)m_{\pm}(u)\Big{|}^{\gamma_{\pm}}\Big{)}$

and note that the condition (C2) remains valid with $g_{u,k}$ instead of $u$.
It now follows directly from the uniform local Pickands Lemma 2 that with
$\mathcal{P}=\mathcal{P}_{[0,\lambda],[0,T]}$ we have

$\mathbb{P}\left\\{\mathcal{P}^{\prime\prime}(\widetilde{Z}_{u,k})>g_{u,k}(\lambda)\right\\}=\mathcal{H}_{2H}^{\mathcal{P}}\left(D^{1/\alpha}\lambda,D^{1/\alpha}T\right)\Psi(u)\exp\left(-A_{\pm}\Big{|}\lambda
u^{-\zeta_{\pm}}k-u^{-\zeta_{\pm}}m_{\pm}(u)\Big{|}^{\gamma_{+}}\right)(1+o(1)),$

where $o(1)$ is uniform in $k\in\\{1,\dots,N^{\pm}(u,\lambda,\Lambda)\\}$.
Thus,

$\bm{\Sigma}_{1}^{\pm}(u,\lambda,\Lambda)\leq\sum_{k=1}^{N^{\pm}(u,\lambda,\Lambda)}\mathbb{P}\left\\{\mathcal{P}(\widetilde{Z}_{u,k})>g_{u,k}(\lambda)\right\\}\sim\\\\[7.0pt]
\sim\mathcal{H}_{2H}^{\mathcal{P}}\left(D^{1/\alpha}\lambda,D^{1/\alpha}T\right)\Psi(u)\sum_{k=1}^{N^{\pm}(u,\lambda,\Lambda)}\exp\left(-A_{\pm}\Big{|}\lambda
u^{-\zeta_{\pm}}k-u^{-\zeta_{\pm}}m_{\pm}(u)\Big{|}^{\gamma_{\pm}}\right)\sim\\\\[7.0pt]
\sim\frac{\mathcal{H}_{2H}^{\mathcal{P}}\left(D^{1/\alpha}\lambda,D^{1/\alpha}T\right)}{\lambda}u^{\zeta_{\pm}}\Psi(u)\int_{0}^{\Lambda}e^{-A_{\pm}x^{\gamma_{\pm}}}\mathop{dx}$

and, letting $u\to\infty$, then $\Lambda\to\infty$ and finally
$\lambda\to\infty$, we see that for large enough $u$, holds

$\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\limsup_{u\to\infty}\frac{\bm{\Sigma}_{1}^{\pm}(u,\lambda,\Lambda)}{u^{\zeta_{\pm}}\Psi(u)}\leq\Gamma\left(\frac{1}{\gamma_{\pm}}+1\right)A_{\pm}^{-1/\gamma_{\pm}}D^{1/\alpha}\mathcal{H}_{\alpha}^{\mathcal{P}}(D^{1/\alpha}T)$

where

$\mathcal{H}_{\alpha}^{\mathcal{P}}(T)=\lim_{\lambda\to\infty}\frac{\mathcal{H}_{\alpha}^{\mathcal{P}}(\lambda,T)}{\lambda}\in(0,\infty).$

Using lower bound in much the same fashion, we obtain

$\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\limsup_{u\to\infty}\frac{\bm{\Sigma}_{1}^{\pm}(u,\lambda,\Lambda)}{u^{\zeta_{\pm}}\Psi(u)}=\Gamma\left(\frac{1}{\gamma_{\pm}}+1\right)A^{-1/\gamma_{\pm}}D^{1/\alpha}\mathcal{H}_{\alpha}^{\mathcal{P}}(D^{1/\alpha}T).$

The same formula obviously holds for $\bm{\Sigma}_{1}^{\prime\pm}$. To
conclude the proof in the Pickands case, it remains to notice that

$\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\lim_{u\to\infty}\frac{\bm{\Sigma}_{1}(u,\lambda,\Lambda)}{u^{\zeta}\Psi(u)}=\\\\[7.0pt]
=\lim_{\lambda\to\infty}\lim_{\Lambda\to\infty}\lim_{u\to\infty}\left(u^{\zeta_{+}-\zeta}\frac{\bm{\Sigma}_{1}^{+}(u,\lambda,\Lambda)}{u^{\zeta_{+}}\Psi(u)}+u^{\zeta_{-}-\zeta}\frac{\bm{\Sigma}_{1}^{-}(u,\lambda,\Lambda)}{u^{\zeta_{-}}\Psi(u)}+u^{-\zeta}\frac{\bm{\Sigma}_{0}(u,\lambda)}{\Psi(u)}\right)=\\\\[7.0pt]
=\left(\Gamma\left(\frac{1}{\gamma_{+}}+1\right)A_{+}^{-1/\gamma_{+}}1_{\zeta=\zeta_{+}}+\Gamma\left(\frac{1}{\gamma_{-}}+1\right)A_{-}^{-1/\gamma_{-}}1_{\zeta=\zeta_{+}}\right)D^{1/\alpha}\mathcal{H}_{\alpha}^{\mathcal{P}}(D^{1/\alpha}T),$

that the same is obviously true for $\bm{\Sigma}_{1}^{\prime}$, and that
$1_{\zeta=\zeta_{\pm}}=1_{\gamma=\gamma_{\pm}}$.

##  4\. Appendix

In this appendix, we recall some known results necessary for the proofs of
previous section.

###  4.1. Parisian functional continuity

Let us show that the Parisian functional $\mathcal{P}\colon C(E\times
F)\to\mathbb{R}$ is continuous in uniform topology. To this end, we take an
arbitrary function $f\in C(E\times F)$ and a family
$\\{f_{\varepsilon},\varepsilon>0\\}\subset C(E\times F)$, which converges to
a function $f\in C(E\times F)$ uniformly

$\sup_{(t,s)\in E\times
F}\Big{|}f(t,s)-f_{\varepsilon}(t,s)\Big{|}<\varepsilon$

as $\varepsilon\to 0$, from which we obtain

$\begin{cases}f(t,s)-f_{\varepsilon}(t,s)<\varepsilon,\\\
f_{\varepsilon}(t,s)-f(t,s)<\varepsilon\end{cases}\quad\text{for
all}\quad(t,s)\in E\times F.$

Hence,

$\sup_{t\in E}\inf_{s\in F}f(t,s)<\varepsilon+\sup_{t\in E}\inf_{s\in
F}f_{\varepsilon}(t,s)\quad\text{and}\quad\sup_{t\in E}\inf_{s\in
F}f_{\varepsilon}(t,s)<\varepsilon+\sup_{t\in E}\inf_{s\in F}f(t,s),$

or, equivalently,

$\Big{|}\sup_{t\in E}\inf_{s\in F}f(t,s)-\sup_{t\in E}\inf_{s\in
F}f_{\varepsilon}(t,s)\Big{|}=\Big{|}\mathcal{P}(f)-\mathcal{P}(f_{\varepsilon})\Big{|}<\varepsilon.$

###  4.2. Uniform local Pickands lemma

The following lemma is from [15], it is reproduced here for the reader’s
convenience. Let

$\xi_{u,\tau_{u}}(t)=\frac{Z_{u,\tau_{u}}(t)}{1+h_{u,\tau_{u}}(t)},\quad t\in
E,\tau_{u}\in K_{u},$

be a family of centered Gaussian random fields with $Z_{u,\tau_{u}}$ a
centered Gaussian random field with unit variance and continuous paths, and
$h_{u,\tau_{u}}$ belonging to $C_{0}(E)$, that is, $h_{u,\tau_{u}}$ is a
continuous function on $E$, such that $h_{u,\tau_{u}}(0)=0$. We assume that
$E$ is a compact subset of $\mathbb{R}^{d}$ and $0\in E$.

The Parisian functional $\Gamma_{E,F}$

$\Gamma_{E,F}(X)=\sup_{t\in E}\inf_{s\in F}X(t+s)$

satisfies the conditions

(F1):

there exists $c>0$ such that $\Gamma(f)\leq c\sup_{t\in E}f(t)$ for any $f\in
C(E)$

(F2):

$\Gamma(af+b)=a\Gamma(f)+b$ for any $f\in C(E)$ and $a>0$, $b\in\mathbb{R}$

of the paper [15]. Therefore, under conditions $(C0)-(C3)$

(C0):

$\lim_{u\to\infty}\inf_{\tau_{u}\in K_{u}}g_{u,\tau_{u}}=\infty$

(C1):

there exists $h\in C_{0}(E)$ such that

$\lim_{u\to\infty}\sup_{\tau_{u}\in K_{u},t\in
E}\left|g_{u,\tau_{u}}^{2}h_{u,\tau_{u}}(t)-h(t)\right|=0$

(C2):

there exists $\theta_{u,\tau_{u}}(t,s)$ such that

$\lim_{u\to\infty}\sup_{\tau_{u}\in K_{u}}\sup_{s\neq t\in
E}\left|g_{u,\tau_{u}}^{2}\frac{\mathbb{E}\left\\{\left|Z_{u,\tau_{u}}(t)-Z_{u,\tau_{u}}(s)\right|^{2}\right\\}}{2\theta_{u,\tau_{u}}(t,s)}-1\right|=0,$

where for some centered Gaussian random field $\eta(t)$, $t\in\mathbb{R}^{d}$
with continuous paths and $\eta(0)=0$,

$\lim_{u\to\infty}\sup_{\tau_{u}\in
K_{u}}\left|\theta_{u,\tau_{u}}(t,s)-\mathbb{E}\left\\{\left|\eta(t)-\eta(s)\right|^{2}\right\\}\right|=0.$

(C3):

there exists $a>0$ such that

$\limsup_{u\to\infty}\sup_{\tau_{u}\in K_{u}}\sup_{s\neq t\in
E}\frac{\theta_{u,\tau_{u}}(t,s)}{\sum_{i=1}^{d}|s_{i}-t_{i}|^{a}}<\infty,$
$\lim_{\varepsilon\to 0}\limsup_{u\to\infty}\sup_{\tau_{u}\in
K_{u}}\sup_{\|t-s\|<\varepsilon,t,s\in
E}g_{u,\tau_{u}}^{2}\mathbb{E}\left\\{\left[Z_{u,\tau_{u}}(t)-Z_{u,\tau_{u}}(s)\right]Z_{u,\tau_{u}}(0)\right\\}=0.$

we have

######  Theorem 2.

Under assumptions (C0)-(C3), if, further,
$\mathbb{P}(\Gamma_{E,F}(\xi_{u,\tau_{u}})>g_{u,\tau_{u}})>0$ for all
$\tau_{u}\in K_{u}$ and all large $u$, then

$\lim_{u\to\infty}\sup_{\tau_{u}\in
K_{u}}\left|\frac{\mathbb{P}\\{\Gamma(\xi_{u,\tau_{u}})>g_{u,\tau_{u}}\\}}{\Psi(g_{u,\tau_{u}})}-\mathcal{H}_{\eta,h}^{\Gamma}(E)\right|=0,$

where

$\mathcal{H}_{\eta,h}^{G}(E)=\mathbb{E}\left\\{e^{\Gamma(\eta^{h})}\right\\},\quad\eta^{h}(t)=\sqrt{2}\eta(t)-\sigma^{2}_{\eta}(t)-h(t).$

###  4.3. Large vicinity cut-off lemma

Next lemma is from [16] (Lemma 5.4), but instead of the version therefrom, we
give a version suitable for our needs. This lemma allows one to get rid of the
complement of the Piterbarg vicinity in all three (Piterbarg, Pickands and
Talagrand) cases (see proof of Theorem 1 below).

######  Lemma 1.

There exist positive constants $C$, $c$, $u_{0}$ and $\Lambda_{0}$ such that
for $\Lambda\geq\Lambda_{0}$ and $u\geq u_{0}$

$\mathbb{P}\left\\{\exists t\in\left[-\delta_{-}(u),\Lambda
u^{-2/\gamma_{-}}\right]\cup\left[\Lambda
u^{-2/\gamma_{+}},\delta_{+}(u)\right]\colon Z(t)>u\right\\}\leq
Ce^{-c\Lambda^{\gamma}}\mathbb{P}\left\\{Z(0)>u\right\\}.$

where $\delta_{\pm}(u)=u^{-2/\gamma_{\pm}}\ln^{2/\gamma_{\pm}}u.$

##  5\. Acknowledgments

Financial support by SNSF Grant 200021-196888 is kindly acknowledged.

## References

  * [1] K. Dȩbicki, E. Hashorva, and L. Ji, ‘‘On Parisian ruin over a finite-time horizon,’’ Sci. China Math., vol. 59, pp. 557–572, Mar. 2016\.
  * [2] K. Kȩpczyński, ‘‘Proportional reinsurance for fractional Brownian risk model,’’ ArXiv201000222 Math, Oct. 2020.
  * [3] M. Chesney, M. Jeanblanc-Picqu, and M. Yor, ‘‘Brownian excursions and Parisian barrier options,’’ Adv. in Appl. Probab., vol. 29, no. 1, pp. 165–184, 1997.
  * [4] A. Dassios and S. Wu, ‘‘Semi-Markov model for excursions and occupation time of Markov processes,’’ 2011.
  * [5] A. Dassios and S. Wu, ‘‘Parisian ruin with exponential claims,’’ LSE Research Online Documents on Economics, 2008.
  * [6] I. Czarna and Z. Palmowski, ‘‘Ruin probability with Parisian delay for a spectrally negative Lévy risk process,’’ Journal of Applied Probability, vol. 48, no. 4, pp. 984–1002, 2011.
  * [7] R. Loeffen, I. Czarna, and Z. Palmowski, ‘‘Parisian ruin probability for spectrally negative Lévy processes,’’ Bernoulli, vol. 19, no. 2, 2011\.
  * [8] I. Czarna and Z. Palmowski, ‘‘Dividend problem with Parisian delay for a spectrally negative Lévy risk process,’’ Theory Appl., vol. 161, pp. 239–256, 2014.
  * [9] I. Czarna, Z. Palmowski, and P. Świątek, ‘‘Binomial discrete time ruin probability with Parisian delay,’’ Scandinavian Actuarial Journal, 2014.
  * [10] K. Dȩbicki, E. Hashorva, and L. Ji, ‘‘Parisian ruin of self-similar Gaussian risk processes,’’ ArXiv14052958 Math, May 2014.
  * [11] V. I. Piterbarg, Asymptotic methods in the theory of Gaussian processes and fields.  American Mathematical Society, 1996.
  * [12] G. Jasnovidov, ‘‘Approximation of ruin probability and ruin time in discrete Brownian risk models,’’ Scand. Actuar. J., no. 8, pp. 718–735, 2020.
  * [13] N. Kriukov, ‘‘Parisian and cumulative Parisian ruin probability for two-dimensional Brownian risk model,’’ An International Journal of Probability and Stochastic Processes, 2021.
  * [14] K. Krystecki, ‘‘Parisian ruin probability for two-dimensional Brownian risk model,’’ 2021.
  * [15] K. Dȩbicki, E. Hashorva, and P. Liu, ‘‘Uniform tail approximation of homogenous functionals of Gaussian fields,’’ Adv. Appl. Probab., vol. 49, pp. 1037–1066, Dec. 2017.
  * [16] K. Dȩbicki, E. Hashorva, and L. Wang, ‘‘Extremes of Vector-Valued Gaussian processes,’’ ArXiv191106350 Math, Nov. 2019.
  * [17] L. Ji and S. Robert, ‘‘Ruin problem of a two-dimensional fractional Brownian motion risk process,’’ Stochastic Models, vol. 34, pp. 73–97, Jan. 2018\.


# Dynamically Improving Branch
Prediction Accuracy Between Contexts

Adam Auten  Department of Electrical and
Computer Engineering
University of Illinois Urbana-Champaign
<EMAIL_ADDRESS>Tanishq Dubey  Department of Electrical and
Computer Engineering
University of Illinois Urbana-Champaign
<EMAIL_ADDRESS>Rohan Mathur  Department of Electrical and
Computer Engineering
University of Illinois Urbana-Champaign
<EMAIL_ADDRESS>
###### Abstract

Branch prediction is a standard feature in most processors, significantly
improving the run time of programs by allowing a processor to predict the
direction of a branch before it has been evaluated. Current branch prediction
methods can achieve excellent prediction accuracy through global tables,
various hashing methods, and even machine learning techniques such as SVMs or
neural networks. Such designs, however, may lose effectiveness when attempting
to predict across context switches in the operating system. Such a scenario
may lead to destructive interference between contexts, therefore reducing
overall predictor accuracy. To solve this problem, we propose a novel scheme
for deciding whether a context switch produces destructive or constructive
interference. First, we present evidence that shows that destructive
interference can have a significant negative impact on prediction accuracy.
Second, we present an extensible framework that keeps track of context
switches and prediction accuracy to improve overall accuracy. Experimental
results show that this framework effectively reduces the effect of destructive
interference on branch prediction.

##  I Introduction

Processors are using speculative techniques more and more to increase the
amount of instruction level parallelism that occurs. This is evident in
current cache designs, where data is prefetched from far memory before actual
values are ready to be used. Prediction employs the same methodology in order
to reduce wasted CPU cycles waiting for a branch to be evaluated. Rather than
wait for the branch direction to be evaluated, a prediction is made on the
direction of the branch. Correct predictions are rewarded with continuous
execution, however, incorrect predictions are given penalized by forcing a
processor to employ some sort of backtracking method, thus wasting cycles
correcting its error. The deeper the processor pipeline, the more serious this
penalty is, hence motivating the need for higher accuracy predictors, usually
through radical new designs. We take a different approach – one that extends
current predictor schemes – by improving the accuracy of predictors between
contexts.

Our work builds on the observation that as a processor switches between the
contexts, the state of the branch predictor may destructively interfere with
the predictions of the upcoming context. It then becomes natural to ask if we
can reduce the effect of these context switches by learning when to clear, or
remove, prior history of a branch predictor.

We propose a framework that uses multiple counters and other bookkeeping data
to efficiently track and analyze branch predictor behavior across context
switches. Our framework not only gives the branch predictor more data to
operate on, but can also consider the effects of a context switch through
longer histories.

This paper makes the following contributions:

  * •

We provide insights into the behavior of branch predictors with respect to
context switches, and show that destructive interference is an issue that can
lead to performance degradation.

  * •

We introduce the context switch accuracy framework (CSAF), the first framework
to use data for contexts as input to a branch predictors, and show that in
general cases it can improve upon existing predictor accuracy. For a single
core ARMv7 processor running a standard Linux 3.13 kernel with 1ms kernel tick
rate simulated in the GEM5 framework, the CSAF improves misprediction on 7 out
of 11 tested benchmarks in a predefiend workload.

  * •

We explain why the CSAF introduces interesting new ideas for future research.

##  II Related Work

###  II-A Dynamic Branch Prediction

Dynamic Branch Prediction has been a well researched topic in recent decades,
with modern advances focusing on the improvement, refinement, or various
schemes of the two level scheme described by Yeh et al. [1] This method uses a
history table full of saturating counters that is indexed using some hash of
the branch address. The action of the prediction is based on the current state
of the counter, which is updated once the outcome of the branch is evaluated.
The problem here is that this scheme, and variations of this scheme, suffer
from a few basic problems.

The first of these problems is the aliasing of branch history addresses, and
while there have been advances [2, 3] to reduce aliasing, the prediction
method remains the same, and thus other problems remain unaddressed.

In addition, history length is a limiting factor in these predictors, first
simply by the number of entries that can be stored in the history table and
secondly the size of each entry itself. Generous improvements in hardware
technology are allowing for larger tables simply through brute force. Table
entry sizes are also limited as a function of the number of entries, and as
the information in the entry decreases, the less information the predictor has
to act upon, including the exclusion of context data from history table
entries.

###  II-B Effects of context switching

Some work has been done on analyzing the effects of context switching and its
effect on a running application or overall workload. This research analyzes
the performance of an application during a specific workload, however, these
papers usually focus on cache performance, with a tangential focus on branch
prediction [4, 5]. They show that the effects of context switching can be
diminished in memory through clever cache size manipulations or optimizations
based on the expected workload, demonstrating that context switches should not
be regarded as trivial to CPU workloads.

Indeed, work has also been done to analyze the effects of context switching on
branch prediction. Such research suggests that context switching does not have
the expected significant effect on branch prediction accuracy as one would
expect, but rather, most accuracies reach a steady state when using realistic
time slices for contexts [6]. However, this is countered by other works which
state that in other, nontrivial, workloads, certain branch predictors, such as
the two level scheme previously discussed, may be susceptible to accuracy loss
when switching contexts [7].

Other work has also been done to develop branch predictor models that
incorporate context data in order to improve prediction accuracy [8]. Results
from these works demonstrate that using context data, appreciable results were
obtained from these schemes. However, the downside remains that these schemes
present wholly new methods for branch prediction, instead of building on top
of existing implementations that have already been shown to perform well.

##  III Motivation

###  III-A Lack of predictors that use contextual data

When looking at the most common branch prediction schemes, we can see that
they are mostly based off the work of Yeh and Patt with their standard two
level design. Of course, systems built off this scheme generally exclude the
idea of contexts, with most of the data for the predictor coming from current
PC address possibly combined with previous accuracy or PC data. This complete
oversight of context data means that in thee case where contexts do play a
large role in workload performance, the branch predictor will not know how the
context data plays a role in the prediction it is about to make. Due to this
lack of data, we feel that a gap has been created in the branch prediction
realm that completely ignores the fact that in modern computing, contexts
switches occur very often and also occur across CPUs in the system. This then
begs the question of what would the performance of a context aware branch
predictor look like and what sort of insight could a context provide to a
prediction scheme.

###  III-B Impact of context switching

In order to further justify the work needed to implement the CSAF, we needed
to quantify the amount of impact a context switch has on branch prediction
accuracy. For this, a representative baseline workload was constructed to run
on the GEM5 simulator with with branch prediction accuracy was recorded.
Specifically, a single core ARMv7 processor running a Linux kernel modified to
have a context switch every millisecond, ran a workload consisting of eight
benchmarks. With this workload, mispredict rate was recorded and graphed. The
results can be seen in Figure 1. As is demonstrated in the graph, there is a
significant spike in the mispredict rate at every context switch. These spikes
average around 200,000 cycles in length before reaching a steady state, which
then leaves room for improvement. In addition to this simulated workload, a
secondary, “worst-case” test was done in order to see what the worst possible
context switch might look like. In order to simulate this, two methods were
used. First, every ten thousand cycles, all branch prediction history was
inverting, meaning every taken was set to not-taken, and every not-taken was
set to taken. With this scenario, which is visualized in Figure 2, it can be
seen that the mispredict rate spikes to 60%, with mispredict rate spikes
decreasing with predictor size. In the second scenario, the entire history was
not inverted, but rather reset to the default predictor state. Similar results
were seen here, with mispredict rates spiking to nearly 40% and decreasing
linearly with predictor size. All in all, it is clear that context switching
does have an effect on the accuracy of the predictor, leaving space to improve
the predictor.

Figure 1: Branch misprediction rate vs. time for a multi-process benchmark
Figure 2: Transient response for Tournament branch predictors in response to
worst-case aliasing and reset

##  IV Description

When analyzing destructive behavior between contexts when using two-level
adaptive branch prediction schemes, we identified that the main source of
initial misprediction stems from saturation counters that have been flipped
since the last time the context ran in its time slice. The error arises when
the new context gets switched in, and begins executing, using the branch
history for a previous context’s execution patterns, instead of its own. This
insight into the poor behavior of branch predictors immediately after a
context switch was one of the guiding factors when developing our novel
approach to this problem.

Another factor that influenced our design was the fact that a reset of the
entire branch predictor’s state is often more detrimental than simply using a
previous processes branch predictor state, meaning that simply resetting the
entire branch predictor upon every context switch would not produce viable
results. Only in specific cases of destructive interference is using a
previous processes state more detrimental than a reset of branch predictor
state, meaning that our design needed to have some adaptive qualities to it.

Using these factors as the primary motivations for our design, we began
formulating our design. First, we recognized that resetting the entire
predictor state was a very destructive operation, especially in larger
predictor tables. During a time slice, we saw that for the benchmarks we ran,
there would often only be 15%-25% pattern history table (PHT) entry usage by a
specific program, with an even fewer number of PHT entries changing their
direction (from taken to not taken, or the other way around). Wiping all of
the PHT entries regardless of whether they actually changed since the last
time that specific program was given a time slice is wasteful. Instead, we
chose a more moderate approach, to only wipe the PHT entries that had changed
direction (meaning, the PHT entries that changed from taken to not-taken, and
vice versa) since the last time the program that is about to run. This ensured
that only branch predictor data that remains in the predictor is either reset
data, or data that pertains to the upcoming program, both of which are better
than the theoretical worst-case state of the predictor.

Secondly, we wanted to control this behavior dynamically, choosing whether to
reset the modified PHT entries if the behavior is deemed to be optimal or not.
To do this, we took inspiration from PHT tables themselves, by utilizing
saturated counters. These counters are each associated with a specific PID to
PID transition, and indicate whether the PHT entries that changed since the
last run should be reset or not. By only updating these counters when we see
behavior that is better or worse by a certain threshold, we remove noise and
random blips of branch misprediction data.

In order to update the saturated counters, we need to classify both good and
bad behavior. Building off of the insight into what causes mispredictions
after a context switch (changed PHT entries), we came up with a metric to
measure whether resetting the modified PHT entries was a desirable action or
not. By simply tracking the number of PHT changes that take place over the
course of a process’ time slice, we can compare this number to the previously
seen number of PHT entry changes. If it has changed by a certain threshold by
getting worse, then we invert the counter.

Updating of the saturated counters and choosing whether to reset the modified
PHT entries are both intrinsically linked to a context switch. Because of
this, both these actions should take place during each context switch that
occurs in the operating system. First, the previous transitions number of PHT
changes should be updated with the new value, and the counter should be
inverted or not, depending on whether it was a unfavorable or favorable
outcome, respectively. After the previous transition’s data is updated, we
look at the new transitions saturated counter, and reset the PHT entries that
were modified since the last time slice if the saturated counters indicate
that they should be.

##  V Methodology

To test this framework out to see how it performs when compared to baseline
runs without the framework, we implemented this framework within GEM5, a
commonly used system simulation tool.

To hold all of the various counters and keep track of the number of modified
PHT entries that changed, we chose to implement this list as a fixed size two
dimensional array. This array is indexed by (current_pid, next_pid) upon every
context switch, with an LRU replacement policy if the array is filled. Every
entry in the two dimensional array holds two items. The first piece of data is
the number of PHT entry changes that occurred after the last time the
corresponding transition was encountered. Secondly, each entry contains a
saturated counter, initialized to strongly not taken at the start.

Every context switch, two updates would occur in the two dimensional array.
Firstly, the previously seen transition’s entry needs to be updated (old_pid
$\rightarrow$ current_pid). If the previous transitions number of changes in
the PHT table was smaller by a certain threshold than the newly found number
of changes in the PHT table, then we classify this as worse behavior than what
we previously saw. In this case, we should invert the counter, so that the
opposite action is taken than previously taken, to see if that method yields
better data. If the behavior of the previously taken action is considered to
be better than what was stored, then we do nothing. We then update the stored
number of modified PHT entries for that particular PID to PID transition entry
in the array.

After we update the previously seen transition’s entry, we have to decide what
to do with the current transition (current_pid $\rightarrow$ next_pid). We
then look up the new transition’s entry in the table. If the counter indicates
taken, then we wipe the modified PHT entries, and continue execution as
normal.

If implemented in a real ARM system, one of the biggest hurdles to go from
simulation to a real working product is being aware of context switches, along
with getting information about process IDs. Luckily, this problem is easily
circumvented with a feature present on newer ARM processors, known as ARM
Software Thread ID registers [9]. By monitoring writes these registers,
notifying the branch prediction framework of a context switch is very
feasible.

##  VI Results

To form a baseline of how context switches effect multiple benchmarks, we
simulate context switching between multiple threads. Using the full-system
emulation mode [10], we simulated a single core ARM system using the
O3_ARM_v7a_3 CPU, running a standard Linux 3.13 kernel with 1 ms kernel tick
rate. Both the instantaneous misprediction for the prediction and the average
misprediction rate for each process was measured. The instantaneous
misprediction rate was measured using a 1000 branch sliding window. Large
spikes in the misprediction rate are observed on context switch boundaries.

To gauge the effectiveness of our algorithm of conditionally resetting the
predictor on destructive context switches, we compared the instantaneous
misprediction rate of our algorithm to baseline. Figure 4 shows the
differential misprediction rate in response to a context switch, with the
results tabulated in table I. The differential rate is calculated by
simulating a multi-process benchmark with and without conditional context
switch resets. For each time, the instantaneous misprediction rate is
subtracted, yielding an instantaneous differential misprediction rate. The
large negative spike in mispredicts show that instantaneous mispredictions are
reduced at a context switch boundary.

The average misprediction rate of each thread over the course of its execution
is shown in figure 3. We see that while some benchmarks see a marginal
improvement, others are not improved. The small improvement seen is likely due
to the infrequency of context switches in our traces (1 every millisecond).
Due to limitations in the Linux kernel, we were unable to simulate smaller
time slices. Interaction between threads determines to a large degree the
effectiveness of our algorithm. Misprediction is only improved if there is
significant aliasing within the predictor between threads. If the degree of
aliasing is small, resetting will do more harm by unnecessarily un-learning
any constructive or neutral aliasing. Additionally, if the predictor footprint
of a thread is small, resetting the predictor affects the performance of non-
contiguously scheduled processes.

Figure 3: Fraction of mispredicted branches per process when all are run together on a single core and context switched with a 1ms time slice Figure 4: Transient response of differential misprediction rate during a context switch Table I: Per-Process Misprediction Rates Benchmark | Branch Mispredictions (%)
---|---
| Baseline | CSAF | Always Reset
Bubblesort | 10.128 | 10.051 | 11.31
FloatMM | 3.095 | 3.136 | 3.572
IntMM | 3.418 | 3.449 | 4.024
Oscar | 3.811 | 3.786 | 4.136
Perm | 18.339 | 18.001 | 17.32
Puzzle | 3.854 | 3.849 | 4.09
Queens | 11.853 | 11.904 | 10.88
Quicksort | 14.638 | 14.679 | 14.699
RealMM | 3.481 | 3.454 | 4.17
Towers | 8.711 | 8.753 | 8.47
Treesort | 8.505 | 8.477 | 8.86

##  VII Future Work

In our simulations, we used a 128-entry BiMode predictor to enhance the
aliasing effects. Future work will involve simulations on larger predictors
that are more performant and prevalent in industry. Additionally, instead of
inverting the saturating counters on bad behavior, we can use the saturated
counters to their full potential by incrementing and decrementing them instead
of simply inverting them. In our trials, we found this did not provide much
benefit, and that the thresholding was enough to reset when destructive
behavior was identified. However, we did not simulate for very long (at a
maximum, 500,000,000 CPU cycles), which only encompassed a relatively small
number of context switches. Perhaps as a program’s behavior got better or
worse, using a saturated counter would have allowed for even less unnecessary
resets, improving the misprediction rate even further.

When looking at the full branch prediction scheme, most modifications that are
made generally change how the table operates or how to fundamentally change
the scheme to be more robust. With our findings, we posit a new methodology
that could be beneficial to prediction. Instead of modifying the branch
predictor by adding hardware components or wholly changing it, we could modify
a critical portion of the scheme. Based on recent work by Kraska et al. titled
“The Case for Learned Index Structures” [11], we can see that there is a
significant performance in using machine learning to learn the hash function
into a map, or the access pattern to an array. In the same lieu, the idea
could be applied to branch predictors. Instead of using a static hash function
for the predictor table, we can learn the best access pattern to the branch
history table. This could potentially significantly improve accuracy as
various parameters could be provided to the learning hash function, such as
context, to improve performance.

## References

  * [1] T. Yeh and Y. Patt, ”Two-Level Adaptive Training Branch Prediction”, in _Proc. 24th Annual International Symposium in Microarchitecture_ , 1991.
  * [2] C. Lee, I. Chen and T. Mudge, ”The Bi-Mode Branch Predictor”, 1997.
  * [3] S. McFarling, ”Combining Branch Predictors”, in _digital_ , Palof Alto, California, 1993.
  * [4] W. Hwu and T. Conte, ”The susceptibility of programs to context switching”, in _IEEE Transactions on Computers_ , 1994, pp. 994-1003.
  * [5] J. Mogul and A. Borg, ”The Effect of Context Switches on Cache Performance”, in _digital_ , Palo Alto, California, 1990.
  * [6] M. Co and K. Skadron, ”The Effects of Context Switching on Branch Predictor Performance”, 2001.
  * [7] J. Chen, M. Smith, C. Young and N. Gloy, ”An Analysis of Dynamic Branch Prediction Schemes on System Workloads”, Philadelphia, 1996.
  * [8] P. Chang, Y. Patt and M. Evers, ”Using Hybrid Branch Predictors to Improve Branch Prediction Accuracy in the Presence of Context Switches”, Philadelphia, 1996.
  * [9] ”ARM Information Center”, Infocenter.arm.com, 2017. [Online]. Available: http://infocenter.arm.com/help/index.jsp. [Accessed: 15- Dec- 2017].
  * [10] ”gem5 Full System Simulation — gem5 Tutorial 0.1 documentation”, Learning.gem5.org, 2017. [Online]. Available: http://learning.gem5.org/book/part4/intro.html. [Accessed: 15- Dec- 2017].
  * [11] T. Kraska, A. Beutel, E. Chi, J. Dean and N. Polyzotis, ”The Case for Learned Index Structures”, 2017.

|  John Doe Hello, here is some text without a meaning. This text should show
what a printed text will look like at this place. If you read this text, you
will get no information. Really? Is there no information? Is there a
difference between this text and some nonsense like “Huardest gefburn”? Kjift
– not at all! A blind text like this gives you information about the selected
font, how the letters are written and an impression of the look. This text
should contain all letters of the alphabet and it should be written in of the
original language. There is no need for special content, but the length of
words should match the language.
---|---


11institutetext: CentraleSupélec, Paris-Saclay University, Mathématiques et
Informatique pour la Complexité et les Systèmes, Gif-sur-Yvette, France.
22institutetext: Radiology Department, Hôpital Cochin AP-HP, Paris, France
33institutetext: Inria Saclay, Gif-sur-Yvette, France 44institutetext: Centre
National de Recherche Scientifique, LIP6, Paris, France 55institutetext:
Therapanacea, Paris, France

# Deep Reinforcement Learning for L3 Slice Localization in Sarcopenia
Assessment

Othmane Laousy 112233 Guillaume Chassagnon 22 Edouard Oyallon 44 Nikos
Paragios 55 Marie-Pierre Revel 22 Maria Vakalopoulou 1133

###### Abstract

Sarcopenia is a medical condition characterized by a reduction in muscle mass
and function. A quantitative diagnosis technique consists of localizing the CT
slice passing through the middle of the third lumbar area (L3) and segmenting
muscles at this level. In this paper, we propose a deep reinforcement learning
method for accurate localization of the L3 CT slice. Our method trains a
reinforcement learning agent by incentivizing it to discover the right
position. Specifically, a Deep Q-Network is trained to find the best policy to
follow for this problem. Visualizing the training process shows that the agent
mimics the scrolling of an experienced radiologist. Extensive experiments
against other state-of-the-art deep learning based methods for L3 localization
prove the superiority of our technique which performs well even with a limited
amount of data and annotations.

###### Keywords:

L3 slice CT slice localization Deep Reinforcement Learning Sarcopenia

##  1 Introduction

Sarcopenia corresponds to muscle atrophy which may be due to ageing,
inactivity, or disease. The decrease of skeletal muscle is a good indicator of
the overall health state of a patient [11]. In oncology, it has been shown
that sarcopenia is linked to outcome in patients treated by chemotherapy [14,
5], immunotherapy [20], or surgery [9]. There are multiple definitions of
sarcopenia [7, 23] and consequently multiple ways of assessing it. On CT
imaging, the method used is based on muscle mass quantification. Muscle mass
is most commonly assessed at a level passing through the middle of the third
lumbar vertebra area (L3), which has been found to be representative of the
body composition [29]. After manual selection of the correct CT slice at the
L3 level, segmentation of muscles is performed to calculate the skeletal
muscle area [8]. In practice, the evaluation is tedious, time-consuming, and
rarely done by radiologists, highlighting the need for an automatic diagnosis
tool that could be integrated into clinical practice. Such automated
measurement of muscle mass could be of great help for introducing sarcopenia
assessment in daily clinical practice.

Muscle segmentation and quantification on a single slice have been thoroughly
addressed in multiple works using simple 2D U-Net like architectures [4, 6].
Few works, however, focus on L3 slice detection. The main challenges for
solving this task rely on the inherent diversity in patient’s anatomy, the
strong resemblance between vertebrae, the variability of CT fields of view as
well as their acquisition and reconstruction protocols.

The most straightforward approach to address L3 localization is by
investigating methods for multiple vertebrae labeling in 3D images using
detection [25] or even segmentation algorithms [22]. Such methods require a
substantial volume of annotations and are computationally inefficient when
dealing with the entire 3D CT scan. In fact, even if our input is 3D, a one-
dimensional output as the $z$-coordinate of the slice is sufficient to solve
the L3 localization problem.

In terms of L3 slice detection, the closest methods leverage deep learning [3,
13] and focus on training simple convolutional neural networks (CNN). These
techniques use maximal intensity projection (MIP), where the objective is to
project voxels with maximal intensity values into a 2D plane. Frontal view MIP
projections contain enough information towards the body and vertebra’s bone
structure differentiation. On the sagittal view, restricted MIP projections
are used to focus solely on the spinal area. In [3] the authors tackle this
problem through regression, training the CNN with parts of the MIP that
contain the L3 vertebra only. More recently, in [13] a UNet-like architecture
(L3UNet-2D) is proposed to draw a 2D confidence map over the position of the
L3 slice.

In this paper, we propose a reinforcement learning algorithm for accurate
detection of the L3 slice in CT scans, automatizing the process of sarcopenia
assessment. The main contribution of our paper is a novel formulation for the
problem of L3 localization, exploiting different deep reinforcement learning
(DRL) schemes that boost the state of the art for this challenging task, even
on scarce data settings. Moreover, in this paper we demonstrate that the use
of 2D approaches for vertebrae detection provides state of the art results
compared to 3D landmark detection methods, simplifying the problem, reducing
the search space and the amount of annotations needed. To the best of our
knowledge, this is the first time a reinforcement learning algorithm is
explored on vertebrae slice localization, reporting performances similar to
medical experts and opening new directions for this challenging task.

##  2 Background

Reinforcement Learning is a fundamental tool of machine learning which allows
dealing efficiently with the exploration/exploitation trade-off [24]. Given
state-reward pairs, a reinforcement learning agent can pick actions to reach
unexplored states or increase its accumulated future reward. Those principles
are appealing for medical applications because they imitate a practitioner’s
behavior and self-learn from experience based on ground-truth. One of the main
issues of this class of algorithm is its sample complexity: a large amount of
interaction with its environment is needed before obtaining an agent close to
an optimal state [26]. However, those techniques were recently combined with
deep learning approaches, which efficiently addressed this issue [17] by
incorporating priors based on neural networks. In the context of highly-
dimensional computer vision applications, this approach allowed RL algorithms
to obtain outstanding accuracy [12] in a variety of tasks and applications.

In medical imaging, model-free reinforcement learning algorithms are highly
used for landmark detection [10] as well as localization tasks [16]. In [1], a
Deep Q-Network (DQN) that automates the view planning process on brain and
cardiac MRI was proposed. This framework takes as an input a single plane and
updates its angle and position during the training process until convergence.
Moreover, in [27] the authors again present a DQN framework for the
localization of different anatomical landmarks introducing multiple agents
that act and learn simultaneously. DRL has also been studied for object or
lesion localization. More recently, in [19] the authors propose a DQN
framework for the localization of $6$ different organs from CT scans achieving
a performance comparable to supervised CNNs. This framework uses a $3$D volume
as input with $11$ different actions to generate bounding boxes for these
organs. Our work is the first to explore and validate a RL scheme on MIP
representations for a single slice detection using the discrete and 2D nature
of the problem.

##  3 Reinforcement Learning Strategy

In this paper, we formulate the slice localization problem as a Markov
Decision Process (MDP), which contains a set of states $S$, actions $A$, and
rewards $R$.

States $S$: For our formulation, the environment $\mathcal{E}$ that we explore
and exploit is a $2$D image representing the frontal MIP projection of the
$3$D CT scans. This projection allows us to reduce our problem’s
dimensionality from a volume of size $512\times 512\times N$ ($N$ being the
varying heights of the volumes) to an image of size $512\times N$. The
reinforcement learning agent is self-taught by interacting with this
environment, executing a set of actions, and receiving a reward linked to the
action taken. An input example is shown in Figure 1. We define a state $s\in
S$ as an image of size $512\times 200$ in $\mathcal{E}$. We consider the
middle of the image to be the slice’s current position on a $z$-axis. To
highlight this, we assign a line of maximum intensity pixel value to the
middle of each image provided as input to our DQN.

Actions $A$: We define a set of discrete actions
$\mathcal{A}=\\{t^{+}_{z},t^{-}_{z}\\}\in\mathbb{R}^{2}$. $t^{+}_{z}$
corresponds to a positive translation (going up by one slice) and $t^{-}_{z}$
corresponds to a negative translation (going down by one slice). These two
actions allow us to explore the entirety of our environment $\mathcal{E}$.

Rewards $R$: In reinforcement learning designing a good reward function is
crucial in learning the goal to achieve. To measure the quality of taking an
action $a\in A$ we use the distance over $z$ between the current slice and the
annotated slice $g$. The reward for non-terminating states is computed with:

$R_{a}(s,s^{\prime})=sign(\mathcal{D}(p,g)-\mathcal{D}(p^{\prime},g))$ (1)

where we denote as $s$ and $s^{\prime}$ the current and next state and $g$ the
ground truth annotation. Moreover, our positions $p$ and $p^{\prime}$ are the
$z$-coordinates of the current and next state respectively. $\mathcal{D}$ is
the Euclidean distance between both coordinates over the $z$-axis. The reward
is non-sparse and binary $r\in\\{-1,+1\\}$ and helps the agent differentiate
between good and bad actions. A good action being when the agent gets closer
to the correct slice. For a terminating state, we assign a reward of $r=0.5$.

Starting States: An episode starts by randomly sampling a slice over the
$z$-axis and ends when the agent has achieved its goal of finding the right
slice. The agent then executes a set of actions and collects rewards until the
episode terminates. When reaching the upper or lower borders of an image, the
current state is assigned to the next state (i.e., the agent does not move),
and a reward of $r=-1$ is appointed to this action.

Final States: During training, a terminal state is defined as a state in which
the agent has reached the right slice. A reward of $r=0.5$ is assigned in this
case, and an episode is terminated. During testing, the termination of an
episode happens when oscillations occur. We adopted the same approach as [1],
and chose actions with the lowest $Q$-value, which have been found to be
closest to the right slice since the DQN outputs higher $Q$-values to actions
when the current slice is far from the ground truth.

###  3.1 Deep Q-Learning

To find the optimal policy $\pi^{*}$ of the MDP, a state-action value function
$Q(s,a)$ can be learned. In Q-Learning, the expected value of the accumulated
discounted future rewards can be estimated recursively using the Bellman
optimality equation:

$Q_{i+1}(s,a)=\mathbb{E}[r+\gamma\underset{a^{\prime}}{\mathrm{max}}\>Q_{i}(s^{\prime},a^{\prime})\mid
s,a]$ (2)

In practice, since the state $S$ is not easily exploitable, we can take
advantage of neural networks as universal function approximators to
approximate $Q(s,a)$ [18]. We utilize an experience replay technique that
consists in storing the agent’s experience
$e_{t}=(s,a,r,s^{\prime},a^{\prime})$ at each time step in a replay memory
$M$. To break the correlation between consecutive samples, we will uniformly
batch a set of experiences from $M$. The Deep Q-Network (DQN) will iteratively
optimize its parameters $\theta$ by minimizing the following loss function:

$L_{i}(\theta_{i})=\mathbb{E}_{(s,a,r,s^{\prime},a^{\prime})}\Big{[}{\big{(}r+\gamma\underset{a^{\prime}}{\mathrm{max}}\>Q_{target}(s^{\prime},a^{\prime};\theta_{i}^{-})-Q_{policy}(s,a;\theta_{i})\big{)}}^{2}\Big{]}$
(3)

with $\theta_{i}$ and $\theta_{i}^{-}$ being the parameters of the policy and
the target network respectively. To stabilize rapid policy changes due to the
distribution of the data and the variations in Q-values, the DQN uses
$Q_{target}(\theta_{i}^{-})$, a fixed version of $Q_{policy}(\theta_{i})$ that
is updated periodically. For our experiments, we update $\theta_{i}^{-}$ every
50 iterations.

###  3.2 Network Architecture

Our Deep Q-Network takes as input the state $s$ and passes it through a
convolutional network. The network contains four convolution layers separated
by parametric ReLU in order to break the linearity of the network, and four
linear layers with LeakyReLU. Contrary to [1], we chose not to add the history
of previously visited states in our case. We opted for this approach since
there is a single path that leads to the right slice. This approach allows us
to simplify our problem even more. Ideally, our agent should learn, just by
looking at the current state, whether to go up or down when the current slice
is respectively below or above the L3 slice. An overview of our framework is
presented in Figure 1.

Figure 1: The implemented Deep Q-Network architecture for L3 slice
localization. The network takes as input an image of size $512\times 200$ with
a single channel. The output is the q-values corresponding to each of the two
actions.

We also explore dueling DQNs from [28]. Dueling DQNs rely on the concept of an
advantage which calculates the benefit that each action can provide. The
advantage is defined as $A(s,a)=Q(s,a)-V(s)$ with $V(s)$ being our state value
function. This algorithm will use the advantage of the Q-values to distinguish
between actions from the state’s baseline values. Dueling DQNs were shown to
provide more robust agents that are wiser in choosing the next best action.
For our dueling DQN, we use the same architecture as the one in Figure 1 but
change the second to last fully connected layer to compute state values on one
side, and action values on the other.

###  3.3 Training

Since our agent is unaware of the possible states and rewards in
$\mathcal{E}$, the exploration step is implemented first. After a few
iterations, our agent can start exploiting what it has learned on
$\mathcal{E}$. In order to balance between exploration and exploitation, we
use an $\epsilon$-greedy strategy. This strategy consists of defining an
exploration rate $\epsilon$, which is initialed to $1$ with a decay of $0.1$,
allowing the agent to become greedy and exploit the environment. A batch size
of $48$ and an experience replay of $17\times 10^{3}$ are used. The entire
framework was developed in Pytorch [21] library using an NVIDIA GTX 1080Ti
GPU. We trained our model for $10^{5}$ episodes, requiring approximately
$20$-$24$ hours. Our source code is available on GitHub: https://git.io/JRyYw.

##  4 Experiments and Results

###  4.1 Dataset

A diverse dataset of $1000$ CT scans has been retrospectively collected for
this study. CT scans were acquired on four different CT models from three
manufacturers (Revolution HD from GE Healthcare, Milwaukee, WI; Brillance 16
from Philips Healthcare, Best, Netherlands; and Somatom AS+ & Somatom Edge
from Siemens Healthineer, Erlangen, Germany). Exams were either abdominal,
thoracoabdominal, or thoraco-abdominopelvic CT scans acquired with or without
contrast media injection. Images were reconstructed using abdominal kernel
with either filtered back-projection or iterative reconstruction. Slice
thickness ranged from $0.625$ to $3$mm, and the number of slices varied from
$121$ to $1407$. The heterogeneity of our dataset highlights the challenges of
the problem from a clinical perspective.

Experienced radiologists manually annotated the dataset, indicating the
position of the middle of the L3 slice. Before computing the MIP, all of the
CT scans are normalized to $1mm$ over the $z$-axis. This normalisation step
harmonises our network’s input, especially since the agent performs actions
along the $z$-axis. After the MIP, we apply a threshold of $100$ HU
(Hounsfield Unit) to $1500$ HU allowing us to eliminate artifacts and foreign
metal bodies while keeping the skeleton structure. The MIP are finally
normalized to [0,1]. From the entire dataset, we randomly selected $100$
patients for testing and the rest $900$ for training and validation. For the
testing cohort, annotations of L3 from a second experienced radiologist have
been provided to measure the interobserver performance.

###  4.2 Results and Discussion

Our method is compared with other techniques from the literature. The error is
calculated as the distance in millimeters ($mm$) between the predicted L3
slice and the one annotated by the experts. In particular, we performed
experiments with the L3UNet-2D [13] approach and the winning SC-Net [22]
method of the Verse2020111https://verse2020.grand-challenge.org/ challenge.
Even if SC-Net is trained on more accurate annotations with 3D landmarks as
well as vertebrae segmentations, and addresses a different problem, we applied
it to our testing cohort. The comparison of the different methods is
summarised in Table 1. SC-Net reports $12$ CT scans with an error higher than
$10$mm. Moreover, L3UNet-2D [13] reports a mean error of $4.24$mm $\pm 6.97$mm
when the method is trained on the entire training set, giving only $7$ scans
with an error higher than $10$mm for the L3 detection. Our proposed method
gives the lowest errors with a mean error of $3.77$mm$\pm 4.71$mm, proving its
superiority. Finally, we evaluated our technique’s performance with a Duel DQN
strategy, reporting higher errors than the proposed one. This observation
could be linked to the small action space that is designed for this study.
Duel DQNs were proven to be powerful in cases with higher action spaces and in
which the computation of the advantage function makes a difference.

Table 1: Quantitative evaluation of the different methods using different number of training samples (metrics in mm). Method | # of samples | Mean | Std | Median | Max | Error $>10mm$
---|---|---|---|---|---|---
Interobserver | - | 2.04 | 4.36 | 1.30 | 43.19 | 1
SC-Net [22] | - | 6.78 | 13.96 | 1.77 | 46.98 | 12
L3UNet-2D [13] | 900 | 4.24 | 6.97 | 2.19 | 40 | 7
Ours (Duel-DQN) | 900 | 4.30 | 5.59 | 3 | 38 | 8
Ours | 900 | 3.77 | 4.71 | 2.0 | 24 | 9
L3UNet-2D [13] | 100 | 145.37 | 161.91 | 32.8 | 493 | 68
Ours | 100 | 5.65 | 5.83 | 4 | 26 | 19
L3UNet-2D [13] | 50 | 108.7 | 97.33 | 87.35 | 392.02 | 86
Ours | 50 | 6.88 | 5.79 | 6.5 | 26 | 11
L3UNet-2D [13] | 10 | 242.85 | 73.07 | 240.5 | 462 | 99
Ours | 10 | 8.97 | 8.72 | 7 | 56 | 33

For the proposed reinforcement learning framework, trained on the whole
training set, 9 CTs had a detection error of more than $10$mm. These scans
were analysed by a medical expert who indicated that $2$ of them have a
lumbosacral transitional vertebrae (LSTV) anomaly [15]. Transitional vertebrae
cases are common and observed in 15-35% of the population [2] highlighting
once again the challenges of this task. For both cases, the localization of
the L3 vertebra for sarcopenia assessment is ambiguous for radiologists and
consequently for the reinforcement learning agent. In fact, the only error
higher than $10mm$ in the interobserver comparison corresponds to an LSTV case
where each radiologist chose a different vertebrae as a basis for sarcopenia
assessment. Even if the interobserver performance is better than the one
reported by the algorithms, our method reports the lowest errors, proving its
potential.

Qualitative results are displayed in Figure 2. The yellow line represents the
medical expert’s annotation and the blue one the prediction of the different
employed models. One can notice that all of the different methods converge to
the correct L3 region with our method reporting great performance. It is
important to note that for sarcopenia assessment, an automatic system does not
need to be at the exact middle of the slice; a few millimeters around will not
skew the end result since muscle mass in the L3 zone does not change
significantly. Concerning prediction times for the RL agent, they depend on
the initial slice that is randomly sampled on the MIP. Computed inference time
for a single step is approximately 0.03 seconds.

Figure 2: Qualitative comparison of different localization methods for two
patients. First left to right represents: interobserver ($4mm$/ $2mm$), SC-Net
($4mm$/$2mm$), L3UNet-2D ($8mm$/$1mm$), Ours ($1mm$/$4mm$). In the parenthesis
we present the reported errors for the first and second row respectively. The
yellow line represents the ground truth and the blue one the prediction.

To highlight the robustness of our network on a low number of annotated
samples, we performed different experiments using 100, 50, and 10 CTs
corresponding respectively to 10%, 5% and 1% of our dataset. We tested those 3
agents on the same 100 patients test set and report results in Table 1. Our
experiments prove the robustness of reinforcement learning algorithms compared
to traditional CNN based ones [13] in the case of small annotated datasets.
One can observe that the traditional methods fail to be trained properly with
a small number of annotations, reporting errors higher than $100$mm for all
three experiments. In our case, decreasing the dataset size does not
significantly affect the performance. In fact, trained on only 10 CTs with the
same number of iterations and memory size, our agent was able to learn a
correct policy and achieve a mean error of $8.97$mm $\pm$ $8.72$mm. Learning a
valid policy from a low number of annotations is one of the strengths of
reinforcement learning. Traditional deep learning techniques rely on pairs of
images and annotations in order to build a robust generalization. Thus, each
pair is exploited only once by the learning algorithm. Reinforcement learning,
however, relies on experiences, each experience
$e_{t}=(s,a,r,s^{\prime},a^{\prime})$ being a tuple of state, action, reward,
next state and next action. Therefore, a single CT scan can provide multiple
experiences to the self-learning agent, making our method ideal for slice
localization problems using datasets with a limited amount of annotations.

##  5 Conclusion

In this paper, we propose a novel direction to address the problem of CT slice
localization. Our experiments empirically prove that reinforcement learning
schemes work very well on small datasets and boost performance compared to
classical convolutional architectures. One limitation of our work lies in the
fact that our agent is always moving $1mm$ independently of the location,
slowing down the process. In the future, we aim to explore different ways to
adapt the action taken depending on the current location, with one possibility
being to incentivize actions with higher increments. Future work also includes
the use of reinforcement learning in multiple vertebrae detection with
competitive or collaborative agents.

## References

  * [1] Alansary, A., Le Folgoc, L., Vaillant, G., Oktay, O., Li, Y., Bai, W., Passerat-Palmbach, J., Guerrero, R., Kamnitsas, K., Hou, B., et al.: Automatic view planning with multi-scale deep reinforcement learning agents. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 277–285. Springer (2018)
  * [2] Apazidis, A., Ricart, P.A., Diefenbach, C.M., Spivak, J.M.: The prevalence of transitional vertebrae in the lumbar spine. The Spine Journal 11(9), 858–862 (2011)
  * [3] Belharbi, S., Chatelain, C., Hérault, R., Adam, S., Thureau, S., Chastan, M., Modzelewski, R.: Spotting l3 slice in ct scans using deep convolutional network and transfer learning. Computers in biology and medicine 87, 95–103 (2017)
  * [4] Blanc-Durand, P., Schiratti, J.B., Schutte, K., Jehanno, P., Herent, P., Pigneur, F., Lucidarme, O., Benaceur, Y., Sadate, A., Luciani, A., Ernst, O., Rouchaud, A., Creze, M., Dallongeville, A., et al.: Abdominal musculature segmentation and surface prediction from ct using deep learning for sarcopenia assessment. Diagnostic and Interventional Imaging 101(12), 789–794 (2020)
  * [5] Bozzetti, F.: Forcing the vicious circle: sarcopenia increases toxicity, decreases response to chemotherapy and worsens with chemotherapy. Annals of Oncology 28(9), 2107–2118 (2017), a focus on esophageal squamous cell carinoma
  * [6] Castiglione, J., Somasundaram, E., Gilligan, L.A., Trout, A.T., Brady, S.: Automated segmentation of abdominal skeletal muscle in pediatric ct scans using deep learning. Radiology: Artificial Intelligence p. e200130 (2021)
  * [7] Cruz-Jentoft, A.J., Bahat, G., Bauer, J., Boirie, Y., Bruyère, O., Cederholm, T., Cooper, C., Landi, F., Rolland, Y., Sayer, A.A., et al.: Sarcopenia: revised european consensus on definition and diagnosis. Age and ageing 48(1), 16–31 (2019)
  * [8] Derstine, B.A., Holcombe, S.A., Goulson, R.L., Ross, B.E., Wang, N.C., Sullivan, J.A., Su, G.L., Wang, S.C.: Quantifying Sarcopenia Reference Values Using Lumbar and Thoracic Muscle Areas in a Healthy Population. J Nutr Health Aging 21(10), 180–185 (2017)
  * [9] Du, Y., Karvellas, C.J., Baracos, V., Williams, D.C., Khadaroo, R.G.: Sarcopenia is a predictor of outcomes in very elderly patients undergoing emergency surgery. Surgery 156(3), 521–527 (2014)
  * [10] Ghesu, F.C., Georgescu, B., Grbic, S., Maier, A., Hornegger, J., Comaniciu, D.: Towards intelligent robust detection of anatomical structures in incomplete volumetric data. Med Image Anal 48, 203–213 (08 2018)
  * [11] Gilligan, L.A., Towbin, A.J., Dillman, J.R., Somasundaram, E., Trout, A.T.: Quantification of skeletal muscle mass: sarcopenia as a marker of overall health in children and adults. Pediatric Radiology 50(4), 455–464 (Apr 2020)
  * [12] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., et al.: Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)
  * [13] Kanavati, F., Islam, S., Aboagye, E.O., Rockall, A.: Automatic l3 slice detection in 3d ct images using fully-convolutional networks (2018)
  * [14] Lee, J., Chang, C.L., Lin, J.B., Wu, M.H., Sun, F.J., Jan, Y.T., Hsu, S.M., Chen, Y.J.: Skeletal muscle loss is an imaging biomarker of outcome after definitive chemoradiotherapy for locally advanced cervical cancer. Clinical Cancer Research 24(20), 5028–5036 (2018)
  * [15] Lian, J., Levine, N., Cho, W.: A review of lumbosacral transitional vertebrae and associated vertebral numeration. European Spine Journal 27(5), 995–1004 (May 2018)
  * [16] Maicas, G., Carneiro, G., Bradley, A.P., Nascimento, J.C., Reid, I.: Deep reinforcement learning for active breast lesion detection from dce-mri. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2017. pp. 665–673. Springer International Publishing, Cham (2017)
  * [17] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
  * [18] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (Feb 2015)
  * [19] Navarro, F., Sekuboyina, A., Waldmannstetter, D., Peeken, J.C., Combs, S.E., Menze, B.H.: Deep reinforcement learning for organ localization in ct. In: Medical Imaging with Deep Learning. pp. 544–554. PMLR (2020)
  * [20] Nishioka, N., Uchino, J., Hirai, S., Katayama, Y., Yoshimura, A., Okura, N., Tanimura, K., Harita, S., Imabayashi, T., Chihara, Y., Tamiya, N., Kaneko, Y., Yamada, T., Takayama, K.: Association of sarcopenia with and efficacy of anti-pd-1/pd-l1 therapy in non-small-cell lung cancer. Journal of Clinical Medicine 8(4) (2019)
  * [21] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)
  * [22] Payer, C., Štern, D., Bischof, H., Urschler, M.: Coarse to fine vertebrae localization and segmentation with spatialconfiguration-net and u-net. In: Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP. vol. 5, pp. 124–133 (2020). https://doi.org/10.5220/0008975201240133
  * [23] Santilli, V., Bernetti, A., Mangone, M., Paoloni, M.: Clinical definition of sarcopenia. Clinical cases in mineral and bone metabolism 11(3), 177 (2014)
  * [24] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)
  * [25] Suzani, A., Seitel, A., Liu, Y., Fels, S., Rohling, R.N., Abolmaesumi, P.: Fast automatic vertebrae detection and localization in pathological ct scans - a deep learning approach. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. pp. 678–686. Springer International Publishing (2015)
  * [26] Tarbouriech, J., Garcelon, E., Valko, M., Pirotta, M., Lazaric, A.: No-regret exploration in goal-oriented reinforcement learning. In: International Conference on Machine Learning. pp. 9428–9437. PMLR (2020)
  * [27] Vlontzos, A., Alansary, A., Kamnitsas, K., Rueckert, D., Kainz, B.: Multiple landmark detection using multi-agent reinforcement learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 262–270. Springer (2019)
  * [28] Wang, Z., de Freitas, N., Lanctot, M.: Dueling network architectures for deep reinforcement learning. CoRR abs/1511.06581 (2015), http://arxiv.org/abs/1511.06581
  * [29] Zopfs, D., Theurich, S., Große Hokamp, N., Naetlitz, J., Gerecht, L., Borggrefe, J., Schlaak, M., Pinto dos Santos, D.: Single-slice ct measurements allow for accurate assessment of sarcopenia and body composition. European radiology 30 (11 2019)