# Oscillations of non-slender tori in the external Hartle-Thorne geometry

Monika Matuszková  1,a Kateřina Klimovičová  1,b Gabriela Urbancová  1,c
Debora Lančová  1,d Eva Šrámková  []1 and Gabriel Török  []1
1Research Centre for Computational Physics and Data Processing  [1]Institute
of Physics  Silesian University in Opava  Bezručovo nám. 13  [1]CZ-746 01
Opava  Czech Republic
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We examine the influence of the quadrupole moment of a slowly rotating neutron
star on the oscillations of non-slender accretion tori. We apply previously
developed methods to perform analytical calculations of frequencies of the
radial epicyclic mode of a torus in the specific case of the Hartle-Thorne
geometry. We present here our preliminary results and provide a brief
comparison between the calculated frequencies and the frequencies previously
obtained assuming both standard and linearized Kerr geometry. Finally, we
shortly discuss the consequences for models of high-frequency quasi-periodic
oscillations observed in low-mass X-ray binaries.

###### keywords:

neutron star – thick accretion disc – Hartle-Thorne metric

##  1 Introduction

Numerous interesting features have been discovered during the long history of
X-ray observations of low-mass X-ray binaries (LMXBs). One of them is the fact
that variability of the X-ray radiation coming from these sources occurs at
frequencies in the order of up to hundreds of Hertz with the highest values
reaching above 1.2 kHz. Even though the discovery of this rapid variability
was made almost 30 years ago, to this day, there is no convincing explanation
of its origin. The phenomenon is called the high-frequency quasi-periodic
oscillations (HF QPOs) and many models have been proposed in the attempt to
explain its nature (see, e.g., Török et al., 2016a; Kotrlová et al., 2020, and
references therein).

It has been noticed that the HF QPOs frequencies are in the same order as
those corresponding to orbital motion in the very close vicinity of a compact
object, such as neutron star (NS) or black hole (BH). This suggests that there
is a relation between the QPO phenomenon and the physics behind the motion of
matter close to the accreting object. Since positions of specific orbits in
the accretion disk (such as its inner edge) and the associated orbital
frequencies depend on the properties of the central object, there is a believe
that it is possible to infer the compact object properties from the QPOs
data.111We often use the shorter term ”QPOs” instead of ”HF QPOs” throughout
the paper.

In the above context, several studies have focused on a possible relation
between the QPOs and an oscillatory motion of an accretion torus formed in the
innermost accretion region (Kluzniak and Abramowicz, 2001; Kluźniak et al.,
2004; Abramowicz et al., 2003a, b; Rezzolla et al., 2003; Bursa, 2005; Török
et al., 2005; Dönmez et al., 2011; Török et al., 2016a; de Avellar et al.,
2018).

Straub and Šrámková (2009) and Fragile et al. (2016) have performed
calculations of frequencies of the epicyclic oscillations of fluid tori
assuming Kerr geometry, which describes rotating BHs. Here we follow their
approach and consider slowly rotating NSs and their spacetimes described by
the Hartle-Thorne geometry (Hartle, 1967; Hartle and Thorne, 1968). We present
the first, preliminary results of our calculations of the radial epicyclic
oscillation frequencies and provide a brief comparison of these to the
frequencies obtained previously for the Kerr and linearized Kerr geometries.
Finally, we discuss some consequences for models of NS QPOs.

##  2 Oscillations of tori in axially symmetric spacetimes

We consider an axially symmetric geometry. The spacetime element may be
expressed in the general form as

$\displaystyle\mathrm{d}s^{2}=g_{tt}\mathrm{d}t^{2}+2g_{t\varphi}\mathrm{d}t\mathrm{d}\varphi+g_{rr}\mathrm{d}r^{2}+g_{\theta\theta}\mathrm{d}\theta^{2}+g_{\varphi\varphi}\mathrm{d}\varphi^{2}.$
(1)

We use the units in which $c=G=1$ with $c$ being the speed of light and $G$
the gravitational constant.

###  2.1 Equilibrium configuration

We assume a perfect fluid torus in the state of pure rotation with constant
specific angular momentum $l$ as described in Abramowicz et al. (2006); Blaes
et al. (2006).

In this case, the fluid forming the torus has a four-velocity $u^{\mu}$ with
only two non-zero components,

$u^{\mu}=A(1,0,0,\Omega),$ (2)

where $A$ is the time component $u^{t}$ and $\Omega$ is the orbital velocity.
One may write

$\displaystyle A=$ $\displaystyle u^{t}=(-g_{tt}-2\Omega
g_{t\varphi}-\Omega^{2}g_{\varphi\varphi})^{-1/2},$ (3) $\displaystyle\Omega=$
$\displaystyle\dfrac{u^{\varphi}}{u^{t}}=\frac{g^{t\varphi}-lg^{\varphi\varphi}}{g^{tt}-lg^{t\varphi}}.$
(4)

The perfect fluid with density $\rho$, pressure $p$ and the energy density $e$
is characterised by the stress-energy tensor

$T^{\mu\nu}=(p+e)u^{\mu}u^{\nu}+pg^{\mu\nu}.$ (5)

For a polytropic fluid, we may write:

$\displaystyle p$ $\displaystyle=K\rho^{\frac{n+1}{n}},$ (6) $\displaystyle e$
$\displaystyle=np+\rho,$ (7)

where $K$ and $n$ denote the polytropic constant and the polytropic index,
respectively. In this work, we use $n=3$, which describes a radiation-
pressure-dominated torus.

The Euler formula is obtained from the energyâmomentum conservation law,
$\nabla_{\mu}T^{\mu}_{\;\ \;\nu}=0$, using the assumption of $l=const.$
(Abramowicz et al., 1978, 2006)

$\nabla_{\mu}(\ln\mathscr{E})=-\frac{\nabla_{\mu}p}{p+e},$ (8)

with $\mathscr{E}$ being the specific energy

$\mathscr{E}=-u_{t}=\left(-g^{tt}+2lg^{t\varphi}-l^{2}g^{\varphi\varphi}\right)^{-1/2}.$
(9)

By integrating (8) we obtain the Bernoulli equation (Fragile et al., 2016;
Horák et al., 2017)

$H\mathscr{E}=const.,$ (10)

where $H=\frac{p+e}{\rho}$ denotes the enthalpy in the form presented by
Fragile et al. (2016) and Horák et al. (2017). From relation (10), we can
derive the equations describing the structure and shape of the torus:

$\displaystyle\frac{p}{\rho}$
$\displaystyle=\frac{p_{0}}{\rho_{0}}f(r,\theta),$ (11) $\displaystyle
f(r,\theta)$
$\displaystyle=\frac{1}{nc_{\mathrm{s},0}^{2}}\left[\left(1+nc_{\mathrm{s},0}^{2}\right)\frac{\mathscr{E}_{0}}{\mathscr{E}}-1\right],$
(12)

where $c_{\mathrm{s}}$ is the sound speed in the fluid defined as (Abramowicz
et al., 2006)222The definition is fully valid for $c_{\mathrm{s}}<<1$, but
this has no significant effect on our results.

$c_{\mathrm{s}}^{2}=\frac{\partial
p}{\partial\rho}=\frac{n+1}{n}\frac{p}{\rho},$ (14)

and the subscript $0$ denotes the quantities evaluated at the torus centre.
From equations (6) and (11), one can obtain the following formulae for
pressure and density of the fluid:

$\displaystyle p$
$\displaystyle=p_{0}\left[f\left(r,\theta\right)\right]^{n+1},$ (15)
$\displaystyle\rho$
$\displaystyle=\rho_{0}\left[f\left(r,\theta\right)\right]^{n}.$ (16)

It is useful to introduce new coordinates $\overline{x}$ and $\overline{y}$ by
relations

$\displaystyle\overline{x}=$
$\displaystyle\frac{\sqrt{g_{rr,0}}}{\beta}\left(\frac{r-r_{0}}{r_{0}}\right),$
(17) $\displaystyle\overline{y}=$
$\displaystyle\frac{\sqrt{g_{\theta\theta,0}}}{\beta}\left(\frac{\frac{\pi}{2}-\theta}{r_{0}}\right).$
(18)

In these coordinates, we have $\overline{x}=0$ and $\overline{y}=0$ at the
torus centre. We furthermore introduce a $\beta$ parameter determining the
torus thickness, which is connected to the sound speed at the torus centre in
the following manner (Abramowicz et al., 2006; Blaes et al., 2006):

$\displaystyle\beta^{2}=$
$\displaystyle\frac{2nc_{\mathrm{S},0}^{2}}{r_{0}^{2}\Omega_{0}^{2}A_{0}^{2}}.$
(19)

The surface of the torus, which coincides with the surface of zero pressure,
is given by the condition $f(r,\theta)=0$. An example of the torus cross-
section is shown in Figure 1 illustrating the character of the equipressure
surfaces for different values of $\beta$. An equilibrium torus is formed when
the perfect fluid fills up a closed equipressure surface. The largest possible
torus arises by filling up the equipressure surface that has a crossing point
– the so-called cusp. We call this structure, for which we have
$\beta=\beta_{\mathrm{cusp}}$, the "cusp torus". Notice that, for
$\beta>\beta_{\mathrm{cusp}}$, the equipressure surfaces are no longer closed
and no torus therefore can be formed.

Figure 1: Meridional cross-section illustrating the shape of the equipressure
surfaces in the Schwarzschild geometry. The red line marks a cusp torus with
$\beta=\beta_{\mathrm{cusp}}$, the blue line corresponds to an equilibrium
torus with $\beta<\beta_{\mathrm{cusp}}$, and the black dot denotes the centre
of the torus (as well as the infinitely slender torus with $\beta\rightarrow
0$).

###  2.2 The oscillating configuration

We assume the effective potential $\mathscr{U}$ (e.g. Abramowicz et al., 2006)
in the form

$\mathscr{U}=g^{tt}-2l_{0}g^{t\varphi}+l_{0}^{2}g^{\varphi\varphi}.$ (20)

An infinitesimally slender torus with $\beta\rightarrow 0$ at $r_{0}$ with
specific angular momentum $l_{0}$ undergoing a small axially symmetric
perturbation in the radial direction will oscillate with the frequency equal
to the radial epicyclic frequency of a free test particle given by (Abramowicz
and Kluźniak, 2005; Aliev and Galtsov, 1981)

$\nu_{r}^{2}=\left.\frac{1}{4\pi^{2}}\frac{\mathscr{E}_{0}^{2}}{2A_{0}^{2}g_{rr,0}}\frac{\partial^{2}\mathscr{U}}{\partial
r^{2}}\right|_{0}.$ (21)

Now let us investigate how the frequency changes when the torus becomes
thicker and/or when the perturbation is not axially symmetric. Assume small
perturbations of all quantities around the equilibrium state in the form
(Abramowicz et al., 2006; Blaes et al., 2006)

$\delta X(t,r,\theta,\varphi)=\delta
X(r,\theta)\mathrm{e}^{i\left(m\varphi-\omega t\right)},$ (22)

where $m$ is the azimuthal number and $\omega$ is the angular frequency of the
oscillations. In this work, we focus on two modes of oscillations: the axially
symmetric ($m=0$) and the first non-axisymmetric ($m=-1$) radial epicyclic
modes.

From the continuity equation $\nabla_{\mu}\left(\rho u^{\mu}\right)=0$, one
can get the relativistic version of the Papaloizou-Pringle equation
(Abramowicz et al., 2006; Fragile et al., 2016), 333For the sake of
simplicity, from now on, we will use $f=f(r,\theta)$.

$\displaystyle\frac{1}{\sqrt{-g}}\partial_{\mu}\frac{\sqrt{-g}g^{\mu\nu}f^{n}\partial_{\nu}W}{nc_{\mathrm{s},0}^{2}f+1}+$
$\displaystyle\left(l_{0}\omega-m\right)^{2}\frac{\Omega
g^{t\phi}-g^{\phi\phi}}{1-\Omega
l_{0}}\frac{f^{n}}{nc_{\mathrm{s},0}^{2}f+1}W=$ $\displaystyle=$
$\displaystyle-\frac{2n\mathscr{A}^{2}\left(\overline{\omega}-m\overline{\Omega}\right)^{2}}{\beta^{2}r_{0}^{2}}f^{n-1}W,$
(23)

where $\\{\mu,\nu\\}\in\\{r,\theta\\}$, $\mathscr{A}\equiv A/A_{0}$,
$\overline{\Omega}\equiv\Omega/\Omega_{0}$,
$\overline{\omega}\equiv\omega/\Omega_{0}$, $g$ is the determinant of the
metric tensor and $W$ equals to (Abramowicz et al., 2006)

$W=-\frac{\delta p}{A\rho\left(\omega-m\Omega\right)}.$ (24)

Equation (23) has no analytical solution except for the limit case of an
infinitely slender torus $\left(\beta\rightarrow 0\right)$. In the case of
non-slender tori $\left(\beta>0\right)$, the equation can be solved using a
perturbation method (see, e.g., Straub and Šrámková (2009)).

####  2.2.1 Solving the Papaloizou-Pringle equation

When the exact solution for a simplified case is known (as for
$\beta\rightarrow 0$), we can use perturbation theory to find the solution for
more complicated cases ($\beta>0$). 444Note the perturbation method gives
reasonable results only for small values of $\beta$ and our results are
therefore valid only for slightly non-slender tori.

By expanding the quantities $\overline{\omega}$, $W$, $\mathscr{A}$,
$\overline{\Omega}$, $f$ in $\beta$ (Straub and Šrámková, 2009)

$Q=Q^{(0)}+\beta Q^{(1)}+\beta^{2}Q^{(2)}+\cdots,\qquad
Q\in\left\\{\overline{\omega},W,\mathscr{A},\overline{\Omega},f\right\\},$
(25)

substituting that into equation (23), and comparing the coefficients of
appropriate order in $\beta$, we obtain the corresponding corrections to $W$
and $\omega$. Note that the zero order corresponds to the slender torus case
($\beta\rightarrow 0$), in which we have $\omega=2\pi\nu_{r}$.

Using this procedure, Straub and Šrámková (2009) have derived the expression
for the radial epicyclic mode frequency with the second order accuracy, which
may be written as

$\omega_{r,\mathrm{m}}=2\pi\,\nu_{r}+m\,\Omega_{0}+P_{\mathrm{m}}\,\beta^{2}+\mathscr{O}\left(\beta^{3}\right),$
(26)

where $P_{\mathrm{m}}$ denotes the second order correction term for which the
explicit form can be found in their paper.

##  3 The Hartle-Thorne geometry

The exterior solution of the Hartle-Thorne metric is characterized by three
parameters: the gravitational mass $M$, angular momentum $J$ and the
quadrupole moment $Q$ of the star. We use this metric assuming dimensionless
forms of the angular momentum and the quadrupole moment, $j=J/M^{2}$ and
$q=Q/M^{3}$, which can be in the Schwarzschild coordinates written as
(Abramowicz et al., 2003)555Note misprints in the original paper.:

$\displaystyle g_{tt}$
$\displaystyle=-\left(1-\frac{2M}{r}\right)\left[1+j^{2}F_{1}(r)+qF_{2}(r)\right],$
(27) $\displaystyle g_{rr}$
$\displaystyle=\left(1-\frac{2M}{r}\right)^{-1}\left[1+j^{2}G_{1}(r)-qF_{2}(r)\right],$
(28) $\displaystyle g_{\theta\theta}$
$\displaystyle=r^{2}\left[1+j^{2}H_{1}(r)+qH_{2}(r)\right],$ (29)
$\displaystyle g_{\varphi\varphi}$
$\displaystyle=r^{2}\sin^{2}\theta\left[1+j^{2}H_{1}(r)+qH_{2}(r)\right],$
(30) $\displaystyle g_{t\varphi}$
$\displaystyle=-\frac{2M^{2}}{r}j\sin^{2}\theta,$ (31)

where (using the $u=\cos\theta$ substitution)

$\displaystyle F_{1}(r)=$ $\displaystyle-\left[8Mr^{4}(r-2M)\right]^{-1}$
$\displaystyle\left[u^{2}\left(48M^{6}-8M^{5}r-24M^{4}r^{2}-30M^{3}r^{3}-60M^{2}r^{4}+135Mr^{5}-45r^{6}\right)\right.$
$\displaystyle+\left.(r-M)\left(16M^{5}+8M^{4}r-10M^{2}r^{3}-30Mr^{4}+15r^{5}\right)\right]+A_{1}(r),$
(32) $\displaystyle F_{2}(r)=$
$\displaystyle\left[8Mr(r-2M)\right]^{-1}\left[5\left(3u^{2}-1\right)\left(r-M\right)\left(2M^{2}+6Mr-3r^{2}\right)\right]-A_{1}(r),$
(33) $\displaystyle G_{1}(r)=$
$\displaystyle\left[8Mr(r-2M)\right]^{-1}\left[\left(L(r)-72M^{5}r\right)-3u^{2}\left(L(r)-56M^{5}r\right)\right]-A_{1}(r),$
(34) $\displaystyle L(r)=$ $\displaystyle
80M^{6}+8M^{4}r^{2}+10M^{3}r^{3}+20M^{2}r^{4}-45Mr^{5}+15r^{6},$ (35)
$\displaystyle A_{1}(r)=$
$\displaystyle\frac{15\left(r^{2}-2M\right)\left(1-3u^{2}\right)}{16M^{2}}\ln\left(\frac{r}{r-2M}\right),$
(36) $\displaystyle H_{1}(r)=$
$\displaystyle\left(8Mr^{4}\right)^{-1}\left(1-3u^{2}\right)\left(16M^{5}+8M^{4}r-10M^{2}r^{3}+15Mr^{4}+15r^{5}\right)$
$\displaystyle+A_{2}(r),$ (37) $\displaystyle H_{2}(r)=$
$\displaystyle\left(8Mr\right)^{-1}5\left(1-3u^{2}\right)\left(2M^{2}-3Mr-3r^{2}\right)-A_{2}(r),$
(38) $\displaystyle A_{2}(r)=$
$\displaystyle\frac{15\left(r^{2}-2M\right)\left(3u^{2}-1\right)}{16M^{2}}\ln\left(\frac{r}{r-2M}\right).$
(39)

While for $j=0$ and $q=0$ the Hartle-Thorne metric coincides with the
Schwarzschild metric, by setting $j=a/M$ and $q=j^{2}$ and performing a
coordinate transformation into the Boyer-Lindquist coordinates (Abramowicz et
al., 2003),

$\displaystyle r_{\mathrm{BL}}$
$\displaystyle=r-\frac{a^{2}}{2r^{3}}\left[(r+2M)(r-2M)+u^{2}(r-2M)(r+3M)\right],$
(40) $\displaystyle\theta_{\mathrm{BL}}$
$\displaystyle=\theta-\frac{a^{2}}{2r^{3}}(r+2M)\cos\theta\sin\theta,$ (41)

we obtain Kerr geometry expanded upon the second order in the dimensionless
angular momentum.

##  4 Oscillations of tori in the vicinity of rotating neutron stars

Let us now study the changes that arise in the torus structure and for the
frequencies of its oscillations when the Hartle-Thorne geometry is assumed to
describe the spacetime geometry.666Following Straub and Šrámková (2009) and
Fragile et al. (2016), we use a Wolfram Mathematica code, which has been
extended to the Hartle-Thorne geometry. The main motivation behind this
analysis is related to models of NS QPOs. While the Kerr geometry is (likely)
proper to be used in the context of BH QPOs (e.g., Kotrlová et al., 2020), its
validity in the case of NS QPOs is limited to very compact NSs only.

###  4.1 The Hartle-Thorne geometry parameters range relevant to rotating NSs

A thorough discussion of the relevance of the Hartle-Thorne geometry for the
calculations of the geodesic orbital motion and QPO models frequencies is
presented in Urbancová et al. (2019). Here we just briefly summarize the
appropriate ranges of the individual parameters that are implied by the
present NS equations of state. The maximum value of the specific angular
momentum of a NS is about $j_{\mathrm{max}}\leavevmode\nobreak\
\sim\leavevmode\nobreak\ 0.7$, the specific quadrupole moment takes values
from $q/j^{2}\leavevmode\nobreak\ \sim\leavevmode\nobreak\ 1.5$ for a very
massive (compact) NS up to $q/j^{2}\leavevmode\nobreak\
\sim\leavevmode\nobreak\ 10$ for a low-mass NS (Urbancová et al., 2019). The
conservative expectations of the NS mass values are about
$1.4-2.5\,M_{\odot}$.

Figure 2: Illustration of some characteristics of tori carried out in
different geometries. The tori are centered at the radial coordinate at which
the radial epicyclic frequency of a free test particle reaches its maximum.
Left panels: Meridional cross-sections of the equipressure surfaces
determining the shape of the tori. From top to bottom, the results correspond
to calculations carried out in the Schwarzschild, linearized Kerr ($j=0.2$),
Kerr ($j=0.2$), and the Hartle-Thorne ($j=0.2,q=j^{2}$ and $j=0.2,q=10j^{2}$)
geometry. Right panels: Plots of the specific angular momentum of the fluid
(which is constant across the torus) along with the Keplerian angular
momentum. The intersection points of the two functions marked by the spots
correspond to the centre of the tori. The coloured segments indicate the
corresponding radial extentions of the tori.  Figure 3: The same as in Figure
2 but for $j=0.4$.

###  4.2 The quadrupole moment influence on the non-oscillating torus shape
and size

In Figures 2 and 3, we present meridional cross-sections of tori carried out
in different geometries, namely the Schwarzschild, Kerr, linearized Kerr, and
the Hatle-Thorne geometry. The figures also show plots of the Keplerian
angular momentum and the angular momentum of the fluid (which is constant
across the torus), and the radial extentions of the tori. For both figures,
the top panels correspond to $j=0$ (a non-rotating NS, i.e., the Schwarzschild
geometry), and the bottom panels to $j=0.2$ (Figure 2) and $j=0.4$ (Figure 3).
The radial coordinate $r_{0}$ is chosen such that the radial epicyclic
frequency of a free test particle defined at this coordinate reaches its
maximum.

In Table 1, we provide a quantitative comparison of the radial extensions of
tori from Figures 2 and 3. It is given in terms of the proper radial distance,
$r_{\mathrm{prop}}$, measured between the minimal, $r_{\mathrm{min}}$, and the
maximal, $r_{\mathrm{max}}$, radial coordinate of the torus surface,

$\Delta
r_{\mathrm{prop}}=\int^{r_{\mathrm{max}}}_{r_{\mathrm{min}}}\sqrt{g_{rr}}\,\,\mathrm{d}r.$
(42)

Table 1: The percentual differences in the proper radial extension $\Delta r_{\mathrm{prop}}$ of tori in the Hartle-Thorne geometry and in the Schwarzschild, Kerr, and linearized Kerr geometries. The displayed values correspond to the situations illustrated in Figures 2 and 3. Geometry | Schwarzschild | Kerr | Lin. Kerr
---|---|---|---
Spin $j$ | $0.2$ | $0.4$ | $0.2$ | $0.4$ | $0.2$ | $0.4$
HT ($q=j^{2}$) | $-\,5\,\%$ | $-\,11\,\%$ | $+\,2\,\%$ | $-\,5\,\%$ | $-\,1\,\%$ | $-\,13\,\%$
HT ($q=10j^{2}$) | $-\,7\,\%$ | $-\,12\,\%$ | $-\,1\,\%$ | $-\,6\,\%$ | $-\,4\,\%$ | $-\,14\,\%$

###  4.3 The quadrupole moment influence on the radial epicyclic oscillations
of non-slender tori

We use equation (26) to derive the radial epicyclic mode frequency as a
function of the radius of the torus centre $r_{0}$. In Figure 4, we plot the
frequencies of both the $m=0$ (left panel) and $m=-1$ (middle panel) radial
epicyclic modes. These are compared for the four different geometries assuming
$j=0.2$. The right panel of this figure illustrates the behaviour of tori
cross-sections corresponding to maxima of the $m=0$ radial epicyclic mode
frequency. Figure 5 then provides the same illustration but for $j=0.4$.

Figure 4: Frequencies of the radial epicyclic mode.
Left panels: The $m=0$ case. From top to bottom: the Schwarzschild, linearized
Kerr, Kerr, and the Hartle-Thorne ($q=j^{2}$ and $q=10j^{2}$) geometry. For
rotating stars, we assume $j=0.2$. The maximal frequencies allowed for the
slender torus and for the cusp torus are denoted by the black and red spots,
respectively.
Middle panels: The same but for the $m=-1$ case. The coloured spots denote the
frequency value corresponding to the radius at which the $m=0$ radial mode
frequency has its maximum.
Right panels: Tori that would oscillate with the maximal value of the $m=0$
radial epicyclic mode frequency for a given torus thickness. Figure 5:  The
same as in Figure 4 but for $a=0.4$.

In Table 2, we provide a quantitative comparison of the maximal frequencies of
the $m=0$ radial epicyclic mode for tori of maximal thicknesses (i.e., the
frequencies denoted by the red dots in the left panels of Figures 4 and 5) for
the Hartle-Thorne and the other three geometries. In Table 3, we then present
the same but for the $m=-1$ radial epicyclic mode (i.e., the frequencies
denoted by the red dots in the middle panels of Figures 4 and 5). The proper
radial extension of tori related to Tables 2 and 3 (i.e., those shown in the
right panels of Figures 4 and 5) are compared in Table 4.

Table 2: The percentual differences in the maximal values of frequencies of the $m=0$ radial epicyclic mode of the cusp tori in the Hartle-Thorne geometry and in the Schwarzschild, Kerr, and linearized Kerr geometries. The displayed values correspond to the situations illustrated in Figures 4 and 5. Geometry | Schwarzschild | Kerr | Lin. Kerr
---|---|---|---
Spin $j$ | $0.2$ | $0.4$ | $0.2$ | $0.4$ | $0.2$ | $0.4$
HT ($q=j^{2}$) | $+\,15\,\%$ | $+\,35\,\%$ | $\,0\,\%$ | $\,0\,\%$ | $-\,1\,\%$ | $-\,4\,\%$
HT ($q=10j^{2}$) | $+\,8\,\%$ | $+\,4\,\%$ | $-\,6\,\%$ | $-\,23\,\%$ | $-\,7\,\%$ | $-\,26\,\%$
Table 3: The percentual differences in the frequency of the $m=-1$ radial epicyclic mode of the cusp tori in the Hartle-Thorne geometry and in the Schwarzschild, Kerr, and linearized Kerr geometries. The frequency is evaluated at the radius at which the $m=0$ radial epicyclic mode frequency has its maximum. The displayed values correspond to the situations illustrated in Figures 4 and 5. Geometry | Schwarzschild | Kerr | Lin. Kerr
---|---|---|---
Spin $j$ | $0.2$ | $0.4$ | $0.2$ | $0.4$ | $0.2$ | $0.4$
HT ($q=j^{2}$) | $+\,20\,\%$ | $+\,49\,\%$ | $\,0\,\%$ | $-\,1\,\%$ | $-\,1\,\%$ | $-\,8\,\%$
HT ($q=10j^{2}$) | $+\,5\,\%$ | $-\,10\,\%$ | $-\,23\,\%$ | $-\,40\,\%$ | $-\,14\,\%$ | $-\,44\,\%$

Table 4: The percentual differences in the proper radial extension $\Delta r_{\mathrm{prop}}$ of the cusp tori relevant to Tables 2 and 3 and shown in the right panels of Figures 4 and 5. Geometry | Schwarzschild | Kerr | Lin. Kerr
---|---|---|---
Spin $j$ | $0.2$ | $0.4$ | $0.2$ | $0.4$ | $0.2$ | $0.4$
HT ($q=j^{2}$) | $-\,7\,\%$ | $-\,17\,\%$ | $0\,\%$ | $-\,4\,\%$ | $0\,\%$ | $+\,1\,\%$
HT ($q=10j^{2}$) | $-\,6\,\%$ | $-\,7\,\%$ | $-\,4\,\%$ | $+\,7\,\%$ | $+\,2\,\%$ | $+\,13\,\%$
Figure 6: Frequency correlations predicted by the CT model vs. data of the 4U
1636-53 atoll source. The fit for $j=0.22$ obtained under the consideration of
the Kerr geometry (the curve marked as Kerr) is compared here to two examples
of predictions obtained under the consideration of the Hartle-Thorne geometry
(the curves marked as HT). Examples of the best fits predicted by the
relativistic precession model for a given $j$ and $q$ are shown as well (the
curves marked as RP model).

##  5 Discussion and conclusions

Our results indicate that, while the shape of the non-oscillating tori is not
much sensitive to the NS quadrupole moment, the frequencies of the radial
epicyclic modes of tori oscillations are affected significantly. Clearly, the
difference of the frequencies of oscillations of tori around BHs and NSs can
reach tens of percents. Although a more detailed analysis is certainly needed
( including the completion of the radial epicyclic mode investigation as well
as the investigation of the vertical epicyclic mode behaviour), we may already
conclude that the consideration of the quadrupole moment induced by the NS
rotation likely should have an impact on the modeling of the high-frequency
quasi-periodic oscillations.

Our conclusion is demonstrated in Figure 6. There we consider a recently
proposed QPO model (CT model; Török et al., 2016a) and compare the frequencies
predicted by the model for several combinations of $M,j,q$ with the
frequencies observed in the 4U 1636-53 atoll source (the data are taken from
Barret et al., 2006; Török, 2009). We include in the figure examples of
correlations predicted by the relativistic precession model (Stella and
Vietri, 1999). This model provides less promising fits of the data than the CT
model while the effects associated to the NS rotation do not imply a
significant improvement (see Török et al., 2012, 2016b, 2016a). It is clear
from the figure that even when we restrict ourselves to values of the Hartle-
Thorne spacetime parameters that are consistent with up-to-date models of
neutron stars, no conceivable smooth curve can reproduce the data in a
significantly better way compared to the CT model.

We acknowledge two internal grants of the Silesian University,
$\mathrm{SGS/12,13/2019}$. We wish to thank the INTER-EXCELLENCE project No.
$\mathrm{LTI17018}$. KK thanks to the INTER-EXCELLENCE project No. LTT17003.
DL thanks the Student Grant Foundation of the Silesian University in Opava,
Grant No. $\mathrm{SGF/1/2020}$, which has been carried out within the EU
OPSRE project entitled “Improving the quality of the internal grant scheme of
the Silesian University in Opava”, reg. number:
$\mathrm{CZ.02.2.69/0.0/0.0/19\\_073/0016951}$. The autors were also supported
by the ESF projects No. $\mathrm{CZ.02.2.69/0.0/0.0/18\\_054/0014696}$.

## References

  * Abramowicz et al. (1978) Abramowicz, M., Jaroszynski, M. and Sikora, M. (1978), Relativistic, accreting disks, _Astronomy and Astrophysics_ , 63, pp. 221–224.
  * Abramowicz et al. (2003) Abramowicz, M. A., Almergren, G. J. E., Kluzniak, W. and Thampan, A. V. (2003), The hartle-thorne circular geodesics, gr-qc/0312070v1.
  * Abramowicz et al. (2006) Abramowicz, M. A., Blaes, O. M., Horák, J., Kluzniak, W. and Rebusco, P. (2006), Epicyclic oscillations of fluid bodies: II. Strong gravity, _Classical and Quantum Gravity_ , 23, pp. 1689–1696, %****␣mat.bbl␣Line␣25␣****astro-ph/0511375.
  * Abramowicz et al. (2003a) Abramowicz, M. A., Bulik, T., Bursa, M. and Kluźniak, W. (2003a), Evidence for a 2:3 resonance in Sco X-1 kHz QPOs, _A &A_, 404, pp. L21–L24, astro-ph/0206490.
  * Abramowicz et al. (2003b) Abramowicz, M. A., Karas, V., Kluzniak, W., Lee, W. H. and Rebusco, P. (2003b), Non-Linear Resonance in Nearly Geodesic Motion in Low-Mass X-Ray Binaries, _PAS_ , 55, pp. 467–466, astro-ph/0302183.
  * Abramowicz and Kluźniak (2005) Abramowicz, M. A. and Kluźniak, W. (2005), Epicyclic Frequencies Derived From The Effective Potential: Simple And Practical Formulae, _Astrophysics and Space Science_ , 300, pp. 127–136, astro-ph/0411709.
  * Aliev and Galtsov (1981) Aliev, A. N. and Galtsov, D. V. (1981), Radiation from relativistic particles in nongeodesic motion in a strong gravitational field, _General Relativity and Gravitation_ , 13, pp. 899–912.
  * Barret et al. (2006) Barret, D., Olive, J.-F. and Miller, M. C. (2006), The coherence of kilohertz quasi-periodic oscillations in the X-rays from accreting neutron stars, _MNRAS_ , 370, pp. 1140–1146, astro-ph/0605486.
  * Blaes et al. (2006) Blaes, O. M., Arras, P. and Fragile, P. C. (2006), Oscillation modes of relativistic slender tori, _Monthly Notices of the Royal Astronomical Society_ , 369, pp. 1235–1252, astro-ph/0601379.
  * Bursa (2005) Bursa, M. (2005), Global oscillations of a fluid torus as a modulation mechanism for black-hole high-frequency QPOs, _Astronomische Nachrichten_ , 326(9), pp. 849–855, astro-ph/0510460.
  * de Avellar et al. (2018) de Avellar, M. G. B., Porth, O., Younsi, Z. and Rezzolla, L. (2018), Kilohertz QPOs in low-mass X-ray binaries as oscillation modes of tori around neutron stars - I, _MNRAS_ , 474, pp. 3967–3975.
  * Dönmez et al. (2011) Dönmez, O., Zanotti, O. and Rezzolla, L. (2011), On the development of quasi-periodic oscillations in Bondi-Hoyle accretion flows, _MNRAS_ , 412(3), pp. 1659–1668, 1010.1739.
  * Fragile et al. (2016) Fragile, P. C., Straub, O. and Blaes, O. (2016), High-frequency and type-c QPOs from oscillating, precessing hot, thick flow, _Monthly Notices of the Royal Astronomical Society_ , 461(2), pp. 1356–1362.
  * Hartle (1967) Hartle, J. B. (1967), Slowly Rotating Relativistic Stars. I. Equations of Structure, _APJL_ , 150, p. 1005.
  * Hartle and Thorne (1968) Hartle, J. B. and Thorne, K. S. (1968), Slowly Rotating Relativistic Stars. II. Models for Neutron Stars and Supermassive Stars, _APJ_ , 153, p. 807.
  * Horák et al. (2017) Horák, J., Straub, O., Šrámková, E., Goluchová, K. and Török, G. (2017), Epicyclic oscillations of thick relativistic disks, in _RAGtime 17-19: Workshops on Black Holes and Neutron Stars_ , pp. 47–59, URL https://ui.adsabs.harvard.edu/abs/2017bhns.work...47H.
  * Kluzniak and Abramowicz (2001) Kluzniak, W. and Abramowicz, M. A. (2001), The physics of kHz QPOs—strong gravity’s coupled anharmonic oscillators, _arXiv e-prints_ , astro-ph/0105057, astro-ph/0105057.
  * Kluźniak et al. (2004) Kluźniak, W., Abramowicz, M. A., Kato, S., Lee, W. H. and Stergioulas, N. (2004), Nonlinear Resonance in the Accretion Disk of a Millisecond Pulsar, _APJL_ , 603(2), pp. L89–L92, astro-ph/0308035.
  * Kotrlová et al. (2020) Kotrlová, A., Šrámková, E., Török, G., Goluchová, K., Horák, J., Straub, O., Lančová, D., Stuchlík, Z. and Abramowicz, M. A. (2020), Models of high-frequency quasi-periodic oscillations and black hole spin estimates in Galactic microquasars, _A &A_, 643, A31, 2008.12963.
  * Rezzolla et al. (2003) Rezzolla, L., Yoshida, S. and Zanotti, O. (2003), Oscillations of vertically integrated relativistic tori - I. Axisymmetric modes in a Schwarzschild space-time, _MNRAS_ , 344, pp. 978–992, astro-ph/0307488.
  * Stella and Vietri (1999) Stella, L. and Vietri, M. (1999), kHz Quasiperiodic Oscillations in Low-Mass X-Ray Binaries as Probes of General Relativity in the Strong-Field Regime, _PRL_ , 82(1), pp. 17–20, astro-ph/9812124.
  * Straub and Šrámková (2009) Straub, O. and Šrámková, E. (2009), Epicyclic oscillations of non-slender fluid tori around Kerr black holes, _Classical and Quantum Gravity_ , 26(5), 055011, 0901.1635.
  * Török (2009) Török, G. (2009), Reversal of the amplitude difference of kHz QPOs in six atoll sources, _A &A_, 497, pp. 661–665, 0812.4751.
  * Török et al. (2005) Török, G., Abramowicz, M. A., Kluźniak, W. and Stuchlík, Z. (2005), The orbital resonance model for twin peak kHz quasi periodic oscillations in microquasars, _A &A_, 436, pp. 1–8.
  * Török et al. (2012) Török, G., Bakala, P., Šrámková, E., Stuchlík, Z., Urbanec, M. and Goluchová, K. (2012), Mass-angular-momentum relations implied by models of twin peak quasi-periodic oscillations, 1408.4220v1.
  * Török et al. (2016a) Török, G., Goluchová, K., Horák, J., Šrámková, E., Urbanec, M., Pecháček, T. and Bakala, P. (2016a), Twin peak quasi-periodic oscillations as signature of oscillating cusp torus, _MNRAS_ , 457, pp. L19–L23, 1512.03841.
  * Török et al. (2016b) Török, G., Goluchová, K., Urbanec, M., Šrámková, E., Adámek, K., Urbancová, G., Pecháček, T., Bakala, P., Stuchlík, Z., Horák, J. and Juryšek, J. (2016b), Constraining Models of Twin-Peak Quasi-periodic Oscillations with Realistic Neutron Star Equations of State, _The Astrophysical Journal_ , 833, 273, 1611.06087.
  * Urbancová et al. (2019) Urbancová, G., Urbanec, M., Török, G., Stuchlík, Z., Blaschke, M. and Miller, J. C. (2019), Epicyclic Oscillations in the Hartle-Thorne External Geometry, _The Astrophysical Journal_ , 877(2), 66, 1905.00730.


11institutetext: University of Piraeus, Pireas 185 34, Greece
22institutetext: Innov-Acts Ltd, City Business Center Office SF09, 27
Michalakopoulou St. 1075, Nicosia, CY 22email<EMAIL_ADDRESS>33institutetext: Netcompany-Intrasoft S.A, 2b, rue Nicolas Bové L-1253
Luxembourg

# XAI for time-series classification leveraging image highlight methods

Georgios Makridis 11 0000-0002-6165-7239 Georgios Fatouros 1122
0000-0001-6843-089X Vasileios Koukos 11  Dimitrios Kotios 11  Dimosthenis
Kyriazis 11 0000-0001-7019-7214 Ioannis Soldatos 33 ht0000-0002-6668-3911

###### Abstract

Although much work has been done on explainability in the computer vision and
natural language processing (NLP) fields, there is still much work to be done
to explain methods applied to time series as time series by nature can not be
understood at first sight. In this paper, we present a Deep Neural Network
(DNN) in a teacher-student architecture (distillation model) that offers
interpretability in time-series classification tasks. The explainability of
our approach is based on transforming the time series to 2D plots and applying
image highlight methods (such as LIME and GradCam), making the predictions
interpretable. At the same time, the proposed approach offers increased
accuracy competing with the baseline model with the trade-off of increasing
the training time.

###### Keywords:

XAI Time-series LIME GradCam Deep Learning.

##  1 Introduction

Figure 1: The proposed high-level architecture. The above pipeline represents
the "traditional" way of applying XAI methods, while the lower one depicts the
proposed model.

The current day and age, also known as the Digital or Information Age, is
characterized by complex computing systems which generate enormous amounts of
data daily. The digital transformation of industrial environments leads to the
fourth industrial revolution -Industry4.0 [15], with Artificial Intelligence
(AI) being the key facilitator of the Industry4.0 era by enabling innovative
tools and processes [27]. At the same time, there has been a growing interest
in eXplainable Artificial Intelligence (XAI) towards human-understandable
explanations for the predictions and decisions made by machine learning
models.

The notion of explaining and expressing a Machine Learning (ML) model is
called interpretability or explainability [5]. This need for interpretability
mainly exists in Deep Neural Networks (DNN), which are defined by large levels
of complexity, thus appearing to be "black boxes" [29]. To address this issue,
researchers have proposed various methods for explaining the predictions made
by deep learning models for image classification tasks, known as XAI for
images [30]. These methods include visualizing the features learned by the
model, feature attribution techniques, and model distillation [25]. For
example, visualizing the features learned by the model can provide insights
into which parts of the input image the model uses to make its prediction,
while feature attribution techniques can highlight the regions of the input
image that are most important for making a prediction. Model distillation, on
the other hand, attempts to convert a complex model into a simpler, more
interpretable one that still preserves the accuracy of the original model. A
knowledge distillation system consists of three principal components: the
knowledge, the distillation algorithm, and the teacher-student architecture
[9]. Model distillation is a technique where a smaller, simpler model is
trained to mimic the predictions of a larger, more complex model. This allows
for the smaller model to be more easily interpretable and can help to provide
insight into the workings of the larger model. For example, in [3] authors
proposed a method called "DeepTaylor" that uses a Taylor series approximation
to distill a deep neural network into a linear model that is more
interpretable.

Figure 2: Schematic explanation of the data aspect to the end-to-end approach
including the main steps to the process, from raw data to the user.

Although much work has been done on explainability in the computer vision and
Natural Language Processing (NLP) fields, there is still much work to be done
to explain methods applied to time series. This might be caused by the nature
of time series, which we can not understand at first sight. Indeed, when a
human looks at a photo or reads a text, he intuitively and instinctively
understands the underlying information in the data [22]. Although temporal
data is ubiquitous in nature, through all forms of sound, humans are not used
to representing this temporal data in the form of a signal that varies as a
function of time. We need expert knowledge or additional methods to leverage
the underlying information present in the data [13]. This problem might
explain why a lot of existing methods focus on highlighting the parts of the
input responsible for a prediction for computer vision tasks (i.e., image
classification). Despite this drawback, timeseries approach play a key role in
industry 4.0 so does XAI for these cases (e.g., XAI for timeseries can enhance
transparency in Industry 4.0 by providing clear and interpretable explanations
for how the predictive maintenance tool works, or for identifying patterns and
anomalies in time series data, providing early detection of potential issues,
and insights into how to mitigate risks).

This paper addresses the emerging challenge of explainability of the "black-
box" DNN models concerning time series as input data by introducing a teacher-
student (distillation model) approach to facilitate user explainability and
reasoning. In this context, the operational objective is twofold:

  1. (i)

Time series categorization based on efficient classification of each time
series, providing a sound approach for time series data along with
unstructured image data (i.e., 2D plot) can be classified using AI-based
methods and represented in the time domain as time-series enables the
exploitation of the image XAI methods.

  2. (ii)

Systematically providing visualization of XAI by applying various XAI methods
on the CNN part of the proposed model.

While the scientific contributions that support and extend the proposed
solution can be briefly summarised in terms of added value, as the proposed
approach enables:

  1. (i)

Combination of the strengths of both networks to achieve better performance.

  2. (ii)

Handling of both time series as arrays and as images, which provides more
flexibility for data inputs and potentially enhances the interpretability of
the model.

  3. (iii)

Explainability by visualizing the important parts of the plot for time series,
which is an open research field with limited state-of-the-art solutions.

In this context, the proposed approach presents a novel approach to XAI for
time series data, in which image highlight techniques are used to visualize
and interpret the predictions of a time series model as depicted in Fig. 1.
While a more detailed view of the proposed approach is provided in Fig. 2,
which depicts the data handling pipeline, in an approach, to apply both the
time series classification and XAI methods. In the first step, the raw data
are pre-processed and fed into the distillation model. Then the model is
trained and finally, the output comprises the predicted "class" and the
heatmap of the time-series representation to be utilized for explainability
purposes.

The remainder of the paper is organized as follows: Section 2 presents related
work done in the areas of study of this paper, while Section 3 delivers an
overview of the proposed methodological approach, introduces the overall
architecture, and offers details regarding the datasets used and how these are
utilized within the models. Section 4 dives deeper into the results of the
conducted research and the implemented algorithms, with the performance of the
proposed mechanisms being depicted in the results and evaluation section.
Section 5 concludes with recommendations for future research and the potential
of the current study.

##  2 Related Work

Time series classification is a challenging task, where DNNs have been shown
to achieve state-of-the-art results in various fields such as finance [8], and
industry [16]. However, the internal workings of these models are often
difficult to interpret. To address this issue, researchers have proposed
various methods for explaining the predictions made by deep learning models
for time series classification, known as XAI for time series. These methods
include attention mechanisms [18] and visualization techniques [6]. For
example, attention mechanisms allow the model to focus on specific parts of
the input time series that are important for making a prediction, while
visualization techniques can provide a graphical representation of the input
time series and the model’s decision-making process.

Attention mechanisms allow the model to focus on specific parts of the input
time series, providing insight into which regions of the input are most
important for making a prediction. For example, in [1] authors proposed an
attention-based LSTM model that learns to weigh different parts of the input
time series when making a prediction.

Furthermore there are post-hoc methods that approach the behavior of a model
by exporting relationships between feature values and predictions, in this
case, feature lags of time series also play a role. While the ante-hoc methods
incorporate the explanation in the structure of the model, which is therefore
already explainable at the end of the training phase. The first category
includes variants of the widely used LIME, k-LIME [11], DLIME [28], LIMEtree
[26], and SHAP (e.g., TimeSHAP [2]), as well as Anchors [21], or LoRE [10].
Surrogate models are built for each prediction sample in most of these
techniques, learning the behavior of the reference model in the particular
instance of interest by adding perturbations (or masking) to the feature
vector variables. The numerous feature disturbance techniques for assessing
the contribution of features to the projected value when they are deleted or
covered are nearly a distinct research topic in this context. These XAI models
may be used for DNN since they are unaffected by the underlying machine-
learning model. The CAM, ConvTimeNet [12], which belongs to this group but
intervenes in the model structure, is particularly interesting. Apart from
these, it is worth mentioning approaches such as RETAIN [4] (Reverse Time
Attention) with application to Electronic Health Records (EHR) data. RETAIN
achieves high accuracy while remaining clinically interpretable and is based
on a two-level neural attention model that detects previous visits. Finally,
NBEATS [19] focuses on an architecture based on residual back-and-forth
connections and a very deep stack of fully connected layers. The architecture
has a number of desirable properties, as it is interpretable and applicable
without modification to a wide range of target areas.

One limitation of XAI models for time series data is that the outputs and
explanations they provide may be difficult for end users to understand. The
models’ complexity and the explanations’ technical nature may make it
challenging for non-experts to interpret the results and take appropriate
actions. Additionally, the effectiveness of XAI models can depend on the end
user’s level of domain knowledge and familiarity with the data. To address
these limitations, it is important to design XAI models with the end user in
mind and provide explanations that are tailored to their needs and level of
expertise. This involve proper visualizations to help users explore and
understand the model’s predictions, as well as providing contextual
information about them following the "human-centricity" perspective that is
the core value behind the evolution of manufacturing towards Industry 5.0
[23].

##  3 Method

###  3.1 Proposed Model Architecture

The main research challenges addressed in this paper enable the accurate
classification of time series and the generation of interpretable XAI
visualizations. These two topics are interconnected as the XAI model is based
on the classification results. To facilitate the usage of a DNN model by
making it explainable, we introduce a form of distillation model (i.e.,
teacher-student architecture) where the teacher is the complex DNN model that
yields state-of-the-art performance while the student model is a CNN model
that can be interpretable by human end users when LIME and GradCam methods are
applied to it. Fig. 3 depicts the architecture of the proposed model, where
the Dense model represents the "teacher" while the CNN represents the
"explainable"-"student".

Figure 3: The architecture of the proposed model, where the Dense model
represents the "teacher" while the CNN represents the "explainable"-"student".
The data are fed as an array to the Dense network while the CNN as 2D plots.

Regarding the DNN model, its layers can be summarized as follows:

  * •

Input layer: This layer will take an image as input.

  * •

Rescaling layer.

  * •

Convolutional layers: to extract features from the input image. The number of
filters and their size can be chosen based on the complexity of the data. In
our experiments, we used 32 filters of size 3x3, and a stride of 1.

  * •

Pooling layers: to reduce the spatial dimensions of the feature maps and
reduce the computational requirements of the network. A common choice is to
use max pooling with a pool size of 2x2 and a stride of 2.

  * •

Flattening layer: to convert the high-dimensional feature maps into a 1D
vector that can be input into a fully connected layer.

  * •

Dense layer: One or multiple dense (fully connected) layers can be added to
make predictions based on the extracted features. The number of nodes in each
dense layer can be chosen based on the complexity of the data and the desired
performance of the model.

  * •

Output layer: this can be a dense layer with a single node and a sigmoid
activation function for binary classification or with as many nodes as the
number of classes in your data for multi-class classification.

###  3.2 Datasets

In the frame of our research, the utilized datasets have been retrieved by [7]
composed of labeled time series. Specifically, we used 2 of the available
datasets with more than 1000 training and test samples:

  * •

Wafer: This dataset was formatted by R. Olszewski as part of his thesis
Generalized feature extraction for structural pattern recognition in time-
series data at Carnegie Mellon University, 2001. Wafer data relates to
semiconductor microelectronics fabrication. A collection of inline process
control measurements recorded from various sensors during the processing of
silicon wafers for semiconductor fabrication constitute the wafer database;
each data set in the wafer database contains the measurements recorded by one
sensor during the processing of one wafer by one tool. The two classes are
normal and abnormal. There is a large class imbalance between normal and
abnormal (10.7% of the train are abnormal, 12.1% of the test). The best
performance in terms of Accuracy till now is 99,8% by using the ST algorithm
[14].

  * •

FordA: This data was originally used in a competition at the IEEE World
Congress on Computational Intelligence, 2008. The classification problem is to
diagnose whether a certain symptom exists or does not exist in an automotive
subsystem. Each case consists of 500 measurements of engine noise and a
classification. There are two separate problems: For FordA the train and test
data set were collected in typical operating conditions, with minimal noise
contamination. The best performance in terms of Accuracy till now is 96,54% by
using the ST algorithm.

###  3.3 Time-series to Image explainability

In order to make the time series interpretable we chose the most comprehensive
representation of a time series (i.e., a 2D plot with the time axis on the
x-axis and the value of the time series on the y-axis.) We first plot the time
series as an image and then input the image into the CNN for classification.
In all of our experiments, we plotted the time series as line-plot figures
with (224,244,1) dimensions.

Once we have plotted the time series as an image, we use them as inputs into a
CNN for classification. The CNN will process the image and use its learned
features to make a prediction about the time series.

Figure 4: Example of LIME important super-pixels

When dealing with image data, feature importance can be translated into the
importance of each pixel to the output forecast. The latter can also be
visualized into a heatmap where the importance of each feature (pixel) can be
displayed with different colors and can work as an excellent explanation for
the human operator. In our approach, we have chosen 2 notable methods of XAI
on image data as the more appropriate for time series plots. The LIME model is
a model-agnostic method based on the work of [20]. While, the Gradient-
weighted Class Activation Mapping (Grad-Cam) model [24] was used as one of the
SotA methods for interpreting the top features (parts of the image) concerning
the "Label" when the underlying model is a CNN. To identify the abstracted
notion of a given image, Grad-Cam utilizes gradients to generate localization
maps and highlight essential parts of the image. Despite the high level of
complexity, Grad-Cam is able to provide intuitive outputs, thus improving the
model’s accuracy and flexibility.

In Fig. 4, we see the output of an XAI model that has been applied to a plot
of a time series based on the LIME method in the study [17], where we can see
the green and red areas of the image. This means that the ML/DL model
classifies the underlying image to a specific category because of these parts
of super-pixels; the size of super-pixels colored in green are the ones that
increase the probability of our image belonging to the predicted class, while
the super-pixels colored in red are the ones that decrease the likelihood.
LIME is a technique for explaining the predictions of any machine learning
model by approximating it locally with an interpretable model. It does this by
perturbing the input data and creating a new, simpler model that is fit only
to the perturbed data. This process is repeated many times, each time with a
different set of perturbations, and the results are aggregated to create a
global explanation of the model’s prediction. In the case of image data, LIME
can be used to explain the prediction of a CNN or other image classification
model by creating a heatmap that shows which parts of the image are most
important for the model’s prediction. The heatmap is created by perturbing the
pixels of the image and creating a new, simpler model that is fit only to the
perturbed data. This process is repeated for different parts of the image, and
the results are aggregated to create the final heatmap.

Figure 5: GradCam heatmap examples in the last layer of the CNN model

An example of applying the Grad-Cam method is depicted in Fig.5. We can see
the original image and the explanation of grad-cam methods where there are
highlighted parts of the image that the CNN model is focusing on in each
layer. This is a technique for visualizing the regions in an image that are
most important for a convolutional neural network (CNN) to classify an image
correctly. It does this by overlaying a heatmap on top of the original image,
where the intensity of the color in the heatmap indicates how important that
region is for the classification. To create the heatmap, Grad-CAM first
computes the gradient of the output of the CNN with respect to the feature
maps of a convolutional layer. It then weighs these gradients by the
importance of the feature maps to the class prediction and produces a weighted
combination of the feature maps. Finally, it resizes the resulting map to the
size of the input image and overlays it on top of the original image to
produce the heatmap. Overall, the heatmap produced by Grad-CAM can be used to
understand which regions of an image are most important for a CNN’s prediction
and to identify potential areas of the image that may be causing the CNN to
make incorrect predictions.

##  4 Evaluation Results

###  4.1 Experimental Setup

We evaluate the performance of our model on a classification task, using a
train, validation, and test dataset split. The task involves classifying a set
of time series into one of two classes. Our distillation model consists of a
teacher network, which is a Dense network, and a student network, which is a
CNN. The goal of the distillation is to transfer the knowledge from the
teacher network to the student network, resulting in an explainable and more
efficient model with similar or better performance. Given that Model
distillation also has its own limitations, one of them is that it may not
always be possible to distill a complex model into a simpler, more
interpretable model. To this end, we provide the results of the teacher-
student network, the teacher-only and the student-only results.

We use the training dataset to train the models, and a validation dataset to
tune the hyperparameters of the network, such as the learning rate and the
number of layers. We perform early stopping if the validation accuracy does
not improve for a certain number of epochs. Finally, we evaluate the
performance of the teacher and student networks on the test dataset.

We compare the performance of the teacher network, the student network, and
our distillation model. We report the accuracy, precision, recall, F1-score
for each model. Additionally, we showcase the interpretability of the student
network using LIME and GradCAM, which highlight the important parts of the
input images that contribute to the classification decision.

We use the TensorFlow deep learning framework to implement the models and
train them in an environment with 20 CPU cores of 2.3GHz, and 64GB of RAM. The
hyperparameters are chosen based on a grid search over a range of values, and
the best set of hyperparameters are chosen based on the validation accuracy.
The models are trained for a maximum of 300 epochs, with early stopping if the
validation accuracy does not improve for 20 epochs. The code and the trained
models are publicly available for reproducibility. Moreover, we make a
qualitative evaluation regarding the interpretability of the XAI result of our
approach. Finally, we present a more fine-grained view of the performance of
the examined method in terms of time. The results are the average outcomes of
ten independently seeded runs for each measurement.

###  4.2 Experimental Results

For both the FordA and Wafer datasets, the Dense network achieved the lowest
precision and F1-score for class 1 compared to the other models. This suggests
that the Dense network may not be as effective at identifying the minority
class in imbalanced datasets. The CNN and proposed models achieved perfect
precision and F1-score for class 0 on both datasets, indicating that they were
able to effectively classify the majority class. The proposed model also
achieved the highest precision and F1-score for class 1 on both datasets,
suggesting that it may be better suited to handling imbalanced datasets. The
overall accuracy of the models varied, with the Dense network achieving the
lowest accuracy on the FordA dataset, and the CNN achieving the lowest
accuracy on the Wafer dataset. The proposed model achieved the highest
accuracy on both datasets

Table 1: Results of the classification tasks with different models Model | (FordA) Dense | (FordA) CNN | (FordA) Proposed
---|---|---|---
Precision (Class 0) | 0.86 | 0.70 | 0.86
Precision (Class 1) | 0.82 | 0.75 | 0.84
F1-Score (Class 0) | 0.84 | 0.73 | 0.86
Accuracy | 0.84 | 0.73 | 0.85
Model | (Wafer) Dense | (Wafer) CNN | (Wafer) Proposed
Precision (Class 0) | 1.0 | 1.0 | 1.0
Precision (Class 1) | 0.98 | 1.0 | 0.99
F1-Score (Class 0) | 1.0 | 1.0 | 1.0
Accuracy | 1.0 | 1.0 | 1.0
Table 2: Execution time of each model for the training task as well as how
many epochs took each model to converge. The RAM column refers to the MB
needed for loading the training datasets to the RAM

| trianing time (sec) | early stopping (epochs) | train-set RAM (MB)
---|---|---|---
Dense (FordA) | 31 | 67 | 1.16
CNN (FordA) | 643 | 76 | 1148.44
Dense + CNN (FordA) | 5106 | 102 | 1149.6
Dense (Wafer) | 186 | 300 | 13.74
CNN (Wafer) | 3539 | 300 | 4135.52
Dense + CNN (Wafer) | 3048 | 257 | 4149.26

Given that, time performance can impact the scalability and cost-effectiveness
of AI systems, as slower models may require more computational resources or
increased processing time, leading to increased costs. Thus, optimizing the
performance of AI models in terms of time is a crucial aspect of AI
development and deployment. To this end, in Table 2 we present the execution
time of each model for the training task as well as how many epochs took each
model to converge. We should highlight that the configurations of the models
were identified as mentioned in the previous subsection. It should also be
mentioned that concerning the inference time (i.e., the time taken by the
model to produce a prediction for a single input) there is no significant
discrepancy.

Moving to the "explainability" part of the model, the model has integrated the
process to provide LIME and Grad-Cam heatmaps helping the interpretation for a
non-data scientist user. LIME helps to understand why a machine learning model
made a certain prediction by looking at the most important parts of the input
data that the model considered when making the prediction. While Grad-CAM
generates a heatmap that shows which areas of an image were most important for
a machine learning model to make a prediction. This helps us to see which
parts of the image the model focused on to make its prediction. In Fig.6 and
Fig.7 we can see some indicative examples of the respective XAI
visualizations.

Figure 6: Example of LIME important superpixels where we can notice the red
areas that seem not relevant to the predicted class

(a) Example of Grad-Cam heatmap and the original image where we can notice the
areas of the image that the model focused

(b) Example of GradCam heatmap and the overlap of heatmap to the original
image where we can notice the areas of the image that the model focused

Figure 7: Grad-Cam example in the last convolution layer of the CNN

##  5 Conclusion

In conclusion, this paper examines an XAI-enhanced DNN for addressing the
problem of interpretability of DNN models when it comes to time series
classification tasks. Specifically, this paper presents a novel distillation
model for time-series classification that leverages image highlight methods
for XAI. The model takes in both the time series data as an array and a 2D
plot, providing a unique approach to time-series classification. The student
model is specifically designed for XAI images, demonstrating the potential of
this method for improving interpretability in time-series classification
tasks. The results of the experiments conducted show that the proposed model
outperforms existing methods and has the potential to be applied to a wide
range of time-series classification problems. A teacher-student architecture
was developed where the student part has as inputs the 2D plot of the time
series where LIME and GradCam model is applied offering XAI capabilities.
Regarding the future steps, we have 2 main objectives to make the model more
human-centered by applying a model to translate the XAI visualizations to
text, of course, we plan to apply the model to time series forecasting tasks
while at the same time examining visual analytics approach, where the human
would inspect the figures and provide some kind of feedback.

As future work we plan to develop a similar approach for time series
forecasting tasks.

####  5.0.1 Acknowledgements

The research leading to the results presented in this paper has received
funding from the Europeans Union’s funded Projects MobiSpaces under grant
agreement no 101070279 and STAR under grant agreement no 956573.

## References

  * [1] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015)
  * [2] Bento, J., Saleiro, P., Cruz, A.F., Figueiredo, M.A., Bizarro, P.: Timeshap: Explaining recurrent models through sequence perturbations. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. pp. 2565–2573 (2021)
  * [3] Chiu, M.T., Xu, X., Wang, K., Hobbs, J., Hovakimyan, N., Huang, T.S., Shi, H.: The 1st agriculture-vision challenge: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 48–49 (2020)
  * [4] Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems 29 (2016)
  * [5] Choo, J., Liu, S.: Visual analytics for explainable deep learning. IEEE computer graphics and applications 38(4), 84–92 (2018)
  * [6] Dahl, M., Mancuso, J., Dupis, Y., Decoste, B., Giraud, M., Livingstone, I., Patriquin, J., Uhma, G.: Private machine learning in tensorflow using secure computation. arXiv preprint arXiv:1810.08130 (2018)
  * [7] Dau, H.A., Bagnall, A., Kamgar, K., Yeh, C.C.M., Zhu, Y., Gharghabi, S., Ratanamahatana, C.A., Keogh, E.: The ucr time series archive. IEEE/CAA Journal of Automatica Sinica 6(6), 1293–1305 (2019)
  * [8] Fatouros, G., Makridis, G., Kotios, D., Soldatos, J., Filippakis, M., Kyriazis, D.: Deepvar: a framework for portfolio risk assessment leveraging probabilistic deep neural networks. Digital Finance pp. 1–28 (2022)
  * [9] Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. International Journal of Computer Vision 129, 1789–1819 (2021)
  * [10] Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., Giannotti, F.: Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018)
  * [11] Hall, P., Gill, N., Kurka, M., Phan, W.: Machine learning interpretability with h2o driverless ai. H2O. ai (2017)
  * [12] Kashiparekh, K., Narwariya, J., Malhotra, P., Vig, L., Shroff, G.: Convtimenet: A pre-trained deep convolutional neural network for time series classification. In: 2019 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2019)
  * [13] Lim, B., Zohren, S.: Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379(2194), 20200209 (2021)
  * [14] Lines, J., Davis, L.M., Hills, J., Bagnall, A.: A shapelet transform for time series classification. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 289–297 (2012)
  * [15] Makridis, G., Kyriazis, D., Plitsos, S.: Predictive maintenance leveraging machine learning for time-series forecasting in the maritime industry. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). pp. 1–8. IEEE (2020)
  * [16] Makridis, G., Mavrepis, P., Kyriazis, D., Polychronou, I., Kaloudis, S.: Enhanced food safety through deep learning for food recalls prediction. In: International Conference on Discovery Science. pp. 566–580. Springer (2020)
  * [17] Makridis, G., Theodoropoulos, S., Dardanis, D., Makridis, I., Separdani, M.M., Fatouros, G., Kyriazis, D., Koulouris, P.: Xai enhancing cyber defence against adversarial attacks in industrial applications. In: 2022 IEEE 5th International Conference on Image Processing Applications and Systems (IPAS). pp. xx–xx (2022). https://doi.org/10.xxxx/xxxx.xxxxxxx
  * [18] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model cards for model reporting. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 220–229 (2019)
  * [19] Oreshkin, B.N., Carpov, D., Chapados, N., Bengio, Y.: N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 (2019)
  * [20] Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)
  * [21] Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: High-precision model-agnostic explanations. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)
  * [22] Rojat, T., Puget, R., Filliat, D., Del Ser, J., Gelin, R., Díaz-Rodríguez, N.: Explainable artificial intelligence (xai) on timeseries data: A survey. arXiv preprint arXiv:2104.00950 (2021)
  * [23] Rožanec, J.M., Novalija, I., Zajec, P., Kenda, K., Tavakoli Ghinani, H., Suh, S., Veliou, E., Papamartzivanos, D., Giannetsos, T., Menesidou, S.A., et al.: Human-centric artificial intelligence architecture for industry 5.0 applications. International Journal of Production Research pp. 1–26 (2022)
  * [24] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [25] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International conference on machine learning. pp. 3145–3153. PMLR (2017)
  * [26] Sokol, K., Flach, P.: Limetree: Interactively customisable explanations based on local surrogate multi-output regression trees. arXiv preprint arXiv:2005.01427 (2020)
  * [27] Soldatos, J., Kyriazis, D.: Trusted Artificial Intelligence in Manufacturing: A Review of the Emerging Wave of Ethical and Human Centric AI Technologies for Smart Production. Now Publishers (2021)
  * [28] Zafar, M.R., Khan, N.M.: Dlime: A deterministic local interpretable model-agnostic explanations approach for computer-aided diagnosis systems. arXiv preprint arXiv:1906.10263 (2019)
  * [29] Zahavy, T., Ben-Zrihem, N., Mannor, S.: Graying the black box: Understanding dqns. In: International conference on machine learning. pp. 1899–1908. PMLR (2016)
  * [30] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. pp. 818–833. Springer (2014)