11institutetext: Max-Planck-Institut für Radioastronomie, Auf dem Hügel 69,
D-53121 Bonn, Germany 22institutetext: Institut de Radioastronomie
Millimétrique, 300 rue de la Piscine, 38406 Grenoble, France

# First 3 mm-VLBI imaging of the two-sided jet in Cygnus A

Zooming into the launching region

B. Boccardi 11 T.P. Krichbaum 11 U. Bach 11 M. Bremer 22 J.A. Zensus 11

(Received 1 March, 2016 / accepted 11 March, 2016)

###### Abstract

Aims. We present for the first time Very-Long-Baseline Interferometry images
of the radio galaxy Cygnus A at the frequency of $86$ $\rm GHz$. Thanks to the
high spatial resolution of only $\sim$200 Schwarzschild radii ($R_{\bf S}$),
such observations provide an extremely detailed view of the nuclear regions in
this archetypal object and allow us to derive important constraints for
theoretical models describing the launching of relativistic jets.

Methods. A pixel-based analysis of the jet outflow, which still appears two-
sided on the scales probed, was performed. By fitting Gaussian functions to
the transverse intensity profiles, we could determine the jet width in the
nuclear region.

Results. The base of the jets appears wide. The minimum measured transverse
width of ${\sim}(227\pm 98)$ $R_{\bf S}$ is significantly larger than the
radius of the Innermost Stable Circular Orbit, suggesting that the outer
accretion disk is contributing to the jet launching. The existence of a faster
and Doppler de-boosted inner section, powered either from the rotation of the
inner regions of the accretion disk or by the spinning black hole, is
suggested by the kinematic properties and by the observed limb brightening of
the flow.

###### Key Words.:

galaxies: jets – galaxies: active – instrumentation: high angular resolution

##  1 Introduction

The launching mechanism of relativistic extragalactic jets is a much debated
and crucial topic for understanding these fascinating objects. There is
currently a broad consensus on the fundamental role that the strong magnetic
fields anchored in the accretion disk play in the process. However, to fully
reproduce the observed properties of such magnetically-driven jets, e.g.
Lorentz factors as high as $\sim$50 and opening angles as small as half a
degree, is still a challenge for theorists. General relativistic MHD
simulations (e.g. Tchekhovskoy et al., 2011) have shown that highly
collimated, relativistic jets can be efficiently powered by the black hole’s
rotational energy extracted through the magnetic field, as described in the
work of Blandford & Znajek (1977). This scenario is supported by a recent
observational study of a large sample of blazars (Ghisellini et al., 2014),
indicating that the contribution of a spinning black hole is necessary to
account for the estimated jet powers. Alternatively, the rotation of the
accretion disk has been proposed as a viable power source for magnetic jet
launching (Blandford & Payne, 1982). Predictions for the terminal Lorentz
factor of such disk winds are diverse, with values ranging from mildly
relativistic ($\Gamma<3$, McKinney & Gammie, 2004; De Villiers et al., 2005)
to relativistic ($\Gamma\sim 10$, Komissarov et al., 2007).

A fundamental tool for discriminating between the two launching mechanisms is
the direct imaging of the jet base through VLBI observations and the
determination of its transverse size. This kind of study has been rarely
performed because the jet base is usually not resolved. Moreover, owing to
synchrotron opacity effects (Königl, 1981), the location of the VLBI core is
often shifted at large distances, ${\sim}10^{4}-10^{6}$ Schwarzschild radii
($R_{\rm S}$), from the black hole (e.g. Marscher et al., 2008). One exception
is M 87, for which a distance of only 14-23 $R_{\rm S}$ has been inferred for
the 43 GHz core (Hada et al., 2011). In the same source, VLBI pilot studies at
1.3 $\rm mm$ (with the Event Horizon Telescope) have determined a core size of
${\sim 5.5}$ $R_{\rm S}$ (Doeleman et al., 2012), which represents an upper
limit for the true size of the jet apex. This important measurement indicates
that the jet of M 87 is likely anchored close to the Innermost Stable Circular
Orbit (ISCO). Performing such an analysis in other sources is clearly
fundamental for testing the variety of possible scenarios. However, while
future instruments with improved capabilities (e.g. phased ALMA) will
certainly enrich the sample of suitable targets, this is currently quite
limited.

In this letter we present, for the first time, a detailed imaging of the jet
base in Cygnus A through Global VLBI observations at 86 $\rm GHz$ (GMVA).
Among the powerful FR II radio galaxies, Cygnus A is one of the very few two-
sided sources whose jet launching region can be probed with sufficient
sensitivity at high radio frequencies. At the source redshift (z=0.056)111$1$
$\rm{mas}{\sim}1.084$ $\rm pc$, assuming a $\rm\Lambda CDM$ cosmology with
${\rm H_{0}}=70.5$ $\rm{h}^{-1}$ $\rm km$ $\rm{s}^{-1}$ $\rm{Mpc}^{-1}$,
$\rm\Omega_{M}=0.27$, $\rm{\Omega_{\Lambda}}=0.73$ and assuming a black hole
mass of $2.5\times 10^{9}\rm M_{\odot}$ (Tadhunter et al., 2003), GMVA
observations allow scales of the order of $10^{2}$ $R_{\rm S}$ to be probed.
The large jet viewing angle enables a prominent counter-jet to be detected in
this source, which represents a fundamental advantage when investigating the
nuclear regions of a jet. In fact, since the transverse width can be
determined continuously along the two-sided flow, the assumption-dependent
back-extrapolations of the properties of the jet apex can be replaced by
actual measurements.

The following analysis complements a previous VLBI study at 43 $\rm GHz$ of
the jet transverse profile and of the kinematic properties (Boccardi et al.,
2016, hereafter B1). This revealed the presence of a pronounced transverse
stratification, both in flux density (limb brightening) and bulk Lorentz
factor ($1\lesssim\Gamma\lesssim 2.5$), typical of a flow with spine-sheath
structure.

Table 1: Log of observations at 86 GHz (3 mm) and characteristics of the clean maps shown in Fig. 1. Col. 1: Date of observation. Col. 2: Array. VLBA_3mm indicates the part of the Very Long Baseline Array (all but the Hancock and St. Croix antennas) which is equipped with 86 GHz receivers; On - Onsala 20 m; Eb - Effelsberg 100-m; PdB - Plateau de Bure Interferometer; PV - IRAM 30-m atop Pico Veleta; Mh - Metsähovi. Col. 3: Beam FWHM and position angle for untapered data with uniform weighting. Date | Array | Beam FWHM
---|---|---
|  | $\mathrm{[mas,deg]}$
08/05/2009 | VLBA_3mm, On, | $0.12\times 0.05,-4.2^{\circ}$
| Eb, PdB, PV |
10/10/2009 | VLBA_3mm, On, | $0.10\times 0.04,-1.9^{\circ}$
| Eb, PdB, PV |
07/05/2010 | VLBA_3mm, On, | $0.11\times 0.04,-7.1^{\circ}$
| Eb, PdB, PV, Mh |

##  2 Observations and VLBI images

The GMVA data set comprises three epochs from observations in 2009/2010 with a
cadence between five and seven months (Table 1). The duration of the
observations was 15-16 hours, ${\sim}$8 of which were spent on Cygnus A. This,
together with the large number of antennas employed and with the favourable
declination of the source, enabled good uv-coverage. Data were recorded in
single polarization mode with a recording rate of 512 $\rm Mb/s$, which
corresponds to an observing bandwidth of 128 $\rm MHz$. The correlation took
place at the MPIfR in Bonn, Germany. The data reduction was carried out in
AIPS (Greisen, 1990). Aside from the standard calibration procedures, special
care was devoted during the fringe fitting. First, as described in Martí-Vidal
et al. (2012), the manual phase calibration was applied. Single sub-band
phases and delays were calculated using one or more scans showing strong
detections from the calibrator J2015+3710. Then, the global fringe fitting was
performed (Schwab & Cotton, 1983). The amplitude calibration was based on the
measured system temperatures and gain curves at each telescope, and it
included a correction for atmospheric opacity.

The imaging and self-calibration of amplitude and phase were performed in
DIFMAP (Shepherd et al., 1994). The final clean maps obtained are presented in
Fig. 1. The nuclear region is detected with a high signal-to-noise ratio S/N
(between $\sim$1200-2000), with the brightest feature – which we refer to as
the “core” – showing a prominent variability over the monitoring period. The
total flux density varies between ${\sim}$0.65 and 1 $\rm Jy$ and is
consistent with the measurement of ${\sim}1$ $\rm Jy$ provided by 86 $\rm GHz$
single-dish observations at IRAM (Agudo et al., 2010).

Figure 1: Clean maps of Cygnus A at 86 $\rm GHz$ from GMVA observations in
2009-2010. Contours represent isophotes at 0.6, 1.2, 2.4, 4.8, 9.6, 19.2, 38.4
$\rm mJy/beam$. The data weighting is uniform. The beam sizes are slightly
larger compared to the values reported in Tab. 1 due to the application of a
Gaussian taper of 0.1 at a uv-radius of 3000 $\rm M\lambda$. The typical noise
level in the images is ${\sim}0.15$ $\rm mJy/beam$.

The amplitude variation during the self-calibration procedure was not greater
than $\sim$30%, thus this percentage can be assumed as the uncertainty
associated with the flux density measurements.

A visual inspection of the maps already shows the high morphological
complexity of Cygnus A at this frequency and at this angular resolution. A
prominent counter-jet is detected in all three epochs. The jet emission is
relatively smooth – i.e. there are no bright features besides the core – but
at the same time rich with sub-structures. It can also be noticed that, due to
the low S/N in the faintest regions, the emission is not detected over the
full jet cross-section in a single epoch. The jet structure can be recovered
much better after stacking the three maps, as shown in Fig. 2. In this image,
characterized by a reduced noise level of ${\sim}0.1$ $\rm mJy/beam$, the jet
limb brightening already observed at 43 GHz also appears to be better defined,
especially in the outer and faintest regions of the approaching jet.

Figure 2: Stacked image of Cygnus A at 86 $\rm GHz$. Each single epoch was
restored with a circular beam of 0.1 $\rm{mas}$. The alignment was based on
the position of the peak of intensity. Contours represent isophotes at 0.4,
0.8, 1.6, 3.2, 6.4, 12.8, 51.6, 102.4, 204.8 $\rm mJy/beam$.

##  3 Analysis and discussion

###  3.1 Transverse width profile

The analysis of the jet transverse structure in Cygnus A was first presented
at the frequency of 43 $\rm GHz$ in B1. In this study it was shown that the
flow emanates parabolically from the position of an emission gap located at
${\sim}0.2$ $\rm{mas}$ east of the brightest feature. A measurement of the jet
transverse width was also provided for the nuclear region. However, the
minimum size of ${\sim}30$ $\rm\mu as$ occurring in the vicinity of the faint
emission gap was about seven times smaller than the beam size in the
transverse direction. In this letter we exploit the improved resolution
achieved by GMVA observations for obtaining a more solid estimate of the
transverse width at the base of the flow. Given the better quality and the
reduced noise level of the stacked image as compared to the single epochs, we
focus our analysis on the former. The stacked image was restored with a
circular beam of 0.1 $\rm mas$, which is approximately the natural resolution
in the transverse direction (Table 1). Then, it was sliced pixel by pixel
(1px=0.02 $\rm mas$) along the jet axis using the AIPS task SLICE. The
transverse intensity distribution in each slice was finally fitted with a
single Gaussian profile using the task SLFIT.

Figure 3 shows the dependence of the de-convolved jet width on the distance
from the core $z$ in the central ${\sim}1.5$ $\rm{mas}$ of the source. The
distance $z$, also expressed in $R_{\bf S}$, was de-projected assuming a
viewing angle $\theta=74.5^{\circ}$ (as calculated in B1). The error bars are
conservatively assumed equal to one fifth of the measured FWHM. The minimum
jet width $d_{\rm min}$ occurs for ${z=-0.02}$ $\rm{mas}$, where the measured
transverse size is $d_{\rm min}=(51\pm 22)$ $\rm\mu as$. This corresponds to
about half of the nominal resolution in the transverse direction. However, as
discussed by several authors (e.g. Fomalont, 1989; Lobanov, 2005; Kovalev et
al., 2005), the limiting resolution of a certain brightness distribution
mainly depends on the S/N of the detection. Following Lobanov (2005), the
resolution limit $d_{\rm lim}$ for the case of a Gaussian profile fitted to a
brightness distribution with uniform weighting is

$d_{\rm lim}=\frac{4}{\pi}\,\sqrt{\pi\ln(2)b\ln\left(\frac{(S_{\rm
p}/\sigma)}{(S_{\rm p}/\sigma)-1}\right)}$ (1)

where $S_{\rm p}$ is the peak flux density of the transverse intensity
profile, $\sigma$ is the rms noise of the map and $b$ the equivalent circular
beam (for uniform weighting) expressed as a function of the major and minor
beam axes, $b_{\rm maj}$ and $b_{\rm min}$, as $b=\sqrt{b_{\rm maj}\cdot
b_{\rm min}}$. The value of $d_{\rm lim}$ was calculated for each slice, and
its profile as a function of $z$ is also reported in Fig. 3. Except for a
faint area in the counter-jet (${z\sim-0.5}$ $\rm{mas}$), the measured width
is everywhere well above the resolution limit, therefore the jet is resolved
in the nuclear region.

Figure 3: Transverse width profile at 86 $\rm GHz$ for the stacked image.
Error bars for the width measurements (green points with label $d_{\rm meas}$)
are assumed equal to 1/5 of each convolved FWHM. The dashed black line with
label $d_{\rm lim}$ indicates the profile of the resolution limit.

###  3.2 Minimum width and implications for the jet launching

The minimum jet width $d_{\rm min}=(51\pm 22)$ $\mu$as measured at 86 $\rm
GHz$ translates into a linear size of $\sim(227\pm 98)$ $R_{\bf S}$. Unlike
the case of M 87 (Doeleman et al., 2012), the jet apex appears to be much
wider than the ISCO radius, which varies between 1 and 9 $R_{\bf S}$ depending
on the BH spin and on the kind of orbit (prograde or retrograde) of the
BH/accretion disk system. These results suggest that at least part of the jet
base in Cygnus A is anchored at large disk radii from the central BH, i.e. the
outer regions of the accretion disk are involved in the jet launching process.
Based on the sub-parsec scale kinematic study presented in B1, we infer that
such a disk wind is characterized by mildly relativistic speeds, with the bulk
Lorentz factor varying from $\Gamma\sim 1$ at the jet edge to $\Gamma\sim 2.5$
in sections of the flow closer to the jet axis.

We point out, however, that our width measurement does not exclude the
existence of both a narrower and an even broader component. Concerning the
former, it should be recalled that in jets seen at large viewing angles
Doppler de-boosting affects not only the receding side but also the
approaching one. For sufficiently high speeds, the Doppler factor can be much
smaller than one, so that the flow is completely invisible. In the case of
Cygnus A, the gradual fading of the accelerating inner-jet components was
directly observed in our kinematic analysis. This, together with the observed
limb brightening, strongly indicate that an invisible faster flow exists in
the central body of the jet. This spine may be driven either from the rotation
of the inner regions of the accretion disk or by the spinning black hole
itself. A hypothetical broader and diffuse disk wind, instead, is likely
resolved out in these high resolution images. Indeed VLBI observations at 5
$\rm GHz$ (Carilli et al., 1991) support the existence of such a component. At
this frequency, the jet appears as an almost cylindrical resolved flow with
transverse width between 2 and 2.2 $\rm mas$, i.e. of the order of 9000
$R_{\bf S}$. Although this large size may partially be the effect of
scattering in the interstellar medium, there is an indication that the jet
base extends to much larger disk radii than measured at 86 $\rm GHz$.

###  3.3 Comparison with results at 43 GHz

The discussion in the previous section has not quite addressed the problem of
truly locating the central engine in Cygnus A. Given the roughly symmetric
expansion of jet and counter-jet shown in Fig. 3, the most natural
interpretation would be that the supermassive black hole is located in the
vicinity of $z\sim 0$. However, although the jet base is assumed to be the
narrowest part of the jet in the simplest models, this may not necessarily be
the case. An interesting argument comes from the comparison of the 86 $\rm
GHz$ width profile with results at 43 $\rm GHz$. For the purposes of the
comparison, the analysis was repeated at the two frequencies following the
method described in Sect. 3.1 after convolving the maps with the same circular
beam of 0.15 $\rm{mas}$. Results are presented in Fig. 4.

Figure 4: Width profiles at 43 and 86 $\rm GHz$. At 43 $\rm GHz$, the analysis
was performed on the stacked image presented in B1, Fig. 2. The arrow
indicates the position of the emission gap in the 43 $\rm GHz$ map.

We first notice that the dip in the 43 $\rm GHz$ profile at ${z\sim-0.25}$
$\rm{mas}$, i.e. in the vicinity of the gap of emission, is not observed at 86
$\rm GHz$. Correspondingly, the 86 $\rm GHz$ map (Fig. 3) does not show a
pronounced emission gap.222In principle, this comparison may be affected by a
misalignment of the maps due to an opacity shift of the cores. However, this
is expected to be negligible at high frequencies, as confirmed by a cross-
correlation of the jet parameters from which we infer a shift $<30$ $\rm\mu
as$. The observed frequency dependence of the properties of this region
supports the idea that the gap is the result of absorption, which becomes very
reduced at 86 $\rm GHz$. If synchrotron opacity is responsible for the it, the
gap most likely marks the true location of the central engine. A detailed
spectral analysis using more sensitive data is in progress with the aim of
clarifying the nature of this feature. Absorption may also explain another
striking aspect of the width profiles in Fig. 4. While results at the two
frequencies are quite consistent in the jet side, the 43 $\rm GHz$ counter-jet
appears narrower than at 86 $\rm GHz$. The puzzling asymmetry between the
opening angles of the approaching and receding side observed at 43 GHz in B1,
almost disappears at 86 GHz. The existence of a parsec scale free-free
absorber covering the counter-jet was suggested by several studies (Krichbaum
et al., 1998; Bach et al., 2004; Struve & Conway, 2010) and may well explain
the observed features. The results presented in this letter hint at the
presence of sub-pc scale material that is still highly opaque at 43 GHz but
not at 86 $\rm GHz$.

##  4 Conclusions

We have presented the first VLBI images of the radio galaxy Cygnus A at the
frequency of 86 $\rm GHz$. The enhanced spatial resolution of these maps
allowed us to investigate the transverse width profile at the onset of the
two-sided flow. The jet base appears wide with a minimum width of
${\sim}(227\pm 98)$ $R_{\bf S}$. This value is much higher than the ISCO
radius, thus we infer that the emission is produced by a mildly relativistic,
parabolically expanding disk wind. The existence of a central and faster jet,
possibly driven by the black hole and invisible due to Doppler de-boosting, is
however not excluded. It is, on the contrary, suggested by the kinematic
properties and by the observed limb brightening of the flow. In the hypothesis
that the Blandford & Znajek (1977) and Blandford & Payne (1982) mechanisms act
simultaneously in relativistic jets, giving rise to a spine-sheath structure
of the flow (e.g. Hardee et al., 2007; Xie et al., 2012), we indeed expect the
slower disk wind to always dominate the emission in radio galaxies and the BH-
driven jet to be the main observable component in blazars due to differential
boosting. The comparison of the width profiles at 43 and 86 $\rm GHz$
indicates that free-free opacity and/or synchrotron opacity are still strong
at 43 $\rm GHz$ but not at 86 $\rm GHz$. Observing at high radio frequencies
is therefore fundamental for unveiling the physical properties of jets at
their onset.

###### Acknowledgements.

We thank the referee for helpful comments. We also thank Andrei Lobanov and
Vassilis Karamanavis for the useful discussions, and Michael Lindqvist for his
support during the observations at Onsala. This research has made use of data
obtained with the Global Millimeter VLBI Array (GMVA), which consists of
telescopes operated by the MPIfR, IRAM, Onsala, Metsahovi, Yebes and the VLBA.
The data were correlated at the correlator of the MPIfR in Bonn, Germany. The
VLBA is an instrument of the National Radio Astronomy Observatory, a facility
of the National Science Foundation operated under cooperative agreement by
Associated Universities, Inc.

## References

  * Agudo et al. (2010) Agudo, I., Thum, C., Wiesemeyer, H., & Krichbaum, T. P. 2010, ApJS, 189, 1
  * Bach et al. (2004) Bach, U., Krichbaum, T. P., Middelberg, E., et al. 2004, in European VLBI Network on New Developments in VLBI Science and Technology, ed. R. Bachiller, F. Colomer, J.-F. Desmurs, & P. de Vicente, 155–156
  * Blandford & Payne (1982) Blandford, R. D. & Payne, D. G. 1982, MNRAS, 199, 883
  * Blandford & Znajek (1977) Blandford, R. D. & Znajek, R. L. 1977, MNRAS, 179, 433
  * Boccardi et al. (2016) Boccardi, B., Krichbaum, T. P., Bach, U., et al. 2016, A&A, 585, A33
  * Carilli et al. (1991) Carilli, C. L., Bartel, N., & Linfield, R. P. 1991, AJ, 102, 1691
  * De Villiers et al. (2005) De Villiers, J.-P., Hawley, J. F., Krolik, J. H., & Hirose, S. 2005, ApJ, 620, 878
  * Doeleman et al. (2012) Doeleman, S. S., Fish, V. L., Schenck, D. E., et al. 2012, Science, 338, 355
  * Fomalont (1989) Fomalont, E. B. 1989, in Astronomical Society of the Pacific Conference Series, Vol. 6, Synthesis Imaging in Radio Astronomy, ed. R. A. Perley, F. R. Schwab, & A. H. Bridle, 213
  * Ghisellini et al. (2014) Ghisellini, G., Tavecchio, F., Maraschi, L., Celotti, A., & Sbarrato, T. 2014, Nature, 515, 376
  * Greisen (1990) Greisen, E. W. 1990, in Acquisition, Processing and Archiving of Astronomical Images, ed. G. Longo & G. Sedmak, 125–142
  * Hada et al. (2011) Hada, K., Doi, A., Kino, M., et al. 2011, Nature, 477, 185
  * Hardee et al. (2007) Hardee, P., Mizuno, Y., & Nishikawa, K.-I. 2007, Ap&SS, 311, 281
  * Komissarov et al. (2007) Komissarov, S. S., Barkov, M. V., Vlahakis, N., & Königl, A. 2007, MNRAS, 380, 51
  * Königl (1981) Königl, A. 1981, ApJ, 243, 700
  * Kovalev et al. (2005) Kovalev, Y. Y., Kellermann, K. I., Lister, M. L., et al. 2005, AJ, 130, 2473
  * Krichbaum et al. (1998) Krichbaum, T. P., Alef, W., Witzel, A., et al. 1998, A&A, 329, 873
  * Lobanov (2005) Lobanov, A. P. 2005, ArXiv Astrophysics e-prints:0503225
  * Marscher et al. (2008) Marscher, A. P., Jorstad, S. G., D’Arcangelo, F. D., et al. 2008, Nature, 452, 966
  * Martí-Vidal et al. (2012) Martí-Vidal, I., Krichbaum, T. P., Marscher, A., et al. 2012, A&A, 542, A107
  * McKinney & Gammie (2004) McKinney, J. C. & Gammie, C. F. 2004, ApJ, 611, 977
  * Schwab & Cotton (1983) Schwab, F. R. & Cotton, W. D. 1983, AJ, 88, 688
  * Shepherd et al. (1994) Shepherd, M. C., Pearson, T. J., & Taylor, G. B. 1994, in Bulletin of the American Astronomical Society, Vol. 26, Bulletin of the American Astronomical Society, 987–989
  * Struve & Conway (2010) Struve, C. & Conway, J. E. 2010, A&A, 513, A10
  * Tadhunter et al. (2003) Tadhunter, C., Marconi, A., Axon, D., et al. 2003, MNRAS, 342, 861
  * Tchekhovskoy et al. (2011) Tchekhovskoy, A., Narayan, R., & McKinney, J. C. 2011, MNRAS, 418, L79
  * Xie et al. (2012) Xie, W., Lei, W.-H., Zou, Y.-C., et al. 2012, Research in Astronomy and Astrophysics, 12, 817


* [100] S. Sun, J. Li, H. Chen, T. Gong, X. Li, B. Hu, A study of resting-state eeg biomarkers for depression recognition, arXiv preprint arXiv:2002.11039 (2020).
  * [101] P. C. Koo, C. Berger, G. Kronenberg, J. Bartz, P. Wybitul, O. Reis, J. Hoeppner, Combined cognitive, psychomotor and electrophysiological biomarkers in major depressive disorder, European archives of psychiatry and clinical neuroscience 269 (7) (2019) 823–832.
  * [102] P. Fernández-Palleiro, T. Rivera-Baltanás, D. Rodrigues-Amorim, S. Fernández-Gil, M. del Carmen Vallejo-Curto, M. Álvarez-Ariza, M. López, C. Rodriguez-Jamardo, J. Luis Benavente, E. de Las Heras, et al., Brainwaves oscillations as a potential biomarker for major depression disorder risk, Clinical EEG and Neuroscience 51 (1) (2020) 3–9.
  * [103] M. M. Caudill, A. M. Hunter, I. A. Cook, A. F. Leuchter, The antidepressant treatment response index as a predictor of reboxetine treatment outcome in major depressive disorder, Clinical EEG and neuroscience 46 (4) (2015) 277–284.
  * [104] C. DeBattista, G. Kinrys, D. Hoffman, C. Goldstein, J. Zajecka, J. Kocsis, M. Teicher, S. Potkin, A. Preda, G. Multani, et al., The use of referenced-eeg (reeg) in assisting medication selection for the treatment of depression, Journal of psychiatric research 45 (1) (2011) 64–75.
  * [105] M. Arns, A. Etkin, U. Hegerl, L. M. Williams, C. DeBattista, D. M. Palmer, P. B. Fitzgerald, A. Harris, R. deBeuss, E. Gordon, Frontal and rostral anterior cingulate (racc) theta eeg in depression: Implications for treatment outcome?, European Neuropsychopharmacology 25 (8) (2015) 1190–1200.
  * [106] A. Wichniak, A. Wierzbicka, W. Jernajczyk, Sleep as a biomarker for depression, International review of psychiatry 25 (5) (2013) 632–645.
  * [107] S. Sun, J. Li, H. Chen, T. Gong, X. Li, B. Hu, A study of resting-state eeg biomarkers for depression recognition, arXiv preprint arXiv:2002.11039 (2020).
  * [108] Z. S. Syed, K. Sidorov, D. Marshall, Depression severity prediction based on biomarkers of psychomotor retardation, in: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, 2017, pp. 37–43.
  * [109] O. J. Robinson, B. J. Sahakian, Cognitive biomarkers in depression. (2013).
  * [110] J. R. Williamson, T. F. Quatieri, B. S. Helfer, G. Ciccarelli, D. D. Mehta, Vocal and facial biomarkers of depression based on motor incoordination and timing, in: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014, pp. 65–72.
  * [111] A. Othmani, D. Kadoch, K. Bentounes, E. Rejaibi, R. Alfred, A. Hadid, Towards robust deep neural networks for affect and depression recognition from speech, arXiv preprint arXiv:1911.00310 (2019).
  * [112] U. R. Acharya, S. L. Oh, Y. Hagiwara, J. H. Tan, H. Adeli, D. P. Subha, Automated eeg-based screening of depression using deep convolutional neural network, Computer methods and programs in biomedicine 161 (2018) 103–113.
  * [113] P. Sandheep, S. Vineeth, M. Poulose, D. Subha, Performance analysis of deep learning cnn in classification of depression eeg signals, in: TENCON 2019-2019 IEEE Region 10 Conference (TENCON), IEEE, 2019, pp. 1339–1344.
  * [114] H. Jebelli, M. M. Khalili, S. Lee, Mobile eeg-based workers’ stress recognition by applying deep neural network, in: Advances in Informatics and Computing in Civil and Construction Engineering, Springer, 2019, pp. 173–180.
  * [115] Y. Mohammadi, M. Hajian, M. H. Moradi, Discrimination of depression levels using machine learning methods on eeg signals, in: 2019 27th Iranian Conference on Electrical Engineering (ICEE), IEEE, 2019, pp. 1765–1769.
  * [116] S. D. Puthankattil, P. K. Joseph, Half-wave segment feature extraction of eeg signals of patients with depression and performance evaluation of neural network classifiers, Journal of Mechanics in Medicine and Biology 17 (01) (2017) 1750006.
  * [117] S. D. Puthankattil, P. K. Joseph, Classification of eeg signals in normal and depression conditions by ann using rwe and signal entropy, Journal of Mechanics in Medicine and biology 12 (04) (2012) 1240019.
  * [118] P. Sandheep, S. Vineeth, M. Poulose, D. Subha, Performance analysis of deep learning cnn in classification of depression eeg signals, in: TENCON 2019-2019 IEEE Region 10 Conference (TENCON), IEEE, 2019, pp. 1339–1344.
  * [119] H. Mallikarjun, H. Suresh, Depression level prediction using eeg signal processing, in: 2014 International Conference on Contemporary Computing and Informatics (IC3I), IEEE, 2014, pp. 928–933.
  * [120] J. Zhu, Y. Wang, R. La, J. Zhan, J. Niu, S. Zeng, X. Hu, Multimodal mild depression recognition based on eeg-em synchronization acquisition network, IEEE Access 7 (2019) 28196–28210.
  * [121] D. Shah, G. Y. Wang, M. Doborjeh, Z. Doborjeh, N. Kasabov, Deep learning of eeg data in the neucube brain-inspired spiking neural network architecture for a better understanding of depression, in: International Conference on Neural Information Processing, Springer, 2019, pp. 195–206.
  * [122] X. Zhang, B. Hu, L. Zhou, P. Moore, J. Chen, An eeg based pervasive depression detection for females, in: Joint International Conference on Pervasive Computing and the Networked World, Springer, 2012, pp. 848–861.
  * [123] H. Cai, X. Sha, X. Han, S. Wei, B. Hu, Pervasive eeg diagnosis of depression using deep belief network with three-electrodes eeg collector, in: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2016, pp. 1239–1246.
  * [124] O. Faust, P. C. A. Ang, S. D. Puthankattil, P. K. Joseph, Depression diagnosis support system based on eeg signal entropies, Journal of mechanics in medicine and biology 14 (03) (2014) 1450035.
  * [125] S. D. Kumar, D. Subha, Prediction of depression from eeg signal using long short term memory (lstm), in: 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), IEEE, 2019, pp. 1248–1253.
  * [126] S. Mahato, S. Paul, Detection of major depressive disorder using linear and non-linear features from eeg signals, Microsystem Technologies 25 (3) (2019) 1065–1076.
  * [127] X. Zhang, B. Hu, L. Zhou, P. Moore, J. Chen, An eeg based pervasive depression detection for females, in: Joint International Conference on Pervasive Computing and the Networked World, Springer, 2012, pp. 848–861.
  * [128] Z. Wan, J. Huang, H. Zhang, H. Zhou, J. Yang, N. Zhong, Hybrideegnet: A convolutional neural network for eeg feature learning and depression discrimination, IEEE Access 8 (2020) 30332–30342.
  * [129] H. Mallikarjun, H. Suresh, Depression level prediction using eeg signal processing, in: 2014 International Conference on Contemporary Computing and Informatics (IC3I), IEEE, 2014, pp. 928–933.
  * [130] X. Li, R. La, Y. Wang, J. Niu, S. Zeng, S. Sun, J. Zhu, Eeg-based mild depression recognition using convolutional neural network, Medical & biological engineering & computing 57 (6) (2019) 1341–1352.
  * [131] S. Mahato, S. Paul, Detection of major depressive disorder using linear and non-linear features from eeg signals, Microsystem Technologies 25 (3) (2019) 1065–1076.
  * [132] H. Cai, Z. Wang, Y. Zhang, Y. Chen, B. Hu, A virtual-reality based neurofeedback game framework for depression rehabilitation using pervasive three-electrode eeg collector, in: Proceedings of the 12th Chinese Conference on Computer Supported Cooperative Work and Social Computing, 2017, pp. 173–176.
  * [133] Y.-j. Li, F.-y. Fan, Classification of schizophrenia and depression by eeg with anns, in: 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, IEEE, 2006, pp. 2679–2682.
  * [134] H. Peng, B. Hu, Q. Liu, Q. Dong, Q. Zhao, P. Moore, User-centered depression prevention: An eeg approach to pervasive healthcare, in: 2011 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, IEEE, 2011, pp. 325–330.
  * [135] Y. Katyal, S. V. Alur, S. Dwivedi, R. Menaka, Eeg signal and video analysis based depression indication, in: 2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies, IEEE, 2014, pp. 1353–1360.
  * [136] Y. Mohan, S. S. Chee, D. K. P. Xin, L. P. Foong, Artificial neural network for classification of depressive and normal in eeg, in: 2016 IEEE EMBS conference on biomedical engineering and sciences (IECBES), IEEE, 2016, pp. 286–290.
  * [137] H. Kwon, S. Kang, W. Park, J. Park, Y. Lee, Deep learning based pre-screening method for depression with imagery frontal eeg channels, International Conference on Information and Communication Technology Convergence (ICTC) (2019).
  * [138] I.-M. Spyrou, C. Frantzidis, C. Bratsas, I. Antoniou, P. D. Bamidis, Geriatric depression symptoms coexisting with cognitive decline: a comparison of classification methodologies, Biomedical Signal Processing and Control 25 (2016) 118–129.
  * [139] J. Zhu, Y. Wang, R. La, J. Zhan, J. Niu, S. Zeng, X. Hu, Multimodal mild depression recognition based on eeg-em synchronization acquisition network, IEEE Access 7 (2019) 28196–28210.
  * [140] L. LIU, R. ZHOU, An important neural indicator of measuring depression: The asymmetry of resting frontal activity, Advances in Psychological Science 23 (6) (2015) 1000–1008.
  * [141] W. Wu, Y. Zhang, J. Jiang, M. V. Lucas, G. A. Fonzo, C. E. Rolle, C. Cooper, C. Chin-Fatt, N. Krepel, C. A. Cornelssen, et al., An electroencephalographic signature predicts antidepressant response in major depression, Nature biotechnology 38 (4) (2020) 439–447.
  * [142] E. Garcia-Ceja, M. Riegler, P. Jakobsen, J. Tørresen, T. Nordgreen, K. J. Oedegaard, O. B. Fasmer, Depresjon: a motor activity database of depression episodes in unipolar and bipolar patients, in: Proceedings of the 9th ACM Multimedia Systems Conference, 2018, pp. 472–477.
  * [143] H. Kristjánsdóttir, P. M. Salkovskis, B. H. Sigurdsson, E. Sigurdsson, A. Agnarsdóttir, J. F. Sigurdsson, Transdiagnostic cognitive behavioural treatment and the impact of co-morbidity: An open trial in a cohort of primary care patients, Nordic journal of psychiatry 70 (3) (2016) 215–223.
  * [144] N. Langer, E. J. Ho, L. M. Alexander, H. Y. Xu, R. K. Jozanovic, S. Henin, A. Petroni, S. Cohen, E. T. Marcelle, L. C. Parra, et al., A resource for assessing information processing in the developing brain using eeg and eye tracking, Scientific data 4 (1) (2017) 1–20.
  * [145] J. F. Cavanagh, A. Napolitano, C. Wu, A. Mueen, The patient repository for eeg data+ computational tools (pred+ ct), Frontiers in neuroinformatics 11 (2017) 67\.
  * [146] L. M. Alexander, J. Escalera, L. Ai, C. Andreotti, K. Febre, A. Mangone, N. Vega-Potler, N. Langer, A. Alexander, M. Kovacs, S. Litke, B. O’Hagan, J. Andersen, B. Bronstein, A. Bui, M. Bushey, H. Butler, V. Castagna, N. Camacho, E. Chan, D. Citera, J. Clucas, S. Cohen, S. Dufek, M. Eaves, B. Fradera, J. Gardner, N. Grant-Villegas, G. Green, C. Gregory, E. Hart, S. Harris, M. Horton, D. Kahn, K. Kabotyanski, B. Karmel, S. P. Kelly, K. Kleinman, B. Koo, E. Kramer, E. Lennon, C. Lord, G. Mantello, A. Margolis, K. R. Merikangas, J. Milham, G. Minniti, R. Neuhaus, A. Levine, Y. Osman, L. C. Parra, K. R. Pugh, A. Racanello, A. Restrepo, T. Saltzman, B. Septimus, R. Tobe, R. Waltz, A. Williams, A. Yeo, F. X. Castellanos, A. Klein, T. Paus, B. L. Leventhal, R. C. Craddock, H. S. Koplewicz, M. P. Milham, An open resource for transdiagnostic research in pediatric mental health and learning disorders, Scientific data 4 (2017) 170181.
  * [147] J. F. Cavanagh, A. W. Bismark, M. J. Frank, J. J. Allen, Multiple dissociations between comorbid depression and anxiety on reward and punishment processing: Evidence from computationally informed eeg, Computational Psychiatry 3 (2019) 1–17.
  * [148] X. Jiang, G.-B. Bian, Z. Tian, Removal of artifacts from eeg signals: a review, Sensors 19 (5) (2019) 987.
  * [149] A. K. Jain, J. Mao, K. M. Mohiuddin, Artificial neural networks: A tutorial, Computer 29 (3) (1996) 31–44.
  * [150] A. ollah Ansari, M. Khalili, Diagnosis of major depressive disorder with neural network models, International Journal of Electronics Communication and Computer Engineering (IJECCE) 5 (5) (2014) 1183–1186.
  * [151] M. Bachmann, L. Päeske, K. Kalev, K. Aarma, A. Lehtmets, P. Ööpik, J. Lass, H. Hinrikus, Methods for classifying depression in single channel eeg using linear and nonlinear signal analysis, Computer methods and programs in biomedicine 155 (2018) 11–17.
  * [152] M. Muzammel, H. Salam, Y. Hoffmann, M. Chetouani, A. Othmani, Audvowelconsnet: A phoneme-level based deep cnn architecture for clinical depression diagnosis, Machine Learning with Applications 2 (2020) 100005.
  * [153] A. Mosavi, S. Ardabili, A. R. Varkonyi-Koczy, List of deep learning models, in: International Conference on Global Research and Education, Springer, 2019, pp. 202–214.
  * [154] A. Craik, Y. He, J. L. Contreras-Vidal, Deep learning for electroencephalogram (eeg) classification tasks: a review, Journal of neural engineering 16 (3) (2019) 031001.
  * [155] X. Li, R. La, Y. Wang, B. Hu, X. Zhang, A deep learning approach for mild depression recognition based on functional connectivity using electroencephalography, Frontiers in Neuroscience 14 (2020).
  * [156] S. Mitra, S. N. Sarbadhikari, S. K. Pal, An mlp-based model for identifying qeeg in depression, International journal of bio-medical computing 43 (3) (1996) 179–187.
  * [157] U. R. Acharya, V. K. Sudarshan, H. Adeli, J. Santhosh, J. E. Koh, A. Adeli, Computer-aided diagnosis of depression using eeg signals, European neurology 73 (5-6) (2015) 329–336.
  * [158] D. Shah, G. Y. Wang, M. Doborjeh, Z. Doborjeh, N. Kasabov, Deep learning of eeg data in the neucube brain-inspired spiking neural network architecture for a better understanding of depression, in: International Conference on Neural Information Processing, Springer, 2019, pp. 195–206.
  * [159] Y. Xiaolong, W. Xiaoyun, J. Xinyi, L. Hongbo, Classification of depression with brain network characteristics based on multiphase map deep neural network equilibrium compensation, Journal of Medical Imaging and Health Informatics 10 (1) (2020) 134–138.
  * [160] P. B. Mitchell, A. Frankland, D. Hadzi-Pavlovic, G. Roberts, J. Corry, A. Wright, C. K. Loo, M. Breakspear, Comparison of depressive episodes in bipolar disorder and in major depressive disorder within bipolar disorder pedigrees, The British Journal of Psychiatry 199 (4) (2011) 303–309.
  * [161] L. Samalin, F. Bellivier, B. Giordana, L. Yon, V. Milhiet, W. El-Hage, P. Courtet, E. Hacques, N. Bedira, A. Dillenschneider, et al., Patients’ perspectives on residual symptoms in bipolar disorder: a focus group study, The Journal of nervous and mental disease 202 (7) (2014) 550–555.
  * [162] T. J. L. Kristeen Cherney, Effects of bipolar disorder on the body (October 25, 2018).
URL https://www.healthline.com/health/bipolar-disorder/effects-on-the-body#1

  * [163] M. De Hert, C. U. Correll, J. Bobes, M. Cetkovich-Bakmas, D. Cohen, I. Asai, J. Detraux, S. Gautam, H.-J. Möller, D. M. Ndetei, et al., Physical illness in patients with severe mental disorders. i. prevalence, impact of medications and disparities in health care, World psychiatry 10 (1) (2011) 52\.
  * [164] J. Angst, A. Gamma, F. Benazzi, V. Ajdacic, D. Eich, W. Rössler, Diagnostic issues in bipolar disorder, European Neuropsychopharmacology 13 (2003) 43–50.
  * [165] R. Young, J. Biggs, V. Ziegler, D. Meyer, Young mania rating scale, Handbook of psychiatric measures (2000) 540–542.
  * [166] J. B. Williams, A structured interview guide for the hamilton depression rating scale, Archives of general psychiatry 45 (8) (1988) 742–747.
  * [167] M. B. First, Structured clinical interview for the dsm (scid), The encyclopedia of clinical psychology (2014) 1–6.
  * [168] F. Kapczinski, F. Dal-Pizzol, A. L. Teixeira, P. V. Magalhaes, M. Kauer-Sant’Anna, F. Klamt, J. C. F. Moreira, M. A. de Bittencourt Pasquali, G. R. Fries, J. Quevedo, et al., Peripheral biomarkers and illness activity in bipolar disorder, Journal of psychiatric research 45 (2) (2011) 156–161.
  * [169] G. Scola, A. C. Andreazza, The role of neurotrophins in bipolar disorder, Progress in Neuro-Psychopharmacology and Biological Psychiatry 56 (2015) 122–128.
  * [170] B. S. Fernandes, M. L. Molendijk, C. A. Köhler, J. C. Soares, C. M. G. Leite, R. Machado-Vieira, T. L. Ribeiro, J. C. Silva, P. M. Sales, J. Quevedo, et al., Peripheral brain-derived neurotrophic factor (bdnf) as a biomarker in bipolar disorder: a meta-analysis of 52 studies, BMC medicine 13 (1) (2015) 289.
  * [171] A. C. Andreazza, M. Kauer-Sant’Anna, B. N. Frey, D. J. Bond, F. Kapczinski, L. T. Young, L. N. Yatham, Oxidative stress markers in bipolar disorder: a meta-analysis, Journal of affective disorders 111 (2-3) (2008) 135–144.
  * [172] F. Benedetti, V. Aggio, M. L. Pratesi, G. Greco, R. Furlan, Neuroinflammation in bipolar depression, Frontiers in Psychiatry 11 (2020) 71.
  * [173] M. İ. Atagün, Brain oscillations in bipolar disorder and lithium-induced changes, Neuropsychiatric disease and treatment 12 (2016) 589\.
  * [174] A. Khaleghi, A. Sheikhani, M. R. Mohammadi, A. M. Nasrabadi, S. R. Vand, H. Zarafshan, M. Moeini, Eeg classification of adolescents with type i and type ii of bipolar disorder, Australasian physical & engineering sciences in medicine 38 (4) (2015) 551–559.
  * [175] S. Z. Metin, T. T. Erguzel, G. Ertan, C. Salcini, B. Kocarslan, M. Cebi, B. Metin, O. Tanridag, N. Tarhan, The use of quantitative eeg for differentiating frontotemporal dementia from late-onset bipolar disorder, Clinical EEG and Neuroscience 49 (3) (2018) 171–176.
  * [176] M. Zelenina, D. Prata, Machine learning with electroencephalography features for precise diagnosis of depression subtypes, arXiv preprint arXiv:1908.11217 (2019).
  * [177] M. B. Fonseca, R. S. de Andrades, S. de Lima Bach, C. D. Wiener, J. P. Oses, et al., Bipolar and schizophrenia disorders diagnosis using artificial neural network, Neuroscience and Medicine 9 (04) (2018) 209.
  * [178] T. T. Erguzel, C. Uyulan, B. Unsalver, A. Evrensel, M. Cebi, C. O. Noyan, B. Metin, G. Eryilmaz, G. H. Sayar, N. Tarhan, Entropy: A promising eeg biomarker dichotomizing subjects with opioid use disorder and healthy controls, Clinical EEG and Neuroscience (2020) 1550059420905724.
  * [179] M. Seeck, L. Koessler, T. Bast, F. Leijten, C. Michel, C. Baumgartner, B. He, S. Beniczky, The standardized eeg electrode array of the ifcn, Clinical Neurophysiology 128 (10) (2017) 2070–2077.
  * [180] J. W. Kam, S. Griffin, A. Shen, S. Patel, H. Hinrichs, H.-J. Heinze, L. Y. Deouell, R. T. Knight, Systematic comparison between a wireless eeg system with dry electrodes and a wired eeg system with wet electrodes, NeuroImage 184 (2019) 119–129.
  * [181] C. Wu, I. MacLeod, A. I. Su, Biogps and mygene. info: organizing online, gene-centric information, Nucleic acids research 41 (D1) (2013) D561–D565.
  * [182] M. L. Phillips, H. A. Swartz, A critical appraisal of neuroimaging studies of bipolar disorder: toward a new conceptualization of underlying neural circuitry and a road map for future research, American Journal of Psychiatry 171 (8) (2014) 829–843.
  * [183] J. Potash, J. Toolan, J. Steele, E. Miller, J. Pearl, P. Zandi, T. Schulze, L. Kassem, S. Simpson, V. Lopez, D. MacKinnon, F. Mcmahon, The bipolar disorder phenome database: A resource for genetic studies, The American journal of psychiatry 164 (2007) 1229–37.  doi:10.1176/appi.ajp.2007.06122045.
  * [184] F. K. Goodwin, K. R. Jamison, Manic-depressive illness: bipolar disorders and recurrent depression, Vol. 1, Oxford University Press, 2007.
  * [185] E. Rejaibi, A. Komaty, F. Meriaudeau, S. Agrebi, A. Othmani, Mfcc-based recurrent neural network for automatic clinical depression recognition and assessment from speech, arXiv preprint arXiv:1909.07208 (2019).
  * [186] N. M. N. Leite, E. T. Pereira, E. C. Gurjao, L. R. Veloso, Deep convolutional autoencoder for eeg noise filtering, in: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2018, pp. 2605–2612.
  * [187] W. Samek, T. Wiegand, K.-R. Müller, Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models, arXiv preprint arXiv:1708.08296 (2017).


Deep Learning Architectures for Time Series Classification dl_models_for_tsc
Time series data refers to data that are ordered in time, such that each data
point is recorded with its time relationship. Time series data can have one
(univariate) or more (multivariate) channels with varying update rates. The
task of time series classification seeks to assign a sequence of data points
to some type of known event.

Within machine learning, deep learning refers to algorithms that use a cascade
of non-linear computational units (neurons) which allows them to learn
hierarchical representations of the data. However, one caveat of deep learning
is that the learned models cannot be understood by inspecting the model’s
internals.

Common Architectures One of the best-known architectures in deep learning is
the multi-layer perceptron (MLP) lippmann1987introduction. An MLP consist of
several layers, where every neuron of a layer is connected to every neuron in
the subsequent layer. Even though MLPs may function as universal function
approximators cybenko1989approximation, they have difficulties modeling the
dependencies between time steps of a time series, since they treat each input
feature independently by training the weights for all the features separately.
Nevertheless, MLPs still have a purpose in time series classification, since
the accuracy that can be achieved with an MLP can be used as a baseline to
evaluate more sophisticated deep learning architectures.

To allow a neural network to model temporal behavior, recurrent neural
networks (RNN) were proposed. An RNN makes sequential predictions for each
individual time step, while taking into account the predictions of the
previous time steps. Since the weights across the neurons in an RNN are
shared, it can generalize better over time, as well as process inputs of
variable length without increasing the model size. Although RNNs were
specifically designed to work on data with sequence dependencies, such as text
or sound, they also have their shortcomings. They are relatively slow to
train, due to the necessary sequential computation for each individual time
step. Additionally, they have limitations in modelling long sequences, for
which variants of RNNs have been proposed, namely LSTMs hochreiter1997long and
GRUs cho2014learning. Furthermore, traditional RNNs do not take into account
future time steps, which for time series classification may prove beneficial.
To address this issue, bidirectional variants of the previously mentioned
networks have been introduced.

An additional deep learning architecture that is suited for time series
classification is a convolutional neural network (CNN)
lecun1989backpropagation. Despite the fact that CNNs were originally proposed
for classifying images, they were shown to be well suited for the task of time
series classification fawaz2019deep. CNNs work by applying convolutional
filters on the whole time series utilizing a sliding window approach. Since
the trained weights of a filter are shared across all the time steps of the
time series, and since the convolutional operation of a time window is
independent of any previous computations, the training of CNNs is very fast,
due to the possible parallelization. An additional benefit of CNNs is that
they are exceptionally noise-resistant, which makes feature engineering less
of an issue. In the case of multivariate time series, they are also able to
capture cross channel correlations.

Recommendations from Literature Wang et al. wang2017time set a baseline for
what deep learning models should be able to achieve. They compared MLPs, as
well as CNN variants, namely fully convolutional networks (FCNs) and residual
networks (ResNets) to non-deep learning state-of-the-art models at that time,
such as COTE bagnall2015time and BOSS schafer2015boss. They showed that deep
learning models can achieve competitive accuracy’s, when compared to the
state-of-the-art models of traditional methods.

Fawaz et al. fawaz2019deep supported this claim in their extensive study,
where they trained 8730 deep learning models on 97 time series datasets, to
evaluate their suitability for time series classification, as well as to
compare them to other state-of-the-art models. The results of the study have
clearly shown that ResNets and FCNs were best suited for the time series
classification task when compared to MLPs and variants of RNNs. Furthermore,
they compared ResNet, the best performing deep learning model in their study,
to the best non-deep learning models for time-series classification at the
time. Their findings were that ResNets can compete, although slightly less
accurate, with the state-of-the-art non-deep learning models. However, the
measured difference in accuracy when comparing the results of ResNet with the
best performing non-deep learning model, namely HIVE-COTE lines2018time, which
is an ensemble of 37 classifiers, was not significant. More recent work by
Fawaz et al. fawaz2019inceptiontime introduced InceptionTime, a CNN that
utilizes inception modules szegedy2015going for time series classification,
achieving higher accuracy than ResNets.

Even though deep learning approaches do not yet beat other state-of-the-art
models for time-series classification, they do come very close, with the
recent work clearly demonstrating substantial potential for improvement. Also,
deep learning exhibits certain advantages which are not shared by traditional
methods. One of the most powerful properties of deep learning models is their
ability to learn meaningful representations from raw data, which is a
necessary property for end-to-end models that do not require manual feature
engineering and prior knowledge about the data. Additionally, due to the heavy
parallelization capabilities, CNNs can classify time series very fast, and
easily beat HIVE-COTE in regards to inference time. Therefore, since deep
learning approaches have proven themselves to excel for other data types (e.g.
image classification or natural language processing), we believe that their
rapid recent advances for time-series classification make them worth of
further investigation.

The Need for Interpretability interpretability

Considering that in high-stake domains a model’s decision may have a
substantial impact on the well-being of a person, as well as considerable
financial and legal consequences for a company, a single metric during the
model development, such as test set accuracy, is not sufficient to rely on
that a model will work as expected when it is in production. While metrics can
give an excellent summary of the overall performance, it may also be necessary
to understand what exact criteria lead a model to make a certain prediction,
by interpreting and analyzing its behavior.

However, what does interpreting a model, or model interpretability in the
context of machine learning mean and why is it actually important? Since this
term was used rather freely in the literature, Doshi-Velez & Kim
doshi2017towards defined interpretability in the context of machine learning
as the ability to explain or to present in understandable terms to a human.

According to Selvaraju et al. selvaraju2017grad, interpretability has
different uses depending on the quality of a model compared to humans. In the
case that a model performs worse than a human, interpretability can be used to
identify failure modes. If a model performs equally good as a human,
interpretability can be used to verify the model’s workings and to foster
trust that the model behaves as it should. When a model outperforms a human,
interpretability can be used to gain new insights and learn from the model.
The need for interpretability and its benefits are therefore evident.

Depending on a model’s degree of interpretability a distinction between white-
box and black-box models can be made, where the former are considered as
inherently interpretable or transparent. Black-box models, of which deep
neural networks are the most notable ones, are typically very complicated so
that by simply observing the input and a model’s internals we can not
understand what the contributing aspects of a model’s prediction have been.
Therefore, the role of interpretability methods (also referred to as XAI
methods) is to present the contributing factors of a model’s prediction in
human understandable form.

State of XAI for Time Series open_up_bb

The literature on interpreting black-box models for time series is rather
scarce when compared to the approaches for images and text. In a recent
evaluation of XAI methods, Guidotti et al. guidotti2018survey specifically
pointed out that there is a lack of literature addressing the problem of
explaining models for time series data, even though they analyzed over 60
methods for opening up black boxes. This is also supported by Adadi et al.
adadi2018peeking and Hohman et al. hohman2018visual in their surveys on the
state of explainable AI methods, where interpretability methods concerning
time series have not even been included, whereas methods for images, text and
tabular data were presented and discussed. Nevertheless, approaches to
interpret deep learning models in the domain of time series classification are
lately getting more traction.

Wang et al. wang2016representation used a deconvolutional network
zeiler2010deconvolutional to learn meaningful representations in an
unsupervised learning manner. They were able to show what the deconvolution
network learned by discretizing and converting the representations to a Markov
Matrix, so that they can be visualized as a graph. When drawing the graph,
they used the force directed placement algorithm which resulted in different
typical graph layouts of the representations depending on the class.

Using an autoencoder, Gee et al. gee2019explaining proposed an approach to
learn class representative prototypes by penalizing prototypes that are too
similar to each other with respect to the L2 distance in the latent space.
They visualize the learned representations by first applying Principal
Component Analysis (PCA) on the learned representations of the data to reduce
the number of dimensions, after which the cosine similarity is calculated
between the reduced vectors so that a similarity matrix is produced. Finally,
using t-distributed stochastic neighbor embedding (t-SNE, an algorithm for
dimensionality reduction suited for visualizing data) the similarity matrix is
reduced to three dimensions for visualization purposes. The prototypes are
visualized from the dimensionally reduced latent space by selecting
representations from the class specific clusters (e.g. from the center of
mass) in the visualized latent space. After projecting a representation of the
reduced vector back to the original latent space, the decoder part of the
autoencoder can be used to visualize it. The authors argue that prototypes are
representative examples that describe influential features in the latent
representations, and provide insight into features across the used training
samples.

Goodfellow et al. goodfellow2018towards trained a CNN to classify univariate
ECG data which represents the heart rhythm of a person. They used class
activation maps (CAMs) zhou2016learning to highlight the most important
aspects of a time series that contributed to a decision. CAMs visualize the
feature importance by projecting the output layer’s weights back on to the
last convolutional layer’s feature maps. While, to a certain degree, CAMs
offer a simple way to identify feature importance, one main disadvantage of
using CAMs is that a global average pooling (GAP) layer is mandatory before
the output layer. This limits the application of this XAI method, given that
CNNs typically have a number of fully connected layers between the
convolutional layers and the output layer, with the purpose of increasing the
networks accuracy. Assaf et al. assaf2019explainable utilized a CNN for a
multivariate time series regression problem, which consisted of giving a
forecast of the average power consumption of a photo-voltaic power plant. To
highlight the most important features of the input they employed Grad-CAM
selvaraju2017grad, which is a generalization of CAM where the GAP layer before
the output layer is not necessary.

While proposing a new neural network architecture, the multilevel Wavelet
Decomposition Network (mWDN), Wang et al. wang2018multilevel additionally
presented an importance analysis method for mWDNs. An mWDN decomposes a time
series in high and low frequency sub-sequences per network layer in an
iterative manner. In the first layer, the input time series is split into a
high and low frequency part, which are classified by an intermediate
classifier. In the next layer, the low frequency sub-sequence of the previous
layer is split again into a high and low frequency part, to which the results
of the intermediate classifier of the previous layer is added. The result of
this addition is then passed to the next intermediate classifier, and the
process is repeated for a defined number of layers. The importance analysis
method resembles sensitivity analysis simonyan2013deep, which is a well-known
model-agnostic XAI method that measures the importance of each feature by
perturbing it and observing the change in the classifiers output. The method
employed by Wang et al. performs the perturbations in the high and low
frequency sub-sequences of the time series in various layers, which makes it
possible to assign a relevance to each sub-sequence and layer by observing the
output. However, this causes the method to become model-specific.

Hsu et al. hsu2019multivariate used occlusion based attention mechanisms and
applied them to time series for interpretable early classification of
multivariate time series. The method uses a sliding window with which non-
overlapping sub-sequences of a time series are extracted and padded with zeros
so that they can be used as an input for a classifier. A model makes a
prediction for each individual sample, which results in a probability
distribution over the input windows that can be used to assign a relevance to
each sub-sequence.

Finally, in a recent work, Arnout et al. arnout2019towards compared multiple
existing interpretability methods, namely saliency maps simonyan2013deep,
Layer-wise Relevance Propagation (LRP) bach2015pixel, DeepLIFT
shrikumar2017learning, Local Interpretable Model-Agnostic Explanations LIME
ribeiro2016should and SHapley Additive exPlanations SHAP lundberg2017unified,
and applied them to LSTMs, fully convolutional networks, as well as ResNets.
They evaluated the explanations by using the perturbation method, which is a
method that perturbs the most relevant input features according to an XAI
method and observes the confidence drop of the classifier compared to the
original prediction.

Considering that Arnout et al. evaluated multiple XAI methods, we will first
briefly describe the idea behind each one. The term saliency maps is often
used interchangeably with sensitivity analysis, whereas saliency maps are a
visualization after the feature importance has been determined, typically
using sensitivity analysis. Nonetheless, it can also be used in combination
with other XAI methods. LRP is a method that takes the output value of the
network for a specific input, and redistributes it throughout the network back
to the input space, while taking into consideration the weights of a neuron
and the activations computed for the specific input. DeepLIFT has a similar
approach to LRP, by propagating the output of the network back to the input
space, however, it redistributes the output value relative to the weights of
the model and the activation difference of the neurons compared to a reference
sample, which has to be specified for each class individually. LIME is a
model-agnostic method, meaning, it can be used to explain any model. LIME
generates an explanation by training a local interpretable classifier, with
respect to a single prediction. The training data that is needed to create
such an interpretable classifier is generated by taking a specific input,
permuting it, and labeling the permutations by using the model. Finally, the
SHAP method calculates the individual feature importance of the input by
calculating the importance of each input feature with respect to an average
prediction of the model. In their experiments, the authors identified that
perturbing the relevant input features identified using DeepLIFT and LRP
resulted in the largest confidence drop for CNNs, while saliency maps and SHAP
worked the best for LSTMs. In the case of ResNets, SHAP identified the most
relevant features, while LIME performed worst for all tested architectures.