$\displaystyle\leq\delta\left(\sqrt{\mathcal{O}(V(\mathcal{G}_{n}))}+\sqrt{\mathcal{O}(V(\mathcal{G}_{n}))+\mathcal{O}(V(\mathcal{G}_{n}))-\log\delta}\right)$
$\displaystyle=\delta\left(\sqrt{\mathcal{O}(V(\mathcal{G}_{n}))}+\sqrt{\mathcal{O}(V(\mathcal{G}_{n}))-\log\delta}\right)$
$\displaystyle\lesssim\delta\mathcal{O}(\sqrt{V(\mathcal{G}_{n})})$

Apply (i) which implies $V(\mathcal{G}_{n})=\mathcal{O}(k_{n}^{2})$ and we
have the displayed result.

∎

Next we state a slightly modified version of the key results of Chernozhukov
et al. (2014) that’s applicable for our context.

######  Theorem A.9 (Gaussian approximation to suprema of empirical processes
index by VC type classes; Chernozhukov et al. (2014)).

Fix $n\geq 1$. Let $(\mathcal{G}_{n},||\cdot||)$ be a subset of a normed
separable space of real functions $f:\mathcal{X}\to\mathbb{R}$ and is equipped
with an envelope $F_{n}$. Suppose:

  1. (i)

$\mathcal{G}_{n}$ is pre-Gaussian. That is, there exists a tight Gaussian
random variable $G_{P,n}$ in $l^{\infty}(\mathcal{G}_{n})$ with mean zero and
covariance function,

$\mathbb{E}[G_{P,n}(f)G_{P,n}(g)]=P(fg)=\mathbb{E}[f(Z_{1})g(Z_{1})],\quad\text{for
all $f,g\in\mathcal{G}_{n}$}$

  2. (ii)

The $\epsilon$-covering number of $\mathcal{G}_{n}$ satisfies
$\sup_{Q}N(\epsilon||F_{n}||_{Q,2},\mathcal{G}_{n},L_{2}(Q))\leq\left(\frac{A_{n}}{\epsilon}\right)^{\nu_{n}}$
for some $\nu_{n}\geq 1$ and $A_{n}>0$, and where the supremum is taken over
all discrete probability measures $Q$ such that $||F_{n}||_{Q,2}>0$;

  3. (iii)

For some $b_{n}\geq\sigma_{n}>0$ and $q\in[4,\infty]$, we have
$\sup_{f\in\mathcal{G}_{n}}P|f|^{k}\leq\sigma_{n}^{2}b_{n}^{k-2}$ for $k=2,3$
and $||F_{n}||_{P,q}\leq b_{n}$.

Let $Z_{n}:=\mathbb{G}_{n}f$. Then for every $\gamma\in(0,1)$, there exists a
random variable $\tilde{Z}_{n}:=\sup_{f\in\mathcal{G}_{n}}G_{P,n}f$ such that

$\mathbb{P}\left(|Z_{n}-\tilde{Z}_{n}|>\frac{b_{n}K_{n}}{\gamma^{1/2}n^{1/2-1/q}}+\frac{(b_{n}\sigma_{n})^{1/2}K_{n}^{3/4}}{\gamma^{1/2}n^{1/4}}+\frac{(b_{n}\sigma_{n}^{2}K_{n}^{2})^{1/3}}{\gamma^{1/3}n^{1/6}}\right)\leq
C\left(\gamma+\frac{\log n}{n}\right)$

where $K_{n}:=c\nu_{n}\max\left\\{\log
n\,,\,\left(\left(1+\sqrt{1+\log\frac{A_{n}b_{n}}{\sigma_{n}}}\right)\right)^{2}\right\\}$
and $c,C>0$ are constants that only depend on $q$.

######  Remark A.3.

This result is nothing more than Corollary 2.2 of Chernozhukov et al. (2014),
which is based on their key result Theorem 2.1. We refer to their paper for
the proof. But let’s remark on what small proof modifications we need to adapt
their result to our Theorem A.9. The major difference between our stated
result and their Corollary 2.2 is the condition on the constant $A$ in the
covering number bound. Their Corollary 2.2 requires $A\geq e$ but we do not
impose this requirement here. Indeed, from Lemma A.8(i), it is unnatural to
require that $A_{n}\geq e$ for all $n$, especially since we only have an upper
bound for $V(\mathcal{G}_{n})$ and not a lower bound. For their proofs, the
authors only require the condition $A\geq e$ to arrive at the uniform entropy
integral condition
$J(\delta,\mathcal{F})\lesssim\delta\sqrt{\nu\log(A/\delta)}$. From Lemma
A.8(iv), we have instead a slightly larger bound of
$J(\delta,\mathcal{G}_{n})\leq\delta\sqrt{\nu_{n}}(1+\sqrt{1+\log(A_{n}/\delta)})$.
Consequently and by inspecting the proofs of their Corollary 2.2, it suffices
to replace their definition of $K_{n}=c\nu(\log n\vee\log\frac{Ab}{\sigma})$
with our slightly larger $K_{n}$, then the remainder of their proof goes
through to our case.

This following result will conclude Step II.

######  Proposition A.10.

Fix any $\gamma\in(0,1)$ and assume $k_{n}/n\to 0$. Then there exists a mean
zero Gaussian process $G_{P,n}$ in $\ell^{\infty}(\mathcal{J}_{n})$ with the
displayed covariance function (17) such that the random variables
$Z_{n}:=\sup_{h\in\mathcal{J}_{n}}\frac{\sqrt{n}}{\sigma_{\varepsilon}t_{n}(h)}\mathcal{R}_{n}(h)$
and $\widetilde{Z}_{n}:=\sup_{h\in\mathcal{J}_{n}}G_{P,n}h$ have,

$\displaystyle|Z_{n}-\widetilde{Z}_{n}|$
$\displaystyle\lesssim\mathcal{O}_{\mathbb{P}}\left(\frac{k_{n}^{13/2}(\log
k_{n})^{9/2}(\log n)}{\gamma^{1/2}n^{1/2}}+\frac{k_{n}^{15/4}(\log
k_{n})^{9/4}(\log n)^{3/4}}{\gamma^{1/2}n^{1/4}}+\frac{k_{n}^{17/6}(\log
k_{n})^{3/2}(\log n)^{2/3}}{\gamma^{1/3}n^{1/6}}\right)$

###### Proof.

We apply Theorem A.9 with the choice of $q=\infty$ and recall the notations
from that theorem statement. Fix any $n\geq 1$ and any $g\in\mathcal{G}_{n}$.
Firstly the second moment is,

$P|g|^{2}=\frac{P|h^{*}|^{2}}{\sigma_{\varepsilon}^{2}(\sqrt{P|h^{*}|^{2}}+a_{n})^{2}}\leq\frac{1}{\sigma_{\varepsilon}^{2}}$

For the third moment,

$\displaystyle P|g|^{3}$
$\displaystyle=\frac{1}{(\sigma_{\varepsilon}\sqrt{P|h^{*}|^{2}}+a_{n})^{3}}P|h^{*}|^{3}$
$\displaystyle\leq\frac{1}{(\sigma_{\varepsilon}\sqrt{P|h^{*}|^{2}}+a_{n})^{3}}P|h^{*}|^{3}$
$\displaystyle\leq\frac{1}{\sigma_{\varepsilon}^{3}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}||h^{*}||+a_{n}\right)^{3}}\sqrt{P|h^{*}|^{6}}$
$\displaystyle\leq\frac{1}{\sigma_{\varepsilon}^{3}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}||h^{*}||+a_{n}\right)^{3}}\sqrt{\sup_{j}\mathbb{E}[\xi_{j}^{6}]\lambda_{1}^{3}f_{n}(\lambda_{k_{n}})^{6}||h^{*}||^{6}}$
$\displaystyle=C\frac{f_{n}(\lambda_{k_{n}})^{3}||h^{*}||^{3}}{\sigma_{\varepsilon}^{3}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}||h^{*}||+a_{n}\right)^{3}}$
$\displaystyle\leq
C\frac{f_{n}(\lambda_{k_{n}})^{3}}{\sigma_{\varepsilon}^{3}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}+a_{n}\right)^{3}}$

Thus it suffices to set,

$\displaystyle\sigma_{n}$ $\displaystyle:=\frac{1}{\sigma_{\varepsilon}},$
$\displaystyle b_{n}$
$\displaystyle:=\max\left\\{\frac{1}{\sigma_{\varepsilon}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}+a_{n}\right)}\,,\,C\frac{f_{n}(\lambda_{k_{n}})^{3}}{\sigma_{\varepsilon}^{3}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}+a_{n}\right)^{3}}\right\\}$
$\displaystyle=\frac{1}{\sigma_{\varepsilon}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}+a_{n}\right)}\max\left\\{1\,,\,C\frac{f_{n}(\lambda_{k_{n}})^{3}}{\sigma_{\varepsilon}^{2}\left(f_{n}(\lambda_{1})\lambda_{k_{n}}^{1/2}+a_{n}\right)^{2}}\right\\}$
$\displaystyle\lesssim(\sqrt{k_{n}\log k_{n}})\max\\{1,(k_{n}\log
k_{n})^{4}\\}$ $\displaystyle\lesssim k_{n}^{9/2}(\log k_{n})^{9/2}$

for which we obtain $\sup_{g\in\mathcal{G}_{n}}P|g|^{2}\leq\sigma_{n}^{2}$ and
$\sup_{g\in\mathcal{G}_{n}}P|g|^{3}\leq\sigma_{n}^{2}b_{n}$ and $F_{n}\leq
b_{n}$. Note that $\frac{b_{n}}{\sigma_{n}}\lesssim k_{n}^{9/2}(\log
k_{n})^{9/2}$.

Let’s now obtain a bound for $K_{n}$. Observe that using Lemma A.8,

$\displaystyle\nu_{n}\log\frac{A_{n}b_{n}}{\sigma_{n}}$
$\displaystyle=\nu_{n}\log\frac{b_{n}}{\sigma_{n}}+\log K+\log
V(\mathcal{G}_{n})+V(\mathcal{G}_{n})\log(16e)$
$\displaystyle\lesssim\mathcal{O}(k_{n}^{2})\mathcal{O}\left(\log
k_{n}+\log\log k_{n}\right)+\mathcal{O}(1)+\mathcal{O}(\log
k_{n})+\mathcal{O}(k_{n}^{2})$ $\displaystyle=\mathcal{O}(k_{n}^{2}\log
k_{n})$

and so

$\displaystyle\nu_{n}\left(1+\sqrt{1+\log\frac{A_{n}b_{n}}{\sigma_{n}}}\right)^{2}$
$\displaystyle=2\nu_{n}+2\sqrt{\nu_{n}}\sqrt{\nu_{n}+\nu_{n}\log\frac{A_{n}b_{n}}{\sigma_{n}}}+\nu_{n}\log\frac{A_{n}b_{n}}{\sigma_{n}}$
$\displaystyle=\mathcal{O}(k_{n}^{2})+\sqrt{\mathcal{O}(k_{n}^{2})}\sqrt{\mathcal{O}(k_{n}^{2})+\mathcal{O}(k_{n}^{2}\log
k_{n})}+\mathcal{O}(k_{n}^{2}\log k_{n})$
$\displaystyle=\mathcal{O}(k_{n}^{2}\log k_{n})$

This implies,

$\displaystyle K_{n}$ $\displaystyle:=c\nu_{n}\max\left\\{\log
n\,,\,\left(1+\sqrt{1+\log\frac{A_{n}b_{n}}{\sigma_{n}}}\right)^{2}\right\\}$
$\displaystyle\lesssim\max\\{\mathcal{O}(k_{n})^{2}\log
n\,,\,\mathcal{O}(k_{n}^{2}\log k_{n})\\}$
$\displaystyle=\mathcal{O}(k_{n}^{2}\log n)$

where we used that $k_{n}/n\to 0$.

Thus by Theorem A.9, for the random variables $Z_{n}$ there exists a random
variable $\widetilde{W}_{n}$ with which we have a mean-zero Gaussian process
$\\{G_{P,n}(g)\\}_{g\in\mathcal{G}_{n}}$ with covariance function

$\mathbb{E}[G_{P,n}(g_{1})G_{P,n}(g_{2})]=\frac{\left\langle\Gamma^{1/2}\Gamma^{\dagger}h_{1},\Gamma^{1/2}\Gamma^{\dagger}h_{2}\right\rangle}{(||\Gamma^{1/2}\Gamma^{\dagger}h_{1}||+a_{n})(||\Gamma^{1/2}\Gamma^{\dagger}h_{2}||+a_{n})},\quad\text{for
all $g_{1},g_{2}\in\mathcal{G}_{n}$}$

such that $g_{i}=\frac{h_{i}^{*}}{\sqrt{P|h_{i}^{*}|^{2}}}$ with
$h_{i}^{*}\in\mathcal{J}_{n}^{*}$, $i=1,2$. Indeed, thanks to again to the
Riesz representation theorem, we can identify this Gaussian process $G_{P,n}$
indexed by $\mathcal{G}_{n}$ with covariance function on the left hand side
with a Gaussian process indexed by $\mathcal{J}_{n}$ having the covariance
function on the right hand side. So with some abuse of notations, we can write
$\widetilde{Z}_{n}=\sup_{g\in\mathcal{G}_{n}}G_{P,n}g=\sup_{h\in\mathcal{J}_{n}}G_{P,n}h$.
Moreover, $Z_{n}$ and $\widetilde{Z}_{n}$ satisfy, for all $\gamma\in(0,1)$

$\displaystyle|Z_{n}-\widetilde{Z}_{n}|$
$\displaystyle=\mathcal{O}_{\mathbb{P}}\left(\frac{k_{n}^{13/2}(\log
k_{n})^{9/2}(\log n)}{\gamma^{1/2}n^{1/2}}+\frac{k_{n}^{15/4}(\log
k_{n})^{9/4}(\log n)^{3/4}}{\gamma^{1/2}n^{1/4}}+\frac{k_{n}^{17/6}(\log
k_{n})^{3/2}(\log n)^{2/3}}{\gamma^{1/3}n^{1/6}}\right)$

∎

We can finally summarize everything and put Steps I and II together.

######  Theorem A.11.

Define $\widetilde{Z}_{n}:=\sup_{h\in\mathcal{J}_{n}}G_{P,n}h$ where
$\\{G_{P,n}(h)\\}_{h\in\mathcal{J}_{n}}$ is a mean zero Gaussian process on
$\ell^{\infty}(\mathcal{J}_{n})$ with covariance function (17). Then for
sufficiently large $n$,

$\left|\sup_{h\in\mathcal{J}_{n}}\left\langle\frac{\sqrt{n}}{{\sigma_{\varepsilon}t_{n}(h)}}(\hat{\rho}-\hat{\Pi}_{k_{n}}\rho),h\right\rangle-\widetilde{Z}_{n}\right|\lesssim\mathcal{O}_{\mathbb{P}}\left(k_{n}^{11/2}(\log
k_{n})^{3/2}+\frac{k_{n}^{13/2}(\log k_{n})^{9/2}(\log n)}{n^{1/6}}\right)$

###### Proof.

By (19), Propositions A.7 and A.10 with an arbitrarily fixed $\gamma\in(0,1)$
and using the notations therein,

$\displaystyle\left|\sup_{h\in\mathcal{J}_{n}}\left\langle\frac{\sqrt{n}}{{\sigma_{\varepsilon}t_{n}(h)}}(\hat{\rho}-\hat{\Pi}_{k_{n}}\rho),h\right\rangle-\widetilde{Z}_{n}\right|$
$\displaystyle\leq\sup_{h\in\mathcal{J}_{n}}\left|\frac{\sqrt{n}}{\sigma_{\varepsilon}t_{n}(h)}\left\langle\mathcal{T}_{n}+\mathcal{S}_{n}+\mathcal{Y}_{n},h\right\rangle\right|+\left|\sup_{h\in\mathcal{J}_{n}}\frac{\sqrt{n}}{\sigma_{\varepsilon}t_{n}(h)}\left\langle\mathcal{R}_{n},x\right\rangle-\widetilde{Z}_{n}\right|$
$\displaystyle\lesssim\sqrt{n}\mathcal{O}_{\mathbb{P}}\left(\frac{k_{n}^{11/2}(\log
k_{n})^{3/2}}{n^{1/2}}\right)$
$\displaystyle\quad+\mathcal{O}_{\mathbb{P}}\left(\frac{k_{n}^{13/2}(\log
k_{n})^{9/2}(\log n)}{\gamma^{1/2}n^{1/2}}+\frac{k_{n}^{15/4}(\log
k_{n})^{9/4}(\log n)^{3/4}}{\gamma^{1/2}n^{1/4}}+\frac{k_{n}^{17/6}(\log
k_{n})^{3/2}(\log n)^{2/3}}{\gamma^{1/3}n^{1/6}}\right)$

The result follows by taking the higher order terms. ∎

The main result of Theorem 2.1 displayed in the main text is thus simply
Theorem A.11 normalized by the appropriate rate.

## References

  * Bosq (2012) Bosq, D. (2012): _Linear Processes in Function Spaces: Theory and Applications_ , vol. 149, Springer Science & Business Media.
  * Cai et al. (2006) Cai, T. T., P. Hall, et al. (2006): “Prediction in functional linear regression,” _The Annals of Statistics_ , 34, 2159–2179.
  * Cardot et al. (2003) Cardot, H., F. Ferraty, A. Mas, and P. Sarda (2003): “Testing hypotheses in the functional linear model,” _Scandinavian Journal of Statistics_ , 30, 241–255.
  * Cardot et al. (1999) Cardot, H., F. Ferraty, and P. Sarda (1999): “Functional linear model,” _Statistics & Probability Letters_, 45, 11–22.
  * Cardot et al. (2007) Cardot, H., A. Mas, and P. Sarda (2007): “CLT in functional linear regression models,” _Probability Theory and Related Fields_ , 138, 325–361.
  * Cardot and Sarda (2011) Cardot, H. and P. Sarda (2011): “Functional linear regression,” in _The Oxford Handbook of Functional Data Analysis_.
  * Chernozhukov et al. (2014) Chernozhukov, V., D. Chetverikov, K. Kato, et al. (2014): “Gaussian approximation of suprema of empirical processes,” _The Annals of Statistics_ , 42, 1564–1597.
  * Conway (1994) Conway, J. B. (1994): _A Course in Functional Analysis_ , Springer, 2nd ed.
  * Crambes and Mas (2013) Crambes, C. and A. Mas (2013): “Asymptotics of prediction in functional linear regression with functional outputs,” _Bernoulli_ , 19, 2627–2651.
  * Cuesta-Albertos et al. (2019) Cuesta-Albertos, J. A., E. García-Portugués, M. Febrero-Bande, and W. González-Manteiga (2019): “Goodness-of-fit tests for the functional linear model based on randomly projected empirical processes,” _The Annals of Statistics_ , 47, 439–467.
  * Dauxois et al. (1982) Dauxois, J., A. Pousse, and Y. Romain (1982): “Asymptotic Theory for the Principal Component Analysis of a Vector Random Function: Some Applications to Statistical Inference,” _Journal of Multivariate Analysis_ , 12, 136–154.
  * Goia and Vieu (2016) Goia, A. and P. Vieu (2016): “An introduction to recent advances in high/infinite dimensional statistics,” .
  * Hilgert et al. (2013) Hilgert, N., A. Mas, N. Verzelen, et al. (2013): “Minimax adaptive tests for the functional linear model,” _Annals of Statistics_ , 41, 838–869.
  * Hörmann et al. (2015) Hörmann, S., Ł. Kidziński, and M. Hallin (2015): “Dynamic functional principal components,” _Journal of the Royal Statistical Society: Series B: Statistical Methodology_ , 319–348.
  * Horváth and Kokoszka (2012) Horváth, L. and P. Kokoszka (2012): _Inference for functional data with applications_ , vol. 200, Springer Science & Business Media.
  * Hsing and Eubank (2015) Hsing, T. and R. Eubank (2015): _Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators_ , vol. 997, John Wiley & Sons.
  * Kato (1995) Kato, T. (1995): _Perturbation Theory for Linear Operators_ , Springer Science & Business Media, 2nd ed.
  * Leung and Tam (2021) Leung, R. C. W. and Y.-M. Tam (2021): “Supplement to “A Small-Uniform Statistic for the Inference of Functional Linear Regressions”,” .
  * Panaretos et al. (2013) Panaretos, V. M., S. Tavakoli, et al. (2013): “Fourier analysis of stationary time series in function space,” _The Annals of Statistics_ , 41, 568–603.
  * Powell (1994) Powell, M. J. (1994): “A direct search optimization method that models the objective and constraint functions by linear interpolation,” in _Advances in optimization and numerical analysis_ , Springer, 51–67.
  * Ramsay and Silverman (2005) Ramsay, J. and B. W. Silverman (2005): _Functional Data Analysis_ , Springer-Verlag New York, 2nd ed.
  * Runarsson and Yao (2005) Runarsson, T. P. and X. Yao (2005): “Search biases in constrained evolutionary optimization,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 35, 233–243.
  * Stancu-Minasian (2012) Stancu-Minasian, I. M. (2012): _Fractional programming: theory, methods and applications_ , vol. 409, Springer Science & Business Media.
  * van der Vaart and Wellner (1996) van der Vaart, A. W. and J. A. Wellner (1996): _Weak Convergence and Empirical Processes: With Applications to Statistics_ , Springer.
  * Wang et al. (2016) Wang, J.-L., J.-M. Chiou, and H.-G. Müller (2016): “Functional Data Analysis,” _Annual Review of Statistics and Its Application_ , 3, 257–295.
  * Yao et al. (2005) Yao, F., H.-G. Müller, and J.-L. Wang (2005): “Functional linear regression analysis for longitudinal data,” _The Annals of Statistics_ , 2873–2903.


11institutetext: Dept. of Electrical and Computer Engineering, Johns Hopkins
University, USA 22institutetext: Dept. of Neuroradiology, Johns Hopkins School
of Medicine, USA 22email<EMAIL_ADDRESS>
# A Multi-Task Deep Learning Framework to Localize the Eloquent Cortex in
Brain Tumor Patients Using Dynamic Functional Connectivity

Naresh Nandakumar 11 Niharika Shimona D’souza Komal Manzoor 1122 Jay J. Pillai
22 Sachin K. Gujar 22 Haris I. Sair 22 and Archana Venkataraman 11

###### Abstract

We present a novel deep learning framework that uses dynamic functional
connectivity to simultaneously localize the language and motor areas of the
eloquent cortex in brain tumor patients. Our method leverages convolutional
layers to extract graph-based features from the dynamic connectivity matrices
and a long-short term memory (LSTM) attention network to weight the relevant
time points during classification. The final stage of our model employs multi-
task learning to identify different eloquent subsystems. Our unique training
strategy finds a shared representation between the cognitive networks of
interest, which enables us to handle missing patient data. We evaluate our
method on resting-state fMRI data from 56 brain tumor patients while using
task fMRI activations as surrogate ground-truth labels for training and
testing. Our model achieves higher localization accuracies than conventional
deep learning approaches and can identify bilateral language areas even when
trained on left-hemisphere lateralized cases. Hence, our method may ultimately
be useful for preoperative mapping in tumor patients.

##  1 Introduction

The eloquent cortex consists of regions in the brain that are responsible for
language comprehension, speech, and motor function. Identifying and
subsequently avoiding these areas during a neurosurgery is crucial for
improving recovery and postoperative quality of life. However, localizing
these networks is challenging due to the varying anatomical boundaries of the
eloquent cortex across people [1, 2]. The language network has especially high
interindividual variability because it can appear on one or both hemispheres
[3]. The gold standard for preoperative functional mapping of eloquent areas
is intraoperative electrocortical stimulation (ECS) of the cerebral cortex
during surgery [4, 5]. While reliable, ECS is highly invasive and requires the
patient to be awake and responsive during surgery.

For these reasons, task-fMRI (t-fMRI) is becoming increasingly popular as a
noninvasive alternative to ECS. Typically, activation maps derived from t-fMRI
are inspected by an expert to determine the regions in the brain that are
recruited during the experimental condition. However, t-fMRI can be unreliable
for certain populations, like children and the cognitively disabled, due to
their inability to complete the paradigm [6, 7]. In contrast to t-fMRI,
resting-state fMRI (rs-fMRI) captures spontaneous fluctuations in the brain
when the subject is lying passively in the scanner. Unlike t-fMRI paradigms,
which are designed to activate a single area, correlations in the rs-fMRI data
can be used to identify multiple cognitive systems [8]. Recent work has moved
towards using rs-fMRI for presurgical mapping to avoid the above issues
associated with t-fMRI [7, 9].

Automatically localizing the eloquent cortex using rs-fMRI is a challenging
problem with limited success in the literature. For example, the authors of
[10, 11] demonstrate that spatial components identified by group ICA on the
rs-fMRI data coincide with the language and motor networks from t-fMRI. While
the result is promising, the spatial accuracy is highly variable across
patients. The work in [12] describes a multi-layer perceptron architecture
that classifies rs-fMRI networks at the voxel level using seed based
correlation maps; this method was extended in [13] to handle tumor cases.
However, the perceptron is trained on healthy subjects and may not accommodate
changes in brain organization due to the tumor. Finally, the method in [9] is
the first end-to-end graph neural network (GNN) to automatically localize
eloquent cortex in tumor patients. While this method achieves good
classification performance, separate GNNs must be trained and tested for each
eloquent network [9], which increases the training time, the overall number of
parameters, and the required training data.

There is growing evidence in the field that functional connectivity patterns
are not static, but evolve over time. In particular, studies have shown that
individual functional systems are more strongly present during specific
intervals of the rs-fMRI scan [14, 15]. Several studies have leveraged these
dynamic connectivity patterns for classification. For example, the work in
[16] uses a long-short term memory (LSTM) cell to learn time dependencies
within the rs-fMRI to discriminate patients with autism from controls. More
recent work by [17] and [18] has shown that combining static and dynamic
connectivity can achieve better patient versus control classification
performance than either set of features alone. However, these works focus on
group-level discrimination. We will leverage similar principles in this paper
to classify ROIs within a single patient.

We propose a novel multi-task deep learning framework that uses both
convolutional nerual networks (CNNs) and an LSTM attention network to extract
and combine dynamic connectivity features for eloquent cortex localization.
The final stage of our model employs multi-task learning (MTL) to implicitly
select the relevant time points for each network and simultaneously identify
regions of the brain involved in language processing and motor functionality.
Our model finds a shared representation between the cognitive networks of
interest, which enables us to handle missing data. This coupling also reduces
the number of model parameters, so that we can learn from limited patient
data. We evaluate our framework on rs-fMRI data from 56 brain tumor patients
while using task fMRI activations as surrogate ground-truth labels for
training and testing. Our model achieves higher localization accuracies than a
variety of baseline techniques, thus demonstrating its promise for
preoperative mapping.

##  2 Eloquent Cortex Localization Using Deep Learning

Figure 1: Top: Specialized convolutional layers identify dynamic patterns that
are shared across the functional systems. Bottom: The dynamic features are
input to an LSTM network to learn attention weights $\mathbf{a}^{l}$
(language) and $\mathbf{a}^{m}$ (motor). Right: MTL to classify the language
($\mathbf{L}$), finger ($\mathbf{M}_{1}$), foot ($\mathbf{M}_{2}$) and tongue
($\mathbf{M}_{3}$) networks.

Our framework makes two underlying assumptions. First, while the anatomical
boundaries of the eloquent cortex may shift across individuals, its functional
connectivity with the rest of the brain will be preserved [10]. Second, the
networks associated with the eloquent cortex phase in and out of synchrony
across the rs-fMRI scan [19]. Hence, isolating these key time points will help
to refine our localization. Fig. 1 illustrates our framework. In the top
branch, we use specialized convolutional filters to capture rs-fMRI co-
activation patterns from the dynamic connectivity matrices. In the bottom
branch, we use an LSTM to identify key time points where the language and/or
motor networks are more synchronous. We tie the activations from the LSTM
branch of our model into our MTL classification problem via our specialized
loss function.

### Input Connectivity Matrices.

We use the sliding window technique to obtain our connectivity matrices [20].
Let $N$ be the number of brain regions in our parcellation, $T$ be the total
number of sliding windows (i.e., time points in our model), and
$\\{\mathbf{W}^{t}\\}_{t=1}^{T}\in\mathbb{R}^{N\times N}$ be the dynamic
similarity matrices. $\mathbf{W}^{t}$ is constructed from the input time
courses $\\{\mathbf{X}^{t}\\}_{t=1}^{T}\in\mathbb{R}^{D\times N}$, where each
$\mathbf{X}^{t}$ is a segment of the rs-fMRI obtained with window size $D$.
The input $\mathbf{W}^{t}\in\mathbb{R}^{N\times N}$ is

$\mathbf{W}^{t}=\exp\Bigg{[}\frac{(\mathbf{X}^{t})^{T}\mathbf{X}^{t}}{\epsilon}-1\Bigg{]}$
(1)

where $\epsilon\geq 1$ is a user-specified parameter that controls decay speed
[9]. Recall that our setup must accommodate the presence of brain tumors that
vary across patients. Since these tumors represent non-functioning areas of
the brain, we follow the approach of [9] and treat the corresponding rows and
columns of the simlarity matrix as “missing data” by fixing them to zero. This
procedure removes the effect of the tumor regions on the downstream
convolution operations.

### Representation Learning for Dynamic Connectivity.

Our network leverages the specialized convolutional layers developed in [21]
for static analysis. The edge-to-edge (E2E) layer in Fig.1 acts across rows
and columns of the input matrix $\mathbf{W}^{t}$. Mathematically, let
$f~{}\in~{}\\{1,\cdots,F\\}$ be the E2E filter index,
$\mathbf{r}^{f}\in\mathbb{R}^{1\times N}$ be the row filter $f$,
$\mathbf{c}^{f}\in\mathbb{R}^{N\times 1}$ be the column filter $f$,
$\mathbf{b}\in\mathbb{R}^{F\times 1}$ be the E2E bias, and $\bm{\phi}(.)$ be
the activation function. For each time point $t$ the feature map
$\mathbf{H}^{f,t}\in\mathbb{R}^{N\times N}$ is computed as follows:

$\mathbf{H}^{f,t}_{i,j}=\bm{\phi}\Bigg{(}\sum_{n=1}^{N}\mathbf{r}^{f}_{n}\mathbf{W}^{t}_{i,n}+\mathbf{c}^{f}_{n}\mathbf{W}^{t}_{n,j}+\mathbf{b}_{f}\Bigg{)}.$
(2)

Effectively, the E2E filter output $\mathbf{H}_{ij}^{f,t}$ for edge $(i,j)$
extracts patterns associated with the neighborhood connectivity of node $i$
and node $j$. The edge-to-node (E2N) filter in Fig. 1 is a 1D convolution
along the columns of each feature map. Mathematically, let
$\mathbf{g}^{f}\in\mathbb{R}^{N\times 1}$ be E2N filter $f$ and
$\mathbf{p}\in\mathbb{R}^{F\times 1}$ be the E2N bias. The E2N output
$\mathbf{h}^{f,t}\in\mathbb{R}^{N\times 1}$ from input $\mathbf{H}^{f,t}$ is
computed as

$\mathbf{h}^{f,t}_{i}=\bm{\phi}\Bigg{(}\sum_{n=1}^{N}\mathbf{g}^{f}_{n}\mathbf{H}^{f,t}_{i,n}+\mathbf{p}_{f}\Bigg{)}.$
(3)

The E2E and E2N layers extract topological graph-theoretic features from the
connectivity data. Following the convolutional layers in the top branch, we
cascade two fully-connected (FC) layers to combine these learned topological
features for our downstream multi-task classification. In the bottom branch,
we use a node-to-graph (N2G) layer to extract features that will be input to
our LSTM network. The N2G filter acts as a 1D convolution along the first
dimension of the E2N output, effectively collapsing the node information to a
low dimensional representation for each time point. Let
$\mathbf{k}^{f}\in\mathbb{R}^{N\times 1}$ be N2G filter $f$ and
$\mathbf{d}\in\mathbb{R}^{F\times 1}$ be the bias. The N2G filter gives a
scalar output $q^{f,t}$ for each input $\mathbf{h}^{f,t}$ by

$q^{f,t}=\bm{\phi}\Bigg{(}\sum_{n=1}^{N}\mathbf{k}_{n}^{f}\cdot\mathbf{h}_{n}^{f,t}+\mathbf{d}_{f}\Bigg{)}.$
(4)

### Dynamic Attention Model.

Per time point, we define $\mathbf{q}^{t}=[q^{1,t}\cdots q^{F,t}]$ and feed
the vectors $\\{\mathbf{q}^{t}\\}_{t=1}^{T}$ into an LSTM module to learn
attention weights for our classification problem. The LSTM adds a cell state
to the basic recurrent neural network to help alleviate the vanishing gradient
problem, essentially by accumulating state information over time [22]. LSTMs
have demonstrated both predictive power for rs-fMRI analysis [16, 18] and the
ability to identify different brain states [23]. We choose $d=2$ as the output
dimension, and perform a softmax over each column of the LSTM output to get
the attention vectors $\mathbf{a}^{l}\in\mathbb{R}^{T\times 1}$ (language) and
$\mathbf{a}^{m}\in\mathbb{R}^{T\times 1}$ (motor). These attention vectors
provide information on which input connectivity matrices are more informative
for identifying the language or motor networks. The attention model outputs
are combined with the classifer during backpropogation in our novel loss
function.

### Multi-task Learning with Incomplete Data.

The black blocks in Fig. 1 show the multi-task FC (MT-FC) layers, where we
have four separate branches to identify the language, finger, foot, and tongue
areas. Up until this point, there has been an entirely shared representation
of the feature weights at each layer. Let
$\mathbf{L}^{t},\mathbf{M}^{t}_{1},\mathbf{M}^{t}_{2},$ and
$\mathbf{M}^{t}_{3}\in\mathbb{R}^{N\times 3}$ be the output of the language,
finger, foot, and tongue MT-FC layers, respectively, at time $t$. The $N\times
3$ matrix represents the region-wise assignment into one of three classes;
eloquent, tumor, and background. As in [9], we introduce the tumor as its own
learned class to remove any bias these regions may have introduced to the
algorithm.

We introduce a novel variant of a modified version of the risk-sensitive
cross-entropy loss function [24, 9], which is designed to handle membership
imbalance in multi-class problems. Let $\delta_{c}$ be the risk factor
associated with class $c$. If $\delta_{c}$ is small, then we pay a smaller
penalty for misclassifying samples that belong to class $c$. Due to a training
set imbalance, we set different values for the language class
($\delta^{l}_{c}$) and motor classes ($\delta^{m}_{c}$) respectively. Let
$\mathbf{Y}^{l},\mathbf{Y}^{m_{1}},\mathbf{Y}^{m_{1}},$ and
$\mathbf{Y}^{m_{3}}\in\mathbb{R}^{N\times 3}$ be one-hot encoding matrices for
the ground-truth class labels of the language and motor subnetworks. Notice
that our framework allows for overlapping eloquent labels, as brain regions
can be involved in multiple cognitive processes. Our loss function is the sum
of four terms:

$\begin{split}\mathcal{L}_{\Theta}(\\{\mathbf{W}^{t}\\}_{t=1}^{T},\mathbf{Y})=\sum_{n=1}^{N}\sum_{c=1}^{3}\Big{[}\underbrace{-\delta^{l}_{c}\log\Big{(}\sigma\Big{(}\sum_{t=1}^{T}\mathbf{L}^{t}_{n,c}\cdot\mathbf{a}^{l,t}\Big{)}\Big{)}\mathbf{Y}^{l}_{n,c}}_{\text{Language
Loss }\mathcal{L}_{l}}\\\
\underbrace{-\delta^{m}_{c}\log\Big{(}\sigma\Big{(}\sum_{t=1}^{T}\mathbf{M}^{t}_{1n,c}\cdot\mathbf{a}^{m,t}\Big{)}\Big{)}\mathbf{Y}^{m_{1}}_{n,c}}_{\text{Finger
Loss
}\mathcal{L}_{m_{1}}}\underbrace{-\delta^{m}_{c}\log\Big{(}\sigma\Big{(}\sum_{t=1}^{T}\mathbf{M}^{t}_{2n,c}\cdot\mathbf{a}^{m,t}\Big{)}\Big{)}\mathbf{Y}^{m_{2}}_{n,c}}_{\text{Foot
Loss }\mathcal{L}_{m_{2}}}\\\
\underbrace{-\delta^{m}_{c}\log\Big{(}\sigma\Big{(}\sum_{t=1}^{T}\mathbf{M}^{t}_{3n,c}\cdot\mathbf{a}^{m,t}\Big{)}\Big{)}\mathbf{Y}^{m_{3}}_{n,c}}_{\text{Tongue
Loss }\mathcal{L}_{m_{3}}}\Big{]}\end{split}$ (5)

where $\sigma(\cdot)$ is the sigmoid function. Our loss in Eq. (5) allows us
to handle missing information during training. For example, if we only have
ground-truth labels for some of the functional systems, then we can freeze the
other branches and just backpropagate the known loss terms. This partial
backpropagation will continue to refine the shared representation, thus
maximizing the amount of information mined from our training data. Note that
our formulation is agnostic to the length of the rs-fMRI scan (i.e. $T$),
which is useful in clinical practice.

### Implementation details.

We implement our network in PyTorch using the SGD optimizer with weight decay
$=5\times 10^{-5}$ for parameter stability, and momentum $=0.9$ to improve
convergence. We train our model with learning rate $=0.002$ and $300$ epochs,
which provides for reliable performance without overfitting. We used $D=45$
and a stride length of $5$ for the sliding window. We specified $F=25$ feature
maps in the convolutional branch, and $2$ layers in our LSTM. The LeakyReLU
with slope $=-0.1$ was used for $\bm{\phi}(.)$. Using cross validation, we set
the cross-entropy weights to $\bm{\delta}^{m}=(1.5,0.5,0.2)$, and
$\bm{\delta}^{l}=(2.25,0.5,0.2)$.

We compare the performance of our model against three baselines:

  * 1.

PCA + Multi-class linear SVM on dynamic connectivity matrices (SVM)

  * 2.

A multi-task GNN on static connectivity (MT-GNN)

  * 3.

A multi-task ANN with LSTM attention model (MT-ANN)

The first baseline is a traditional machine learning SVM approach to our
problem. The MT-GNN operates on static connectivity and does not have an LSTM
module. We include the MT-GNN to observe the difference in performance with
and without using dynamic information. The MT-ANN maintains the same number of
parameters as our model but has fully-connected layers instead of
convolutional layers. Therefore, the MT-ANN does not consider the network
organization of the input dynamic connectivity matrices.

##  3 Experimental Results

Figure 2: Top: Tumor boundaries for three patients. Left: One sagital and
axial view of a language network. Right: Coronal views of the motor sub-
networks for one patient.

### Dataset and Preprocessing.

We evaluate the methods on rs-fMRI data from $56$ brain tumor patients who
underwent preoperative mapping at our institution. These patients also
underwent t-fMRI scanning, which we use to derive pseudo ground-truth labels
for training and validation. Our dataset includes three different motor
paradigms that are designed to target distinct parts of the motor homunculus
[25]: finger tapping, tongue moving, and foot tapping. It also includes two
language paradigms, sentence completion and silent word generation. Since the
t-fMRI data was acquired for clinical purposes, not all patients performed
each task. The number of subjects that performed the language, finger, foot,
and tongue tasks are displayed in the left column of Table 1.

The fMRI data was acquired using a $3.0$ T Siemens Trio Tim (TR $=2000$ ms, TE
$=30$ ms, FOV $=24$ cm, res $=3.59\times 3.59\times 5$ mm). Preprocessing
steps include slice timing correction, motion correction and registration to
the MNI-152 template. The rs-fMRI was further bandpass filtered from $0.01$ to
$0.1$ Hz, spatially smoothed with a $6$ mm FWHM Gaussian kernel, scrubbed
using the ArtRepair toolbox [26] in SPM8, linearly detrended, and underwent
nuisance regression using the CompCor package [27]. We used the Craddocks
atlas to obtain $N$=384 brain regions [28]. Tumor boundaries for each patient
were manually delineated by a medical fellow using the MIPAV software package
[29]. An ROI was determined as belonging to the eloquent class if a majority
of its voxel membership coincided with that of the t-fMRI activation map.
Tumor labels were determined in a similar fashion according to the MIPAV
segmentations. A general linear model implemented in SPM8 was used to obtain
t-fMRI activation maps. Fig. 2 shows representative examples of the tumor
boundaries and each of the four cognitive networks of interest obtained from
t-fMRI.

Table 1: Class accuracy, overall accuracy, and ROC statistics. The number in the first column indicates number of patients who performed the task. Task | Method | Eloquent | Overall | AUC
---|---|---|---|---
Language (56) | SVM | $0.49$ | $0.59$ | $0.55$
| MT-ANN | $0.70$ | $0.71$ | $0.70$
| MT-GNN | $0.73$ | $0.74$ | $0.74$
| Proposed | $\mathbf{0.85}$ | $\mathbf{0.81}$ | $\mathbf{0.80}$
Finger (36) | SVM | $0.54$ | $0.61$ | $0.57$
| MT-ANN | $0.73$ | $0.75$ | $0.74$
| MT-GNN | $\underline{0.87}$ | $\mathbf{0.86}$ | $\mathbf{0.84}$
| Proposed | $\mathbf{0.88}$ | $\underline{0.85}$ | $\mathbf{0.84}$
Foot (17) | SVM | $0.58$ | $0.63$ | $0.60$
| MT-ANN | $0.72$ | $0.77$ | $0.74$
| MT-GNN | $0.82$ | $0.79$ | $0.79$
| Proposed | $\mathbf{0.86}$ | $\mathbf{0.85}$ | $\mathbf{0.82}$
Tongue (39) | SVM | $0.54$ | $0.60$ | $0.58$
| MT-ANN | $0.74$ | $0.76$ | $0.73$
| MT-GNN | $0.85$ | $0.81$ | $0.82$
| Proposed | $\mathbf{0.87}$ | $\mathbf{0.83}$ | $\mathbf{0.84}$

### Localization.

We use 8-fold cross validation (CV) to quantify our eloquent cortex
localization performance. Table 2 reports the eloquent per-class accuracy and
the area under the receiver operating characteristic curve (AUC) for detecting
the eloquent class on the testing data. Each MT-FC branch has separate
metrics. Our proposed method has the best overall performance, as highlighted
in bold. Even with attention from the LSTM layer, we observe that a fully-
connected ANN still is sub-par for our task compared to using the specialized
E2E, E2N, and N2G layers. Furthermore, our performance gains are most notable
when classifying the language and foot networks. The former is particularly
relevant for preoperative mapping, due to the difficulties in identifying the
language network even with ECS [1, 2]. Figure 3 shows the language (left) and
motor (right) attention vectors for all patients across time. We observe that
both systems phase in and out, such that when one system is more active, the
other is less active. This pattern lends credence to our hypothesis that
identifying the critical intervals is key for localization. Hence, our model
outperforms the static MT-GNN.

Figure 3: Language (L) and motor (R) attention weights for all patients.

### Bilateral Language Identification.

Finally, we test whether our model can recover a bilateral language network,
even when this case is not present in the training data. Here, we trained the
model on $51$ left-hemisphere language network patients and tested on the
remaining $5$ bilateral patients. Our model correctly predicted bilateral
parcels in all five subjects. Fig. 4 shows ground truth (blue) and predicted
language maps (yellow) for two example cases. The mean language class accuracy
for these five cases was $\mathbf{0.72}$. This is slighly lower than reported
in Table 1 likely due to the mismatch in training information.

Figure 4: Ground truth (Blue) and predicted (Yellow) language labels for two
subjects.

##  4 Conclusion

We have demonstrated a novel multi-task learning framework that uses dynamic
functional connectivity to identify separate sub-systems of the eloquent
cortex in brain tumor patients. Our model is extendable to adding more
eloquent sub-classes, as it finds a shared representation of the eloquent
cortex that can subsequently classify sub-regions of interest. Going one step
further, we show that our model can correctly identify bilateral language
networks even when trained on only unilateral cases. Finally, our attention
features suggest that using dynamic connectivity could be preferred to the
traditional static case. Our results demonstrate promise for using rs-fMRI
analysis in the preoperative phase for tumor resection procedures.

Acknowledgements: This work was supported by the National Science Foundation
CAREER award 1845430 (PI: Venkataraman) and the Research & Education
Foundation Carestream Health RSNA Research Scholar Grant RSCH1420.

## References

  * [1] G. A. Ojemann and H. A. Whitaker, “Language localization and variability,” Brain and language, vol. 6, no. 2, pp. 239–260, 1978.
  * [2] D. Tomasi and N. Volkow, “Language network: segregation, laterality and connectivity,” Molecular psychiatry, vol. 17, no. 8, p. 759, 2012.
  * [3] N. Tzourio-Mazoyer, G. Josse, F. Crivello, and B. Mazoyer, “Interindividual variability in the hemispheric organization for speech,” Neuroimage, vol. 21, no. 1, pp. 422–435, 2004.
  * [4] D. K. Gupta, P. Chandra, B. Ojha, B. Sharma, A. Mahapatra, and V. Mehta, “Awake craniotomy versus surgery under general anesthesia for resection of intrinsic lesions of eloquent cortex—a prospective randomised study,” Clinical neurology and neurosurgery, vol. 109, no. 4, pp. 335–343, 2007.
  * [5] M. S. Berger, J. Kincaid, G. A. Ojemann, and E. Lettich, “Brain mapping techniques to maximize resection, safety, and seizure control in children with brain tumors,” Neurosurgery, vol. 25, no. 5, pp. 786–792, 1989.
  * [6] S.-M. Kokkonen, J. Nikkinen, J. Remes, J. Kantola, T. Starck, M. Haapea, J. Tuominen, O. Tervonen, and V. Kiviniemi, “Preoperative localization of the sensorimotor area using independent component analysis of resting-state fmri,” Magnetic resonance imaging, vol. 27, no. 6, pp. 733–740, 2009.
  * [7] M. H. Lee, M. M. Miller-Thomas, T. L. Benzinger, D. S. Marcus, C. D. Hacker, E. C. Leuthardt, and J. S. Shimony, “Clinical resting-state fmri in the preoperative setting: are we ready for prime time?,” Topics in magnetic resonance imaging: TMRI, vol. 25, no. 1, p. 11, 2016.
  * [8] M. P. Van Den Heuvel and H. E. H. Pol, “Exploring the brain network: a review on resting-state fmri functional connectivity,” European neuropsychopharmacology, vol. 20, no. 8, pp. 519–534, 2010.
  * [9] N. Nandakumar, K. Manzoor, J. J. Pillai, S. K. Gujar, H. I. Sair, and A. Venkataraman, “A novel graph neural network to localize eloquent cortex in brain tumor patients from resting-state fmri connectivity,” in International Workshop on Connectomics in Neuroimaging, pp. 10–20, Springer, 2019.
  * [10] H. I. Sair et al., “Presurgical brain mapping of the language network in patients with brain tumors using resting-state fmri: Comparison with task fmri,” Human brain mapping, vol. 37, no. 3, pp. 913–923, 2016.
  * [11] Y. Tie et al., “Defining language networks from resting-state fmri for surgical planning—a feasibility study,” Human brain mapping, vol. 35, no. 3, pp. 1018–1030, 2014.
  * [12] C. D. Hacker, T. O. Laumann, N. P. Szrama, A. Baldassarre, A. Z. Snyder, E. C. Leuthardt, and M. Corbetta, “Resting state network estimation in individual subjects,” Neuroimage, vol. 82, pp. 616–633, 2013.
  * [13] E. C. Leuthardt, G. Guzman, S. K. Bandt, C. Hacker, A. K. Vellimana, D. Limbrick, M. Milchenko, P. Lamontagne, B. Speidel, J. Roland, et al., “Integration of resting state functional mri into clinical practice-a large single institution experience,” PloS one, vol. 13, no. 6, p. e0198349, 2018.
  * [14] N. C. Dvornek, X. Li, J. Zhuang, and J. S. Duncan, “Jointly discriminative and generative recurrent neural networks for learning from fmri,” in International Workshop on Machine Learning in Medical Imaging, pp. 382–390, Springer, 2019.
  * [15] W. Yan, H. Zhang, J. Sui, and D. Shen, “Deep chronnectome learning via full bidirectional long short-term memory networks for mci diagnosis,” in International conference on medical image computing and computer-assisted intervention, pp. 249–257, Springer, 2018.
  * [16] N. C. Dvornek, P. Ventola, K. A. Pelphrey, and J. S. Duncan, “Identifying autism from resting-state fmri using long short-term memory networks,” in International Workshop on Machine Learning in Medical Imaging, pp. 362–370, Springer, 2017.
  * [17] B. Rashid, M. R. Arbabshirani, E. Damaraju, M. S. Cetin, R. Miller, G. D. Pearlson, and V. D. Calhoun, “Classification of schizophrenia and bipolar patients using static and dynamic resting-state fmri brain connectivity,” Neuroimage, vol. 134, pp. 645–657, 2016.
  * [18] A. El-Gazzar, M. Quaak, L. Cerliani, P. Bloem, G. van Wingen, and R. M. Thomas, “A hybrid 3dcnn and 3dc-lstm based model for 4d spatio-temporal fmri data: An abide autism classification study,” in OR 2.0 Context-Aware Operating Theaters and Machine Learning in Clinical Neuroimaging, pp. 95–102, Springer, 2019.
  * [19] J. M. Kunert-Graf, K. Eschenburg, D. Galas, J. N. Kutz, S. Rane, and B. W. Brunton, “Extracting reproducible time-resolved resting state networks using dynamic mode decomposition,” Frontiers in computational neuroscience, vol. 13, p. 75, 2019.
  * [20] R. M. Hutchison, T. Womelsdorf, E. A. Allen, P. A. Bandettini, V. D. Calhoun, M. Corbetta, S. Della Penna, J. H. Duyn, G. H. Glover, J. Gonzalez-Castillo, et al., “Dynamic functional connectivity: promise, issues, and interpretations,” Neuroimage, vol. 80, pp. 360–378, 2013.
  * [21] J. Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, and G. Hamarneh, “Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment,” NeuroImage, vol. 146, pp. 1038–1049, 2017.
  * [22] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in Advances in neural information processing systems, pp. 802–810, 2015.
  * [23] H. Li and Y. Fan, “Brain decoding from functional mri using long short-term memory recurrent neural networks,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 320–328, Springer, 2018.
  * [24] S. Suresh et al., “Risk-sensitive loss functions for sparse multi-category classification problems,” Information Sciences, vol. 178, no. 12, pp. 2621–2638, 2008.
  * [25] C. R. Jack Jr et al., “Sensory motor cortex: correlation of presurgical mapping with functional mr imaging and invasive cortical mapping.,” Radiology, vol. 190, no. 1, pp. 85–92, 1994.
  * [26] P. K. Mazaika, F. Hoeft, G. H. Glover, A. L. Reiss, et al., “Methods and software for fmri analysis of clinical subjects,” Neuroimage, vol. 47, no. Suppl 1, p. S58, 2009.
  * [27] Y. Behzadi, K. Restom, J. Liau, and T. T. Liu, “A component based noise correction method (compcor) for bold and perfusion based fmri,” Neuroimage, vol. 37, no. 1, pp. 90–101, 2007.
  * [28] R. C. Craddock et al., “A whole brain fmri atlas generated via spatially constrained spectral clustering,” Human brain mapping, vol. 33, no. 8, pp. 1914–1928, 2012.
  * [29] M. J. McAuliffe, F. M. Lalonde, D. McGarry, W. Gandler, K. Csaky, and B. L. Trus, “Medical image processing, analysis and visualization in clinical research,” in Proceedings 14th IEEE Symposium on Computer-Based Medical Systems. CBMS 2001, pp. 381–386, IEEE, 2001.