# DSAM: A Deep Learning Framework for Analyzing Temporal and Spatial Dynamics
in Brain Networks

Bishal Thapaliya 1, 2 Robyn Miller 5 Jiayu Chen 1, 2 Yu-Ping Wang 4 Esra Akbas
1 Ram Sapkota 1, 2
Bhaskar Ray 1, 2 Pranav Suresh 1, 2 Santosh Ghimire 6 Vince Calhoun 1, 2, 3
Jingyu Liu 1,2
1Georgia State University 2TReNDs Center 3Georgia Institute of Technology
4Tulane University 5Los Alamos National Laboratory 6Tribhuvan University
Corresponding Author: Bishal Thapaliya<EMAIL_ADDRESS>
###### Abstract

Resting-state functional magnetic resonance imaging (rs-fMRI) is a noninvasive
technique pivotal for understanding human neural mechanisms of intricate
cognitive processes. Most rs-fMRI studies compute a single static functional
connectivity matrix across brain regions of interest, or dynamic functional
connectivity matrices with a sliding window approach. These approaches are at
risk of oversimplifying brain dynamics and lack proper consideration of the
goal at hand. While deep learning has gained substantial popularity for
modeling complex relational data, its application to uncovering the
spatiotemporal dynamics of the brain is still limited. In this study we
propose a novel interpretable deep learning framework that learns goal-
specific functional connectivity matrix directly from time series and employs
a specialized graph neural network for the final classification. Our model,
DSAM , leverages temporal causal convolutional networks to capture the
temporal dynamics in both low- and high-level feature representations, a
temporal attention unit to identify important time points, a self-attention
unit to construct the goal-specific connectivity matrix, and a novel variant
of graph neural network to capture the spatial dynamics for downstream
classification. To validate our approach, we conducted experiments on the
Human Connectome Project dataset with 1075 samples to build and interpret the
model for the classification of sex group, and the Adolescent Brain Cognitive
Development Dataset with 8520 samples for independent testing. Compared our
proposed framework with other state-of-art models, results suggested this
novel approach goes beyond the assumption of a fixed connectivity matrix, and
provides evidence of goal-specific brain connectivity patterns, which opens up
potential to gain deeper insights into how the human brain adapts its
functional connectivity specific to the task at hand. Our implementation can
be found on https://github.com/bishalth01/DSAM.

_Keywords_ Graph Neural Networks $\cdot$ Temporal Convolutional Networks
$\cdot$ Resting-State fMRI Data $\cdot$ Attention

##  1 Introduction

Resting-state functional magnetic resonance imaging (rs-fMRI) has emerged as a
noninvasive technique pivotal for unraveling the intricate function of the
human brain. This imaging modality has not only revolutionized our
understanding of brain function but has also showcased its potential as a
diagnostic tool for assessing brain disorders Fornito et al. (2015). By
capturing intrinsic neural fluctuations during periods of rest, rs-fMRI
enables researchers to delve into the dynamic interplay between brain regions,
offering a gateway to comprehend functional connectivity patterns. The
importance of understanding brain dynamics and related disorders has prompted
significant efforts from fields such as traditional machine learning Farahdel
et al. (2021); Ray et al. (2023); Thapaliya et al. (2021, 2023b) and
especially deep learning Heinsfeld et al. (2018); Sapkota et al. (2024);
Suresh et al. (2023).

The analysis of rs-fMRI data relies on graph-theoretical metrics to abstract
complex brain networks Azevedo et al. (2022). Dimensionality reduction is a
crucial step due to the high-dimensional nature of the data. This can be
achieved through (1) temporal dimension reduction by constructing static
connectivity matrices; (2) spatial dimension reduction by aggregating voxel-
level signals into a predefined region of interests (ROIs) Wang et al. (2019),
and (3) combined temporal and spatial reduction methods like group independent
component analysis Beckmann et al. (2005). These steps help handle the large
data volume and address the signal-to-noise ratio challenges Smith and Nichols
(2018). However, such reduction methods risk losing valuable information. For
example, collapsing the temporal dimension oversimplifies brain dynamics,
treating it as static, contrary to emerging evidence of continuous
connectivity evolution over time Avena-Koenigsberger et al. (2017); Liao et
al. (2017). Additionally, common association measures often rely on linear
models, despite the known nonlinear characteristics of brain signals and
interactions Duggento et al. (2018); Goelman et al. (2018).

A better understanding of brain connectivity variations necessitates the
mathematical models capable of identifying brain disorder-specific
irregularities Arslan et al. (2018); Ma et al. (2018). Existing methods often
compute functional connectivity (FC) matrices independently of the prediction
task, limiting their adaptability. A notable attempt by Kim et al. (2021a)
computes FC based on learned data representations, yet it lacks adaptability
in connectivity estimation. We propose using deep learning (DL) models with
learnable weights to gauge goal-specific connectivity matrices. This approach
bridges connectivity analysis and predictive modeling, enabling us to uncover
nuanced FC patterns in association with brain function and disorders Mahmood
et al. (2022).

Furthermore, FC estimates often follow static or dynamic computation
approaches. Static connectivity assumes stationary brain activities, while
dynamic FC, on the other hand, reveals recurring patterns that evade static
approaches Allen et al. (2012). Using a static graph-based method to
understand dynamic systems can lead to reduced classification performance, as
demonstrated by Xu et al. (2020). Notably, Kipf et al. (2018) demonstrates
improved outcomes by dynamically re-evaluating the learned static graph during
testing, emphasizing the importance of leveraging dynamic nature of brain
function. Modeling dynamic connectivity is therefore crucial Yaesoubi et al.
(2018). Yet, current methods frequently employs predefined sliding window
techniques with window sizes and strides Armstrong et al. (2016); Damaraju et
al. (2014). In this study, we propose to use deep neural networks to present
spatial and temporal dynamics and to build a goal-specific FC matrix. This
approach addresses the limitations of static methods and predefined sliding
window techniques, offering insights into the dynamic nature of brain
function.

Graph neural networks (GNN) are the most frequently used DL methods to study
FC matrix. Most GNNs learn embeddings of nodes by just considering their
neighborhood throughout the whole graph. However, this could be problematic
with brain connectome due to the sub-network nature of the brain. Recently,
BrainGNN Li et al. (2021) proposed a new GNN architecture that tackled this
limitation by proposing a clustering-based embedding method in the graph
convolutional layer, which allows nodes to consider their cluster assignment
(representing different regions of the brain) while learning embeddings.
Similarly, BrainRGIN Thapaliya et al. (2023a) proposed an isomorphism-based
clustered convolution with attention-based readouts to improve the prediction
power of GNNs. However, the major limitation of BrainGNN and BrainRGIN is that
it did not capture the temporal dynamics of the brain. Beyond static FC GNN
models, FBNetGen (Kan et al. (2022a)) delves into the trainable generation of
brain networks while investigating their interpretability for downstream
tasks, utilizing a combination of FC features and learned feature
representations at the final linear layer for the prediction. STAGIN (Kim et
al. (2021b)) employs GNNs with spatio-temporal attention to model dynamic
brain networks extracted from fMRI data using dynamic FCs as input to the
model. Graph Transformer Dwivedi and Bresson (2020) is a recent transformer-
based method that uses static FC as features and applies multi-head attention
to perform downstream tasks. (Azevedo et al. (2022)) introduces a slightly
different idea of using raw rsfMRI timepoints as features instead of FC, and
uses temporal blocks including (temporal) convolutional networks (TCN) or Long
Short Term Memory (LSTM) to capture temporal information, and graph block to
capture spatial information. A much more recent model, Brain Network
Transformer (BNT) (Kan et al. (2022b)) effectively used transformer-scaled dot
product attention and introduced an orthonormal clustering readout function
that leads to cluster-aware node embeddings. On the other hand, IBGNN (Cui et
al. (2022)) initializes a mask based on edges and refines it with sparse
control during training to identify top connections and regions for a specific
task, aiming to create a task-specific sub-network for interpretation.
Similarly, DICE Mahmood et al. (2022) proposed a novel idea of directed task-
based connectivity matrix generation using temporal attention and self-
attention. It is noteworthy that only a handful of papers like Mahmood et al.
(2022); Azevedo et al. (2022) make use of only rsfMRI timepoints without
making use of correlation based FC matrix.

In light of these challenges and opportunities, our paper proposes a new
built-in interpretable model, Dynamic Spatio-Temporal Attention Model (DSAM ),
that leverages the power of TCNs, self-attention, and GNNs to unfold the
intricate spatiotemporal dynamics of brain directly from rsfMRI timepoints.
Through this approach, we strive to overcome the limitations of conventional
methodologies and pave the way for a deeper comprehension of how the human
brain learns and adapts its functional connectivity for various complex
conditions. We want to point out that although we validate the model via sex
classification, it can be used to predict/classify various phenotypes ranging
from age, intelligence, and cognitive processes.

Overall, the main contribution of the paper lies in the built-in
interpretability and end-to-end training, including (a) temporal feature
extraction directly from raw time series of the brain activity using a novel
multilevel adaptation of TCNs, (b) shared temporal attention mechanisms to
select informative time points, (c) node-node self-attention to build goal-
specific brain connectivity matrix, (d) ROI-Aware GNNs for the spatial
dynamics of the brain.

The remainder of this paper is organized as follows: Section 2 describes our
data and preprocessing step, Our DSAM model is also presented in Section 2.
Experiments and results are discussed in Sections 3 and 4. Finally, Section 5
concludes the paper with a discussion.

##  2 Materials and Methods

###  2.1 Data and Preprocessing

####  2.1.1 Human Connectome Project (HCP) young adult data

HCP rsfMRI data Essen et al. (2013) is first minimally pre-processed following
the pipeline described in Glasser et al. (2013). The preprocessing includes
gradient distortion correction, motion correction, and field map
preprocessing, followed by registration to T1 weighted image. The registered
EPI image is then normalized to the standard MNI152 space. To reduce noise
from the data, FIX-ICA based denoising is applied Griffanti et al. (2014);
Salimi-Khorshidi et al. (2014). To minimize the effects of head motion,
subject scans with framewise displacement (FD) over 0.3mm at any time of the
scan were discarded. The FD is computed with fsl motion outliers function of
the FSL Jenkinson et al. (2012). There are 152 discarded scans after filtering
out with the FD, and 1075 scans are left. For all experiments, the scans from
the first run of HCP subjects released under S1200 are used. We use Schaefer
et al. (2017) atlas for brain parcellation, with 100 regions. For each region,
the average value is computed for all the voxels falling inside a region, thus
resulting in a single time series for each region. After dividing data into
regions, each time series is standardized by their score having zero mean and
unit variance.

####  2.1.2 Adolescent Brain Cognitive Development (ABCD) Data

ABCD is a large ongoing study following youths from age 9-10 into late
adolescence to understand factors that increase the risk of physical and
mental health problems. Participants were recruited from 21 sites across the
US to represent various demographic variables. Data used in this study are
from 8520 children aged 9–10 at baseline, including resting state fMRI image
and fluid intelligence, crystallized intelligence, and total composite scores.
Data are splited into training (n = 5964), validation (n = 1278), and test (n
= 1278) subsets. There are 4430 male and 4089 female subjects in this study.

We conduct preprocessing on the raw resting-state fMRI data utilizing a
combination of the FMRIB Software Library (FSL) v6.0 toolbox and Statistical
Parametric Mapping (SPM) 12 toolbox within the MATLAB 2019b environment. The
preprocessing encompasses several key steps, namely: 1) correction for rigid
body motion; 2) distortion correction; 3) removal of dummy scans; 4)
standardization to the Montreal Neurological Institute (MNI) space; and 5)
application of a 6 mm Gaussian kernel for smoothing. Similar to HCP dataset,
we use Schaefer et al. (2017) atlas for brain parcellation, with 100 regions.

###  2.2 The overview framework for DSAM

The Dynamic Spatio-Temporal Attention Model (DSAM) as presented in Figure 1 is
designed to effectively capture the intricate spatio-temporal dynamics
inherent in complex systems such as brain networks. The DSAM architecture is
comprised of several key components that work cohesively to extract
informative features from spatio-temporal data and perform classification
tasks. It begins with a Temporal Convolutional Network (TCN), capturing
temporal dependencies in time series data. The TCN architecture consists of
multiple blocks, each aimed at extracting different levels of feature from the
input rsfMRI data. Following the TCN, the model incorporates a Temporal
Attention Block, utilizing a Transformer-based multi-head attention mechanism,
selectively considering informative temporal features in critical time points.
Subsequently, the model employs a Self-Attention Block to calculate a learned
connectivity matrix across nodes, representing the spatial network as input to
the graph neural network model. The graph structure is constructed either
dynamically or statically by Pearson correlation of the output from the
Temporal Attention Block or original time series. The learned connectivity
matrix, is passed as an input to the Relational Graph Isomorphism Network
(RGIN) model to capture the spatial dynamics of the data. Finally, the output
from RGIN is fed into a linear model for the classification task, leveraging
the learned features and interactions within the graph structure. The
architecture is further clearly elucidated in the subsections below.

Figure 1:  Overall Architecture of DSAM . 3 TCN blocks are used to extract the
temporal features with different levels of abstraction (low level, medium
level, and high level). Temporal attention uses a shared multi head attention
module to filter the important time points. Self Attention Block is used to
learn the directed FNC matrix, which is used as an input to the Graph Block,
followed by Graph Readout and a fully connected layer for classification.

###  2.3 Temporal Convolutional Networks

Temporal Convolutional Networks form the first component of our spatio-
temporal model and are responsible for extracting temporal features from the
1D time-series data of each brain node. It is well documented that TCNs (Bai
et al. (2018)) can perform equally well or sometimes even better than
Recurrent Neural Networks (RNNs) and LSTMs for many kinds of sequential data.
Some advantages of the convolutional operator are, for instance, (1) low
memory requirement for long input sequences, especially compared to LSTMs and
Gated Recurrent Units (GRUs), (2) better parallelization because a TCN layer
is processed as a whole instead of sequentially as in RNNs, and (3) easier to
train (e.g., it is known that LSTM training can commonly encounter issues with
vanishing gradients). TCNs are built on dilated causal convolutions, which are
unique 1D filters whose receptive field size quickly grows across the temporal
dimension of the input as the network’s depth increases. To maintain temporal
order, the padding of the convolution is "causal" in the sense that output at
a particular time step is convolved exclusively with elements from earlier
time steps from the previous layers.

Formally, given a single ROI time series $t_{i}\in R^{T}$ , i is the index of
node, T is the number of time points, and a filter $f\in R^{K}$, K is the
kernel size, the dilated casual convolution operation with $f$ at time $t$ is
represented as:

$(t_{i}*f)(t)=\sum_{s=0}^{K-1}f(s)\times t_{i}(t-d*s),$ (1)

where $d$ is the dilation factor. When d = 1, a dilated convolution reduces to
a regular convolution. Using larger dilation enables an output at the top
level to effectively represent a wider receptive field of the input.

We propose a new multi-level adaptation of TCNs that includes three TCN
blocks, and each block consists of two temporary convolution layers with
multiple channels/kernels, batch normalization, and a dropout. Similar to Bai
et al. (2018), we use batch normalization instead of weight normalization
because of its stable performance during training. Despite employing the same
kernel size across all blocks, the dilation levels progressively increase
along with the TCN blocks, as in Equation 2.

$\tilde{T}_{nb}=TCN_{3}(TCN_{2}(TCN_{1}(t_{i},d=1),d=2),d=4)$ (2)

where $\tilde{T}_{nb}\in\mathbb{R}^{N\times T}$ and N is the number of nodes.
The three TCN blocks present three levels of features (high-level features,
medium-level features, and low-level features) from the time series. At each
of the three blocks, we employ distinct 1D convolutions across channels to
obtain a single channel representation for that block.

####  2.3.1 Shared temporal attention block

To identify crucial time points, we introduce a shared attention  mechanism
applied across the features extracted from different levels of TCN. This
single attention mechanism filters important time points independently across
different blocks while sharing weights. This design choice enforces the
attention network to select a similar set of time points across all blocks,
enabling a coherent selection of temporal patterns. Subsequently, we apply a
top-k approach to filter out unimportant time points.

Initially, the outputs from three TCN blocks, $\tilde{T}_{1},\tilde{T}_{2}$
and $\tilde{T}_{3}$ represented as $\tilde{T}_{nb}\in\mathbb{R}^{(N\times
T)}$across all nodes are individually fed to a shared attention block to
compute the individual attention scores as demonstrated below:

$\displaystyle K_{T}=\tilde{T}_{nb}W_{K},\quad Q_{T}=\tilde{T}_{nb}W_{Q},\quad
V_{T}=\tilde{T}_{nb}W_{V}$ (3)
$\displaystyle\text{AttentionMatrix}(Q_{T},K_{T})=\text{softmax}\left(\frac{Q_{T}K_{T}^{T}}{\sqrt{d_{k_{T}}}}\right)$
(4)
$\displaystyle\text{Attention}(Q_{T},K_{T},V_{T})=\left(\text{AttentionMatrix}\right)V_{T}$
(5)

where, $\textit{AttentionMatrix}\in\mathbb{R}^{T\times T}$,
$Attention\in\mathbb{R}^{N\times T}$, and $W_{Q},W_{K}$, $W_{V}$ are the
parameters of the attention block. We implemented multi-head attention(MHA)
for this module, followed by topK approach to extract the important time
points based on topK threshold $th$ within each block.

$\displaystyle\hat{T}_{nb}=\textit{topK}\left(\text{MHA}(\tilde{T}_{nb}),th\right)\quad\text{for
}nb=1,2,3$ (6) $\displaystyle\hat{T}=\oplus_{nb=1}^{3}\hat{T}_{nb}$ (7)

Here, $\hat{T}_{nb}\in\mathbb{R}^{(N\times(T\times th))}$ , and $\oplus$
represents concatenation, thus, the outputs are concatenated to get
$\hat{T}\in\mathbb{R}^{(N\times 3\times(T\times th))}$ . The application of
the above attention mechanism along the time dimension enables the
identification of important time points. By focusing on these crucial time
points, we can uncover temporal patterns that contribute to downstream
classification tasks.

###  2.4 Spatial Self-Attention Block

Each brain ROI can be considered as a node in the graph, and the FC between
ROIs shows how brain regions are linked with each other. To capture the
directed FC between brain ROIs, we use a spatial self-attention module.

For each node $n\in N$, a sequence of $\hat{T}$ time point vector, denoted as
$\hat{t}_{n}\in\mathbb{R}^{T}$, is fed into the self-attention module to
compute the attention matrix $Att_{FC}\in\mathbb{R}^{N\times N}$ along with
multi-head attention. In a simpler term, we’re evaluating how each node
relates to others over a sequence of time points. The following set of
equations summarizes the process, where $\top$ represents transpose and
$\oplus$ represents concatenation.

$\displaystyle k_{n}$ $\displaystyle=\hat{t}_{n}^{\top}W(k),\quad
q_{n}=\hat{t}_{n}^{\top}W(q)$ (8) $\displaystyle\ K_{A}$
$\displaystyle=\oplus_{n=1}^{N}k_{n},\quad Q_{A}=\oplus_{n=1}^{N}q_{n}$ (9)
$\displaystyle\ Att_{FC}$
$\displaystyle=\text{softmax}(Q_{A}K_{A}^{\top}/\sqrt{d_{K_{A}}})$ (10)

Here, $k_{n}$ and $q_{n}$ represent the key and query embeddings respectively
for each input $\hat{t}_{n}$. The matrices $K$ and $Q$ are formed by
concatenating all key and query embeddings across all nodes. Here, the
resulting $Att_{FC}\in\mathbb{R}^{N\times N}$ is the connectivity matrix
between $N$ nodes in the graph, learned FC matrices for downstream
classification. This enforces the model to estimate connectivity differences
between classification groups (e.g., male and female or healthy controls and
patients). The Directed Connectivity (DC) estimated by the model represents an
interpretable graph to provide insights into task-dependent nodes and their
connectivity.

###  2.5 Graph Neural Network Block

We make use of Relational Graph Isomorphism Network (RGIN) along with TopK
Pooling to extract the spatial information from rs-fMRI data, which are
explained further below.

####  2.5.1 Constructing a connectivity graph

The brain is spatially divided into N ROIs, which represent graph nodes
indexed by the set $V=1,...,N$. The input node features are represented as a
matrix $H=[h_{1},.....,h_{N}]^{T}$ where $h_{i}$ is the feature vector of the
node $V_{i}$. An edge set $E$ represents the functional connections between
ROIs with each edge $e_{ij}$ linking two nodes $(i,j)\in E$. The adjacency
matrix $A_{k}\in R^{N\times N}$ is computed is formed by thresholded element
$e_{i,j}$ of correlation matrix computes as Pearson’s correlation coefficient
between nodes’ feature representation to achieve either fully connected or
sparse graph. correlation matrix is computed in two different ways:

  1. 1.

Pearson’s correlation coefficient between nodes whose feature representation
is the output of shared attention block with dimension $\mathbb{R}^{(N\times
3\times(T\times th))}$, which is dynamic in nature and changes during
training,

  2. 2.

the Pearson’s correlation coefficient between nodes whose feature
representation is the original fMRI data series with dimension
$\mathbb{R}^{N\times T}$, constant throughout the training.

The input node features, $h_{i}$ $\in\mathbb{R}^{1\times N}$ , are the output
of the self attention block which represents the directed connectivity
strengths between node i to all other nodes.

The resultant undirected graph is denoted by the tuple $G=(V,E)$. Let
$(H=Att_{FC})\in R^{N*N},E\in R^{|E|*1}$, and $A\in R^{N*N}$ symbolize the
nodes features, edge features, and adjacency for the graph structure G.

####  2.5.2 Relational Graph Isomorphism Network (RGIN)

The RGIN convolution, introduced in our recent work Thapaliya et al. (2023a),
offers a novel approach to effectively capture both node and edge features in
a graph by merging the concepts of Relational Graph Convolutional Networks
(RGCN) and Graph Isomorphism Network (GIN). This method is particularly
tailored for understanding brain function, where edge relations signify
distinct functional clusters within the brain network. In simpler terms, nodes
in the brain are categorized into functional clusters, and different clusters
give rise to unique types of edge relations.

Unlike conventional approaches that fix cluster assignments for ROIs, the
proposed RGIN model autonomously learns cluster formations. This means that
nodes within the same cluster collaborate to achieve optimal performance for
the given task. The integration of GIN aids in learning node embeddings for
enhanced graph discriminative expression.

In RGIN, the aggregation function of GIN is replaced with that of RGCN,
allowing the model to learn distinct mechanisms for different clusters. This
design choice is motivated by the fact that GIN employs Multilayer Perceptrons
(MLPs) to adapt to the complexity of brain connectivity, while RGCN excels in
modeling the sub-network clustering nature of brain connectivity.

The forward propagation function of RGIN is defined as follows:

$h_{i}^{(l)}=\text{MLP}^{(l)}\left((1+\epsilon^{(l)})\cdot W_{i}^{(l)}\cdot
h_{i}^{(l-1)}+\sum_{j\in N^{(l)}_{(i)}}W_{j}^{(l)}\cdot e_{i,j}^{(l-1)}\cdot
h_{j}^{(l-1)}\right)$ (11)

In standard GNN models, $W_{i}$ and $W_{j}$ are learnable weight matrices.
However, in RGIN, they are defined as functions with parameters determined by
the cluster assignment of nodes, making it an ROI-aware convolution layer. The
model is trained end-to-end with a 2-layered MLP. The parameter
$\epsilon^{(l)}$ is learnable and influences the importance of a node compared
to its neighbors.

$\textbf{W}^{(l)}_{i}=\theta_{2}^{(l)}\cdot\text{relu}(\theta_{1}^{(l)}\textbf{r}_{i}+\textbf{b}^{(l)})$
(12)

where ${W}^{(l)}_{i}$ is a function of the position encoding $r_{i}$, with
parameters $\theta_{1}$ and $\theta_{2}$ to build a output of dimension
$d(l+1)\times d(l)$. ReLU is the rectified linear unit activation function and
$\textbf{b}^{(l)}$ is the bias term in the MLP. We represent each node’s
location information, $\textbf{r}_{i}$, using one-hot encoding instead of
coordinates, assuming that the ROIs are aligned in the same order for all the
brain graphs.

Additionally, we assume that
$\theta^{(l)}_{1}=[\alpha^{(l)}_{1},\ldots,\alpha^{(l)}_{N^{(l)}}]$, where
$N^{(l)}$ is the number of ROIs in the $l^{th}$ layer,
$\alpha^{(l)}_{i}=[\alpha^{(l)}_{i1},\ldots,\alpha^{(l)}_{iK(l)}]^{T}$,
$\forall i\in\\{1,\ldots,N^{(l)}\\}$, and $K(l)$ is the number of clusters. In
this study, $K$ is selected as 7. $\alpha^{(l)}_{i}$ is the non-negative
assignment scores of ROI $i$ to clusters. Assume
$\mathbf{W}^{(l)}=[\beta^{(l)}_{1},\ldots,\beta^{(l)}_{K(l)}]$ with
$\beta^{(l)}_{u}\in\mathbb{R}^{d(l+1)\times d(l)}$, $\forall
u\in\\{1,\ldots,K(l)\\}$, where $\beta^{(l)}_{u}$ is a basis matrix. Then, we
can rewrite equation (4) as:

$\textbf{W}^{(l)}_{i}=\sum_{u=1}^{K(l)}{\alpha}_{iu}^{(l)}\beta^{(l)}_{u}+\textbf{b}^{(l)}$
(13)

The incorporation of position encoding $r_{i}$ in ${W}^{(l)}_{i}$ is defined
using parameters $\theta_{1}$ and $\theta_{2}$ to make the convolution layer
specific to clustering. The final forward propagation function of RGIN
Convolution can be expressed as:

$\begin{split}h_{i}^{(l)}=\text{MLP}^{(l)}\Biggl{(}&(1+\epsilon^{(l)})\cdot\left(\sum_{u=1}^{K(l)}\alpha_{iu}^{(l)}\beta^{(l)}_{u}+\textbf{b}^{(l)}\right)\cdot
h_{i}^{(l-1)}\\\ &+\sum_{j\in
N^{(l)}_{(i)}}h_{j}^{(l-1)}\cdot\left(\sum_{u=1}^{K(l)}\alpha_{ju}^{(l)}\beta^{(l)}_{u}+\textbf{b}^{(l)}\right)\cdot
e_{i,j}^{(l-1)}\Biggr{)}\end{split}$ (14)

This formulation reduces the number of learnable parameters while still
allowing a separate embedding kernel for each ROI, contributing to the model’s
efficiency.

####  2.5.3 Pooling Layers

Via RGIN convolution, node-wise representations are generated. However, for
the prediction task discussed in this paper, which requires graph-level
prediction instead of node-level, these node-wise representations must be
collected or pooled together. We apply ROI-aware TopK Pooling as it improves
interpretability for rsfMRI data, by keeping the most indicative ROIs and
removing the noisy and uninformative nodes (Li et al. (2021)), along with
enforcing sparsity on the network. To select the most indicative ROIs, the
choice of which nodes to drop is determined based on the scores of nodes
obtained by projecting the node features to 1D via a learnable projection
vector $\omega^{(l)}\in\mathbb{R}^{d^{(l)}}$. The pooled graph
$(V^{(l+1)},E^{(l+1)})$ is computed as follows:

  1. 1.

Calculate the scores of nodes $s^{(l)}$ with node feature matrix as follows:

$\displaystyle s^{(l)}=\frac{{H}^{(l)}w^{(l)}}{\|w^{(l)}\|_{2}}$ (15)

  2. 2.

Normalize the score $s^{(l)}$ by subtracting its mean and dividing by its
standard deviation, yielding final scores $\tilde{s}^{(l)}$:

$\displaystyle\tilde{s}^{(l)}=\frac{(s^{(l)}-\mu(s^{(l)}))}{\sigma(s^{(l)})}$
(16)

  3. 3.

Find the top $k$ elements in the normalized score vector $\tilde{s}^{(l)}$
with $k$\- largest values and get their indices as $i$:

$\displaystyle i=\text{topk}(\tilde{s}^{(l)},k)$ (17)

  4. 4.

Create the pooled node features $H^{(l+1)}$ by element-wise multiplying the
original node features $H^{(l)}$ with the sigmoid of the normalized scores and
selecting only the elements indexed by $i$:

$\displaystyle H^{(l+1)}=(H^{(l)}\odot\text{sigmoid}(\tilde{s}^{(l)}))_{i:}$
(18)

  5. 5.

Create new adjacency matrix as the new pooled graph $E^{(l+1)}$ by returning
the edges between selected nodes $i$ from previous edge matrix $E^{(l)}$

$E^{(l+1)}=E^{(l)}_{i,i}$

In this description, $\|\cdot\|_{2}$ represents the $L_{2}$ norm, $\mu$ and
$\sigma$ are functions that calculate the mean and standard deviation of a
vector, topK identifies the indices of the largest $k$ elements in a vector,
$\odot$ denotes element-wise multiplication, and $(\cdot)_{i,:}$ selects
elements in the $i^{th}$ row and all columns.

####  2.5.4 Readout

Previous studies have proven the effectiveness of the mean and max element-
wise pooling operation(Li et al. (2021)). For a network, $G$ with $l$
convolution and respective pooling layers, the output graph of the $l^{th}$
pooling block is summarized using a mean and max pooling operation element-
wise on $H(l)=[h(l)_{i}:i=1,...,N(l)]$. The resulting vector is obtained by
concatenating both the mean and max summaries. To obtain a graph-level
representation, the summary vectors from each layer of the RGIN block are
concatenated together, and a 2-layered MLP is used for the final
classification.

##  3 Experiments Setup and Model Overview

###  3.1 Model Overview

Our model as presented in Fig. 1 begins with the utilization of a TCN
architecture to extract the temporal dynamics of rs-fMRI time series for each
node in the network. Within the three blocks, a kernel of size 7, denoted as
$K=7$, is employed, and the number of channels is 8, 16, and 32 respectively.
The dilation factor $d$ adopts the form $d=2^{(l-1)}$, where $l$ signifies the
block (i.e., $l\in\\{1,2,3\\}$).

Following this, the model transitions into a Temporal Attention Block
featuring a Transformer-based Shared Attention block Vaswani et al. (2017).
This block incorporates a single-layer attention component with
$\textit{number\\_heads}=8$. After extracting the attention scores, we did a
hyperparameter search to finalize the optimal threshold for selecting the
top-$k$ time points ($th$) from the provided range of values:
$[1,0.8,0.6,0.4,0.2,0.1]$, which resulted in 0.1 providing enough informative
feature representations to be used further. The resultant feature set assumes
dimensions of $\mathbb{R}^{(N\times 3\times(T\times th))}$. The th as 0.1 or
just 10% significantly reduces the time points (using only 360 time points
from a total of 3600 time points from three blocks of TCN) and thus the total
parameter count later on.

Subsequently, this feature set is passed to the self-attention block to
calculate the learned connectivity matrix. The self-attention Block comprises
W matrices for key, value, and query operations with the embedding size of 64.
The number of attention heads, denoted as num_attention_heads, is fixed at 8.
The resulting learned connectivity matrix emerges as a node feature
representation with dimensions of $\mathbb{R}^{1\times N}$ for each node,
which subsequently serves as input to an RGIN model.

Adjacency matrix within the graph structure is constructed using one of two
methods: dynamic edges from TCN outputs, and constant edges from original time
series. As a result of the hyperparameter search, a fixed threshold of 30% is
applied to build a sparse edge representation for both dynamic and constant
edge scenarios. We used two layers of RGIN convolution block of sizes [32,32]
with a pooling ratio of 0.5, and a 2-layered fully connected layer with sizes
[32, 512] with ReLU activation functions in between. The number of clustered
communities for RGIN block is set to 7, and the motivation for this comes from
the seven functional networks defined by (Yeo et al. (2011)) to show key brain
functional domains. In the final step, the output from the RGIN feeds into a
linear model for the classification task.

###  3.2 Training the Model

The DSAM architecture was implemented using Pytorch Paszke et al. (2019), and
Pytorch Geometric Fey and Lenssen (2019) for the specific graph neural network
components. The number of nodes was 100 (corresponding to Schaefer Atlas), and
the number of node features per node was the number of time points for each
node (i.e., 1200). We tested different versions of the architecture with
dynamic or constant edges. To evaluate the performance and validity of our
model, we conducted proof-of-concept experiments using a widely recognized
binary sex prediction task Jiang et al. (2019); Weis et al. (2019). Our
approach employed a 5-fold stratified cross-validation procedure on the HCP
dataset. In each fold, we divided the dataset into training and test sets,
with the test set comprising 20% of the original data.

The neural network training of our model was performed for 150 epochs,
utilizing the Adam optimizer Kingma and Ba (2014) and the Cross-Entropy loss
function Zhang and Sabuncu (2018). An early stopping mechanism was
incorporated, terminating training if the validation loss failed to decrease
over 40 consecutive epochs. Additionally, the learning rate was subject to
reduction by a factor of 0.1 with a patience setting of 30. We utilized a
batch size of 32. Within each random run, we preserved the model with the
smallest validation loss. The model exhibiting the lowest validation loss was
evaluated on the respective test set. This process was independently executed
for each of the five test sets. Metrics derived from these evaluations were
subsequently averaged to produce the final results.

We also compared the performance of DSAM  with popular recent models along
with traditional robust ML models as baseline models. The recent baselines
includes BrainGNN (Li et al. (2021)), BrainRGIN (Thapaliya et al. (2023a)),
BrainNetCNN (Kawahara et al. (2017)), Brain Network Transformer Kan et al.
(2022b), FBNetGen Kan et al. (2022a), Graph Transformers Dwivedi and Bresson
(2020) etc. Furthermore, we also compared the performance of our model with a
stable deep spatiotemporal model Azevedo et al. (2022) that also uses TCNs as
the initial block to capture the temporal information, but afterward makes use
of node and edge models to capture the spatial dependencies. Furthermore, we
compared with the baseline models like GAT, and SVM. Based on performance,
setting the learning rate of both BrainRGIN and BrainGNN initially to 0.001
and reducing every 30 epochs achieved the best performance for all prediction
tasks. For BrainGNN, we used two convolutional layers of size 32, as presented
in the original paper (Li et al. (2021)), with a fully connected layer of size
512. We utilized the authors’ provided open-source codes for BrainGNN Li et
al. (2021), FBNetGen Kan et al. (2022a) and Brain Network Transformer (BNT)
Kan et al. (2022b), conducting grid searches to fine-tune essential
hyperparameters based on the recommended best configurations. Specifically,
for BrainGNN, we also explored various learning rates (0.01, 0.005, 0.001)
while for FBNetGen, we investigated how varying hidden dimensions (8, 12, 16)
impact the performance. For BNT, we used different hidden dimensions (256,
512, 1024) during scaled dot product attention, along with varying fully
connected layers (2 and 3). In the case of BrainNetCNN Kawahara et al. (2017),
we experimented with different dropout rates (0.3, 0.5, 0.7). Regarding GT
(Graph Transformer) Dwivedi and Bresson (2020), we examined the impact of
different number of transformer layers (1, 2, 4) and the number of attention
heads (2, 4, 8) over 50 epochs of training. For the training of the model by
Azevedo et al. (2022), we used a hyperparameter sweep referencing the
parameters suggested in the original paper. The input feature used for this
model was also the direct rs-fMRI time points. The hyperparameters used were:
batch size = 32, optimizers = [Adam, Rmsprop], dropout a range of values
between 0 and 0.5, number of nodemodel layers = [2,3], temporal embedding
dimension = [16,32], lr a range of values between 0.01 and 0.00001. The
learning rate was scheduled to reduce by a factor of 0.1, same as above with a
patience of 30. Other experimental settings were fixed as in the original
paper Azevedo et al. (2022). Besides, we experimented with different numbers
of heads = [2,4,6] for GAT, where we also tested with FC features as input
features. For SVM, we used Polyssifier
(https://github.com/sergeyplis/polyssifier). This tool is widely used to
perform baseline model comparisons and has a feature to automatically perform
hyperparameter search and identify the best parameters of the model for the
highest performance. The input features for SVM was directly the cross-
correlation between each of the nodes, which is termed the Functional
Connectivity (FC) features.

##  4 Results

In this section, we present the results of our experiments comparing DSAM with
the baseline models, and evaluating different forms of the DSAM architecture
with varying components. We aim to assess the impact of different
architectural choices on the performance of DSAM in capturing dynamic
connectivity patterns in brain networks.

###  4.1 Comparison with Baseline Models

Table 1 provides a comparison of DSAM with baseline models in terms of various
evaluation metrics. We report the number of edges, Area Under the ROC Curve
(AUC), accuracy, sensitivity, and specificity.

Table 1: Comparison with baseline models on HCP Dataset

Model | AUC | Accuracy | Sensitivity | Specificity
---|---|---|---|---
DSAM (Ours) - Dynamic Edges | $83.02\pm 1.5$ | $82.05\pm 0.83$ | $85\pm 0.8$ | $85.18\pm 0.5$
DSAM (Ours) - Constant Edges | $81.12\pm 1.7$ | $80.06\pm 0.4$ | $84.12\pm 1.3$ | $84.34\pm 0.8$
Brain Network Transformer Kan et al. (2022b) | $82.82\pm 0.7$ | $81.5\pm 0.8$ | $85.06\pm 1.7$ | $85.34\pm 0.4$
BrainNetCNN Kawahara et al. (2017) | $74.3\pm 2.31$ | $75.21\pm 2.1$ | $74.12\pm 2.7$ | $77.34\pm 1.2$
FBNetGen | $80.12\pm 1.8$ | $81.12\pm 1.8$ | $83.75\pm 1.3$ | $79.34\pm 0.9$
BrainRGIN Thapaliya et al. (2023a) | $77.51\pm 0.9$ | $79.96\pm 2.5$ | $84.12\pm 1.3$ | $74.34\pm 0.2$
N+E (Concat) Azevedo et al. (2022) | $77.12$ | $77.01$ | $79.32$ | $80.16$
GAT Veličković et al. (2017) | $78.12\pm 3.34$ | $77.13\pm 2.7$ | $68.12$ | $69.32$
SVM | $75.32$ | $73.21$ | $72.56$ | $74.34$

Our DSAM architecture, both in dynamic and constant settings, outperforms the
baseline models across all metrics. In the dynamic setting, DSAM achieves an
AUC of 83.02%, an accuracy of 82.05%, and strong sensitivity and specificity
scores of 85% and 86%, respectively. Even in a constant setting, DSAM
maintains competitive performance with an AUC of 81.12% and an accuracy of
80.06%. The comparison with the recent model by Azevedo et al. (2022) baseline
reveals significant improvement over existing methods that do not learn the
connectivity matrix. Furthermore, we observed stable and comparable
performances from baseline models such as BrainNetCNN Kawahara et al. (2017),
FBNetGen Kan et al. (2022a), and GT Dwivedi and Bresson (2020), indicating the
reliability of these models. However, the Brain Network Transformer (BNT) Kan
et al. (2022b) exhibited robust performance with stable metrics. These results
demonstrate the effectiveness of DSAM in capturing and utilizing dynamic
connectivity patterns in brain networks, providing superior classification
performance compared to traditional baseline models.

###  4.2 Experiment with the Adolescent Brain Cognitive Development (ABCD)
Dataset

####  4.2.1 Hyperparameter settings for ABCD Dataset

Similar to the HCP dataset, we performed a binary sex prediction task to
evaluate the performance and validity of our model. The configuration the
model is same, while the time points of input node was 360. We performed a
hyperparameter search to finalize the optimal threshold for selecting the
top-k time points (tk) from the provided range of values:[1, 0.8, 0.6, 0.4],
similar to HCP. We employed a 5-fold cross-validation procedure. All other
hyperparameter settings remain the same.

####  4.2.2 Results from ABCD Dataset

In the context of ABCD, our architecture DSAM  outperforms most of the
baseline models as can be seen in Table 2. In the dynamic setting, DSAM
achieves an AUC of 84.12%, an accuracy of 83.27%, an AUC of 81.12%, and an
accuracy of 82.06% in the constant setting. Similar to the experiment with HCP
datasets, baseline models such as BrainNetCNN (Kawahara et al. (2017)),
FBNetGen (Kan et al. (2022a)), Brain Network Transformer (BNT) (Kan et al.
(2022b)), and GT (Dwivedi and Bresson (2020)) exhibited stable and comparable
performances indicating the reliability of these models in the analysis.

Table 2: Comparison with baseline models on ABCD Dataset

Model | AUC | Accuracy | Sensitivity | Specificity
---|---|---|---|---
DSAM (Ours)- Dynamic Edges | 85.12 ±1.5 | 83.27 ±1.75 | 85 ±1.8 | 86 ±0.9
DSAM (Ours)- Constant Edges | 81.12±1.2 | 82.06 ±1.1 | 84.12±1.3 | 84.34 ±0.9
Brain Network Transformer Kan et al. (2022b) | 84.98±2.1 |  84.12 ±1.73 | 86.32±1.9 | 87.12 ±1.2
BrainNetCNN Kawahara et al. (2017) | 78.21±1.3 | 79.22 ±1.8 | 78.23±1.3 | 80.43 ±1.1
FBNetGen | 82.12±2.1 | 82.90 ±1.4 | 84.91±1.1 | 82.79 ±0.7
BrainRGIN Thapaliya et al. (2023a) | 78.23±1.6 | 80.19 ±2.5 | 81.48±0.8 | 83.19 ±0.6
N+E (Concat) (Azevedo et al. (2022)) | 79.26 | 80.92 | 82.13 | 83.66

###  4.3 Ablation Studies

####  4.3.1 Question 1: How important are each blocks of DSAM  architecture ?

Table 3: Ablation studies with different forms of DSAM architecture on HCP
dataset

Architecture | Edges | GCN Features | AUC | Accuracy
---|---|---|---|---
| DSAM (TCN + Temporal
---
Attention+Self Attention+RGIN)
Dynamic | Learned FC | 83.02 | 82.05
| TCN+Self Attention+RGIN
---
Dynamic | Learned FC | 79.52 | 79.05
| TCN+Temporal Attention+
---
RGIN
Dynamic | TCN outputs | 73.29 | 73.98
| TCN + Temporal Attention+
---
Self Attention + 2 Layered MLP
- | - | 63.29 | 66.98
RGIN (Only GNN) | Constant | Static FC | 77.51 | 79.96

Our ablation studies (performed on the HCP dataset), detailed in Table 3, shed
light on the nuanced relationships among various components within the DSAM
architecture and their influence on performance metrics, particularly AUC and
accuracy. The comprehensive DSAM architecture, encompassing TCNs, Temporal
Attention, Self Attention, and GNNs, exhibits notable efficacy with an AUC of
83.02% and an accuracy of 82.05% in dynamic scenarios. When specific
components are selectively removed or modified, a discernible decline in
performance is observed. For instance, the exclusion of Temporal Attention
resulted in a drop of accuracy to 79.52% which as a similar case with 100%
time points and with a tremendous amount of timepoints features in the self-
attention block with huge performance load. Similarly, leaving out the Self-
Attention component and directly using the output of Temporal Attention as
input features to GNN, we saw a notable drop in AUC to 73.29% and accuracy to
73.98%. This underscores the pivotal role played by Temporal and Self-
attention in extracting essential temporal features and refining the model’s
understanding of complex relationships in dynamic settings. Furthermore, the
role of GNN (RGIN) in capturing the spatial dynamics is also vividly evident
from Table 3, where dropping GNN and using 2-layered MLP directly for
prediction after Self-Attention resulted in a significant drop in accuracy.

In summary, our ablation studies highlight the intricate interplay among TCN,
Temporal Attention, Self Attention, and GCN within the DSAM architecture. The
findings underscore the necessity for a holistic integration of these
components to effectively capture dynamic connectivity patterns, showcasing
the significance of each element in maintaining the model’s overall
performance integrity. These insights provide valuable guidance for refining
and optimizing the DSAM architecture for applications involving dynamic
learning scenarios.

####  4.3.2 Question 2: What percentage of original time points are
informative for the prediction?

In our investigation of the HCP dataset, we aimed to understand how the
selection of time points influences our model’s predictive performance.
Through a hyperparameter search across the threshold (th) ranges of 10%, 20%,
40%, 60%, 80%, and 100%, we made a surprising discovery: our model achieved
its optimal performance with just 10% of the original time points. This
selection dramatically reduces the number of features processed by the
subsequent self-attention block, with only 120-time points retained per level
of TCN from 1200 time points. Considering three levels of TCN, this amounts to
a total of merely 360-time points utilized for prediction. Despite this
reduction, these selected time points encapsulate crucial information, thanks
to feature concatenation occurring at multiple levels within the model
architecture. Notably, this reduction in the number of time points also
corresponds to a decrease in the total number of parameters, further
streamlining our model’s computational load.

Figure 2:  Regions of significant differences between male and female (HCP)

####  4.3.3 Question 3: How does performance alter when we select a larger
number of nodes?

We also aim to understand how the number of ROI selections impacts the
performance of the model, and if the model learns better with more nodes on
the HCP dataset. For this, we used the same atlas Schaefer et al. (2017) atlas
for brain parcellation, but with 200 and 400 regions. As this drastically
increases the parameters of the model, we reperformed the hyperparameter
optimization for just the dropout selecting a different number of parameters,
where dropout of 0.65 for 200 regions and 0.68 for 400 regions gave the best
results of 82.15% and 83.01% accuracy. The results reveal that the results are
pretty similar, and when optimized, reveal very similar performance.

####  4.3.4 Question 4: How many blocks of TCN should be used for the best
results??

In this section, we answer the idea behind selecting 3 blocks of TCN. The aim
of using multiple blocks in TCN is to have diverse representations such that
the network can learn a more comprehensive and diverse set of features.
Because the dilation increases exponentially, the first block captures low-
level temporally local patterns, while subsequent blocks can learn more high-
level, large-range, complex features. This hierarchy allows the network to
understand the data at different levels of abstraction, potentially improving
the model’s ability to capture intricate patterns present in resting state
fMRI data. Therefore, we experimented with different numbers of blocks (2, 3,
and 4) to see what number of blocks in TCN presented us with the best results.
Using the same model configurations, 2,3 and 4 blocks reported an accuracy of
79.12%, 82.05%, and 81.95%, which is an evidence that three-level of feature
dilation is enough to capture enough temporal patterns in rs-fMRI data. We did
not experiment with blocks larger than 4 blocks because the complexity and
parameter of the whole model increased significantly.

(a) Comparison of Learned Directed Connectivity via Self-Attention for Males
vs Females

(b)  Visualizing the Direction of Significance of Nodes in Relation to Sex

Figure 3: (a) Comparison of learned directed FC between males and females,
highlighting regions of interest. (b) Visualization of the direction of
significance of nodes concerning sex differences.

###  4.4 Directed Functional Connectivity for Sex for HCP dataset

Using the model’s inbuilt interpretable power, we plot the learned functional
connectivity matrix. The output of the trained self-attention block was
extracted from a holdout test set and normalized along the column dimension.
The male and female connectivity matrixes were extracted, and the correlation
heatmaps are presented in Figure 3. The 100 ROIs are organized into seven
functional networks: Visual network; Sensorimotor network; Dorsal attention
network; Ventral attention network; Limbic network; Control network; Default
mode network.

After extracting the connectivity matrix for both males and females, we
performed t-test on all pairs of connectivity between males and female to
analyze the results. Multiple comparison with Bonferroni correction was done
for both tests.

For the t-test between all pairs of 100 nodes, we found connectivity between
52 pairs of nodes (18 nodes involved ) to be significant (p-value $\leq
5\times 10^{-5}$). The most involved nodes included the right postcentral
gyrus, left calcarine sulcus, left middle frontal gyrus, right and left
posterior cingulate cortex (PCC), left cingulum bundle, and right insular
cortex. These regions represent key components of the brain involved in
somatosensory processing, visual perception, executive functions, default mode
network activity, emotional regulation, and social cognition, which we
elaborate further in discussion section below. Furthermore, upon grouping the
nodes within the 7 connectivity networks, the default mode network had the
highest number of nodes (n=5) to be sex differential, whereas the limbic
network had zero significant nodes. The complete number of nodes per networks
are represented in Figure 4.

Figure 4:  Significant nodes within each network (HCP)

##  5 Discussion and Conclusion

In this research study, we have demonstrated the effectiveness of our model,
DSAM , in capturing the spatiodynamics of the brain and learning connectivity
matrices relevant to prediction tasks. Our model leverages TCN, temporal
attention, self-attention, and ROI-aware GNNs to comprehensively capture the
spatiotemporal dynamics of brain connectivity directly from raw and noisy
resting-state fMRI (rs-fMRI) data. Significantly, this approach enhances
interpretability and offers a deeper understanding of the mechanisms
underlying brain function.

One of the key strengths of our model lies in its direct utilization of raw
resting-state fMRI time points, bypassing the need for handcrafted FC features
commonly employed by existing models such as BNT, FBNetGen, BrainRGIN, and
BrainGNN. This approach represents a departure from conventional methods and
presents a more challenging task of seamlessly integrating both temporal and
spatial blocks, thus contributing to the model’s "heaviness." Moreover, our
model offers a significant advantage in its ability to learn directed goal-
specific connectivity matrices. The directed FC matrices are different from
correlation-based static FC (see the Supplementary figure for static FC),
suggesting that the model is able to learn new patterns as well as learn
effectively critical nodes and connections within the brain for the task. This
capability holds immense promise for uncovering relevant phenotypes and
advancing our understanding of brain function. Though having differences, we
have some similarities in static FC vs learned directed FC for sex where in
both t-tests, default mode had the highest number of significant nodes (23
nodes in static FC and 5 nodes in learned FC), and the limbic network had the
lowest number of nodes (3 for static FC, 0 for learned FC). This comparison
highlights the nuanced differences between static FC and our goal-oriented
learned FC, underscoring the relevance and specificity of our approach in
uncovering brain connectivity patterns tailored to the task at hand.

In this research study, our decision to focus more on the HCP dataset over the
ABCD dataset is rooted in several factors. Firstly, the HCP dataset offers a
sample size almost ten times smaller than the ABCD dataset. This smaller
sample size makes it more manageable for training and running experiments
efficiently, allowing us to explore our model’s capabilities more effectively
within the constraints of computational resources.

The results obtained from the interpretation perfectly align with existing
research findings in the field of neuroscience. Specifically, research has
consistently demonstrated differential activation of the right postcentral
gyrus during mental rotation tasks, indicating potential sex-specific
cognitive strategies or neural mechanisms Butler et al. (2006). Furthermore,
sex differences observed in regions like the Calcarine have been linked to
affective disorders, reinforcing the significance of considering sex-specific
neurobiological factors in understanding these conditions Chaudhary et al.
(2023); Tu et al. (2022); Piani et al. (2022). Similarly, investigations into
middle frontal volume and connectivity have underscored associations with
antisocial personality and autism, emphasizing the importance of sex-specific
considerations in elucidating the neurobiological underpinnings of these
disorders Piani et al. (2022); Raine et al. (2009). Moreover, the role of the
posterior cingulate cortex (PCC) within the Default Mode Network (DMN) has
been highlighted, with findings suggesting significant sex-by-aging
interaction and implications for conditions like autism and substance use
disorders Scheinost et al. (2014); Jung et al. (2015); Ritchay et al. (2021).
Similarly, investigations into the insula and cingulum, components of the
saliency network, have revealed sex differences in connectivity patterns,
particularly in the context of autism spectrum disorders, further emphasizing
the relevance of sex-specific neural mechanisms in understanding these
conditions Cummings et al. (2020). Additionally, sex differences in
morphological measures, particularly those related to interoception, provide
further insight into the complex interplay between brain structure, function,
and the manifestation of cognitive and affective processes Longarzo et al.
(2021). Integrating findings from these diverse brain regions and networks not
only enhances our understanding of sex-specific neural mechanisms but also
holds promise for personalized approaches to diagnosis and treatment across
various neurological and psychiatric conditions.

As we observe the difference in results when using dynamic edge weights as
compared to static edge weights, we assume this is because of the graph being
adapted with the temporal features over time, which extracts more relevant
edge features as compared to using static edges. This difference helps to
adjust the graph over time more specifically to the task at hand. As the main
focus of the model is to learn goal-specific connectivity of the brain, we
also believe having dynamic edges did open more flexibility for the model to
be more oriented towards a specific task.

We want to point out the limitations of the model being "heavy" and
performance-resource tradeoffs. Fully aware of its computational weights, each
component of our model’s architecture is carefully designed to satisfy the
task at hand. That said, we also optimized the model for a lighter computation
load by utilizing only a fraction of the original time points from the TCN.
Specifically, we retain just 10% of the original time points, significantly
reducing the computational load and parameter count, reflecting our deliberate
efforts to streamline the analysis process while maintaining performance.
Despite the computational demands, the performance gains achieved over simpler
models warrant focused and merit attention. The decision to employ a simpler
or heavier model depends on case-by-case careful consideration of performance-
resource trade-offs. Our purpose here is to present a novel model architecture
meticulously designed to align with the requirements of the task at hand,
modeling temporal-spatial dynamics of brain function for a specific task.

In conclusion, our model represents a significant step forward in
computational neuroscience, offering a powerful tool for analyzing and
interpreting brain connectivity dynamics directly from raw rs-fMRI data. As a
preliminary version of this study, future work will include additional
experiments on specific tasks (such as classifying patients with schizophrenia
or Alzheimer’s, predicting cognitive ability). Its utility and effectiveness
opens up great potential to gain deeper insights into how the human brain
adapts its functional connectivity specific to the task.

## Data and Code availability for replication

The code is openly available at https://github.com/bishalth01/DSAM. The
important parameters for replication are mentioned above in Section 3, and
also in readme.txt file inside the GitHub. Data cannot be open-sourced due to
restrictions but can be provided upon special request.

## Acknowledgments

This study was funded part by NIH grant R01DA049238 and NSF grant 2112455. We
thank the Adolescent Brain Cognitive Development (ABCD) participants and their
families for their time and dedication to this project. Data used in the
preparation of this article were obtained from the Adolescent Brain Cognitive
Development (ABCD) Study (https://abcdstudy.org), held in the NIMH Data
Archive (NDA). This is a multisite, longitudinal study designed to recruit
more than 10,000 children aged 9-10 and follow them over 10 years into early
adulthood. The ABCD Study is supported by the National Institutes of Health
and additional federal partners under award numbers U01DA041048, U01DA050989,
U01DA051016, U01DA041022, U01DA051018, U01DA051037, U01DA050987, U01DA041174,
U01DA041106, U01DA041117, U01DA041028, U01DA041134, U01DA050988, U01DA051039,
U01DA041156, U01DA041025, U01DA041120, U01DA051038, U01DA041148, U01DA041093,
U01DA041089, U24DA041123, U24DA041147. A full list of supporters is available
at https://abcdstudy.org/federal-partners.html. A listing of participating
sites and a complete listing of the study investigators can be found at
https://abcdstudy.org/consortium_members. ABCD consortium investigators
designed and implemented the study and/or provided data but did not
necessarily participate in analysis or writing of this report. This manuscript
reflects the views of the authors and may not reflect the opinions or views of
the NIH or ABCD consortium investigators.

## Author contributions statement

B.T. conducted the primary data analysis, coding implementation, experiments,
writing, and reviewing of the manuscript. R.M, Y.P.W, and E.A. actively
participated in reviewing the design of the model and provided insightful
suggestions while reviewing the manuscript. R.S, B.R., and P.S were involved
in the in-depth discussion of the analyses and reviewing the manuscript. J.C.
assisted in analyzing the results, and also assisted in interpretation. S.G
contributed to the theoretical analysis of the overall architecture and
participated in reviewing the manuscript. V.D.C was involved in the design of
the project and the manuscript review. J.L. was involved in the design, data
analysis, and revising the manuscript. All authors reviewed the manuscript.

## Financial Disclosures Section

All authors declare that they have no conflicts of interest.

## References

  * Allen et al. (2012) Allen, E.A., Damaraju, E., Plis, S.M., Erhardt, E.B., Eichele, T., Calhoun, V.D., 2012.  Tracking whole-brain connectivity dynamics in the resting state.  Cerebral Cortex 24, 663–676.  URL: https://doi.org/10.1093/cercor/bhs352, doi:doi:10.1093/cercor/bhs352.
  * Armstrong et al. (2016) Armstrong, C.C., Moody, T.D., Feusner, J.D., McCracken, J.T., Chang, S., Levitt, J.G., Piacentini, J.C., O’Neill, J., 2016.  Graph-theoretical analysis of resting-state fMRI in pediatric obsessive–compulsive disorder.  Journal of Affective Disorders 193, 175–184.  URL: https://doi.org/10.1016/j.jad.2015.12.071, doi:doi:10.1016/j.jad.2015.12.071.
  * Arslan et al. (2018) Arslan, S., Ktena, S.I., Glocker, B., Rueckert, D., 2018.  Graph saliency maps through spectral convolutional networks: Application to sex classification with brain connectivity.  URL: https://arxiv.org/abs/1806.01764, doi:doi:10.48550/ARXIV.1806.01764.
  * Avena-Koenigsberger et al. (2017) Avena-Koenigsberger, A., Misic, B., Sporns, O., 2017.  Communication dynamics in complex brain networks.  Nature Reviews Neuroscience 19, 17–33.  URL: https://doi.org/10.1038/nrn.2017.149, doi:doi:10.1038/nrn.2017.149.
  * Azevedo et al. (2022) Azevedo, T., Campbell, A., Romero-Garcia, R., Passamonti, L., Bethlehem, R.A., Liò, P., Toschi, N., 2022.  A deep graph neural network architecture for modelling spatio-temporal dynamics in resting-state functional MRI data.  Medical Image Analysis 79, 102471.  URL: https://doi.org/10.1016/j.media.2022.102471, doi:doi:10.1016/j.media.2022.102471.
  * Bai et al. (2018) Bai, S., Kolter, J.Z., Koltun, V., 2018.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  ArXiv abs/1803.01271.
  * Beckmann et al. (2005) Beckmann, C.F., DeLuca, M., Devlin, J.T., Smith, S.M., 2005.  Investigations into resting-state connectivity using independent component analysis.  Philosophical Transactions of the Royal Society B: Biological Sciences 360, 1001–1013.  URL: https://doi.org/10.1098/rstb.2005.1634, doi:doi:10.1098/rstb.2005.1634.
  * Butler et al. (2006) Butler, T., Imperato-McGinley, J., Pan, H., Voyer, D., Cordero, J., Zhu, Y.S., Stern, E., Silbersweig, D., 2006.  Sex differences in mental rotation: Top–down versus bottom–up processing.  NeuroImage 32, 445–456.  URL: http://dx.doi.org/10.1016/j.neuroimage.2006.03.030, doi:doi:10.1016/j.neuroimage.2006.03.030.
  * Chaudhary et al. (2023) Chaudhary, S., Hu, S., Hu, K., Dominguez, J.C., Chao, H.H., Li, C.S.R., 2023.  Sex differences in the effects of trait anxiety and age on resting-state functional connectivities of the amygdala.  Journal of Affective Disorders Reports 14, 100646.  URL: http://dx.doi.org/10.1016/j.jadr.2023.100646, doi:doi:10.1016/j.jadr.2023.100646.
  * Cui et al. (2022) Cui, H., Dai, W., Zhu, Y., Li, X., He, L., Yang, C., 2022.  Interpretable graph neural networks for connectome-based brain disorder analysis.  URL: https://arxiv.org/abs/2207.00813, doi:doi:10.48550/ARXIV.2207.00813.
  * Cummings et al. (2020) Cummings, K.K., Lawrence, K.E., Hernandez, L.M., Wood, E.T., Bookheimer, S.Y., Dapretto, M., Green, S.A., 2020.  Sex differences in salience network connectivity and its relationship to sensory over-responsivity in youth with autism spectrum disorder.  Autism Research 13, 1489–1500.  URL: http://dx.doi.org/10.1002/aur.2351, doi:doi:10.1002/aur.2351.
  * Damaraju et al. (2014) Damaraju, E., Allen, E., Belger, A., Ford, J., McEwen, S., Mathalon, D., Mueller, B., Pearlson, G., Potkin, S., Preda, A., Turner, J., Vaidya, J., van Erp, T., Calhoun, V., 2014.  Dynamic functional connectivity analysis reveals transient states of dysconnectivity in schizophrenia.  NeuroImage: Clinical 5, 298–308.  URL: https://doi.org/10.1016/j.nicl.2014.07.003, doi:doi:10.1016/j.nicl.2014.07.003.
  * Duggento et al. (2018) Duggento, A., Passamonti, L., Valenza, G., Barbieri, R., Guerrisi, M., Toschi, N., 2018.  Multivariate granger causality unveils directed parietal to prefrontal cortex connectivity during task-free MRI.  Scientific Reports 8.  URL: https://doi.org/10.1038/s41598-018-23996-x, doi:doi:10.1038/s41598-018-23996-x.
  * Dwivedi and Bresson (2020) Dwivedi, V.P., Bresson, X., 2020.  A generalization of transformer networks to graphs.  URL: https://arxiv.org/abs/2012.09699, doi:doi:10.48550/ARXIV.2012.09699.
  * Essen et al. (2013) Essen, D.C.V., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K., 2013.  The WU-minn human connectome project: An overview.  NeuroImage 80, 62–79.  URL: https://doi.org/10.1016/j.neuroimage.2013.05.041, doi:doi:10.1016/j.neuroimage.2013.05.041.
  * Farahdel et al. (2021) Farahdel, B., Thapaliya, B., Suresh, P., Ray, B., Calhoun, V.D., Liu, J., 2021.  Confirmatory factor analysis on mental health status using abcd cohort, in: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 3540–3547.  doi:doi:10.1109/BIBM52615.2021.9669378.
  * Fey and Lenssen (2019) Fey, M., Lenssen, J.E., 2019.  Fast graph representation learning with pytorch geometric.  URL: https://arxiv.org/abs/1903.02428, doi:doi:10.48550/ARXIV.1903.02428.
  * Fornito et al. (2015) Fornito, A., Zalesky, A., Breakspear, M., 2015.  The connectomics of brain disorders.  Nature Reviews Neuroscience 16, 159–172.  URL: https://doi.org/10.1038/nrn3901, doi:doi:10.1038/nrn3901.
  * Glasser et al. (2013) Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., Essen, D.C.V., Jenkinson, M., 2013.  The minimal preprocessing pipelines for the human connectome project.  NeuroImage 80, 105–124.  URL: https://doi.org/10.1016/j.neuroimage.2013.04.127, doi:doi:10.1016/j.neuroimage.2013.04.127.
  * Goelman et al. (2018) Goelman, G., Dan, R., Keadan, T., 2018.  Characterizing directed functional pathways in the visual system by multivariate nonlinear coherence of fMRI data.  Scientific Reports 8.  URL: https://doi.org/10.1038/s41598-018-34672-5, doi:doi:10.1038/s41598-018-34672-5.
  * Griffanti et al. (2014) Griffanti, L., Salimi-Khorshidi, G., Beckmann, C.F., Auerbach, E.J., Douaud, G., Sexton, C.E., Zsoldos, E., Ebmeier, K.P., Filippini, N., Mackay, C.E., Moeller, S., Xu, J., Yacoub, E., Baselli, G., Ugurbil, K., Miller, K.L., Smith, S.M., 2014.  ICA-based artefact removal and accelerated fMRI acquisition for improved resting state network imaging.  NeuroImage 95, 232–247.  URL: https://doi.org/10.1016/j.neuroimage.2014.03.034, doi:doi:10.1016/j.neuroimage.2014.03.034.
  * Heinsfeld et al. (2018) Heinsfeld, A.S., Franco, A.R., Craddock, R.C., Buchweitz, A., Meneguzzi, F., 2018.  Identification of autism spectrum disorder using deep learning and the abide dataset.  NeuroImage: Clinical 17, 16–23.  URL: http://dx.doi.org/10.1016/j.nicl.2017.08.017, doi:doi:10.1016/j.nicl.2017.08.017.
  * Jenkinson et al. (2012) Jenkinson, M., Beckmann, C.F., Behrens, T.E., Woolrich, M.W., Smith, S.M., 2012.  FSL.  NeuroImage 62, 782–790.  URL: https://doi.org/10.1016/j.neuroimage.2011.09.015, doi:doi:10.1016/j.neuroimage.2011.09.015.
  * Jiang et al. (2019) Jiang, R., Calhoun, V.D., Fan, L., Zuo, N., Jung, R., Qi, S., Lin, D., Li, J., Zhuo, C., Song, M., Fu, Z., Jiang, T., Sui, J., 2019.  Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores.  Cerebral Cortex 30, 888–900.  URL: https://doi.org/10.1093/cercor/bhz134, doi:doi:10.1093/cercor/bhz134.
  * Jung et al. (2015) Jung, M., Mody, M., Saito, D.N., Tomoda, A., Okazawa, H., Wada, Y., Kosaka, H., 2015.  Sex differences in the default mode network with regard to autism spectrum traits: A resting state fmri study.  PLOS ONE 10, e0143126.  URL: http://dx.doi.org/10.1371/journal.pone.0143126, doi:doi:10.1371/journal.pone.0143126.
  * Kan et al. (2022a) Kan, X., Cui, H., Lukemire, J., Guo, Y., Yang, C., 2022a.  Fbnetgen: Task-aware gnn-based fmri analysis via functional brain network generation.  URL: https://arxiv.org/abs/2205.12465, doi:doi:10.48550/ARXIV.2205.12465.
  * Kan et al. (2022b) Kan, X., Dai, W., Cui, H., Zhang, Z., Guo, Y., Yang, C., 2022b.  Brain network transformer.  URL: https://arxiv.org/abs/2210.06681, doi:doi:10.48550/ARXIV.2210.06681.
  * Kawahara et al. (2017) Kawahara, J., Brown, C.J., Miller, S.P., Booth, B.G., Chau, V., Grunau, R.E., Zwicker, J.G., Hamarneh, G., 2017.  Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment.  NeuroImage 146, 1038–1049.  URL: http://dx.doi.org/10.1016/j.neuroimage.2016.09.046, doi:doi:10.1016/j.neuroimage.2016.09.046.
  * Kim et al. (2021a) Kim, B.H., Ye, J.C., Kim, J.J., 2021a.  Learning dynamic graph representation of brain connectome with spatio-temporal attention.  URL: https://arxiv.org/abs/2105.13495, doi:doi:10.48550/ARXIV.2105.13495.
  * Kim et al. (2021b) Kim, B.H., Ye, J.C., Kim, J.J., 2021b.  Learning dynamic graph representation of brain connectome with spatio-temporal attention.  URL: https://arxiv.org/abs/2105.13495, doi:doi:10.48550/ARXIV.2105.13495.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  URL: https://arxiv.org/abs/1412.6980, doi:doi:10.48550/ARXIV.1412.6980.
  * Kipf et al. (2018) Kipf, T., Fetaya, E., Wang, K.C., Welling, M., Zemel, R., 2018.  Neural relational inference for interacting systems.  URL: https://arxiv.org/abs/1802.04687, doi:doi:10.48550/ARXIV.1802.04687.
  * Li et al. (2021) Li, X., Zhou, Y., Dvornek, N., Zhang, M., Gao, S., Zhuang, J., Scheinost, D., Staib, L.H., Ventola, P., Duncan, J.S., 2021.  BrainGNN: Interpretable brain graph neural network for fMRI analysis.  Medical Image Analysis 74, 102233.  URL: https://doi.org/10.1016/j.media.2021.102233, doi:doi:10.1016/j.media.2021.102233.
  * Liao et al. (2017) Liao, X., Cao, M., Xia, M., He, Y., 2017.  Individual differences and time-varying features of modular brain architecture.  NeuroImage 152, 94–107.  URL: https://doi.org/10.1016/j.neuroimage.2017.02.066, doi:doi:10.1016/j.neuroimage.2017.02.066.
  * Longarzo et al. (2021) Longarzo, M., Mele, G., Alfano, V., Salvatore, M., Cavaliere, C., 2021.  Gender brain structural differences and interoception.  Frontiers in Neuroscience 14.  URL: http://dx.doi.org/10.3389/fnins.2020.586860, doi:doi:10.3389/fnins.2020.586860.
  * Ma et al. (2018) Ma, G., Ahmed, N.K., Willke, T., Sengupta, D., Cole, M.W., Turk-Browne, N.B., Yu, P.S., 2018.  Similarity learning with higher-order graph convolutions for brain network analysis.  URL: https://arxiv.org/abs/1811.02662, doi:doi:10.48550/ARXIV.1811.02662.
  * Mahmood et al. (2022) Mahmood, U., Fu, Z., Ghosh, S., Calhoun, V., Plis, S., 2022.  Through the looking glass: Deep interpretable dynamic directed connectivity in resting fMRI.  NeuroImage 264, 119737.  URL: https://doi.org/10.1016/j.neuroimage.2022.119737, doi:doi:10.1016/j.neuroimage.2022.119737.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  Pytorch: An imperative style, high-performance deep learning library.  URL: https://arxiv.org/abs/1912.01703, doi:doi:10.48550/ARXIV.1912.01703.
  * Piani et al. (2022) Piani, M.C., Maggioni, E., Delvecchio, G., Ferro, A., Gritti, D., Pozzoli, S.M., Fontana, E., Enrico, P., Cinnante, C.M., Triulzi, F.M., Stanley, J.A., Battaglioli, E., Brambilla, P., 2022.  Sexual dimorphism in the brain correlates of adult-onset depression: A pilot structural and functional 3t mri study.  Frontiers in Psychiatry 12.  URL: http://dx.doi.org/10.3389/fpsyt.2021.683912, doi:doi:10.3389/fpsyt.2021.683912.
  * Raine et al. (2009) Raine, A., Yang, Y., Narr, K.L., Toga, A.W., 2009.  Sex differences in orbitofrontal gray as a partial explanation for sex differences in antisocial personality.  Molecular Psychiatry 16, 227–236.  URL: http://dx.doi.org/10.1038/mp.2009.136, doi:doi:10.1038/mp.2009.136.
  * Ray et al. (2023) Ray, B., Chen, J., Fu, Z., Suresh, P., Thapaliya, B., Farahdel, B., Calhoun, V.D., Liu, J., 2023.  Replication and refinement of brain age model for adolescent development URL: https://doi.org/10.1101/2023.08.16.553472, doi:doi:10.1101/2023.08.16.553472.
  * Ritchay et al. (2021) Ritchay, M.M., Huggins, A.A., Wallace, A.L., Larson, C.L., Lisdahl, K.M., 2021.  Resting state functional connectivity in the default mode network: Relationships between cannabis use, gender, and cognition in adolescents and young adults.  NeuroImage: Clinical 30, 102664.  URL: http://dx.doi.org/10.1016/j.nicl.2021.102664, doi:doi:10.1016/j.nicl.2021.102664.
  * Salimi-Khorshidi et al. (2014) Salimi-Khorshidi, G., Douaud, G., Beckmann, C.F., Glasser, M.F., Griffanti, L., Smith, S.M., 2014.  Automatic denoising of functional MRI data: Combining independent component analysis and hierarchical fusion of classifiers.  NeuroImage 90, 449–468.  URL: https://doi.org/10.1016/j.neuroimage.2013.11.046, doi:doi:10.1016/j.neuroimage.2013.11.046.
  * Sapkota et al. (2024) Sapkota, R., Thapaliya, B., Suresh, P., Ray, B., Calhoun, V.D., Liu, J., 2024.  Multimodal imaging feature extraction with reference canonical correlation analysis underlying intelligence, in: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2071–2075.  doi:doi:10.1109/ICASSP48485.2024.10448219.
  * Schaefer et al. (2017) Schaefer, A., Kong, R., Gordon, E.M., Laumann, T.O., Zuo, X.N., Holmes, A.J., Eickhoff, S.B., Yeo, B.T.T., 2017.  Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI.  Cerebral Cortex 28, 3095–3114.  URL: https://doi.org/10.1093/cercor/bhx179, doi:doi:10.1093/cercor/bhx179.
  * Scheinost et al. (2014) Scheinost, D., Finn, E.S., Tokoglu, F., Shen, X., Papademetris, X., Hampson, M., Constable, R.T., 2014.  Sex differences in normal age trajectories of functional brain networks.  Human Brain Mapping 36, 1524–1535.  URL: http://dx.doi.org/10.1002/hbm.22720, doi:doi:10.1002/hbm.22720.
  * Smith and Nichols (2018) Smith, S.M., Nichols, T.E., 2018.  Statistical challenges in “big data” human neuroimaging.  Neuron 97, 263–268.  URL: https://doi.org/10.1016/j.neuron.2017.12.018, doi:doi:10.1016/j.neuron.2017.12.018.
  * Suresh et al. (2023) Suresh, P., Ray, B., Thapaliya, B., Farahdel, B., Kazemivash, B., Chen, J., Duan, K., Calhoun, V.D., Liu, J., 2023.  Effective training strategy for nn models of working memory classification with limited samples, in: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1–4.  doi:doi:10.1109/ISBI53787.2023.10230722.
  * Thapaliya et al. (2023a) Thapaliya, B., Akbas, E., Chen, J., Sapkota, R., Ray, B., Suresh, P., Calhoun, V., Liu, J., 2023a.  Brain networks and intelligence: A graph neural network based approach to resting state fmri data.  URL: https://arxiv.org/abs/2311.03520, doi:doi:10.48550/ARXIV.2311.03520.
  * Thapaliya et al. (2021) Thapaliya, B., Calhoun, V.D., Liu, J., 2021.  Environmental and genome-wide association study on children anxiety and depression, in: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 2330–2337.  doi:doi:10.1109/BIBM52615.2021.9669291.
  * Thapaliya et al. (2023b) Thapaliya, B., Ray, B., Farahdel, B., Suresh, P., Sapkota, R., Holla, B., Mahadevan, J., Chen, J., Vaidya, N., Perrone-Bizzozero, N., Benegal, V., Schumann, G., Calhoun, V.D., Liu, J., 2023b.  Cross-continental environmental and genome-wide association study on children and adolescent anxiety and depression URL: http://dx.doi.org/10.1101/2023.02.06.23285530, doi:doi:10.1101/2023.02.06.23285530.
  * Tu et al. (2022) Tu, Z., Wu, F., Jiang, X., Kong, L., Tang, Y., 2022.  Gender differences in major depressive disorders: A resting state fmri study.  Frontiers in Psychiatry 13.  URL: http://dx.doi.org/10.3389/fpsyt.2022.1025531, doi:doi:10.3389/fpsyt.2022.1025531.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017.  Attention is all you need.  URL: https://arxiv.org/abs/1706.03762, doi:doi:10.48550/ARXIV.1706.03762.
  * Veličković et al. (2017) Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y., 2017.  Graph attention networks.  URL: https://arxiv.org/abs/1710.10903, doi:doi:10.48550/ARXIV.1710.10903.
  * Wang et al. (2019) Wang, L., Li, K., Chen, X., Hu, X.P., 2019.  Application of convolutional recurrent neural network for individual recognition based on resting state fMRI data.  Frontiers in Neuroscience 13.  URL: https://doi.org/10.3389/fnins.2019.00434, doi:doi:10.3389/fnins.2019.00434.
  * Weis et al. (2019) Weis, S., Patil, K.R., Hoffstaedter, F., Nostro, A., Yeo, B.T.T., Eickhoff, S.B., 2019.  Sex classification by resting state brain connectivity.  Cerebral Cortex 30, 824–835.  URL: https://doi.org/10.1093/cercor/bhz129, doi:doi:10.1093/cercor/bhz129.
  * Xu et al. (2020) Xu, D., Ruan, C., Korpeoglu, E., Kumar, S., Achan, K., 2020.  Inductive representation learning on temporal graphs.  URL: https://arxiv.org/abs/2002.07962, doi:doi:10.48550/ARXIV.2002.07962.
  * Yaesoubi et al. (2018) Yaesoubi, M., Adalı, T., Calhoun, V.D., 2018.  A window-less approach for capturing time-varying connectivity in fmri data reveals the presence of states with variable rates of change.  Human Brain Mapping 39, 1626–1636.  URL: https://doi.org/10.1002/hbm.23939, doi:doi:10.1002/hbm.23939.
  * Yeo et al. (2011) Yeo, B.T.T., Krienen, F.M., Sepulcre, J., Sabuncu, M.R., Lashkari, D., Hollinshead, M., Roffman, J.L., Smoller, J.W., Zöllei, L., Polimeni, J.R., Fischl, B., Liu, H., Buckner, R.L., 2011.  The organization of the human cerebral cortex estimated by intrinsic functional connectivity.  Journal of Neurophysiology 106, 1125–1165.  URL: https://doi.org/10.1152/jn.00338.2011, doi:doi:10.1152/jn.00338.2011.
  * Zhang and Sabuncu (2018) Zhang, Z., Sabuncu, M.R., 2018.  Generalized cross entropy loss for training deep neural networks with noisy labels.  URL: https://arxiv.org/abs/1805.07836, doi:doi:10.48550/ARXIV.1805.07836.