# Cross-Network Learning with Partially Aligned Graph Convolutional Networks

Meng Jiang  Department of Computer Science and Engineering, University of
Notre Dame384 Fitzpatrick HallNotre DameIndianaUSA<EMAIL_ADDRESS>
###### Abstract.

Graph neural networks have been widely used for learning representations of
_nodes_ for many downstream tasks on graph data. Existing models were designed
for the nodes on a single graph, which would not be able to utilize
information across multiple graphs. The real world does have multiple graphs
where the nodes are often _partially aligned_. For examples, knowledge graphs
share a number of named entities though they may have different relation
schema; collaboration networks on publications and awarded projects share some
researcher nodes who are authors and investigators, respectively; people use
multiple web services, shopping, tweeting, rating movies, and some may
register the same email account across the platforms. In this paper, I propose
partially aligned graph convolutional networks to learn node representations
across the models. I investigate multiple methods (including model sharing,
regularization, and alignment reconstruction) as well as theoretical analysis
to _positively_ transfer knowledge across the (small) set of partially aligned
nodes. Extensive experiments on real-world knowledge graphs and collaboration
networks show the superior performance of our proposed methods on relation
classification and link prediction.

Transfer Learning, Graph Neural Network, Representation Learning

##  1\. Introduction

Figure 1. Cross-GCN learning uses network alignment information to bridge the
separated model parameters and representation spaces and transfer knowledge
across the models.

Graph learning represents or encodes graph structure to perform tasks such as
classification, clustering, and regression on graphs. It is important and
ubiquitous with applications ranging from knowledge graph completion to movie
recommendation. It is originally studied as dimensionality reduction in
adjacency matrix factorization methods (Cai et al., 2010). Then network
embedding methods aim at preserving particular structure, property, or side
information in graph or networked data (Cui et al., 2018; Dong et al., 2017).
Recently, graph convolutional networks (GCNs) have been introduced to specify
model parameters (e.g., a set of weight matrices) on how to aggregate
information from a node’s local neighborhood (Kipf and Welling, 2017; Hamilton
et al., 2017b). Unlike the network embedding methods, these parameters are
shared across nodes and allow the approaches to generate node representations
that do not necessarily rely on the entire graph (Veličković et al., 2017;
Hamilton et al., 2017a; Wang et al., 2019c).

The sparsity problem in graph data is a major bottleneck for most graph
learning methods. For example, knowledge graphs are incomplete, causing
inaccuracy in downstream tasks such as fact retrieval and question answering,
especially when knowledge is represented under the Open World Assumption
(Schlichtkrull et al., 2018). In real-world recommender systems, users can
rate a very limited number of items, so the user-item bipartite graph is
always extremely sparse (Ying et al., 2018; Fan et al., 2019). The observed
data that can be used for adjacency matrix factorization or neural net graph
learning are radially insufficient.

Figure 2. Examples of partially aligned networks in real world for two
different tasks.

Although we cannot fabricate more observations, we may borrow useful knowledge
from graph data on different sources, domains, or platforms. Consider the
following case: A new UberPool-like carpooling App has launched. Due to lack
of usage at the beginning, machine learning on the very sparse user-user ride-
sharing graph cannot be effective for ride recommendation. Now suppose that a
small number of the users register the App with their social media accounts,
and we have social graph data available from the social media platform. Then
we ask, though the percentage of the “aligned” users is very small (10%, 5%,
or even less than 1%), can we use the alignment as a bridge and transfer
knowledge from the social graph to the ride-sharing graph? Since ride sharing
and social relationship are somewhat related, the _partially_ aligned users on
different graphs (platforms) can share similar behavioral patterns. Thus,
learning across the graphs can be beneficial. When the graph learning models
are state-of-the-art, learning across graph convolutional networks can
transfer knowledge via the partially aligned nodes.

Figure 1 illustrates the idea of cross-GCN learning. Originally, the three GCN
models are trained _separately_ to generate low-dimensional representations of
nodes $U^{[\cdot]}$ with a set of model parameters $\\{W^{[\cdot]}_{(l)}\\}$.
We seek for effective methods to transfer knowledge across the GCN models
through the partial alignment information between $\mathcal{G}^{[\cdot]}$ into
their node representations.

In this paper, I study how to alleviate the sparsity problem in graph learning
by transferring knowledge across multiple GCN models, when the graphs are
partially aligned.

The contributions of this work can be described in three aspects:

  1. (1)

Cross-GCN learning methods: I design three different types of methods: (a)
sharing model parameters, (b) aligning representation spaces with
regularizations, and (c) simultaneously reconstructing the alignment and
learning representations. I present how to generalize them for more than two
GCNs and for relational graphs.

  2. (2)

Theoretical foundations: I present theories on (a) the choice of the number of
dimensions of the representation spaces to force knowledge transfer and (b)
the conditions of positive transfer across GCN models, regularizations, and
alignment reconstruction.

  3. (3)

Extensive experiments: As shown in Figure 2, I perform experiments on two
types of graphs datasets and graph learning tasks: (a) link prediction on
weighted graphs (e.g., collaboration social networks) and (b) relation
classification on knowledge graphs. Both graphs are partially aligned (see the
number of aligned nodes in red). Cross-GCN learning performs consistently
better than individual GCNs on all the datasets and tasks.

The rest of this paper is organized as follows. Section gives definitions of
basic concepts (e.g., graph convolutional network) and the proposed goal.
Section presents multiple kinds of methods to achieve the goal of cross-
network learning. Section provides theoretical analysis on choice of number of
dimensions and positive transfer. Section presents experimental results on
knowledge graphs and collaboration networks for relation classification and
link prediction, respectively. Section reviews the related papers, and Section
concludes the work.

##  2\. Preliminaries

###  2.1. Graphs and Learning Tasks

We study on two types of graphs and related tasks about learning, classifying,
or predicting links between nodes.

###### Definition 2.1 (Weighted graph).

A weighted graph, denoted as
$\mathcal{G}=(\mathcal{U},\mathcal{E}\subset\mathcal{U}\times\mathcal{U})$,
contains a set of nodes $\mathcal{U}$ and a set of links $\mathcal{E}$, with a
weight mapping function $w:\mathcal{E}\rightarrow\mathbb{R}$.

For example, collaboration networks are weighted graphs. The weight is the
frequency of collaborations between two person nodes. So, it is a positive
integer when the link exists. The task of _link prediction_ is to predict
whether the value of $w(u_{i},u_{j})$ is positive or what the value is, given
a pair of nodes $u_{i},u_{j}\in\mathcal{U}$.

###### Definition 2.2 (Relational graph).

A relational graph, denoted as
$\mathcal{G}_{rel}=(\mathcal{U},\mathcal{E}\subset\mathcal{U}\times\mathcal{U}\times\mathcal{R},\mathcal{R})$,
contains a set of nodes $\mathcal{U}$, a set of relational links
$\mathcal{E}$, and a relation schema $\mathcal{R}$. A relation link
$e=(u_{i},u_{j},r)\in\mathcal{E}$ indicates that $u_{i}$ and $u_{j}$ have
relation $r$.

Knowledge graphs are typical relational graphs, where the relational links
need to be learned and completed. The task of _relation classification_ is to
predict the type of relation between two nodes (i.e., entities) $u_{i},u_{j}$
from the relation schema $\mathcal{R}$ so that a link $e=(u_{i},u_{j},r)$ is
likely to exist in the graph.

The fundamental problem of both tasks is _node representation learning_ that
learns low-dimensional vectors of nodes to preserve the graph’s information.
Take _link prediction_ as an example – formally, it is to learn a mapping
function: $\vec{u}=f(u,\mathcal{G},\Theta)\in\mathbb{R}^{d}$, where $\Theta$
is the node representation learning model’s parameters and $d$ is the number
of dimensions of node representations. Then the value of $w(u_{i},u_{j})$ is
assigned as $\sigma(\vec{u_{i}}^{\top}\vec{u_{j}})$, where $\sigma$ is
sigmoid. Simple predictive functions can be specified for the other two tasks.

###  2.2. Graph Convolutional Network

Graph convolutional network (GCN) is one of the most popular models for node
representation learning (Kipf and Welling, 2017). Let us continue using _link
prediction_ as the example. We denote the number of nodes in graph
$\mathcal{G}$ by $n=|\mathcal{U}|$. Suppose $A\in\mathbb{R}^{n\times n}$ is
the adjacency matrix of $\mathcal{G}$ and $X\in\mathbb{R}^{n\times m}$ has the
nodes’ raw attribute values, where $m$ is the number of the raw attributes. A
two-layer GCN generates the matrix of node representations as follows:

(1)
$U=\mathrm{softmax}\left(\hat{A}~{}\mathrm{ReLU}(\hat{A}~{}X~{}W_{(1)})~{}W_{(2)}\right)\in\mathbb{R}^{n\times
d},$

where $\Theta=\\{W_{(1)}\in\mathbb{R}^{m\times
d},W_{(2)}\in\mathbb{R}^{d\times d}\\}$ has layer-specific weight matrices. We
choose two layers as most of existing studies suggest. The pre-processing step
includes (1) $\tilde{A}=A+I_{n}$, (2) $\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}$,
and (3)
$\hat{A}=\tilde{D}^{\frac{1}{2}}~{}\tilde{A}~{}\tilde{D}^{\frac{1}{2}}$. Graph
reconstruction is used to inform the GCN to learn $\Theta$ to generate
effective $U$:

(2) $g(X,A,W_{(1)},W_{(2)})=\sigma(UU^{\top})\in{[0,1]}^{n\times n}.$

The loss function can be written as follows:

(3) $\displaystyle f(W_{(1)},W_{(2)})$ $\displaystyle=$
$\displaystyle\mathcal{L}\left(g(X,A,W_{(1)},W_{(2)}),~{}A\right)$
$\displaystyle=$ $\displaystyle-\sum_{i,j\in\\{1\dots
n\\}}A_{ij}~{}\mathrm{log}\left(\sigma(\vec{u_{i}}^{\top}\vec{u_{j}})\right)$

where $\vec{u_{i}}$ and $\vec{u_{j}}$ are the $i$-th and $j$-th rows in matrix
$U$. It is worth noting that one can easily extend our study from two-layer
GCN to $l$-layer GCN ($l>2$).

###  2.3. Cross-Network Learning

One GCN can be built for one graph. What if we have multiple graphs which
overlap each other to perform the same type of tasks? The overlapping parts,
referred as _network alignment_ serve as bridge for transferring knowledge
across multiple GCNs so that they can mutually enhance each other. Without
loss of generality, we focus on the cases that have two graphs
$\mathcal{G}^{[A]}$ and $\mathcal{G}^{[B]}$. We denote by
$A^{[A,B]}\in{\\{0,1\\}}^{n_{A}\times n_{B}}$ the adjacency matrix of network
alignment between the two graphs: $A^{[A,B]}_{ij}=1$, if node $u^{[A]}_{i}$
and node $u^{[B]}_{j}$ are aligned; where $n_{A}=|\mathcal{U}^{[A]}|$ and
$n_{B}=|\mathcal{U}^{[B]}|$. Usually the graphs are _partially aligned_ ,
leading to missing alignment links and the sparsity of the alignment matrix.

###### Definition 2.3 (Separated GCNs).

Traditionally, the network alignment was ignored in building GCNs. So, two
separated GCNs would be built for the tasks on graphs $\mathcal{G}^{[A]}$ and
$\mathcal{G}^{[B]}$:

$\displaystyle
f(W^{[A]}_{(1)},W^{[A]}_{(2)})=\mathcal{L}\left(g(X^{[A]},A^{[A]},W^{[A]}_{(1)},W^{[A]}_{(2)}),~{}A^{[A]}\right),$
(4) $\displaystyle
f(W^{[B]}_{(1)},W^{[B]}_{(2)})=\mathcal{L}\left(g(X^{[B]},A^{[B]},W^{[B]}_{(1)},W^{[B]}_{(2)}),~{}A^{[B]}\right),$

where no model parameters (between ${\Theta}^{[A]}$ and ${\Theta}^{[B]}$) is
shared and no alignment information is used.

###### Definition 2.4 (Cross-network learning).

Given two graphs $\mathcal{G}^{[A]}$ and $\mathcal{G}^{[B]}$ as well as the
network alignment information $A^{[A,B]}$, build cross-network learning models
that have the properties below:

  * •

generate node representations $U^{[A]}\in\mathbb{R}^{n_{A}\times d_{A}}$ and
$U^{[B]}\in\mathbb{R}^{n_{B}\times d_{B}}$ through graph convolutions, where
$d_{A}$ and $d_{B}$ are the numbers of dimensions, respectively – $d_{A}\ll
m_{A}$, $d_{B}\ll m_{B}$, and they do not have to be equivalent;

  * •

perform better than separated GCNs by sharing model parameters and/or
incorporating the network alignment information.

##  3\. Proposed Methods

In this section, we present multiple types of methods for transferring
knowledge across GCNs. The first type trains two GCNs by sharing their model
parameters (Sections 3.1 and 3.2). The second uses the network alignment
information to regularize the output representation spaces (Section 3.3). And
the third type uses network alignment reconstruction as an extra task for
knowledge transfer (Section 3.4). Generalizations to multiple GCNs and
particular types of graphs are discussed (Sections 3.5 to
LABEL:sec:bipartite).

###  3.1. Sharing Model Parameters across GCNs

Recall that a two-layer GCN on graph $\mathcal{G}^{[A]}$ has parameters
$\Theta^{[A]}=\\{W^{[A]}_{1}\in\mathbb{R}^{m_{A}\times
d_{A}},W^{[A]}_{2}\in\mathbb{R}^{d_{A}\times d_{A}}\\}$. Same goes for graph
$\mathcal{G}^{[B]}$.

Suppose $d_{A}\neq d_{B}$. None of the parameters can be directly shared.

Suppose $d_{A}=d_{B}$. The second-layer weight parameters can be shared:
$W_{2}=W^{[A]}_{2}=W^{[B]}_{2}$. So, we can perform joint GCN training:

(5)

where $\alpha^{[A]}$ and $\alpha^{[B]}$ are per-graph weight of the training
procedure. When we have two graphs, $\alpha^{[A]}+\alpha^{[B]}=1$.

Suppose $m_{A}=m_{B}$ and $d_{A}=d_{B}$. The weight parameters of both layers
can be shared across GCNs. We omit the equation for space – it simply replaces
$W^{[A]}_{(1)}$ and $W^{[B]}_{(1)}$ in Eq.(5) by $W_{(1)}$. However, the
multiple graphs can hardly have the same raw attribute space. So, when
$d=d_{A}=d_{B}$ but $m_{A}\neq m_{B}$, we assume that linear factorized
components of $W^{[A]}_{(1)}$ and $W^{[B]}_{(1)}$ can be shared. We denote the
shared component by $W_{(1)}\in\mathbb{R}^{\hat{m}\times d}$, where
$\hat{m}=\mathrm{min}\\{m_{A},m_{B}\\}$ or a smaller value. Then the loss
function of joint training is as follows:

(6)

where $Q^{[A]}\in\mathbb{R}^{m_{A}\times\hat{m}}$ and
$Q^{[B]}\in\mathbb{R}^{m_{B}\times\hat{m}}$ are linear transformation matrices
on raw feature spaces for aligning the first-layer weight parameter matrices.

When the number of graphs is two, we can perform alternatives to save one
linear transformation matrix. Alternative 1 is to use $Q^{[A\rightarrow
B]}\in\mathcal{R}^{m_{A}\times m_{B}}$ to align $\mathcal{G}^{[A]}$’s raw
attribute space to $\mathcal{G}^{[B]}$’s:

(7)

And alternative 2 is to use $Q^{[B\rightarrow A]}\in\mathcal{R}^{m_{B}\times
m_{A}}$ to align $\mathcal{G}^{[B]}$’s raw attribute space to
$\mathcal{G}^{[A]}$’s.

###  3.2. Training Strategies across GCNs

The training strategy presented in the above section is _joint training_ or
referred as multi-task learning. The weights of the tasks, i.e.,
$\alpha^{[A]}$ and $\alpha^{[B]}$, are determined prior to the training
procedure. The other popular training strategy is called _pre-training_ , that
is to define one task a time as the target task and all the other tasks as
source tasks. Take the method of sharing $W_{(2)}$ as an example. The loss
function of joint training was given in Eq.(5). Now if learning graph
$\mathcal{G}^{[B]}$ is the target task, then learning graph
$\mathcal{G}^{[A]}$ is the source task. The pre-training procedure has two
steps. Their loss functions are:

$\displaystyle
f_{1}(W^{[A]}_{(1)};W_{(2)})=\mathcal{L}\left(g(X^{[A]},A^{[A]},W^{[A]}_{(1)},W_{(2)}),~{}A^{[A]}\right),$
(8) $\displaystyle
f_{2}(W^{[B]}_{(1)};W_{(2)})=\mathcal{L}\left(g(X^{[B]},A^{[B]},W^{[B]}_{(1)},W_{(2)}),~{}A^{[B]}\right).$

Each step still minimizes Eq.(5): specifically, the first step uses
$\alpha^{[A]}=1$ and $\alpha^{[B]}=0$; and the second step uses
$\alpha^{[A]}=0$ and $\alpha^{[B]}=1$. The first step warms up the training of
the shared $W_{(2)}$ on $\mathcal{G}^{[A]}$ so that the “fine-tuning” step on
$\mathcal{G}^{[B]}$ can find more effective model parameters than starting
from pure randomization. When the source and target are swapped, the two steps
are swapped.

###  3.3. Aligning Representation Spaces with Regularizations

Besides sharing model parameters, an idea of bridging the gap between two GCNs
is to align their output representation spaces. Given GCNs trained on two
graphs $\mathcal{G}^{[A]}$ and $\mathcal{G}^{[B]}$, if two nodes $u^{[A]}_{i}$
and $u^{[B]}_{j}$ are aligned, we assume that their representations,
$\vec{u}^{[A]}_{i}\in\mathbb{R}^{d_{A}}$ and
$\vec{u}^{[B]}_{j}\in\mathbb{R}^{d_{B}}$ are highly correlated. For example,
one named entity (person, location, or organization) in two different
knowledge graphs should have very similar representations. The same goes for
one researcher in two collaboration networks (as an investigators in projects
or an author in papers).

We add the network alignment information as two types of regularization terms
into the loss function. The first type is _Hard regularization_. It assumes
that the aligned nodes have exactly the same representations in the two output
spaces. This requires the two spaces to have the same number of dimensions:
$d_{A}=d_{B}$. The term is written as:

(9)
$h(W^{[A]}_{(1)},W^{[A]}_{(2)},W^{[B]}_{(1)},W^{[B]}_{(2)})={\left\|U^{[A]}-A^{[A,B]}~{}U^{[B]}\right\|}^{2}_{\mathrm{F}}.$

The entire loss function is as below:

(10) $\displaystyle
f({\Theta}^{[A]},{\Theta}^{[B]})=f(W^{[A]}_{(1)},W^{[A]}_{(2)},W^{[B]}_{(1)},W^{[B]}_{(2)})$
$\displaystyle=$
$\displaystyle(1-\beta)\cdot\alpha^{[A]}\cdot\mathcal{L}\left(g(X^{[A]},A^{[A]},W^{[A]}_{(1)},W^{[A]}_{(2)}),~{}A^{[A]}\right)$
$\displaystyle+~{}(1-\beta)\cdot\alpha^{[B]}\cdot\mathcal{L}\left(g(X^{[B]},A^{[B]},W^{[B]}_{(1)},W^{[B]}_{(2)}),~{}A^{[B]}\right)$
$\displaystyle+~{}\beta\cdot
h(W^{[A]}_{(1)},W^{[A]}_{(2)},W^{[B]}_{(1)},W^{[B]}_{(2)}),$

where $\beta$ is the weight of the regularization task.

The second type is _Soft regularization_ which is designed for more common
cases that $d_{A}\neq d_{B}$. It assumes that the aligned nodes’
representations from one GCN can be linearly transformed into the ones in the
other GCN’s output space:

(11) $h({\Theta}^{[A]},{\Theta}^{[B]},R^{[A\rightarrow
B]})={\left\|U^{[A]}R^{[A\rightarrow
B]}-A^{[A,B]}~{}U^{[B]}\right\|}^{2}_{\mathrm{F}},$

where $R^{[A\rightarrow B]}\in\mathbb{R}^{d_{A}\times d_{B}}$ is the
transformation matrix. Note that $R^{[A\rightarrow B]}$ is treated as model
parameters in the loss function.

###  3.4. Alignment Reconstruction

When the two graphs are partially aligned, the network alignment information
is usually sparse and incomplete. To address this issue, we treat the
alignment information as a bipartite graph $\mathcal{G}^{[A,B]}$ and learn to
reconstruct and complete the graph. Given the node representations
$U^{[A]}\in\mathbb{R}^{n_{A}\times d_{A}}$ and
$U^{[B]}\in\mathbb{R}^{n_{B}\times d_{B}}$, can we reconstruct the adjacency
matrix of observed alignment data $\mathcal{G}^{[A,B]}$, i.e., $I^{[A,B]}\odot
A^{[A,B]}\in\mathcal{R}^{n_{A}\times n_{B}}$, where $I^{[A,B]}$ is the
indicator matrix of the observations? We introduce
$R^{[A,B]}\in\mathbb{R}^{d_{A}\times d_{B}}$ and minimize the following
optimization term:

(12)
$h={\left\|\sigma\left(U^{[A]}~{}R^{[A,B]}~{}U^{[B]\top}\right)-I^{[A,B]}\odot
A^{[A,B]}\right\|}^{2}_{\mathrm{F}}.$

The entire loss function optimizes on three graph reconstruction tasks
($\mathcal{G}^{[A]}$, $\mathcal{G}^{[B]}$, and $\mathcal{G}^{[A,B]}$), by
replacing $h$ in Eq.(10) with the above equation. The learning procedure
transfers knowledge across the three tasks to find effective representations
$U^{[A]}$ and $U^{[B]}$.

###  3.5. Beyond Two: Extend to Multiple GCNs

Suppose we have $K>2$ GCNs. All the above methods (i.e., loss functions) can
be easily extended from $2$ to $K$ GCNs. For example, the loss of joint multi-
GCN training with _shared $W_{2}$_ can be written as below (extended from
Eq.(5)):

(13)

$f({\\{W^{[i]}_{(1)}\\}}|^{K}_{i=1};W_{(2)})=\sum^{K}_{i=1}\alpha^{[i]}\mathcal{L}\left(g(X^{[i]},A^{[i]},W^{[i]}_{(1)},W_{(2)}),~{}A^{[i]}\right).$

And the loss of joint multi-GCN training with _shared $W_{1}$ and $W_{2}$_ can
be written as below (extended from Eq.(6)):

(14)

$f({\\{Q^{[i]}\\}}|^{K}_{i=1};W_{(1)},W_{(2)})=\sum^{K}_{i=1}\alpha^{[i]}\mathcal{L}\left(g(X^{[i]}Q^{[i]},A^{[i]},W_{(1)},W_{(2)}),~{}A^{[i]}\right).$

When using the network alignment information, the _soft regularization_ term
is written as below (extended from Eq.(11)):

(15) $\displaystyle
h\left({\\{{\Theta}^{[i]}\\}}|^{K}_{i=1},~{}{\\{R^{[i\rightarrow
j]}\\}}|_{i<j;~{}i,j\in\\{1\dots n\\}}\right)$ $\displaystyle=$
$\displaystyle\sum_{i<j;~{}i,j\in\\{1\dots n\\}}\gamma^{[i\rightarrow
j]}{\left\|U^{[i]}R^{[i\rightarrow
j]}-A^{[i,j]}~{}U^{[j]}\right\|}^{2}_{\mathrm{F}},$

where $\gamma^{[i\rightarrow j]}$ is the regularization weight of aligning
representation spaces between graphs $\mathcal{G}^{[i]}$ and
$\mathcal{G}^{[j]}$.

The optimization term for alignment reconstruction and completion can be
written as below (extended from Eq.(16)):

(16) $h=\sum_{i<j}\gamma^{[i\rightarrow
j]}{\left\|\sigma\left(U^{[i]}~{}R^{[i,j]}~{}U^{[j]\top}\right)-I^{[i,j]}\odot
A^{[i,j]}\right\|}^{2}_{\mathrm{F}}.$

###  3.6. Implementation on Relational Graphs: Relation Classification across
GCNs

Schlichtkrull _et al._ proposed neural relational modeling that uses GCNs for
relation classification on relational graphs (Schlichtkrull et al., 2018). The
output node representations from two-layer GCN are given as below:

(17)
$U=\mathrm{softmax}\left(\sum_{r\in\mathcal{R}}\frac{1}{c_{r}}A_{:,:,r}~{}\mathrm{ReLU}\left(\sum_{r\in\mathcal{R}}\frac{1}{c_{r}}A_{:,:,r}~{}X~{}W_{r,(1)}\right)~{}W_{r,(2)}\right).$

Clearly, it can be derived from Eq.(1) by making these changes:

  * •

The relational graph
$\mathcal{G}_{rel}=\\{\mathcal{U},\mathcal{E},\mathcal{R}\\}$ is denoted as a
three-way tensor $A\in{\\{0,1\\}}^{n\times n\times n_{R}}$, where
$n=|\mathcal{U}|$ is the number of entity nodes and $n_{R}=|\mathcal{R}|$ is
the number of relation types;

  * •

The model parameters are two three-way tensors:
$\Theta=\\{{W_{r,(1)}}|_{r\in\mathcal{R}}\in\mathbb{R}^{m\times
d},{W_{r,(2)}}|_{r\in\mathcal{R}}\in\mathbb{R}^{d\times d}\\}$;

  * •

$c_{r}$ controls the weight of relation type $r$.

Relation classification can be considered as relational link prediction if we
apply the prediction on all the possible entity-relation triples. The full
prediction matrix is:

(18) $g(X,A,W_{(1)},W_{(2)},D)=\sigma(U\otimes D\otimes
U^{\top})\in{[0,1]}^{n\times n\times n_{R}},$

where $D\in\mathbb{R}^{d\times d\times n_{R}}$ is a three-way core tensor. For
each relation type $r$, $D_{:,:,r}$ is a $d\times d$-diagonal matrix. The loss
function is below:

(19) $\displaystyle
f(W_{(1)},W_{(2)},D)=\mathcal{L}\left(g(X,A,W_{(1)},W_{(2)},D),~{}A\right)$
$\displaystyle=$ $\displaystyle-\sum_{i,j\in\\{1\dots
n\\};~{}r\in\mathcal{R}}A_{i,j,r}~{}\mathrm{log}\left(\sigma(\vec{u_{i}}^{\top}~{}D_{:,:,r}~{}\vec{u_{j}})\right)$

All the proposed methods for partially aligned GCNs in the previous sections
can be applied here. The key differences are (1) a three-way tensor data for
reconstruction and (2) additional parameters $D$.

##  4\. Theoretical Analysis

In this section, we present theoretical results for partially aligned GCNs.
First, we study the choices of the number of dimensions of node
representations that may perform no knowledge transfer across GCNs. Then, we
study two factors of positive transfer across GCNs. Note that many details are
given in Appendix to save space.

###  4.1. Choices of the Number of Dimensions $d$

The task of graph reconstruction is to use the node representations
$U\in\mathbb{R}^{n\times d}$ to recover the adjacency matrix
$A\in\mathbb{R}^{n\times n}$. With singular value decomposition, we have
$A\simeq\hat{U}~{}\Sigma~{}\hat{U}^{\top}$, where
$\hat{U}\in\mathbb{R}^{n\times\mathrm{rank}(A)}$ and $\Sigma$ is square
diagonal of size $\mathrm{rank}(A)\times\mathrm{rank}(A)$. So, the loss
function in Eq.(3) can be approximated as the squared loss:

(20)
$f={\left\|~{}U~{}B-\hat{U}~{}\sqrt{\Sigma}~{}\right\|}^{2}_{\mathrm{F}},$

where $B\in\mathcal{R}^{d\times\mathrm{rank}(A)}$. It transforms the $n\times
n$ classification tasks (for link prediction) into $\mathrm{rank}(A)$
regression tasks.

###### Proposition 4.1.

Suppose we have $K$ GCNs. In a linear setting for every model in the partially
aligned GCNs, the loss function becomes:

(21)

$f\left({\\{\Theta_{i}=W^{[i]}_{(1)}W^{[i]}_{(2)}B^{[i]}\\}}|^{K}_{i=1}\right)=\sum^{K}_{i=1}{{\left\|~{}\hat{A}^{[i]~{}2}X^{[i]}\Theta_{i}-\hat{U}^{[i]}~{}\sqrt{{\Sigma}^{[i]}}~{}\right\|}^{2}_{\mathrm{F}}}.$

The optimal solution for the $i$-th GCN is

(22)
${\Theta}_{i}={\left(X^{[i]\top}\hat{A}^{[i]~{}4}X^{[i]}\right)}^{\dagger}X^{[i]\top}\hat{A}^{[i]~{}2}\hat{U}^{[i]}\sqrt{{\Sigma}^{[i]}}\in\mathcal{R}^{m_{i}\times\mathrm{rank}(A)},$

where $X^{\dagger}$ denotes $X$’s pseudoinverse. Hence a capacity of
$\mathrm{rank}(A^{[i]})$ suffices for the $i$-th GCN. If
$d\geq\sum^{K}_{i=1}\mathrm{rank}(A^{[i]})$, then $B^{[i]}$ can be an identity
matrix and there is _no transfer_ between any two GCNs. In that case, no
matter ${\\{W^{[i]}_{(1)}\\}}|^{K}_{i=1}$ or
${\\{W^{[i]}_{(2)}\\}}|^{K}_{i=1}$ are shared or not, these exists an optimum
$W^{[i]*}_{(1)}W^{[i]*}_{(2)}={\Theta}_{i}$, for all $i=1\dots K$.

The illustration of this idea and extension to nonlinear settings are
discussed in Appendix.

###  4.2. Positive Transfer across GCN Models

We apply the theory of _positive transfer_ in multi-task learning (MTL) (Wu et
al., 2019) into cross-GCNs learning. If the cross-GCNs training improves over
just training the target GCN on one particular graph, we say the source GCNs
on other graphs _transfer positively_ to the target GCN. (Wu et al., 2019)
proposed a theorem to quantify how many data points are needed to guarantee
positive transfer between two ReLU model tasks. By the above section, it is
necessary to limit the capacity of the shared model parameters to enforce
knowledge transfer. Here we consider $d=1$ and $m=m_{A}=m_{B}$ to align with
the theorem in (Wu et al., 2019). We have a theorem for positive transfer
between two _one-layer_ GCNs that share the first-layer model parameters
$W_{(1)}$:

###### Theorem 4.2.

Let $(\hat{A}^{[i]}X^{[i]}\in\mathbb{R}^{n_{i}\times
m},\vec{\hat{u}}^{[i]}\sqrt{\lambda^{[i]}_{1}})\in\mathbb{R}^{n_{i}}$ denote
the data of graph $\mathcal{G}^{[i]}$ ($i=\\{A,B\\}$). Suppose the output is
generated by $ReLU(\hat{A}^{[i]}X^{[i]}~{}{\theta}^{[i]})$. And suppose
$c\geq\mathrm{sin}({\theta}^{[A]},{\theta}^{[B]})/\kappa(\hat{A}^{[B]}X^{[B]})$.
Denote by $W^{\star}_{(1)}$ the optimal solution of the one-layer setting of
Eq.(6):
$f(W_{(1)})=\sum_{i\in\\{A,B\\}}\mathcal{L}\left(g(X^{[i]},A^{[i]},W_{(1)}),~{}A^{[i]}\right)$.
With probability $1-\delta$ over the randomness of
$(\hat{A}^{[A]}X^{[A]},\vec{\hat{u}}^{[A]}\sqrt{\lambda^{[A]}_{1}})$, when

(23) $n_{i}\geq\mathrm{max}\left(\frac{m\log m}{c^{2}}(\frac{1}{c^{2}}+\log
m),\frac{{\|\vec{\hat{u}}^{[B]}\sqrt{\lambda^{[B]}_{1}}\|}^{2}}{c^{2}}\right),$

we have that the estimation error is at most:

(24)
$\mathrm{sin}(W^{\star}_{(1)},{\theta}^{[A]})\leq\mathrm{sin}({\theta}^{[A]},{\theta}^{[B]})+O(c/\kappa(\hat{A}^{[B]}X^{[B]})).$

_Notations for the above theorem._
$\kappa(A)=\lambda_{\mathrm{max}}(A)/\lambda_{\mathrm{min}}(A)$ denotes the
condition number of $A\in\mathbb{R}^{n\times m}$, where
$\lambda_{\mathrm{max}}(A)$ denotes its largest singular value and
$\lambda_{\mathrm{min}}(A)$ denotes its min$\\{n,m\\}$-th largest singular
value. As MTL is effective over “similar” tasks (Xue et al., 2007), the cross-
GCNs learning can be effective across similar GCN models. A natural definition
of GCN similarity is $\mathrm{cos}(\theta^{[A]},\theta^{[B]})$: higher means
more similar. So
$\mathrm{sin}(W^{\star},\theta)=\sqrt{1-\mathrm{cos}^{2}(W^{\star},\theta)}$
can denote the estimation error. Proof of the theorem can be referred in (Wu
et al., 2019).

###  4.3. Positive Transfer with Network Alignment

Soft regularization: We transfer knowledge across two GCNs’ representation
spaces by optimizing Eq.(10) where $h$ is specified as Eq.(11). The optimal
solution is

$\displaystyle U^{[A]\star}={\left((1-\beta)\cdot\alpha^{[A]}~{}I+\beta
R^{[A\rightarrow B]}R^{[A\rightarrow B]\top}\right)}^{-1}\cdot$ (25)
$\displaystyle\cdot{\left((1-\beta)\cdot\alpha^{[A]}~{}\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}+\beta~{}A^{[A,B]}U^{[B]\star}R^{[A\rightarrow
B]\top}\right)}.$

So, if $\beta=0$, $U^{[A]\star}=\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}$ may overfit
the graph data $\mathcal{G}^{[A]}$; if $\beta\in(0,1]$, the information
transferred from $\mathcal{G}^{[B]}$ via the alignment $A^{[A,B]}$, i.e.,
$A^{[A,B]}U^{[B]\star}R^{[A\rightarrow B]}$, can have positive impact on link
prediction on $\mathcal{G}^{[A]}$.

Alignment reconstruction: Suppose we optimize Eq.(10 where $h$ is in Eq.(16).
The optimal solution is

$\displaystyle
U^{[A]\star}={\left((1-\beta)\cdot\alpha^{[A]}~{}\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}+\beta
A^{[A,B]}U^{[B]\star}R^{[A,B]\top}\right)}$ (26)
$\displaystyle\cdot{\left((1-\beta)\cdot\alpha^{[A]}~{}I+\beta
R^{[A,B]}U^{[B]\star\top}U^{[B]\star}R^{[A,B]\top}\right)}^{-1}.$

So, if $\beta=0$, $U^{[A]\star}=\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}$ may overfit
the graph data $\mathcal{G}^{[A]}$; if $\beta\in(0,1]$, the information
transferred from $\mathcal{G}^{[B]}$ via the alignment $A^{[A,B]}$, i.e.,
$A^{[A,B]}U^{[B]\star}R^{[A,B]\top}$, can have positive impact on link
prediction on $\mathcal{G}^{[A]}$.

Table 1. Performance of VGAE (Kipf and Welling, 2016), (Separated) GCNs (Kipf
and Welling, 2017), and 13 cross-GCN learning models on link prediction in
NSFAward and DBLP academic graphs ($\mathcal{G}^{[A]}$ and
$\mathcal{G}^{[B]}$). The models take various options on model sharing,
regularizations, and alignment reconstruction. Equations can be found in
Sections 2 and 3. Higher AUC or AP means better performance. We investigated
the standard deviation of all the cells: all the values are smaller than
0.0016 and thus omitted for space. Underlined numbers are targeted values in
pre-training strategies. Bolded numbers are the highest on the columns.

##  5\. Experiments

In this section, I perform cross-network learning on three types of graph
learning tasks. For each task, I present datasets (stats can be found in
Figure 2), experimental settings, results, and analysis.

###  5.1. Link Prediction

####  5.1.1. Datasets

Below are two collaboration networks I use that can be considered as weighted
graphs between researcher nodes.

  * •

NSFAward: It is publicly available on the National Science Foundation (NSF)
Awards database from 1970 to 2019.111NSF Award Search:
https://www.nsf.gov/awardsearch/download.jsp Nodes are investigators of the
awards. Two investigator nodes have a link if they are participated in the
same award(s) and the weight of the link is the number of their shared
award(s).

  * •

DBLP: I expand the DBLP-4Area (Huang et al., 2016) from 5,915 authors to
14,387 authors by adding their co-authors in the graph. The weight of the link
between two author nodes is the number of their co-authored papers.

Figure 3. Sensitivity analysis on M13 (sharing $W_{(1)}$ and $W_{(2)}$, plus
alignment reconstruction): (a) on the weight of the alignment reconstruction
task $\beta$, (b) on the weight of learning on graph $\mathcal{G}^{[A]}$:
$\alpha^{[A]}$. And (c) running time analysis in seconds per epoch.

####  5.1.2. Experimental settings

For each graph, I randomly split the observed links into training, validation,
and test sets by 8:1:1. Then I randomly select unobserved links as negative
examples in the validation and test sets. Positives and negatives are
balanced. I use Area Under the ROC Curve (AUC) and Average Precision (AP) as
evaluation metrics. I perform the evaluation 10 times, calculate mean value
and standard deviation. I also report the “overall” performance by calculating
the harmonic mean of the AUC/AP on the two datasets. All models are developed
by PyTorch and one NVIDIA GeForce RTX 2080 Ti graphic card. I set
$d_{A}=d_{B}=64$. I use grid search on the hyperparameters in
$\\{0.1,0.2,\dots,0.9\\}$ and find that the best combination is
$\alpha^{[A]}=\alpha^{[B]}=\beta=0.5$.

####  5.1.3. Results

Table 1 presents the performance of as many as 13 cross-GCN learning models I
implemented. Models M3–M8 share $W_{(2)}$ across the NSFAward and DBLP
academic graphs. Models M9–M13 share both $W_{(1)}$ and $W_{(2)}$. M6 uses
hard regularization; M1, M7, and M12 use soft regularization; and M2, M8, and
M13 use alignment reconstruction. I have the following observations:

O1. _Cross-network models perform better than baselines and the best is M13:_
M13 shares $W_{(1)}$ and $W_{(2)}$ between two GCNs, and uses alignment
reconstruction to bridge the two representation spaces. In terms of the
overall AUC on the two test sets, M13 achieves 0.865, which is significantly
higher than VGAE’s 0.782 (Kipf and Welling, 2016) and (Separated) GCNs’ 0.803
(Kipf and Welling, 2017). The AP is improved from VGAE’s 0.773 and GCN’s 0.793
to 0.870. Clearly, the knowledge is successfully transferred across the GCN
models on two academic graphs. Performance on test sets is worse than but very
close to that on validation sets, which indicates no or very little
overfitting in the GCN models. Both models need information beyond the
corresponding graph data for effective learning and decision-making on the
test data.

O2. _Sharing weight matrices is beneficial, and sharing those of both layers
is beneficial:_ Suppose we do not use the alignment information at all: the
only possible bridge across two GCNs is weight matrices $W_{(1)}$ and
$W_{(2)}$. M3–M5 share $W_{(2)}$. Compared with separated GCNs, M5 improves
the overall AUC on test sets from 0.803 to 0.836. And M9–M11 share both
$W_{(1)}$ and $W_{(2)}$. Compared with M5, M11 improves the AUC from 0.836 to
0.847. When soft regularization is adopted, we can compare M7 and M12 with M1;
when alignment reconstruction is adopted, we can compare M8 and M13 with M2.
The conclusions are the same.

Figure 4. Empirical analysis on the choice of $d$.

O3. _If $W_{(2)}$ is shared, joint training is slightly better than pre-
training:_ M3 pre-trains the model with $\mathcal{G}^{[A]}$ and then fine-
tunes on $\mathcal{G}^{[B]}$; M4 switches the use of the graphs. So we focus
on M3’s performance on $\mathcal{G}^{[B]}$ and M4’s on $\mathcal{G}^{[A]}$.
Compared with separated GCNs, M3 improves AUC on the test set from 0.840 to
0.854; M4 improves it from 0.769 to 0.806. However, with a joint training
strategy, M5 achieves an AUC on $\mathcal{G}^{[B]}$ of 0.860 over M3’s 0.854
and achieves an AUC on $\mathcal{G}^{[A]}$ of 0.812 over M4’s 0.806.

O4. _If $W_{(1)}$ is shared, learning linear transformation matrices on both
graphs is slightly better than learning a matrix on one of them:_ M11 learns
$Q^{[A]}\in\mathbb{R}^{m_{A}\times\hat{m}}$ and
$Q^{[B]}\in\mathbb{R}^{m_{B}\times\hat{m}}$. M9 learns $Q^{[A\rightarrow B]}$
only, and M10 learns $Q^{[B\rightarrow A]}$ only. In terms of the overall AUC
on the two test sets, compared with M5 that shares $W_{(2)}$ only, M9 improves
AUC from M5’s 0.836 to 0.840, M10 achieves an AUC of 0.845, and M11 achieves
an AUC of 0.847.

O5. _Regularizations, hard or soft, are beneficial; and alignment
reconstruction is better:_ (a) M6 adds hard regularization to M5. M6 improves
the overall AUC on the two test sets from M5’s 0.836 to 0.842. (b) M1, M7, and
M12 add soft regularization to separated GCNs, M5, and M11, respectively. M1
improves the AUC from 0.803 to 0.805, M7 improves from 0.836 to 0.849, and M13
improves from 0.847 to 0.863. With more model parameters (e.g., weight
matrices) shared, the improvements by regularizations become bigger. (c) M8
and M13 add alignment reconstruction to M5 and M11, respectively. M8 improves
the AUC from 0.836 to 0.856. M13 improves from 0.847 to 0.865. The AUC scores
are quite significantly improved by alignment reconstruction, slightly higher
than regularizations.

####  5.1.4. Hyperparameter sensitivity

In Figure 3(a) and (b) we investigate the effect of hyperparameters $\beta$
and $\alpha^{[A]}$ on the performance of model M13. When $\beta=0$, M13 is
equivalent with M11. With $\beta$ increasing, the performance becomes better
by learning to reconstruct the network alignment. It is the best when
$\beta\in[0.4,0.6]$. Then a too big $\beta$ will overfit the model on the
alignment reconstruction, so the AUC/AP decreases quickly when $\beta$ goes
from 0.6 to 1. Given $\beta=0.5$, a too big or too small $\alpha^{[A]}$ will
overfit one of the graph data, $\mathcal{G}^{[A]}$ or $\mathcal{G}^{[B]}$.
When $\alpha^{[A]}$ is around 0.5, the performance is the best. So, all the
optimization terms are important.

####  5.1.5. Time complexity

Figure 3(c) presents the time cost per epoch of separated GCNs, M1, M2, M5,
M11–M13. The models that share both $W_{(1)}$ and $W_{(2)}$ (M11), share
$W_{(2)}$ only (M5), and share neither of them (Separated) do not show
significant difference of time cost. Adopting soft regularizations (M1 vs
Separated, M12 vs M11) increases the time cost slightly. And adopting
alignment reconstruction (M2 vs M1, M13 vs M12) take slightly more time. The
time complexity remains at the same level in the proposed models.

####  5.1.6. On the choice of $d$

Figure 4 presents the overall AUC on test sets of Separated GCNs, M5 (sharing
$W_{(2)}$), and M11 (sharing $W_{(1)}$ and $W_{(2)}$), when the number of
dimensions $d$ is $\\{2,2^{2},\dots,2^{8}\\}$. When $d$ becomes bigger, the
improvements by model parameters sharing become smaller. As presented in
theoretical analysis, small $d$ would force the knowledge transfer across the
GCN models; when $d$ is too big, there might be no transfer. We choose $d=64$
as the default number of dimensions of the latent spaces.

###  5.2. Relation Classification

####  5.2.1. Datasets

Below are four knowledge graphs I use that can be considered as relational
graphs between entity nodes.

  * •

FB15K: It is a subset of the relational database Freebase (Schlichtkrull et
al., 2018).

  * •

WN18: It is a subset of WordNet containing lexical relations.

  * •

YAGO: The open source knowledge base developed by MPI.

  * •

DBpedia500: A subset of structured content in the Wikipedia.

I will investigate whether incorporating partially-aligned graphs YAGO and
DBPedia500 can improve the performance of relation classification on FB15K and
WN18, respectively. Entities are aligned if they have the same surface name.
Resolving the inaccuracy of entity alignment is important but out of this
work’s scope.

Table 2. Performance of existing methods (e.g., RGCN (Schlichtkrull et al.,
2018)) and cross-R-GCN learning models on relation classification.

####  5.2.2. Experimental settings

Both FB15K and WN18 have standard splits for training, validation, and test. I
provide results using two common evaluation metrics: mean reciprocal rank
(MRR) and Hits at $k$ (Hits@$k$). I report both raw and filtered MRR (with
filtered MRR typically considered more reliable) and filtered Hits@1 and
Hits@3, following TransE (Bordes et al., 2013). Other baseline methods
include:

  * •

HOlE (Nickel et al., 2016): it replaces vector-matrix product with circular
correlation to compute knowledge graph embeddings.

  * •

DistMult (Yang et al., 2014): a neural embedding approach that learns
representations of entities and relations.

  * •

ComplEx (Trouillon et al., 2016): it models asymmetric relations by
generalizing DistMult to the complex domain.

  * •

R-GCN (Schlichtkrull et al., 2018): it is the first work of relation graph
convolutional network, considered the state of the art. And I use it as the
basic GCN model for cross-GCN learning.

  * •

CrossE (Zhang et al., 2019a): its “interaction embeddings” are learned from
both knowledge graph connections and link explanations.

I use the Adam optimizer with a learning rate of 0.01. Implementation details
can be found in Appendix.

####  5.2.3. Results

Table 2 presents the results of the baselines and six cross-GCN learning
models (M1, M2, M5, M11–M13). Model M5 shares $W_{(2)}$ across two knowledge
graphs. M11–M13 share both $W_{(1)}$ and $W_{(2)}$. M1 and M12 use soft
regularization; and M2 and M13 use alignment reconstruction. Our main
observation is that _cross-network models perform better than baselines and
the best model is M13_. With the large-scale YAGO, M13 improves the filtered
MRR from 0.696 to 0.742 and the filtered Hits@1 from 0.601 to 0.641 for
relation classification on FB15K, compared to a sole R-GCN model. With
DBPedia500 which is bigger than YAGO, M13 improves the filtered MRR from 0.819
to 0.843 and the filtered Hits@1 from 0.697 to 0.743 on WN18, compared to
R-GCN. Both M12 and M13 that share two-layers parameters and alignment
information achieve higher performance than the best baseline CrossE on all
the metrics.

##  6\. Related Work

###  6.1. Graph Neural Networks

Graph neural network (GNN) is a type of neural network that operates on graph
structure to capture the dependence of graphs via message passing between the
nodes of graphs (Zhang et al., 2020; Wu et al., 2020). Kipf _et al._ proposed
graph convolutional networks that use a spectral graph convolution to capture
the spatial dependency (Kipf and Welling, 2017). Velivckovic _et al._
considered the interaction between nodes in the graph convolution measured as
graph attention weight (Veličković et al., 2017). Hamilton _et al._
implemented the graph convolution as an efficient aggregation and sampling
algorithm for inductive node representation learning (Hamilton et al., 2017a).

Recently, GNN techniques have been applied for learning representations on
relational graphs, specifically knowledge graphs (Sadeghian et al., 2019;
Zhang et al., 2019c; Ren and Leskovec, 2020). Relational GNNs perform
significantly better than conventional knowledge graph embeddings
(Schlichtkrull et al., 2018). Most techniques suffer from the issue of data
sparsity and our proposed methods address it with knowledge transfer across
networks.

In this paper, we discussed weighted graph, relational graph, and bipartite
graphs; however, heterogeneous graphs are ubiquitous and have unique essential
challenges for modeling and learning (Wang et al., 2020; Luo et al., 2020; Liu
et al., 2020; Wang et al., 2019a; Wang et al., 2019b). We have been seeing
heterogeneous GNNs (Dong et al., 2017; Liu et al., 2018; Zhang et al., 2019b;
Fu et al., 2020) and heterogeneous attention networks for graphs (Wang et al.,
2019c; Yang et al., 2020; Hu et al., 2020a; Yao et al., 2020; Hu et al.,
2020b). Extending our proposed cross-network learning to heterogeneous graphs
and enabling inductive learning in the framework are very important. These two
points are out of the scope of our work but worth being considered as future
directions.

###  6.2. Network Alignment and Modeling

The alignment of nodes between two or more than two graphs has been created
and/or utilized in at least three categories of methods (Jiang et al., 2012,
2015; Vijayan and Milenković, 2017; Chen et al., 2017; Qin et al., 2020; Gu
and Milenković, 2020). First, Kong _et al._ inferred anchor links between two
social networks by matching the subgraphs surrounding the linked nodes (Kong
et al., 2013). Emmert _et al._ wrote a comprehensive survey on the algorithms
of graph matching (Emmert-Streib et al., 2016). Second, Zhang _et al._ used
graph pattern mining algorithms for node alignment on incomplete networks
(Zhang et al., 2017) and multi-level network (Zhang et al., 2019d). Nassar _et
al._ leveraged the multimodal information for network alignment (Nassar and
Gleich, 2017). The third line of work learned node representations and trained
the supervised models with node pairing labels (if available) (Du et al.,
2019; Ye et al., 2019; Nassar et al., 2018). A matrix factorization-based
model has demonstrated that cross-network or cross-platform modeling can
alleviate the data sparsity on a single platform (Jiang et al., 2016).

##  7\. Conclusions

In this paper, I proposed partially aligned GCNs that jointly learn node
representations across graphs. I provided multiple methods as well as
theoretical analysis to positively transfer knowledge across the set of
partially aligned nodes. Extensive experiments on real-world knowledge graphs
and collaboration networks show the superior performance of the proposed
models on relation classification and link prediction.

## Acknowledgment

This research was supported by National Science Foundation award IIS-1849816.

## References

  * (1)
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ 26 (2013), 2787–2795.
  * Cai et al. (2010) Deng Cai, Xiaofei He, Jiawei Han, and Thomas S Huang. 2010\.  Graph regularized nonnegative matrix factorization for data representation.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 8 (2010), 1548–1560.
  * Chen et al. (2017) Zheng Chen, Xinli Yu, Bo Song, Jianliang Gao, Xiaohua Hu, and Wei-Shih Yang. 2017\.  Community-based network alignment for large attributed network. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 587–596.
  * Cui et al. (2018) Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. 2018\.  A survey on network embedding.  _IEEE Transactions on Knowledge and Data Engineering_ 31, 5 (2018), 833–852.
  * Dong et al. (2017) Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017\.  metapath2vec: Scalable representation learning for heterogeneous networks. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 135–144.
  * Du et al. (2019) Xingbo Du, Junchi Yan, and Hongyuan Zha. 2019.  Joint Link Prediction and Network Alignment via Cross-graph Embedding.. In _IJCAI_. 2251–2257.
  * Emmert-Streib et al. (2016) Frank Emmert-Streib, Matthias Dehmer, and Yongtang Shi. 2016\.  Fifty years of graph matching, network alignment and network comparison.  _Information sciences_ 346 (2016), 180–197.
  * Fan et al. (2019) Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019\.  Graph neural networks for social recommendation. In _The World Wide Web Conference_. 417–426.
  * Fu et al. (2020) Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020\.  MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding. In _Proceedings of The Web Conference 2020_. 2331–2341.
  * Gu and Milenković (2020) Shawn Gu and Tijana Milenković. 2020.  Data-driven network alignment.  _PloS one_ 15, 7 (2020), e0234978.
  * Hamilton et al. (2017a) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a.  Inductive representation learning on large graphs. In _Advances in neural information processing systems_. 1024–1034.
  * Hamilton et al. (2017b) William L Hamilton, Rex Ying, and Jure Leskovec. 2017b.  Representation learning on graphs: Methods and applications.  _arXiv preprint arXiv:1709.05584_ (2017).
  * Hu et al. (2020b) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020b.  Gpt-gnn: Generative pre-training of graph neural networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1857–1867.
  * Hu et al. (2020a) Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020a.  Heterogeneous graph transformer. In _Proceedings of The Web Conference 2020_. 2704–2710.
  * Huang et al. (2016) Zhipeng Huang, Yudian Zheng, Reynold Cheng, Yizhou Sun, Nikos Mamoulis, and Xiang Li. 2016\.  Meta structure: Computing relevance in large heterogeneous information networks. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 1595–1604.
  * Jiang et al. (2015) Meng Jiang, Peng Cui, Xumin Chen, Fei Wang, Wenwu Zhu, and Shiqiang Yang. 2015\.  Social recommendation with cross-domain transferable knowledge.  _IEEE transactions on knowledge and data engineering_ 27, 11 (2015), 3084–3097.
  * Jiang et al. (2012) Meng Jiang, Peng Cui, Fei Wang, Qiang Yang, Wenwu Zhu, and Shiqiang Yang. 2012\.  Social recommendation across multiple relational domains. In _Proceedings of the 21st ACM international conference on Information and knowledge management_. 1422–1431.
  * Jiang et al. (2016) Meng Jiang, Peng Cui, Nicholas J Yuan, Xing Xie, and Shiqiang Yang. 2016.  Little Is Much: Bridging Cross-Platform Behaviors through Overlapped Crowds. In _Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI)_. 13–19.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Variational graph auto-encoders.  _arXiv:1611.07308_ (2016).
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks. In _International Conference on Learning Representations_.
  * Kong et al. (2013) Xiangnan Kong, Jiawei Zhang, and Philip S Yu. 2013\.  Inferring anchor links across multiple heterogeneous social networks. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 179–188.
  * Liu et al. (2020) Siwei Liu, Iadh Ounis, Craig Macdonald, and Zaiqiao Meng. 2020\.  A Heterogeneous Graph Neural Model for Cold-Start Recommendation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2029–2032.
  * Liu et al. (2018) Ziqi Liu, Chaochao Chen, Xinxing Yang, Jun Zhou, Xiaolong Li, and Le Song. 2018\.  Heterogeneous graph neural networks for malicious account detection. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. 2077–2085.
  * Luo et al. (2020) Wenjuan Luo, Han Zhang, Xiaodi Yang, Lin Bo, Xiaoqing Yang, Zang Li, Xiaohu Qie, and Jieping Ye. 2020\.  Dynamic Heterogeneous Graph Neural Network for Real-time Event Prediction. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3213–3223.
  * Nassar and Gleich (2017) Huda Nassar and David F Gleich. 2017.  Multimodal network alignment. In _Proceedings of the 2017 SIAM International Conference on Data Mining_. SIAM, 615–623.
  * Nassar et al. (2018) Huda Nassar, Nate Veldt, Shahin Mohammadi, Ananth Grama, and David F Gleich. 2018.  Low rank spectral network alignment. In _Proceedings of the 2018 World Wide Web Conference_. 619–628.
  * Nickel et al. (2016) Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016\.  Holographic embeddings of knowledge graphs. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Qin et al. (2020) Kyle K Qin, Flora D Salim, Yongli Ren, Wei Shao, Mark Heimann, and Danai Koutra. 2020\.  G-CREWE: Graph CompREssion With Embedding for Network Alignment. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 1255–1264.
  * Ren and Leskovec (2020) Hongyu Ren and Jure Leskovec. 2020.  Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Sadeghian et al. (2019) Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019.  Drum: End-to-end differentiable rule mining on knowledge graphs. In _Advances in Neural Information Processing Systems_. 15347–15357.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _European Semantic Web Conference_. Springer, 593–607.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction. International Conference on Machine Learning (ICML).
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks. In _International Conference on Learning Representations_.
  * Vijayan and Milenković (2017) Vipin Vijayan and Tijana Milenković. 2017.  Multiple network alignment via multiMAGNA++.  _IEEE/ACM transactions on computational biology and bioinformatics_ 15, 5 (2017), 1669–1682.
  * Wang et al. (2020) Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020.  Heterogeneous Graph Neural Networks for Extractive Document Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Wang et al. (2019a) Shen Wang, Zhengzhang Chen, Ding Li, Zhichun Li, Lu-An Tang, Jingchao Ni, Junghwan Rhee, Haifeng Chen, and Philip S Yu. 2019a.  Attentional heterogeneous graph neural network: Application to program reidentification. In _Proceedings of the 2019 SIAM International Conference on Data Mining_. SIAM, 693–701.
  * Wang et al. (2019b) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019b.  Kgat: Knowledge graph attention network for recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 950–958.
  * Wang et al. (2019c) Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019c.  Heterogeneous graph attention network. In _The World Wide Web Conference_. 2022–2032.
  * Wu et al. (2019) Sen Wu, Hongyang R Zhang, and Christopher Ré. 2019.  Understanding and Improving Information Transfer in Multi-Task Learning. In _International Conference on Learning Representations_.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020\.  A comprehensive survey on graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ (2020).
  * Xue et al. (2007) Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. 2007.  Multi-task learning for classification with dirichlet process priors.  _Journal of Machine Learning Research_ 8, Jan (2007), 35–63.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_ (2014).
  * Yang et al. (2020) Xu Yang, Cheng Deng, Tongliang Liu, and Dacheng Tao. 2020\.  Heterogeneous Graph Attention Network for Unsupervised Multiple-Target Domain Adaptation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Yao et al. (2020) Shaowei Yao, Tianming Wang, and Xiaojun Wan. 2020.  Heterogeneous Graph Transformer for Graph-to-Sequence Learning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7145–7154.
  * Ye et al. (2019) Rui Ye, Xin Li, Yujie Fang, Hongyu Zang, and Mingzhong Wang. 2019.  A Vectorized Relational Graph Convolutional Network for Multi-Relational Network Alignment.. In _IJCAI_. 4135–4141.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018\.  Graph convolutional neural networks for web-scale recommender systems. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 974–983.
  * Zhang et al. (2019b) Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019b.  Heterogeneous graph neural network. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 793–803.
  * Zhang et al. (2019c) Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019c.  Quaternion knowledge graph embeddings. In _Advances in Neural Information Processing Systems_. 2735–2745.
  * Zhang et al. (2019d) Si Zhang, Hanghang Tong, Ross Maciejewski, and Tina Eliassi-Rad. 2019d.  Multilevel network alignment. In _The World Wide Web Conference_. 2344–2354.
  * Zhang et al. (2017) Si Zhang, Hanghang Tong, Jie Tang, Jiejun Xu, and Wei Fan. 2017.  ineat: Incomplete network alignment. In _2017 IEEE International Conference on Data Mining (ICDM)_. IEEE, 1189–1194.
  * Zhang et al. (2019a) Wen Zhang, Bibek Paudel, Wei Zhang, Abraham Bernstein, and Huajun Chen. 2019a.  Interaction embeddings for prediction and explanation in knowledge graphs. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 96–104.
  * Zhang et al. (2020) Ziwei Zhang, Peng Cui, and Wenwu Zhu. 2020.  Deep learning on graphs: A survey.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).

##  Appendix A Additional Information for Theoretical Analysis

###  A.1. Choices of the Number of Dimensions $d$

The total capacities (i.e., sum of ranks of adjacency matrices) can be
implemented as
$\sum^{K}_{i=1}\mathrm{argmax}_{p}~{}\lambda_{p}(A^{[i]})\geq\epsilon$, the
sum of the minimum positions of no-smaller-than-$\epsilon$ singular value. To
illustrate the idea, suppose $\\{W^{[i]}_{(2)}\\}$ are shared across GCNs,
which means $W_{(2)}=W^{[1]}_{(2)}=\dots=W^{[K]}_{(2)}$. As long as the shared
$W^{*}_{(2)}$ contains ${{\Theta}_{i}}|^{K}_{i=1}$ in its column span, there
exits $W^{[i]*}_{(1)}$ such that $W^{[i]*}_{(1)}W^{*}_{(2)}={\Theta}_{i}$,
which is optimal for Eq.(21) with minimum error. But this means no transfer
across any GCNs. This can hurt generalization if a GCN has limited data
$\mathcal{G}^{[i]}$, in which its single-GCN solution overfits training data,
whereas the cross-network solution can leverage other graphs’ data to improve
generalization.

Extension to the ReLU/softmax setting. If the capacity of the shared model
parameters (i.e., the number of dimensions $d$) is larger than the total
capacities, then we can share both ${\\{W^{[i]}_{(1)}\\}}|^{K}_{i=1}$ and
${\\{W^{[i]}_{(2)}\\}}|^{K}_{i=1}$. This remains an optimal solution to the
joint training problem in the ReLU/softmax setting. Furthermore, there is _no
transfer_ between any GCNs through the shared model parameters.

###  A.2. Positive Transfer with Network Alignment

Soft regularization: The loss function can be re-written as:

(27) $\displaystyle f$ $\displaystyle=$
$\displaystyle(1-\beta)\cdot\alpha^{[A]}~{}{\|U^{[A]}-\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}\|}^{2}_{\mathrm{F}}$
$\displaystyle+~{}(1-\beta)\cdot\alpha^{[B]}~{}{\|U^{[B]}-\hat{U}^{[B]}\sqrt{\Sigma^{[B]}}\|}^{2}_{\mathrm{F}}$
$\displaystyle+~{}\beta~{}{\|U^{[A]}R^{[A\rightarrow
B]}-A^{[A,B]}U^{[B]}\|}^{2}_{\mathrm{F}}.$

The gradient is

$\displaystyle\frac{\mathrm{d}~{}f}{\mathrm{d}~{}U^{[A]}}=2~{}(1-\beta)\cdot\alpha^{[A]}~{}(U^{[A]}-\hat{U}^{[A]}\sqrt{\Sigma^{[A]}})$
(28) $\displaystyle+2~{}\beta~{}(U^{[A]}R^{[A\rightarrow
B]}-A^{[A,B]}U^{[B]})R^{[A\rightarrow B]~{}\top}.$

Alignment reconstruction: The loss function is:

(29) $\displaystyle f$ $\displaystyle=$
$\displaystyle(1-\beta)\cdot\alpha^{[A]}~{}{\|U^{[A]}-\hat{U}^{[A]}\sqrt{\Sigma^{[A]}}\|}^{2}_{\mathrm{F}}$
$\displaystyle+~{}(1-\beta)\cdot\alpha^{[B]}~{}{\|U^{[B]}-\hat{U}^{[B]}\sqrt{\Sigma^{[B]}}\|}^{2}_{\mathrm{F}}$
$\displaystyle+~{}\beta~{}{\|U^{[A]}R^{[A,B]}U^{[B]\top}-A^{[A,B]}\|}^{2}_{\mathrm{F}}.$

The gradient is

$\displaystyle\frac{\mathrm{d}~{}f}{\mathrm{d}~{}U^{[A]}}=2~{}(1-\beta)\cdot\alpha^{[A]}~{}(U^{[A]}-\hat{U}^{[A]}\sqrt{\Sigma^{[A]}})$
(30)
$\displaystyle+2~{}\beta~{}(U^{[A]}R^{[A,B]}U^{[B]\top}-A^{[A,B]})U^{[B]}R^{[A,B]~{}\top}.$


# Multi-site Diagnostic Classification Of Schizophrenia Using 3D CNN On
Aggregated Task-based fMRI Data

Vigneshwaran S, V Bhaskaran
Sri Sathya Sai Institute of Higher Learning, India
<EMAIL_ADDRESS><EMAIL_ADDRESS>}

###### Abstract

In spite of years of research, the mechanisms that underlie the development of
schizophrenia, as well as its relapse, symptomatology, and treatment, continue
to be a mystery. The absence of appropriate analytic tools to deal with the
variable and complicated nature of schizophrenia may be one of the factors
that contribute to the development of this disorder. Deep learning is a
subfield of artificial intelligence that was inspired by the nervous system.
In recent years, deep learning has made it easier to model and analyse
complicated, high-dimensional, and nonlinear systems. Research on
schizophrenia is one of the many areas of study that has been revolutionised
as a result of the outstanding accuracy that deep learning algorithms have
demonstrated in classification and prediction tasks. Deep learning has the
potential to become a powerful tool for understanding the mechanisms that are
at the root of schizophrenia. In addition, a growing variety of techniques
aimed at improving model interpretability and causal reasoning are
contributing to this trend. Using multi-site fMRI data and a variety of deep
learning approaches, this study seeks to identify different types of
schizophrenia. Our proposed method of temporal aggregation of the 4D fMRI data
outperforms existing work. In addition, this study aims to shed light on the
strength of connections between various brain areas in schizophrenia
individuals.

##  1 Introduction

The application of functional magnetic resonance imaging (fMRI) to research
human brain function and disorders has evolved as a strong neuroimaging
method. It is one of the most crucial methods for determining how brain areas
communicate with one another to complete certain tasks. It offers a potential
method for studying the interaction between geographically distant separate
brain areas that are engaged in a task at the same time.

The blood oxygenated level-dependent (BOLD) signal in fMRI is not random, but
rather temporally coherent between geographically distant functionally
consistent areas. Functional brain networks are formed by functionally
connected areas (Biswal ., 1995; Cordes ., 2000). The identification of
functional activation in fMRI data during task or task-free ”resting state” is
crucial in neuroscience for pre-surgical planning and the diagnosis of
neuropsychiatric diseases such as Alzheimer’s (Jones ., 2012), autism (Starck
., 2013), and schizophrenia (Sakoğlu ., 2010).

Several ways have been tested to diagnose diseases or classify persons as
normal or ill. In general, the low SNR of fMRI data presents difficulties in
using this data to diagnose neuropsychiatric disorders. To the best of our
knowledge, no standard automated fMRI instrument is available in hospitals for
illness diagnosis. This has prompted researchers to investigate the
development of such systems that can assist doctors. Researchers, for example,
have used stationary functional connectivity (FC) as a biomarker to
differentiate patients with neurological and psychiatric diseases such as
Alzheimer’s (Huang ., 2010) or Schizophrenia from normal subjects (Fox
Greicius, 2010) because FC has been discovered to be changed in several
neuropsychological diseases (Jones ., 2012; X. Li ., 2014; Leonardi ., 2013;
Leonardi Van De Ville, 2013).

In this study, we first collected fMRI raw datasets from two publicly
available repositories including 300+ participants and developed a 3D
convolutional network to train on the images by aggregating information over
the timesteps for every subject for the automatic diagnosis of individuals
with schizophrenia. We compare our model with various baseline models and
record the performance.

##  2 Related Work

Machine learning has had a significant impact on schizophrenia diagnosis
studies. Along with MRI and fMRI, the researchers analysed data from genetics,
electroencephalography, and even audio interviews (Cortes-Briones ., 2021). A
combination of fMRI and genomics data from a single location was utilized to
perform canonical correlation analysis using two fully linked, sparse
autoencoders followed by SVM (G. Li ., 2020). There were also several studies
that combined fMRI and structural MRI; the fMRI tasks included resting state,
letter-n back task, auditory oddball, and audiovisual stimuli tasks that
elicited negative and neutral emotion (Salvador ., 2019; Kim ., 2016; Dakka .,
2017; Oh ., 2019; Yan ., 2019). These studies used a variety of strategies to
differentiate between classes, including 3D activation maps, functional
connectomes, ICA decomposition, and independent polygenic risk scores along
with modelling techniques such as artificial neural networks and decision tree
classifiers.

Our research was inspired by a publication in which the authors attempted to
discover consistent patterns and connections between brain areas and also
categorized participants using a variety of machine learning models
(Gheiratmand ., 2017). However, their study was focused on a single site, and
we desired to incorporate data from numerous locations, which is where (Zeng
., 2018) came in handy. The writers of this article combined private and
publicly available datasets. That, we believe, is the only effort in this area
that utilized such a vast sample space for model training. We were unable to
locate many open fMRI datasets, particularly those that included both T1
weighted and resting-state fMRI. Finally, we chose two locations, COBRE and
UCLA.

##  3 Materials and Methods

###  3.1 Participants

The dataset is made up of 120 schizophrenia patients and 206 healthy people
who were gathered from two different imaging resources. Refer to Table 1. The
first subgroup was developed at the University of California, Los Angeles
(UCLA: 50 schizophrenics and 122 healthy controls; patients were allowed to
use stable medications) (Bilder ., 2016). The second group comes from the
Center for Biomedical Research Excellence (COBRE) (COBRE: 69 Schizophrenia
Strict Spectrum patients, 11 Schizoaffective Spectrum patients, and 84 healthy
controls; all of the patients were on antipsychotic medications) (Aine .,
2017).

All individuals were screened and rejected if they had a history of
neurological illness, mental retardation, serious head trauma resulting in a
loss of consciousness lasting more than 5 minutes, or a history of substance
addiction or dependency within the previous 12 months. The Structured Clinical
Interview for DSM-IV Disorders was utilized to obtain diagnostic information
(SCID).

Study | Age(mean ${\pm}$ std) | Age | M | F | Control | Shcz Strict | Schz Affective | Schizophrenia
---|---|---|---|---|---|---|---|---
UCLA | 33.00 ${\pm}$ 9.07 | 21 - 50 | 103 | 69 | 122 | 0 | 0 | 50
COBRE | 38.28 ${\pm}$ 12.60 | 18 - 66 | 125 | 39 | 84 | 69 | 11 | 0
Combined | 33.58 ${\pm}$ 11.24 | 18 - 66 | 228 | 108 | 206 | 69 | 11 | 50
Table 1: Demographics of two sites of data

###  3.2 Image Acquisition

On a Siemens Erlangen 3.0 Tesla Trim Trio scanner, MR images from both
locations were collected. There were, however, modest changes in the methods
used to capture the photos. The COBRE dataset was collected in complete
k-space EPI sequences in a single shot. The time repetitions were set to 2000
milliseconds, and the echo time was set to 29 milliseconds. With a field of
vision of 192 mm and a thickness of 4 mm, there is no gap between the 32
slices. A resting state run was performed on each participant, yielding
150-time steps. An asymmetrical spin-echo echo-planar sequence was used to
collect the UCLA dataset. The time repetitions were set to 2000 milliseconds,
and the echo time was set to 30 milliseconds. With a field of vision of 192 mm
and a thickness of 4 mm, there is no gap between the 34 slices. A resting
state run was performed on each participant, yielding 152-time steps.

###  3.3 Data Preprocessing

We performed an extensive 7-step pipeline to preprocess the resting-state fMRI
data. It has been recorded by research that enhanced preprocessing of the fMRI
data improves the performance of modelling (Churchill ., 2015). First, we
apply slice-time correction to each voxel’s time series using SPM known as
Hanning-Windowed Sinc Interpolation (HWSI). Afterwards, we subtract 7
timesteps from the beginning of each subject to account for magnetic
saturation. Using FSL-MCFLIRT (Jenkinson ., 2002), we were able to correct the
fMRI for motion artefacts induced by low-frequency drifts, which might have a
detrimental influence on the time course decomposition of the data. We
eliminated the skull and neck voxels from the structural T1 weighted image
using the FSL-BET (Smith, 2002) corresponding to each fMRI time series from
the structural T1 weighted image. As a result, we coregistered the structural
image with the functional image, and the process was completed. Following
that, we matched the registered brains to the Montreal Neurological Institute
standard 3mm brain template (MNI152) (Fonov ., 2011). Using a Gaussian kernel
with a full-width half-maximum of 4mm, spatial smoothing was performed to a
time series of data (FWHM). The final step was to downscale the images to the
integer format in order to decrease the amount of disc space consumed.

###  3.4 Functional Connectivity Measure

In order to extract the functional connectomes, we used two brain atlases for
two different purposes. One was to use it for classification purposes whereas
the other one is for understanding brain connectivity to understand which
regions of interest are the biomarkers for schizophrenia. For the purpose of
training the classification model, we leaned towards using the Harvard-Oxford
cortical parcellation (Kennedy ., 2016) sampled at 2mm with a threshold of a
probability of 0.25. For every subject, we initially computed the time series
for each of the ROIs provided by the atlas by standardizing and transforming
the original fMRI space into the atlas space without performing confounds
regression. Following this, we were able to compute the Pearson correlation
coefficient between the voxels over the time step. Finally, we had a 96 X
96-shaped correlation matrix for each of the subjects. The next brain atlas we
used is MSDL (Varoquaux ., 2011) which has 39 regions of interest. We repeated
the same procedures on this atlas as well. Once we computed the functional
connectomes, we used those to visualize the results.

Figure 1: ROI plot of Harvard-Oxford cortical parcellation atlas

###  3.5 Random Forest

Functional connectomes were first flattened to make it into a 1D vector of
length 9216. To find the best hyperparameters, a grid search was performed.
The best parameter turned out to be an entropy-based loss, with a max depth of
6, maximum features of 96 at every node, and 500 estimators. As the dataset is
comparatively small, the classifier overfitted the training set. The
performance and metrics are discussed in the results section.

###  3.6 Vanilla Neural Network

The input to the model is a flattened connectome matrix just like the random
forest input. We tried to keep the model as simple as possible. We started
with 213 neurons and gradually reduced it to 28 neurons before having a final
layer with a single unit. All the hidden layers were activated using ReLU
functions. As the problem is a binary classification, the last layer is a
sigmoid-activated neuron. We played with various dropout strategies before
settling with 30 % between the layers.

###  3.7 Sparse Autoencoder

The paper (Zeng ., 2018) we took inspiration from the multi-site approach and
used sparse autoencoders with discriminant features. We tried to replicate the
work with the two-site data pool to which we had access. We again flattened
the connectomes to be passed as inputs. Our latent representation was of
dimension 1024 which was reduced from 9216. We used 4096 and 2048 units
respectively in two layers between the input and latent representation. We
observed that SeLU worked better than ReLU in retaining information. We added
l1 regularization with a penalty term of 0.001. We used a stochastic gradient
descent optimizer with a learning rate of 0.1 and trained for 30 epochs.

Then we used the encoder to reduce the data of all the training samples and
trained a simple vanilla neural network to classify. This network had 4 layers
of neurons(512, 128, 64, 32) with ReLU activations and a final output layer
with sigmoid activation. We followed the same drop strategy as in the Vanilla
Neural Network model. We used an Adam optimizer as it was generalizing better.

###  3.8 3D CNN

Majority of the neuroimaging research that involves task-based fMRI data for
classification use some method to transform into either functional connectomes
or other forms of dimensionality reduction. Analysing a time series of 3D
vectors is computationally expensive and there are very limited available
methods that try to capture the temporal cohesion. These methods come from
different domains and have their own set of challenges when we try to adapt
them to neuroimaging problems. We still wanted to analyse the images. So
instead of processing the 4D data, we tried to capture the information in a 3D
plane. It is recorded in multiple pieces of research that schizophrenia is
caused due to reduced neuronal activity. So we aggregated the voxel values to
try and record either maximal values or minimal values. The intuition behind
this is that the network might be able to observe reduced patterns in certain
parts of the brain.

Figure 2: A single subject aggregated over the $4^{th}$ axis

We first started by aggregating data over the 4th axis by their maximum values
and minimum values for all the subjects. So we reduced the data size by many
folds and captured the data in a 3D space. Now we had a lot of techniques that
could be applied to 3D data. A lot of work has been performed in the domain of
CT scans. So we adopted a 3D convolutional neural network which was commonly
used to classify lung images with pneumonia. We performed minimal
argumentation by rotating the 3D image with some angle. The random angle was
selected from (-20, -10, -5, 5, 10, 20) every time it was augmented.

Figure 3: Architecture of 3D CNN model

The input is a 61 X 73 X 61 matrix. A series of 3D convolutions was applied
with 64, 128, and 256 filter sizes respectively all with a kernel size of 3
with ReLU activation function before passing it through fully connected
layers. In between the convolutions layers, we performed max pooling with a
size of 2 followed by batch normalization to account for overfitting. See
Figure 3 for a detailed representation.

The performance was better than what was recorded in (Gheiratmand ., 2017).
The best model of that paper, which was a linear support vector machine-scored
74% in terms of accuracy. Even though our maximum aggregation was giving this
performance, the minimum aggregation did not perform well and scored an
accuracy score of 66%.

##  4 Results

###  4.1 Model Performance

The random forest model on the connectome matrice scored an accuracy of 69.5%.
The sensitivity and specificity were 0.86 and 0.38 respectively. There was no
difference in terms of model performance in both the vanilla neural network
and also the sparse autoencoder. Both of them recorded an accuracy of 71%
which is greater than the random forest. The type 1 errors and type2 errors
were also improved in terms of ratio as the sensitivity and specificity were
0.77 and 0.62 respectively. Finally, our proposed methods of aggregated time
series scored an accuracy of 77% which is significantly better than the ones
reported in (Gheiratmand ., 2017). The sensitivity was great as it was 0.91
and the specificity metric was 0.54. Figure 4 has better visualization of the
model performance metrics.

Figure 4: Performance Metrics

###  4.2 Brain Connectivity

As the objective of this study was to also provide explainability that would
result in a better understanding of the disorder, we wanted to take a deeper
look at the interactions between the brain regions between the healthy
subjects and schizophrenia patients. So we started by taking the functional
connectomes of both the groups and averaging them. This way, now it was
possible to have a single connectome that captured the variance of the entire
group. This functional connectome was used to perform a connectivity plot.

Figure 5 was rendered computed using the MSDL atlas that we discussed in the
functional connectivity measure section. At first sight, we could not make out
much of a difference by looking at it. Therefore, this matrix was used to
visualize a brain mask to bring out the connectivity between regions. We also
visualized the connection in 3D space but due to the confines of this 2D
document, it isn’t recorded in this paper. However, the 3D visualizations are
available in the repository.

Figure 5: Brain connectivity of health controls and schizophrenic subjects

##  5 Discussion

According to the findings of this study, a deep learning approach for reliable
and early identification of Schizophrenia illness using brain scans has been
proposed. When it comes to distinguishing schizophrenic individuals from
normal people, the suggested approach uses a modified version of the Sparse
Autoencoder algorithm in conjunction with a 3D Convolutional Neural Network.
This aids in the correct and timely diagnosis of schizophrenia, as well as the
early intervention necessary to combat the disorder. The suggested approach
solves the limits of manual clinical elements as well as the limitations of
existing procedures, which were discussed in the preceding sections.

The suggested approach primarily contributes to improving the accuracy with
which Schizophrenia illness may be detected using brain scans such as
functional magnetic resonance imaging (fMRI). As a result, it may be used to
aid in the diagnosis of Schizophrenia illness using various modalities of
brain imaging, and it can also be used as a CAD for the diagnosis of the
disorder.

###  5.1 Future Scope

The work reported in this study is a first step toward directly exploiting
image data for classification purposes (i.e., converting 4D to 3D space). The
poor results obtained indicate that more investigation is required to fully
comprehend this issue. The next step in our study in this sector is to
explicitly incorporate the many kinds of noise known to corrupt fMRI results,
such as magnetic field inhomogeneity or temporal signal drift. These
parameters can be determined using phantoms. In a second strategy, classifiers
for a single scanning site would be developed, but they would be trained to
utilize priors learned from the other scanning sites as priors instead.
Several organizations are already looking at the application of multitask
learning to achieve this aim, and their first findings indicate that this
would be a good option to follow in the future. Significant advancements in
the field of 3D environment segmentation would benefit the neuroimaging
community significantly since concepts might be borrowed from the field. A
study of graph algorithms on connectome brain graphs may reveal insights to
detect schizophrenia or provide explainability.

###  5.2 Conflicts of interest

The authors report no conflicts of interest.

###  5.3 Supplementary Data

We have made all the code, resources, and side notes required to reproduce the
work publicly available at this GitHub repository:
https://github.com/s-vigneshwaran/Schizophrenia-Classification-from-multi-
site-fMRI-data.

## References

  * Aine . (2017) 21Aine, C., Bockholt, HJ., Bustillo, JR., Cañive, JM., Caprihan, A., Gasparovic, C.others  2017\.  Multimodal neuroimaging in schizophrenia: description and dissemination Multimodal neuroimaging in schizophrenia: description and dissemination.  Neuroinformatics154343–364.
  * Bilder . (2016) 20Bilder, R., Poldrack, R., Cannon, T., London, E., Freimer, N., Congdon, E.Sabb, F.  2016\.  UCLA consortium for neuropsychiatric phenomics LA5c Study Ucla consortium for neuropsychiatric phenomics la5c study.
  * Biswal . (1995) 1Biswal, B., Yetkin, FZ., Haughton, VM. Hyde, JS.  199510\.  Functional connectivity in the motor cortex of resting human brain using echo-planar mri Functional connectivity in the motor cortex of resting human brain using echo-planar mri.  Magnetic Resonance in Medicine344537–541.  https://doi.org/10.1002/mrm.1910340409 10.1002/mrm.1910340409
  * Churchill . (2015) 22Churchill, NW., Spring, R., Afshin-Pour, B., Dong, F. Strother, SC.  2015\.  An automated, adaptive framework for optimizing preprocessing pipelines in task-based functional MRI An automated, adaptive framework for optimizing preprocessing pipelines in task-based functional mri.  PloS one107e0131520.
  * Cordes . (2000) 2Cordes, D., Haughton, VM., Arfanakis, K., Wendt, GJ., Turski, PA., Moritz, CH.Meyerand, ME.  2000Oct.  Mapping functionally related regions of brain with functional connectivity MR imaging Mapping functionally related regions of brain with functional connectivity mr imaging.  AJNR Am J Neuroradiol2191636–1644.
  * Cortes-Briones . (2021) 11Cortes-Briones, JA., Tapia-Rivas, NI., D’Souza, DC. Estevez, PA.  2021\.  Going deep into schizophrenia with artificial intelligence Going deep into schizophrenia with artificial intelligence.  Schizophrenia Research.
  * Dakka . (2017) 15Dakka, J., Bashivan, P., Gheiratmand, M., Rish, I., Jha, S. Greiner, R.  2017\.  Learning neural markers of schizophrenia disorder using recurrent neural networks Learning neural markers of schizophrenia disorder using recurrent neural networks.  arXiv preprint arXiv:1712.00512.
  * Fonov . (2011) 25Fonov, V., Evans, AC., Botteron, K., Almli, CR., McKinstry, RC., Collins, DL.others  2011\.  Unbiased average age-appropriate atlases for pediatric studies Unbiased average age-appropriate atlases for pediatric studies.  Neuroimage541313–327.
  * Fox Greicius (2010) 7Fox, MD. Greicius, M.  2010\.  Clinical applications of resting state functional connectivity Clinical applications of resting state functional connectivity.  Frontiers in systems neuroscience419.
  * Gheiratmand . (2017) 18Gheiratmand, M., Rish, I., Cecchi, GA., Brown, MR., Greiner, R., Polosecki, PI.Dursun, SM.  2017\.  Learning stable and predictive network-based patterns of schizophrenia and its clinical symptoms Learning stable and predictive network-based patterns of schizophrenia and its clinical symptoms.  NPJ schizophrenia311–12.
  * Huang . (2010) 6Huang, S., Li, J., Sun, L., Ye, J., Fleisher, A., Wu, T.others  2010\.  Learning brain connectivity of Alzheimer’s disease by sparse inverse covariance estimation Learning brain connectivity of alzheimer’s disease by sparse inverse covariance estimation.  NeuroImage503935–949.
  * Jenkinson . (2002) 23Jenkinson, M., Bannister, P., Brady, M. Smith, S.  2002\.  Improved optimization for the robust and accurate linear registration and motion correction of brain images Improved optimization for the robust and accurate linear registration and motion correction of brain images.  Neuroimage172825–841.
  * Jones . (2012) 3Jones, DT., Vemuri, P., Murphy, MC., Gunter, JL., Senjem, ML., Machulda, MM.others  2012\.  Non-stationarity in the “resting brain’s” modular architecture Non-stationarity in the “resting brain’s” modular architecture.  PloS one76e39731.
  * Kennedy . (2016) 26Kennedy, D., Haselgrove, C., Fischl, B., Breeze, J., Frazier, J., Seidman, L.Kosofsky, B.  2016\.  Harvard-Oxford cortical and subcortical structural atlases Harvard-oxford cortical and subcortical structural atlases.  Harvard Center for Morphometric Analysis.
  * Kim . (2016) 14Kim, J., Calhoun, VD., Shim, E. Lee, JH.  2016\.  Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia.  Neuroimage124127–146.
  * Leonardi . (2013) 9Leonardi, N., Richiardi, J., Gschwind, M., Simioni, S., Annoni, JM., Schluep, M.Van De Ville, D.  2013\.  Principal components of functional connectivity: a new approach to study dynamic brain connectivity during rest Principal components of functional connectivity: a new approach to study dynamic brain connectivity during rest.  NeuroImage83937–950.
  * Leonardi Van De Ville (2013) 10Leonardi, N. Van De Ville, D.  2013\.  Identifying network correlates of brain states using tensor decompositions of whole-brain dynamic functional connectivity Identifying network correlates of brain states using tensor decompositions of whole-brain dynamic functional connectivity.  2013 International Workshop on Pattern Recognition in Neuroimaging 2013 international workshop on pattern recognition in neuroimaging ( 74–77).
  * G. Li . (2020) 12Li, G., Han, D., Wang, C., Hu, W., Calhoun, VD. Wang, YP.  2020\.  Application of deep canonically correlated sparse autoencoder for the classification of schizophrenia Application of deep canonically correlated sparse autoencoder for the classification of schizophrenia.  Computer methods and programs in biomedicine183105073.
  * X. Li . (2014) 8Li, X., Zhu, D., Jiang, X., Jin, C., Zhang, X., Guo, L.Liu, T.  2014\.  Dynamic functional connectomics signatures for characterization and differentiation of PTSD patients Dynamic functional connectomics signatures for characterization and differentiation of ptsd patients.  Human brain mapping3541761–1778.
  * Oh . (2019) 16Oh, K., Kim, W., Shen, G., Piao, Y., Kang, NI., Oh, IS. Chung, YC.  2019\.  Classification of schizophrenia and normal controls using 3D convolutional neural network and outcome visualization Classification of schizophrenia and normal controls using 3d convolutional neural network and outcome visualization.  Schizophrenia research212186–195.
  * Sakoğlu . (2010) 5Sakoğlu, Ü., Pearlson, GD., Kiehl, KA., Wang, YM., Michael, AM. Calhoun, VD.  2010\.  A method for evaluating dynamic functional network connectivity and task-modulation: application to schizophrenia A method for evaluating dynamic functional network connectivity and task-modulation: application to schizophrenia.  Magnetic Resonance Materials in Physics, Biology and Medicine235351–366.
  * Salvador . (2019) 13Salvador, R., Canales-Rodríguez, E., Guerrero-Pedraza, A., Sarró, S., Tordesillas-Gutiérrez, D., Maristany, T.Pomarol-Clotet, E.  2019\.  Multimodal integration of brain images for MRI-based diagnosis in schizophrenia Multimodal integration of brain images for mri-based diagnosis in schizophrenia.  Frontiers in neuroscience1203.
  * Smith (2002) 24Smith, SM.  2002\.  Fast robust automated brain extraction Fast robust automated brain extraction.  Human brain mapping173143–155.
  * Starck . (2013) 4Starck, T., Nikkinen, J., Rahko, J., Remes, J., Hurtig, T., Haapsamo, H.others  2013\.  Resting state fMRI reveals a default mode dissociation between retrosplenial and medial prefrontal subnetworks in ASD despite motion scrubbing Resting state fmri reveals a default mode dissociation between retrosplenial and medial prefrontal subnetworks in asd despite motion scrubbing.  Frontiers in human neuroscience7802.
  * Varoquaux . (2011) 27Varoquaux, G., Gramfort, A., Pedregosa, F., Michel, V. Thirion, B.  2011\.  Multi-subject dictionary learning to segment an atlas of brain spontaneous activity Multi-subject dictionary learning to segment an atlas of brain spontaneous activity.  Biennial International Conference on information processing in medical imaging Biennial international conference on information processing in medical imaging ( 562–573).
  * Yan . (2019) 17Yan, W., Calhoun, V., Song, M., Cui, Y., Yan, H., Liu, S.others  2019\.  Discriminating schizophrenia using recurrent neural network applied on time courses of multi-site FMRI data Discriminating schizophrenia using recurrent neural network applied on time courses of multi-site fmri data.  EBioMedicine47543–552.
  * Zeng . (2018) 19Zeng, LL., Wang, H., Hu, P., Yang, B., Pu, W., Shen, H.others  2018\.  Multi-site diagnostic classification of schizophrenia using discriminant deep learning with functional connectivity MRI Multi-site diagnostic classification of schizophrenia using discriminant deep learning with functional connectivity mri.  EBioMedicine3074–85.