# DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback
Alignment

Gongpei Zhao, Tao Wang, Congyan Lang, Yi Jin, Yidong Li
School of Computer Science and Technology
Beijing Jiaotong University
Beijing
{csgpzhao<EMAIL_ADDRESS>
&Haibin Ling
Department of Computer Science
Stony Brook University
Stony Brook
<EMAIL_ADDRESS>

###### Abstract

Graph neural networks (GNNs) are recognized for their strong performance
across various applications, with the backpropagation (BP) algorithm playing a
central role in the development of most GNN models. However, despite its
effectiveness, BP has limitations that challenge its biological plausibility
and affect the efficiency, scalability and parallelism of training neural
networks for graph-based tasks. While several non-backpropagation (non-BP)
training algorithms, such as the direct feedback alignment (DFA), have been
successfully applied to fully-connected and convolutional network components
for handling Euclidean data, directly adapting these non-BP frameworks to
manage non-Euclidean graph data in GNN models presents significant challenges.
These challenges primarily arise from the violation of the independent and
identically distributed (i.i.d.) assumption in graph data and the difficulty
in accessing prediction errors for all samples (nodes) within the graph. To
overcome these obstacles, in this paper we propose DFA-GNN, a novel forward
learning framework tailored for GNNs with a case study of semi-supervised
learning. The proposed method breaks the limitations of BP by using a
dedicated forward training mechanism. Specifically, DFA-GNN extends the
principles of DFA to adapt to graph data and unique architecture of GNNs,
which incorporates the information of graph topology into the feedback links
to accommodate the non-Euclidean characteristics of graph data. Additionally,
for semi-supervised graph learning tasks, we developed a pseudo error
generator that spreads residual errors from training data to create a pseudo
error for each unlabeled node. These pseudo errors are then utilized to train
GNNs using DFA. Extensive experiments on 10 public benchmarks reveal that our
learning framework outperforms not only previous non-BP methods but also the
standard BP methods, and it exhibits excellent robustness against various
types of noise and attacks.

##  1 Introduction

As a class of neural networks (NNs) specifically designed to process and learn
from graph data, graph neural networks (GNNs) [1, 2] have gained significant
popularity in addressing graph analytical challenges. They have demonstrated
outstanding success in various applications, including recommendation systems
[3], drug discovery [4] and question answering [5]. The impressive
accomplishments of GNNs, as well as other neural network models, are largely
attributed to the backpropagation (BP) algorithm [6], which has emerged as the
standard technique for training deep neural networks.

The backpropagation algorithm adjusts neural network weights based on the loss
between the prediction and the ground truth, and allows the network to learn
and improve over time. However, despite its effectiveness, BP draws concerns
on its biological plausibility for two main reasons [7, 8]: (1) it uses the
same weights in reverse order for both feedforward and feedback paths,
creating the weight symmetry problem [8]; and (2) its parameter updating
relies on the activity of all downstream layers, leading to the update locking
problem [9]. These limitations may as well impact the efficiency, scalability
and parallel processing capabilities of neural network training.

To address these limitations, direct feedback alignment (DFA) [10] offers an
effective alternative to BP by training neural networks through a single
forward pass. DFA uses fixed random feedback connections to project output
errors directly onto hidden neurons, allowing for parallel gradient
computation and eliminating the need for sequential backward error
propagation. While demonstrated a limited accuracy penalty compared with BP,
DFA aligns with brain-like learning mechanisms through its use of global error
modulation and local synaptic activity, making it a notable non-BP method
applicable in areas such as image classification [11] and privacy protection
[12].

Directly applying DFA to GNNs, however, faces two challenges: (1) graph data
often violates the independent and identically distributed (i.i.d.) assumption
and thus ties the supervision gradients with the graph structure, making the
straightforward error projection of DFA inadequate; and (2) DFA requires the
prediction errors of all the input samples, while for graph data, especially
under the semi-supervised setting, samples (nodes) without ground truth meet
problems for the error calculation, complicating the deployment of DFA to
GNNs.

To tackle these challenges, in this paper we propose DFA-GNN, a non-BP
learning framework tailored for graph neural networks. Our primary
contribution is to improve and extend DFA to graph neural networks.
Specifically, we redesign the random feedback strategy for graph data to make
the DFA portable to GNNs. The information from the graph topology, in the form
of an adjacency matrix, is incorporated into the feedback links to accommodate
the non-Euclidean characteristics of graph data. We take graph convolutional
network (GCN) [13] as a case study, and derive the specific formula for
updating parameters in each GCN layer. Furthermore, for the semi-supervised
graph learning task, we develop a novel pseudo error generator that spreads
residual errors from training data to generate a pseudo error for each
unlabeled node. Such pseudo errors are then used for the training of graph
neural networks by DFA.

Figure 1: Illustrations of BP, FF, FORWARDGNN and proposed DFA-GNN.

In summary, our proposed learning procedure for GNNs contributes in three
significant folds:

  * •

We introduce DFA-GNN, a non-BP training algorithm that extends DFA to GNN
architectures. It offers a more biologically plausible alternative to
traditional BP methods.

  * •

For semi-supervised graph learning tasks, we develop a novel pseudo error
generator that propagates residual errors from the training data to create
pseudo errors for unlabeled nodes.

  * •

We prove the convergence of our DFA-GNN, and validate its effectiveness on 10
benchmarks. The experimental results demonstrate the superiority of our DFA-
GNN against both traditional BP and the state-of-the-art non-BP approaches.

##  2 Related Work

The biological implausibility of BP mainly lies in weight transport and update
locking issues. feedback alignment (FA) [8] addresses the weight transport by
using fixed random weights to convey error gradients. Building on this, direct
feedback alignment (DFA) [10], direct random target projection (DRTP) [14] and
PEPITA [9] further tackle the update locking problem with non-BP update
methods. Prompted by the recent critiques of [7], the forward-forward (FF)
algorithm emerges as a more neurophysiologically aligned alternative, using
dual forward passes with positive and negative data to simplify the training
process and accommodate non-differentiable elements. The recently proposed
cascaded forward algorithm (CaFo) [11] attaches a class predictor to each
layer, where only the layer-wise predictors are locally trained, with each
neural block being randomly initialized and remaining fixed throughout.

Our work aims to push the frontier of the non-BP training algorithm for GNNs,
which is a field still in its infancy. A remarkable recent work along the line
is FORWARDGNN [15] inspired by the forward-forward algorithm. FORWARDGNN
avoids the constraints imposed by BP via an effective layer-wise local forward
training. It trains GNNs using a single forward pass with the assistant of a
data augmentation strategy. The augmented graph structure integrates virtual
nodes linked only to labeled nodes, leaving the local topology of unlabeled
nodes unchanged. The augmentation strategy makes it possible to operate
without generating negative inputs. Despite of its advantages, FORWARDGNN
still suffers from the greed-based training strategy, and thus results in
inferiority in prediction performance in comparison with traditional BP
algorithm.

In contrast, our method does not necessitate data augmentation for graph data;
instead, it directly utilizes the discrepancy between predictions and actual
ground truth to update each layer. Our approach directly outputs the
prediction of the multi-class distribution, eliminating the need to calculate
the goodness between unlabeled nodes and virtual nodes. As a result, our
method offers convenience for multi-class prediction tasks and gains
improvements in prediction performance.

##  3 Preliminaries

###  3.1 Problem Definition

An attributed relational graph of $n$ nodes can be represented by
$G=(\mathcal{V},\mathcal{E},\textbf{X})$, where
$\mathcal{V}=\\{v_{1},v_{2},\cdots,v_{n}\\}$ represents the set of n nodes,
and $\mathcal{E}=\\{e_{ij}\\}$ signifies the set of edges.
$\textbf{X}=\\{\textbf{x}_{1}^{\text{T}};\textbf{x}_{2}^{\text{T}};\cdots;\textbf{x}_{n}^{\text{T}}\\}\in\mathbb{R}^{n\times
d}$ is the attribute set for all nodes, with $\textbf{x}_{i}$ being the
$d$-dimensional attribute vector for node $v_{i}$. The adjacency matrix
$\textbf{A}=\\{a_{ij}\\}\in\mathbb{R}^{n\times n}$ denotes the topological
structure of graph G, where $a_{ij}>0$ if there exists an edge $e_{ij}$
between nodes $v_{i}$ and $v_{j}$ and $a_{ij}=0$ otherwise.

For semi-supervised node classification, the node set $\mathcal{V}$ can be
split into a labeled node set $\mathcal{V}_{L}\subset\mathcal{V}$ with
attributes $\textbf{X}_{L}\subset\textbf{X}$ and an unlabeled one
$\mathcal{V}_{U}=\mathcal{V}/\mathcal{V}_{L}$ with attributes
$\textbf{X}_{U}=\textbf{X}/\textbf{X}_{L}$.111For notation conciseness, we
abuse the set notation and matrix notation interchangeably whenever
appropriate. For example, X represents both a set of $n$ attributes and a
matrix. We assume that each node belongs to exactly one class, and denote
$\textbf{y}_{L}=\\{y_{i}\\}$ as the ground-truth labels of node set
$\mathcal{V}_{L}$ where $y_{i}$ denotes the class label of node
$v_{i}\in\mathcal{V}_{L}$.

The objective of semi-supervised node classification is to train a classifier
using the graph and the known labels $\textbf{y}_{L}$, and then apply this
classifier to predict the labels for the unlabeled nodes $\textbf{v}_{U}$.
Define a classifier
$f_{\theta}:(\tilde{\textbf{y}}_{L},\tilde{\textbf{y}}_{U})\leftarrow
f_{\theta}(\textbf{X},\textbf{A},\textbf{y}_{L})$, where $\theta$ is the
parameters of model. $\tilde{\textbf{y}}_{L}$ and $\tilde{\textbf{y}}_{U}$ are
the predicted labels of nodes $\textbf{v}_{L}$ and $\textbf{v}_{U}$
respectively. Generally, the goal is to make the predicted labels
$\tilde{\textbf{y}}_{L}$ align as closely as possible with the actual ground-
truth labels $\textbf{y}_{L}$ in favor
of:$\theta^{*}=\arg\min_{\theta}d(\tilde{\textbf{y}}_{L},\textbf{y}_{L})=\arg\min_{\theta}d(f_{\theta}(\textbf{X},\textbf{A},\textbf{y}_{L}),\textbf{y}_{L})$,
where $d(\cdot,\cdot)$ represents a measure of some type of distance between
two sets of labels.

###  3.2 Direct Feedback Alignment

While BP relies on symmetric weights for error propagation to hidden layers,
there is evidence suggesting that symmetrical weight distribution may not be
crucial for learning. For example, the study of feedback alignment (FA) shows
that learning can still occur when errors are back propagated using randomly
fixed weights. Direct feedback alignment (DFA) advances in this direction by
directly transmitting output errors to each hidden layer through fixed linear
feedback links. Specifically, for an $L$ layer network, feedback matrices
$\textbf{B}^{(l)}\in\mathbb{R}^{n_{L}\times n_{l}}$ are employed to replace
the derivatives $\partial\textbf{x}^{(L)}/\partial\textbf{x}^{(l)}$ of output
neurons with respect to hidden neurons in the $l$-th layer. The approximate
gradient $\delta\textbf{W}^{(l)}$ for the weights of the $l$-th hidden layer
is then computed as:
$\delta\textbf{W}^{(l)}=\frac{\partial\mathcal{L}}{\partial\textbf{x}^{(L)}}\textbf{B}^{(l+1)}\frac{\partial\textbf{x}^{(l+1)}}{\partial\textbf{W}^{(l)}}$,
where $\textbf{x}^{(l)}$ represents the latent representation of a sample at
the $l$-th layer, and $\mathcal{L}$ the loss value. In DFA, the feedback
matrices assigned to hidden layers are randomly selected and remain unchanged
throughout the training process. The effectiveness of DFA hinges on the
alignment between forward weights and feedback matrices, leading to a
congruence between the estimated and the actual gradient. When the angle
between these gradients remains below 90 degrees, the update direction points
to a downward trajectory. DFA has been successfully implemented in popular
deep learning architectures such as fully-connected neural network (FC) [16]
and convolutional neural network (CNN) [17]. However, extending DFA to graph
neural networks remains unexplored.

##  4 Proposed DFA-GNN

As depicted in Fig. 1, our DFA-GNN aims to train GNN models in non-BP
framework by extending the DFA algorithm. Different from the original DFA
algorithm designed on FC for Euclidean data, two key issues should be solved
when extending it to GNN for graph data: (1) the original random feedback
operations need to be reformulated to handle the dependence between samples
(nodes); and (2) high-quality pseudo errors for test samples are required as
they are not isolated from the training procedure. To this end, in Sec. 4.1 we
redesign the random feedback strategy specified for graph data, and in Sec.
4.2 we develop a novel pseudo error generator for semi-supervised graph
learning tasks. Finally, we provide deep insight of our DFA-GCN about its
convergence and optimization in Sec. 4.3.

###  4.1 Generalizing DFA to GNN

The training process of traditional BP for GNN is listed in Algo. 1 of Appx.
A.1, and it uses both a forward propagation and a backward one in each epoch.
Generally, different GNN models may differ in the operations of aggregation
and combination, and we provide a typical implementation in Algo. 1. We take
graph convolutional network (GCN) [13], one of the most classic and successful
GNN models, as a case study to integrate DFA for graph learning.

For illustrative purpose, we consider a three-layer GCN model with ReLU for
hidden activation and sigmoid for output activation. The forward propagation
process could be written as:

$\displaystyle\begin{aligned}
\mathbf{Layer~{}1}:\textbf{H}^{(0)}&=\textbf{SX}^{(0)},\quad\textbf{X}^{(1)}=\mathrm{relu}(\textbf{H}^{(0)}\textbf{W}^{(0)}),\\\
\mathbf{Layer~{}2}:\textbf{H}^{(1)}&=\textbf{SX}^{(1)},\quad\textbf{X}^{(2)}=\mathrm{relu}(\textbf{H}^{(1)}\textbf{W}^{(1)}),\\\
\mathbf{Output~{}layer}:\textbf{H}^{(2)}&=\textbf{SX}^{(2)},\quad\textbf{X}^{(3)}=\textbf{H}^{(2)}\textbf{W}^{(2)},\quad\tilde{\textbf{Y}}=\mathrm{sigmoid}(\textbf{X}^{(3)}),\
\ \end{aligned}$ (1)

where
$\textbf{S}=\tilde{\textbf{D}}^{-\frac{1}{2}}\tilde{\textbf{A}}\tilde{\textbf{D}}^{-\frac{1}{2}}$,
$\tilde{\textbf{A}}=\textbf{A}+\textbf{I}$ is the adjacency matrix of graph
$G$ after adding self loop.
$\tilde{\textbf{D}}=\mathrm{diag}(\tilde{\textbf{A}})$ the diagonal matrix of
$\tilde{\textbf{A}}$, $\textbf{W}^{(l-1)}$ a trainable weight matrix of the
$l$-th layer and $\sigma$ a non-linear activation function.
$\textbf{X}^{(l)}\in\mathbb{R}^{n\times d}$ denotes the latent representation
matrix of the $l$-th layer and $\textbf{X}^{(0)}=\textbf{X}$. If we choose
sigmoid activation function in the output layer and a binary cross-entropy
(BCE) loss function, the loss $J$ for a graph with $n$ nodes and the gradient
E at the output layer are calculated as:

$\displaystyle J$ $\displaystyle=$
$\displaystyle-\frac{1}{N}\sum_{m,k}\textbf{Y}_{m,k}\log\tilde{\textbf{Y}}_{m,k}+(1-\textbf{Y}_{m,k})\log(1-\tilde{\textbf{Y}}_{m,k}),$
(2) E $\displaystyle=$ $\displaystyle\delta\textbf{X}^{(3)}=\frac{\partial
J}{\partial\textbf{X}^{(3)}}=\tilde{\textbf{Y}}-\textbf{Y},$ (3)

where $\tilde{\textbf{Y}}$ and $\textbf{Y}\in\mathbb{R}^{n\times c}$ are
respectively prediction and one-hot ground truth; $m$ and $k$ are respectively
sample index and output unit. It is important to highlight that E represents
the exact error between the prediction and the ground truth. For GCN, the
gradients for hidden layers are calculated according to Algo. 1 as:
$\delta\textbf{X}^{(2)}=\textbf{S}^{\text{T}}\delta\textbf{H}^{(2)}=\textbf{S}^{\text{T}}\textbf{E}\textbf{W}^{(2)\text{T}},\delta\textbf{X}^{(1)}=\textbf{S}^{\text{T}}\delta\textbf{H}^{(1)}=\textbf{S}^{\text{T}}\delta\textbf{X}^{(2)}\textbf{W}^{(1)\text{T}}$.

As demonstrated by the work of FF [8] and DFA [10], learning can be effective
when errors are back propagated using randomly fixed weights. Similarly, we
establish parallel direct feedback links for each layer. We approximate the
update directions for the hidden layers as follows:

$\displaystyle\delta\textbf{X}^{(2)}=\textbf{S}^{\text{T}}\textbf{E}\textbf{B}^{(2)},\quad\delta\textbf{X}^{(1)}=\textbf{S}^{\text{T}}\delta\textbf{H}^{(1)}=\textbf{S}^{\text{T}}\delta\textbf{X}^{(2)}\textbf{B}^{(1)},$
(4)

where $\textbf{B}^{(i)}$ is a fixed random weight matrix with appropriate
dimension. $\delta\textbf{X}^{(1)}$ can be then written as:

$\displaystyle\delta\textbf{X}^{(1)}=\textbf{S}^{\text{T}}\delta\textbf{X}^{(2)}\textbf{B}^{(1)}=\textbf{S}^{\text{T}}(\textbf{S}^{\text{T}}\textbf{E}\textbf{B}^{(2)})\textbf{B}^{(1)}=(\textbf{S}^{\text{T}})^{2}\textbf{E}\textbf{B}^{(1)},$
(5)

and the weight updates for all layers are calculated as:

$\displaystyle\delta\textbf{W}^{(0)}=\textbf{H}^{(0)\text{T}}\delta\textbf{X}^{(1)},\quad\delta\textbf{W}^{(1)}=\textbf{H}^{(1)\text{T}}\delta\textbf{X}^{(2)},\quad\delta\textbf{W}^{(2)}=\textbf{H}^{(2)\text{T}}\textbf{E}.$
(6)

The above derivations can be easily extended to GCNs with more layers, as done
in our experiments.

###  4.2 Pseudo Error Generator by Spreading Residual Errors

In the computation of gradient (Eq. 6), the labels of all nodes are required,
but not all nodes are labeled in the semi-supervised learning task. To address
this issue, we introduce a mechanism to generate errors, assigning a pseudo
error to each unlabeled node. Inspired by the ingenious work of [18] which
makes correction for the prediction of simple base predictor, we adopt the
simlilar approach for the generation of pseudo error for each unlabeled node.
The underlying principle is the expectation that errors in initial predictions
are likely to propagate along the graph edges. That is, an error at a given
node $v$ suggests a higher likelihood of similar errors to its the neighbor
nodes. This concept of error propagation across the graph is supported by
previous studies [19]. Our approach draws inspiration from the strategy of
residual propagation used in node regression tasks, and more broadly, from the
frameworks of generalized least squares and correlated error models [20].

In semi-supervised graph learning, the error matrix
$\textbf{E}=\\{\textbf{e}_{1}^{\text{T}};\textbf{e}_{2}^{\text{T}};\cdots;\textbf{e}_{n}^{\text{T}}\\}\in\mathbb{R}^{n\times
c}$ as described in Eq. 3 is modified as the residual on the training nodes,
while being set to zero for all other nodes. This adjustment entails
initializing $\textbf{e}_{i}$ as a zero vector for all nodes
$v_{i}\in\mathcal{V}_{U}$.

The residuals in the rows of E for the training nodes are zero only when the
forward process achieves perfect predictions. We utilize the label spreading
technique [21] to smooth the error with the goal of optimizing the following
objective:

$\displaystyle\textbf{Z}^{*}=\arg\min_{\textbf{Z}\in\mathbb{R}^{n\times
c}}tr(\textbf{Z}^{\text{T}}(\textbf{I}-\textbf{S})\textbf{W})+\mu\|\textbf{Z}-\textbf{E}\|_{F}^{2},$
(7)

where $tr(\cdot)$ denotes the trace of a matrix. The first term enhances the
smoothness of the error estimation throughout the graph, while the second term
ensures that the final solution stays consistent with the initial error
estimate E. Following the optimization methodology in [21], the solution to
Eq. 7 can be obtained through iterative processing

$\displaystyle\textbf{Z}^{(t+1)}=(1-\alpha)\textbf{E}+\alpha\textbf{S}\textbf{Z}^{(t)},\quad\alpha=\frac{1}{1+\mu},\quad\textbf{Z}^{(0)}=\textbf{E}.$
(8)

This process represents the diffusion of error, and such propagation is
demonstrably appropriate within the context of regression problems under a
Gaussian assumption. Nevertheless, for classification tasks like ours, the
smoothed error $\textbf{Z}^{*}$ might not align with the correct scale.
Typically,
$\|\textbf{Z}^{(t+1)}\|_{2}\leq(1-\alpha)\|\textbf{E}\|_{2}+\alpha\|\textbf{S}\|_{2}\|\textbf{E}^{(t)}\|_{2}=(1-\alpha)\|\textbf{E}\|_{2}+\alpha\|\textbf{Z}^{(t)}\|_{2}$.
Starting with $\textbf{Z}^{(0)}=\textbf{E}$, we find that
$\|\textbf{Z}^{(t)}\|_{2}\leq\|\textbf{E}\|_{2}$, indicating a need to adjust
the scale of residuals adaptively. The aim is to match the magnitude of error
in $\textbf{Z}^{*}$ to that in E as closely as possible. Given that we only
have accurate error information for labeled nodes, we use the average error
across these nodes to estimate the appropriate scale. Specifically, with
$\textbf{e}_{i},\textbf{z}^{*}_{i}\in\mathbb{R}^{c}$ representing the $i$-th
row of E and $\textbf{Z}^{*}$ respectively, the adjusted error for an
unlabeled node $j$ is calculated as:
$\hat{\textbf{e}}_{j}=\eta/\|\textbf{z}^{*}_{j}\|_{1}\cdot\textbf{z}^{*}_{j}$,
in which
$\eta=\frac{1}{|\mathcal{V}_{L}|}\sum_{v_{i}\in\mathcal{V}_{L}}\|\textbf{e}_{i}\|_{1}$.

Give the rescaled pseudo error $\hat{\textbf{e}}$ for each unlabeled node, we
define $\hat{\textbf{E}}\in\mathbb{R}^{n\times c}$, where the $i$-th row is
set to $\textbf{e}_{i}^{\text{T}}$ for nodes $v_{i}\in\mathcal{V}_{L}$ and to
$\hat{\textbf{e}}_{i}^{\text{T}}$ for other nodes. The matrix
$\hat{\textbf{E}}$ can then be directly utilized in Eqs. 5 and 6 for GCN
training. Nevertheless, not all rescaled error of unlabeled nodes are accurate
and useful. To address this, a mask is implemented to filter out these nodes.
Define $\hat{\textbf{Y}}=\tilde{\textbf{Y}}-\hat{\textbf{E}}$ as the corrected
prediction. For labeled nodes, this corrected prediction equals to the one-hot
ground truth. We introduce a mask vector $\textbf{p}\in\mathbb{R}^{n}$ to
facilitate this process. Formally, the setup is as follows:

$\displaystyle p_{i}=\begin{cases}1,&\text{ if
$\mathrm{count}(\hat{\textbf{y}}_{i}>\epsilon)=1,$ }\\\ 0,&\text{ otherwise,
}\end{cases}$ (9)

where $p_{i}$ is the $i$-th element ($1\leq i\leq n$) of the vector p,
$\epsilon$ a manually set threshold for controlling filtering and
$\mathrm{count}(\cdot)$ a counting function. Eq. 9 focuses on retaining only
those well predicted nodes, characterized by a single category being
identified as positive and the rest as negative. With the mask, the filtering
operations could be performed on $\hat{\textbf{E}}$ and S by row through the
mask. The weight updates in Eq. 6 are modified as:

$\displaystyle\delta\textbf{W}^{(0)}=\textbf{H}^{(0)\text{T}}\textbf{S}_{\textit{f}}^{(2)\text{T}}\hat{\textbf{E}}_{\textit{f}}\textbf{B}^{(1)},\quad\delta\textbf{W}^{(1)}=\textbf{H}^{(1)\text{T}}\textbf{S}_{\textit{f}}^{(1)\text{T}}\hat{\textbf{E}}_{\textit{f}}\textbf{B}^{(2)},\quad\delta\textbf{W}^{(2)}=\textbf{H}_{\textit{f}}^{(2)\text{T}}\hat{\textbf{E}}_{\textit{f}},$
(10)

where $\textbf{S}_{\textit{f}}^{(k)}$, $\hat{\textbf{E}}_{\textit{f}}$,
$\textbf{H}_{\textit{f}}^{(k)}$ represent the row filtering of
$\textbf{S}^{k}$, $\hat{\textbf{E}}$ and $\textbf{H}^{(k)}$ respectively,
according to the mask p. As $(\textbf{S}^{\text{T}})^{k}\hat{\textbf{E}}$
exactly denotes the accumulation of errors related to the $k$-hop neighbors of
each node, the difference between Eq. 6 and Eq. 10 lies in the partial
sampling of neighbors to approximate the update directions, as compared with
using all neighbors. Given that only poorly predicted nodes are excluded and
they constitute a small fraction, the update directions in both approaches
generally remain aligned. The formal description of our algorithm is provided
in Appx. A.2.

###  4.3 Insights of DFA-GNN

DFA-GNN provides a non-BP training procedure for GNN. For GCN we provide a
detailed analysis on how such an asymmetric feedback path introduced in Sec.
4.1 can provide learning by aligning the gradients of backward propagation and
forward propagation with its own. [10] has originally proved the conclusion
for fully-connected layer architectures. We can show that this conclusion is
equally valid for the GCN architecture, and provide the detailed proof in
Appx. A.3.

Experiments in Fig. 2(a) validate the dynamic process of alignment between B
and W during training, and the weight alignment leads to gradient alignment
because for weight alignment of DFA:

$\displaystyle\textbf{W}^{(0<l<L)}\propto\textbf{B}^{(l)\text{T}}\textbf{B}^{(l+1)},\
\ \textbf{W}^{(L)}\propto\textbf{B}^{(L)\text{T}},$ (11)

where the symbol $\propto$ represents a positive scalar multiple relationship.
As gradient alignment requires
$\delta\textbf{X}^{(l)}_{\text{DFA}}\propto\delta\textbf{X}^{(l)}_{\text{BP}}$,
i.e.,
$(\textbf{S}^{\text{T}})^{L-l}\textbf{E}\textbf{B}^{(l)}\propto\textbf{S}^{\text{T}}\delta\textbf{X}^{(l+1)}\textbf{W}^{(l)\text{T}}$,
the weight alignment directly implies gradient alignment if the feedback
matrices are assumed right-orthogonal, i.e.,
$\textbf{B}\textbf{B}^{\text{T}}=\textbf{I}$. This assumption holds if the
feedback matrices elements are sampled i.i.d. from a Gaussian distribution
since $\mathbb{E}[\textbf{B}\textbf{B}^{\text{T}}]\propto\textbf{I}$, hence
Eq. 11 induces the weights, by the orthogonality condition, to cancel out by
pairs of two:

$\displaystyle\textbf{S}^{\text{T}}\delta\textbf{X}^{(l+1)}\textbf{W}^{(l)\text{T}}\propto(\textbf{S}^{\text{T}})^{L-l}\textbf{E}\textbf{B}^{(L)}\textbf{B}^{(L)\text{T}}\cdots\textbf{B}^{(l+1)\text{T}}\textbf{B}^{(l)}=(\textbf{S}^{\text{T}})^{L-l}\textbf{E}\textbf{B}^{(l)}.$
(12)

The alignments of weights and gradients make our method trained with DFA tend
to converge to a specific region within the landscape, guided by the structure
of the feedback matrices, while the optimization paths trained with BP
according to stochastic gradient descent often exhibit divergent within the
loss landscape, as shown in Fig. 2 (d).

For deeper understanding of the training mechanism in our method, we divide
the entire training process into three stages: Stage 1 (train layers 1 and 2
while freezing layer 3), Stage 2 (freeze layers 1 and 2 while training layer
3) and Stage 3 (train layers 1 and 2 while freezing layer 3). The results in
Figs. 2(b,c) show a strong correlation between weight alignment and the
fitting degree of the model, as indicated by the loss. Notably, even though
our method updates parameters of each layer in parallel, the effective update
follows a backward-to-forward manner. As shown in Fig. 2 (b), when the
parameters of layer $l$ are not effectively learned, updating the preceding
layers does not enhance fitting ability of the model. This behavior contrasts
with the characteristics of traditional BP, which indicates that the alignment
of weights and gradients also adhere to a backward-to-forward sequence.

Figure 2: For a three-layer GCN model trained by DFA-GNN on Cora, (a) the
accuracy and angle between W and B; (b) the change in loss across different
stages; (c) the change in angle between W and B across different stages; (d)
difference in optimization direction between BP and our method.

##  5 Experiments

###  5.1 Comparison with Baseline Training Algorithms

We evaluate our method on 10 benchmark datasets across various domains and
compare it with the BP [22], PEPITA [9], two versions of the FF (abbr. FF+LA,
FF+VN) [7, 15], two versions of the CaFo (abbr. CaFo+MSE, CaFo+CE) [11] and
the FORWARDGNN-SF (abbr. SF) [15]. Detailed datasets and experimental setup
information can be found in Appx. A.4 and Appx. A.5, respectively. The
comparative analysis of various algorithms on benchmark datasets is summarized
in Tab. 1. While the non-BP methods such as PEPITA, CaFo and FF have proven
effective for architectures involving fully-connected and convolutional layers
with Euclidean data, they exhibit weaker performance with non-Euclidean graph
data. This is primarily due to the unique challenges posed by graph data.

Table 1: Results on datasets: mean accuracy (%) $\pm$ 95% confidence interval.
The best result on each dataset is indicated with bold.

| BP | PEPITA | CaFo+MSE | CaFo+CE | FF+LA | FF+VN | SF | Ours
---|---|---|---|---|---|---|---|---
Cora |  | 86.04
---
$\pm$0.62
| 25.78
---
$\pm$6.24
| 71.79
---
$\pm$1.76
| 71.78
---
$\pm$1.71
| 84.20
---
$\pm$0.85
| 74.50
---
$\pm$1.54
| 84.54
---
$\pm$0.77
| 87.72
---
$\pm$1.63
CiteSeer |  | 78.20
---
$\pm$0.57
| 21.24
---
$\pm$2.63
| 65.43
---
$\pm$0.99
| 63.12
---
$\pm$1.15
| 75.25
---
$\pm$1.09
| 69.97
---
$\pm$1.08
| 73.84
---
$\pm$1.02
| 80.49
---
$\pm$0.41
PubMeb |  | 85.24
---
$\pm$0.28
| 36.13
---
$\pm$10.88
| 77.66
---
$\pm$0.82
| 78.29
---
$\pm$0.64
| 83.68
---
$\pm$0.38
| 79.60
---
$\pm$0.62
| 84.68
---
$\pm$0.61
| 86.28
---
$\pm$0.67
Photo |  | 93.03
---
$\pm$0.59
| 70.63
---
$\pm$7.13
| 89.48
---
$\pm$0.33
| 90.59
---
$\pm$0.26
| 86.39
---
$\pm$3.46
| 15.56
---
$\pm$9.38
| 92.48
---
$\pm$0.33
| 93.04
---
$\pm$0.31
Computer |  | 89.48
---
$\pm$0.37
| 63.25
---
$\pm$8.78
| 83.02
---
$\pm$0.59
| 82.94
---
$\pm$0.73
| 75.87
---
$\pm$4.55
| 12.27
---
$\pm$1.60
| 84.04
---
$\pm$0.65
| 86.72
---
$\pm$0.68
Texas |  | 54.26
---
$\pm$3.44
| 39.67
---
$\pm$18.02
| 56.39
---
$\pm$3.44
| 31.14
---
$\pm$3.44
| 59.67
---
$\pm$3.28
| 19.67
---
$\pm$12.30
| 37.71
---
$\pm$2.95
| 79.51
---
$\pm$1.97
Cornell |  | 71.31
---
$\pm$4.43
| 50.49
---
$\pm$20.66
| 28.85
---
$\pm$2.95
| 36.72
---
$\pm$5.73
| 45.90
---
$\pm$8.04
| 18.52
---
$\pm$12.46
| 60.66
---
$\pm$5.25
| 75.24
---
$\pm$4.92
Actor |  | 31.94
---
$\pm$0.88
| 21.55
---
$\pm$3.86
| 23.83
---
$\pm$0.76
| 23.83
---
$\pm$0.64
| 33.58
---
$\pm$1.54
| 15.05
---
$\pm$7.26
| 28.33
---
$\pm$1.38
| 34.07
---
$\pm$0.75
Chameleon |  | 41.28
---
$\pm$2.29
| 36.97
---
$\pm$2.45
| 37.36
---
$\pm$1.91
| 36.48
---
$\pm$1.44
| 33.21
---
$\pm$1.99
| 24.76
---
$\pm$2.61
| 42.35
---
$\pm$2.27
| 41.19
---
$\pm$1.56
Squirrel |  | 37.81
---
$\pm$0.71
| 33.99
---
$\pm$1.24
| 31.00
---
$\pm$1.18
| 31.00
---
$\pm$0.92
| 33.66
---
$\pm$0.87
| 18.91
---
$\pm$4.28
| 36.00
---
$\pm$1.50
| 38.17
---
$\pm$2.21

Firstly, in typical fully-connected and convolutional layers, shallow layers
capture coarse-grained features while deep layers handle fine-grained
features, with these two types of features usually being highly correlated.
However, in GNNs, different layers aggregate information from varying
neighborhood ranges, resulting in layers that often contain uncorrelated
information. Particularly in heterophilic graphs, the information extracted by
deep and shallow layers may be entirely unrelated or even have opposing
effects on predictions. This lack of correlation complicates the application
of layer-wise optimization strategies, which rely on greedy strategies and
local loss calculations common in traditional networks. This is a key reason
for the underperformance of methods like PEPITA, CaFo and FF in GNNs, as
evidenced in Tab. 1, particularly on datasets characterized by low homophily.
Secondly, since graph data do not adhere to the i.i.d. assumption, sampling
positive and negative samples based on features (FF+LA) and topology (FF+VN)
for the FF algorithm can be unreliable, potentially leading to inconsistent
results. Notably, FF+VN modifies the original graph topology by introducing
virtual nodes into both positive and negative graphs, which results in overall
unsatisfactory performance in benchmarks. For CaFo, the rigidity in fixing the
parameters of each block, with only the predictors being learnable, further
constrains its adaptability. As the most recently proposed non-BP GNN training
approach, SF shows a performance that is superior to PEPITA, CaFo and FF but
still lags behind the traditional BP method on most datasets. The reason lies
in that SF also introduces virtual nodes that disrupt graph topology and
employs a layer-wise training strategy. By contrast, our method well adapts to
graph data and gains significant improvement in testing accuracy in comparison
with the baseline algorithms, achieving the best or second-best results across
all datasets.

The training times for each method are shown in Appx. A.6. Our approach
demonstrates a general time advantage over CaFo, FF and SF. Our method
includes a forward propagation and a parameter update where all layers execute
in parallel during each iteration, which offers greater parallelism compared
with layer-wise update methods like CaFo, FF and SF. Our method has a higher
training time consumption compared with BP, primarily due to the additional
time needed for generating pseudo errors and filtering masks, as discussed in
Sec. 4.2.

###  5.2 Ablation Study

We ablate the proposed method to study the importance of designs in DFA-GNN.
Two designs are studied including the pseudo error generator (abbr. EG) and
the node filter (abbr. NF). For trials with EG, the pseudo error generator is
applied according to Eq. 7 to assign a pseudo error for each unlabeled node.
It worth noting that when ER is removed from the method, only the errors of
labeled nodes are used for the updating of parameters according to Eq. 6. For
trials with NF, the mask calculated as Eq. 9 is introduced in training process
and the parameters are updated according to Eq. 10. The ablation results are
included in Tab. 2. We note that even the most naive version of DFA-GNN
elaborated in Sec. 4.1 achieves comparable results in comparison with BP.
Furthermore, both the two designs introduced in Sec. 4.2 contribute to our
training framework, making significant enhancement to DFA-GNN to outperform BP
method.

Table 2: Ablation study results on different datasets with proposed designs.

EG | NF | Cora | CiteSeer | PubMed | Actor | Chameleon | Squirrel
---|---|---|---|---|---|---|---
✗ | ✗ | 83.02$\pm$1.36 | 78.17$\pm$0.71 | 82.92$\pm$0.42 | 30.72$\pm$1.72 | 39.09$\pm$1.17 | 34.61$\pm$1.01
✓ | ✗ | 86.70$\pm$1.00 | 79.26$\pm$0.90 | 84.01$\pm$0.23 | 31.97$\pm$1.46 | 39.62$\pm$1.31 | 34.87$\pm$1.78
✓ | ✓ |  87.72$\pm$1.63 |  80.49$\pm$0.41 |  86.28$\pm$0.67 |  34.07$\pm$0.75 |  41.19$\pm$1.56 |  38.17$\pm$2.21

###  5.3 Robustness Analysis

We focus on over-smoothing and random structural attack, two common sources of
perturbation that reduce GNN performance. Over-smoothing [23] is a problematic
issue in GNNs, stemming from the aggregation mechanism within GNNs, which
hinders the expansion of GNN models to a large number of layers. We test the
robustness of our method against over-smoothing in Fig. 3 (a). Our method
demonstrates greater robustness compared with BP, particularly when dealing
with architectures that have a large number of layers. This enhanced
robustness is due to the fact that the global loss directly contributes to the
optimization of each individual layer. SF is less effected by over-smoothing
due to its layer-wise optimization with local loss. However, its performance
largely depends on shallow layers and the best performance on each dataset is
inferior to ours.

For random structural attack [24], three random attack types are implemented
on the original graph topology with a perturbation rate $\lambda$ from 0.2 to
0.8. To better compare the robustness of different methods, we employ a more
challenging experimental setup with a sparse supervision pattern, where each
class has only 20 labeled nodes. The node split follows [13]. The detailed
operating description for attacking type is summarized as follows: (1) add:
randomly add $\lambda|\mathcal{E}|$ edges to the original graph for a denser
topology; (2) remove: randomly remove $\lambda|\mathcal{E}|$ edges from the
original graph for a sparser topology; and (3) flip: randomly choose
$\lambda|\mathcal{E}|$ node pairs, and remove the edge if there exists one
between the pair, and otherwise add an edge to connect the pair.

Our method is less sensitive to all types of perturbations as shown in Figs. 3
(b-d), consistently outperforms two approaches on each trial, and exhibits
exciting robustness even under a high perturbation rate. As the pseudo error
generator derives pseudo error for each unlabeled node to update each layer,
this supervision generation mechanism helps enhance robustness of the model
against noise and attacks. Interestingly, a comparison of results across three
different types of attacks shows that removing edges has the least adverse
effect, suggesting that injecting incorrect topological information could be
more detrimental to GNN performance than losing valuable original topology.

Figure 3: (a) Test accuracy with the model layers increasing. (b-d) Test
accuracy with the perturbation rate (b: add, c: remove, d: flip) increasing.
Table 3: Performance of our method integrated with different GNN models.

| Cora | CiteSeer | PubMeb | Photo | Computer
---|---|---|---|---|---
SGC+BP | 85.30$\pm$0.79 | 79.45$\pm$0.87 | 79.72$\pm$0.35 | 91.71$\pm$0.44 | 85.07$\pm$0.27
SGC+ours |  88.60$\pm$0.85 |  81.02$\pm$0.78 |  80.40$\pm$0.18 |  92.01$\pm$0.29 |  85.94$\pm$0.31
GAT+BP | 85.35$\pm$0.62 | 79.09$\pm$1.31 | 85.78$\pm$0.34 | 93.16$\pm$0.43 |  88.91+0.78
GAT+ours |  86.96$\pm$1.08 |  79.77$\pm$1.15 |  86.18$\pm$0.32 |  93.24$\pm$0.51 | 87.38$\pm$0.66
GraphSage+BP | 87.04$\pm$0.84 | 79.65$\pm$1.11 |  88.32$\pm$0.28 | 92.89$\pm$0.51 | 88.00$\pm$0.29
GraphSage+ours |  87.88$\pm$1.00 |  79.69$\pm$1.33 | 87.65$\pm$0.28 |  93.95$\pm$0.33 |  88.14$\pm$0.39
APPNP+BP | 85.75$\pm$0.89 |  80.46$\pm$0.37 |  85.81$\pm$0.28 | 91.55$\pm$0.81 | 85.50$\pm$0.57
APPNP+ours |  85.81$\pm$1.11 | 79.47$\pm$0.80 | 85.75$\pm$0.32 |  91.66$\pm$0.39 |  85.68$\pm$0.43
ChebNet+BP | 83.45$\pm$1.07 | 76.93$\pm$0.71 | 87.09$\pm$0.31 | 90.89$\pm$0.74 | 86.51$\pm$0.78
ChebNet+ours |  85.53$\pm$1.36 |  77.85$\pm$0.96 | 86.43$\pm$0.37 |  92.65$\pm$0.49 |  87.02$\pm$0.62

###  5.4 Portability Analysis

We apply our training algorithm to five popular GNN models [25, 26, 27, 28,
29] and report the mean accuracy across ten random splits in Tab. 3. Each of
the testing models is modified to fit our framework. Specifically, for SGC,
which only has a single learnable linear output layer, the training of our
framework involves no direct feedback but only incorporates the pseudo error
generator and node filter. For GraphSage, we utilize a mean-aggregator for
message aggregation. Our observations indicate that our method can be
effectively ported to mainstream GNN models. All test models integrated with
our algorithm work well and surpass the performance of traditional BP in most
scenarios. It demonstrates the effectiveness of our method across various GNN
models and underscores its excellent portability and potential generalization
ability to other innovative GNN models.

##  6 Conclusion

In this paper, we investigate the potential of non-backpropagation training
methods within the context of graph learning. We adapt the direct feedback
alignment algorithm for training graph neural networks on graph data and
introduce DFA-GNN. This new approach incorporates a meticulously designed
random feedback strategy, a pseudo error generator, and a node filter to
effectively spread residual errors. Through mathematical formulations, we
demonstrate that our method can align with backpropagation in terms of
parameter update gradients and perform effective training. Extensive
experiments on real-world datasets confirm the effectiveness, efficiency,
robustness and versatility of our proposed forward graph learning framework.

## References

  * [1] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI open, 1:57–81, 2020.
  * [2] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * [3] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui.  Graph neural networks in recommender systems: a survey.  ACM Computing Surveys, 55(5):1–37, 2022.
  * [4] Jiacheng Xiong, Zhaoping Xiong, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng.  Graph neural networks for automated de novo drug design.  Drug discovery today, 26(6):1382–1393, 2021.
  * [5] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec.  Qa-gnn: Reasoning with language models and knowledge graphs for question answering.  arXiv preprint arXiv:2104.06378, 2021.
  * [6] Robert Hecht-Nielsen.  Theory of the backpropagation neural network.  In Neural networks for perception, pages 65–93. Elsevier, 1992\.
  * [7] Geoffrey Hinton.  The forward-forward algorithm: Some preliminary investigations.  arXiv preprint arXiv:2212.13345, 2022.
  * [8] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman.  Random synaptic feedback weights support error backpropagation for deep learning.  Nature communications, 7(1):13276, 2016.
  * [9] Giorgia Dellaferrera and Gabriel Kreiman.  Error-driven input modulation: Solving the credit assignment problem without a backward pass.  In International Conference on Machine Learning, pages 4937–4955. PMLR, 2022.
  * [10] Arild Nøkland.  Direct feedback alignment provides learning in deep neural networks.  Advances in neural information processing systems, 29, 2016.
  * [11] Gongpei Zhao, Tao Wang, Yidong Li, Yi Jin, Congyan Lang, and Haibin Ling.  The cascaded forward algorithm for neural network training.  arXiv preprint arXiv:2303.09728, 2023.
  * [12] Ruben Ohana, Hamlet Medina, Julien Launay, Alessandro Cappelli, Iacopo Poli, Liva Ralaivola, and Alain Rakotomamonjy.  Photonic differential privacy with direct feedback alignment.  Advances in Neural Information Processing Systems, 34:22010–22020, 2021.
  * [13] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations, 2016.
  * [14] Charlotte Frenkel, Martin Lefebvre, and David Bol.  Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks.  Frontiers in neuroscience, 15:629892, 2021.
  * [15] Namyong Park, Xing Wang, Antoine Simoulin, Shuai Yang, Grey Yang, Ryan A Rossi, Puja Trivedi, and Nesreen K Ahmed.  Forward learning of graph neural networks.  In The Twelfth International Conference on Learning Representations, 2023.
  * [16] Yuchen Zhang, Jason Lee, Martin Wainwright, and Michael I Jordan.  On the learnability of fully-connected neural networks.  In Artificial Intelligence and Statistics, pages 83–91. PMLR, 2017\.
  * [17] Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou.  A survey of convolutional neural networks: analysis, applications, and prospects.  IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021.
  * [18] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson.  Combining label propagation and simple models out-performs graph neural networks.  In International Conference on Learning Representations, 2020.
  * [19] Junteng Jia and Austion R Benson.  Residual correlation in graph neural network regression.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 588–598, 2020.
  * [20] Cosma Shalizi.  Advanced data analysis from an elementary point of view.  2013\.
  * [21] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf.  Learning with local and global consistency.  Advances in neural information processing systems, 16, 2003.
  * [22] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  nature, 323(6088):533–536, 1986.
  * [23] Nicolas Keriven.  Not too little, not too much: a theoretical analysis of graph (over) smoothing.  Advances in Neural Information Processing Systems, 35:2268–2281, 2022.
  * [24] Yaxin Li, Wei Jin, Han Xu, and Jiliang Tang.  Deeprobust: a platform for adversarial attacks and defenses.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 16078–16080, 2021.
  * [25] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger.  Simplifying graph convolutional networks.  In International conference on machine learning, pages 6861–6871. PMLR, 2019.
  * [26] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In International Conference on Learning Representations, 2018.
  * [27] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  Advances in neural information processing systems, 30, 2017.
  * [28] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann.  Predict then propagate: Graph neural networks meet personalized pagerank.  In International Conference on Learning Representations, 2018.
  * [29] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  Advances in neural information processing systems, 29, 2016.
  * [30] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad.  Collective classification in network data.  AI magazine, 29(3):93–93, 2008.
  * [31] Zhilin Yang, William Cohen, and Ruslan Salakhudinov.  Revisiting semi-supervised learning with graph embeddings.  In International conference on machine learning, pages 40–48. PMLR, 2016.
  * [32] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel.  Image-based recommendations on styles and substitutes.  In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52, 2015.
  * [33] Benedek Rozemberczki, Carl Allen, and Rik Sarkar.  Multi-scale attributed node embedding.  Journal of Complex Networks, 9(2):cnab014, 2021.
  * [34] Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang.  Geom-gcn: Geometric graph convolutional networks.  In International Conference on Learning Representations, 2019.
  * [35] Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova.  A critical look at the evaluation of gnns under heterophily: Are we really making progress?  In The Eleventh International Conference on Learning Representations, 2022.
  * [36] Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J. Smola, and Zheng Zhang.  Deep graph library: Towards efficient and scalable deep learning on graphs.  In ICLR workshop on representation learning on graphs and manifolds, 2019.
  * [37] Matthias Fey and Jan Eric Lenssen.  Fast graph representation learning with pytorch geometric.  arXiv preprint arXiv:1903.02428, 2019.

##  Appendix A APPENDIX

Here we provide some implementation details of our methods to help readers
further understand the algorithms and experiments in this paper.

###  A.1 GNN Training Process of BP

Algorithm 1 GNN Training Process of BP

Input: $G$, X, W, $\textbf{y}_{L}$, max_epoch.
Output: $\tilde{\textbf{y}}$, W.

1: for epoch in [0, 1, …, max_epoch-1] do

2: for $l$ in [0, 1, …, $L-1$] do {Forward propagation}

3: for $v$ in v do

4:
$\textbf{h}_{v}^{(l)}=\sum_{u\in\tilde{N}(v)}e_{vu}\cdot\textbf{x}_{u}^{(l)}$;
{Aggregation}

5: end for

6: $\textbf{X}^{(l+1)}=\sigma(\textbf{H}^{(l)}\textbf{W}^{(l)})$;
{Combination}

7: end for

8: $\textit{loss}=Loss\\_computing(\textbf{X}^{(L)},\textbf{y}_{L})$

9: for $l$ in [$L-1$, $L-2$, …, 0] do {Backward propagation}

10:
$\delta\textbf{W}^{(l)}=\textbf{H}^{(l)T}\delta\textbf{X}^{(l+1)}$;{Combination}

11:
$\delta\textbf{H}^{(l)}=\delta\textbf{X}^{(l+1)}\textbf{W}^{(l)T}$;{Combination}

12: for $v$ in v do

13:
$\delta\textbf{x}_{v}^{(l)}=\sum_{u\in\tilde{N}(v)}e_{uv}\cdot\delta\textbf{h}_{u}^{(l)}$;
{Aggregation}

14: end for

15: end for

16:
$Weight\\_updating(epoch,\textbf{W}^{(0)},\delta\textbf{W}^{(0)},...,\textbf{W}^{(L-1)},\delta\textbf{W}^{(L-1)})$;

17: end for

18: $\tilde{\textbf{y}}=\arg\max\tilde{\textbf{Y}}=\arg\max\textbf{X}^{(L)}$.

###  A.2 GCN Training Process of Our Method

Algorithm 2 GCN Training Process

Input: $G$, X, W, $\textbf{y}_{L}$, max_epoch.
Output: $\tilde{\textbf{y}}$, W.

1: Initialize {$\textbf{B}^{(1)}$, $\textbf{B}^{(2)}$, …,
$\textbf{B}^{(L-1)}$};

2: for epoch in [0, 1, …, max_epoch-1] do

3: Compute the prediction $\tilde{\textbf{Y}}$ given by Eq. 1; {Forward
propagation}

4: Compute the error E using $\tilde{\textbf{Y}},\textbf{y}_{L}$ for labeled
nodes according to Eq. 3;

5: Compute pseudo error $\textbf{Z}^{*}$ for unlabeled nodes by Eqs. 8, and
rescale it to get rescaled error $\hat{\textbf{E}}$;

6: Compute mask vector p given by Eq. 9;

7: for $l$ in [0, 1, …, $L-1$] do

8: if $l==L-1$ then

9: Compute $\delta\textbf{W}^{(l)}$ using $\hat{\textbf{E}}$, p according to
Eq. 10; {Direct feedback}

10: else

11: Compute $\delta\textbf{W}^{(l)}$ using $\hat{\textbf{E}}$, p and
$\textbf{B}^{(l+1)}$ according to Eq. 10; {Direct feedback}

12: end if

13: end for

14:
$Weight\\_updating(epoch,\textbf{W}^{(0)},\delta\textbf{W}^{(0)},...,\textbf{W}^{(L-1)},\delta\textbf{W}^{(L-1)})$;

15: end for

16: $\tilde{\textbf{y}}=\arg\max\tilde{\textbf{Y}}$.

###  A.3 Proof of Theorem in Sec. 4.3

Theorem. For a GCN model with two hidden layers $k$ and $k+1$ where $k$
connects to $k+1$ in sequence, we have
$\textbf{x}^{(k+1)}=\sigma(\textbf{a}^{(k+1)})$ and
$\textbf{a}^{(k+1)}=g(\textbf{W}\textbf{x}^{(k)})$, where $\sigma$ is the
activation function and $g(\cdot)$ the aggregation operation in Algo. 1. Let
the layers be updated according to the non-zero update directions
$\delta\textbf{x}^{(k)}$ and $\delta\textbf{x}^{(k+1)}$ where
$\frac{\delta\textbf{x}^{(k)}}{\|\delta\textbf{x}^{(k)}\|}$ and
$\frac{\delta\textbf{x}^{(k+1)}}{\|\delta\textbf{x}^{(k+1)}\|}$ are constant
for each data point. The negative update directions will minimize the
following layer-wise criterion:

$\displaystyle\textbf{P}=\textbf{P}^{(k)}+\textbf{P}^{(k+1)}=\frac{\delta\textbf{x}^{(k)\text{T}}\textbf{x}^{(k)}}{\|\delta\textbf{x}^{(k)}\|}+\frac{\delta\textbf{x}^{(k+1)\text{T}}\textbf{x}^{(k+1)}}{\|\delta\textbf{x}^{(k+1)}\|}.$
(13)

Minimizing P will lead to an increase in the gradient, thereby enhancing the
alignment criterion:

$\displaystyle\textbf{Q}=\textbf{Q}^{(k)}+\textbf{Q}^{(k+1)}=\frac{\delta\textbf{x}^{(k)\text{T}}\textbf{c}^{(k)}}{\|\delta\textbf{x}^{(k)}\|}+\frac{\delta\textbf{x}^{(k+1)\text{T}}\textbf{c}^{(k+1)}}{\|\delta\textbf{x}^{(k+1)}\|},$
(14)

where

$\displaystyle\begin{aligned}
\textbf{c}^{(k)}&=\frac{\partial\textbf{x}^{(k+1)}}{\partial\textbf{x}^{(k)}}\delta\textbf{x}^{(k+1)}=\textbf{W}^{\text{T}}g^{\prime}(\delta\textbf{x}^{(k+1)}\odot\sigma^{\prime}(\textbf{a}^{(k+1)})),\\\\[6.0pt]
\textbf{c}^{(k+1)}&=\frac{\partial\textbf{x}^{(k+1)}}{\partial\textbf{x}^{(k)\text{T}}}\delta\textbf{x}^{(k)}=g^{\prime}(\textbf{W}\delta\textbf{x}^{(k)})\odot\sigma^{\prime}(\textbf{a}^{(k+1)}).\end{aligned}$
(15)

$g^{\prime}(\cdot)$ is the aggregation of gradients in Algo. 1, (line 13). If
$\textbf{Q}^{(k)}>0$, then $-\delta\textbf{x}^{(k)}$ serves as a direction of
descent to minimize $\textbf{P}^{(k+1)}$.

Proof. Let $i$ be the any of the layers $k$ or $k+1$, and prescribed update
$-\delta\textbf{x}^{(i)}$ is the steepest descent direction to minimize
$\textbf{P}^{(i)}$. Since any partial derivative of
$\frac{\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}$ is zero, we have:

$\displaystyle-\frac{\partial\textbf{P}^{(i)}}{\partial\textbf{x}^{(i)}}=-\frac{\partial}{\partial\textbf{x}^{(i)}}[\frac{\delta\textbf{x}^{(i)\text{T}}\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}]=-\frac{\partial}{\partial\textbf{x}^{(i)}}[\frac{\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}]\textbf{x}^{(i)}-\frac{\partial\textbf{x}^{(i)}}{\partial\textbf{x}^{(i)}}\frac{\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}=-\alpha^{(i)}\delta\textbf{x}^{(i)},$
(16)

where $\alpha^{(i)}=\frac{1}{\|\delta\textbf{x}^{(i)}\|}>0$. As
$\delta\textbf{a}^{(i)}=\frac{\partial\textbf{x}^{(i)}}{\partial\textbf{a}^{(i)}}\delta\textbf{x}^{(i)}=\delta\textbf{x}^{(i)}\odot\sigma^{\prime}(\textbf{a}^{(i)})$,
the gradients maximizing $\textbf{Q}^{(k)}$ and $\textbf{Q}^{(k+1)}$ are:

$\displaystyle\begin{aligned}
\frac{\partial\textbf{Q}^{(i)}}{\partial\textbf{c}^{(i)}}&=\frac{\partial}{\partial\textbf{c}^{(i)}}[\frac{\delta\textbf{x}^{(i)\text{T}}\textbf{c}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}]=\frac{\partial}{\partial\textbf{c}^{(i)}}[\frac{\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}]\textbf{c}^{(i)}+\frac{\partial\textbf{c}^{(i)}}{\partial\textbf{c}^{(i)}}\frac{\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|}=\alpha^{(i)}\delta\textbf{x}^{(i)},\\\\[6.0pt]
\frac{\partial\textbf{Q}^{(k+1)}}{\partial\textbf{W}}&=\frac{\partial\textbf{Q}^{(k+1)}}{\partial\textbf{c}^{(k+1)}}\frac{\partial\textbf{c}^{(k+1)}}{\partial\textbf{W}}=\alpha^{(k+1)}g^{\prime}(\delta\textbf{x}^{(k+1)}\odot\sigma^{\prime}(\textbf{a}^{(k+1)}))\delta\textbf{x}^{(k)\text{T}}=\alpha^{(k+1)}g^{\prime}(\delta\textbf{a}^{(k+1)})\delta\textbf{x}^{(k)\text{T}},\\\\[6.0pt]
\frac{\partial\textbf{Q}^{(k)}}{\partial\textbf{W}}&=\frac{\partial\textbf{c}^{(k)}}{\partial\textbf{W}^{\text{T}}}\frac{\partial\textbf{Q}^{(k)}}{\partial\textbf{c}^{(k)T}}=g^{\prime}(\delta\textbf{x}^{(k+1)}\odot\sigma^{\prime}(\textbf{a}^{(k+1)}))\alpha^{(k)}\delta\textbf{x}^{(k)\text{T}}=\alpha^{(k)}g^{\prime}(\delta\textbf{a}^{(k+1)})\delta\textbf{x}^{(k)\text{T}}.\end{aligned}$
(17)

When ignoring the magnitude of the gradients we have
$\frac{\partial\textbf{Q}}{\partial\textbf{W}}\approx\frac{\partial\textbf{Q}^{(k)}}{\partial\textbf{W}}\approx\frac{\partial\textbf{Q}^{(k+1)}}{\partial\textbf{W}}$.
If $\textbf{x}^{(i)}$ is projected onto $\delta\textbf{x}^{(i)}$ we have
$\textbf{x}^{(i)}=\frac{\textbf{x}^{(i)\text{T}}\delta\textbf{x}^{(i)}}{\|\delta\textbf{x}^{(i)}\|^{2}}\delta\textbf{x}^{(i)}+\textbf{x}^{(i)}_{res}$.
The prescribed update for W is:

$\displaystyle\begin{aligned}
\delta\textbf{W}&=-\delta\textbf{x}^{(k+1)}\frac{\partial\textbf{x}^{(k+1)}}{\partial\textbf{W}}=-g^{\prime}(\delta\textbf{x}^{(k+1)}\odot\sigma^{\prime}(\textbf{a}^{(k+1)}))\textbf{x}^{(k)\text{T}}\\\\[6.0pt]
&=-g^{\prime}(\delta\textbf{a}^{(k+1)})\textbf{x}^{(k)\text{T}}=-g^{\prime}(\delta\textbf{a}^{(k+1)})(\alpha^{(k)}\textbf{P}^{(k)}\delta\textbf{x}^{(k)}+\textbf{x}^{(k)}_{res})^{\text{T}}\\\\[6.0pt]
&=-\alpha^{(k)}\textbf{P}^{(k)}g^{\prime}(\delta\textbf{a}^{(k+1)})\delta\textbf{x}^{(k)\text{T}}-g^{\prime}(\delta\textbf{a}^{(k+1)})\textbf{x}^{(k)\text{T}}_{res}=-\textbf{P}^{(k)}\frac{\partial\textbf{Q}^{(k)}}{\partial\textbf{W}}-g^{\prime}(\delta\textbf{a}^{(k+1)})\textbf{x}^{(k)\text{T}}_{res}\end{aligned}$
(18)

It is obvious that $\textbf{Q}^{(k)}$ and $\textbf{Q}^{(k+1)}$ can be
maximized when the component of
$\frac{\partial\textbf{Q}^{(k)}}{\partial\textbf{W}}$ in $\delta\textbf{W}$ is
maximized by minimizing $\textbf{P}^{(k)}$. The gradient to minimize
$\textbf{P}^{(k)}$ is the prescribed update $-\delta\textbf{x}^{(k)}$. The
angle between $\delta\textbf{x}^{(k)}$ and the gradient of BP
$\textbf{c}^{(k)}$ is within 90∘ if $\textbf{Q}^{(k)}>0$ because the cosine of
the two vector is $\frac{\textbf{Q}^{(k)}}{\|\textbf{c}^{(k)}\|}>0$, and it
also indicates that $\textbf{c}^{(k)}$ is nonzero and therefore in a
descending trend. Consequently, $\delta\textbf{x}^{(k)}$ will be oriented
towards a descending direction since any vector that lies within 90∘ of the
steepest descent direction will similarly point downwards, in other words, for
GCN, a broad spectrum of asymmetric feedback paths can offer a descending
gradient direction for a hidden layer as long as $\textbf{Q}^{(i)}>0$.

From Eq. 5, it is obvious one advantage of our method is that
$\delta\textbf{x}^{(i)}$ is non-zero for any non-zero error e, as a randomly
generated matrix $\textbf{B}^{(i)}$ is highly likely to be of full rank.
Ensuring $\delta\textbf{x}^{(i)}$ is non-zero is crucial for achieving
$\textbf{Q}^{(i)}>0$. Maintaining static feedback across training helps
preserve the characteristic, and also simplifies the process of maximizing
$\textbf{Q}^{(i)}$ due to the more consistent direction to
$\delta\textbf{x}^{(i)}$. Our method shows better biological plausibility in
GNN training, which introduces asymmetric feedback paths to take place of BP,
not only solving the weight transport problem and partially solving the update
locking problem, but also releasing the requirement to store neural
activations and accumulated gradients for backward propagation.

###  A.4 Datasets Statistics

We evaluate our method on 10 benchmark datasets across domains: citation
networks (Cora, CiteSeer, PubMed) [30, 31], Amazon co-purchase graph (Photo,
Computer) [32], Wikipedia graphs (Chameleon, Squirrel) [33], actor co-
occurrence graph (Actor) [34] and webpage graphs from WebKB (Texas, Cornell)
[34]. The datasets adopted are representative which describe diverse real-life
scenarios. Some of them are highly homophilic while others are heterophilic.
Note that for Chameleon and Squirrel, we use the filtered version from [35],
as the original version from [33] may contain duplicated nodes. The detailed
statistics of the datasets are summarized in Tab. 4. We compute the node
homophily for each dataset using the method proposed by [34], referred to as
the homophily value.

Table 4: Datasets statistics. | Cora | CiteSeer | PubMed | Computer | Photo | Chameleon | Squirrel | Actor | Texas | Cornell
---|---|---|---|---|---|---|---|---|---|---
Nodes | 2708 | 3327 | 19717 | 13752 | 7650 | 890 | 2223 | 7600 | 183 | 183
Edges | 5278 | 4552 | 44324 | 245861 | 119081 | 17708 | 93996 | 26659 | 279 | 277
Features | 1433 | 3703 | 500 | 767 | 745 | 2325 | 2089 | 932 | 1703 | 1703
Classes | 7 | 6 | 5 | 10 | 8 | 5 | 5 | 5 | 5 | 5
Homophily | 0.825 | 0.706 | 0.792 | 0.785 | 0.836 | 0.244 | 0.190 | 0.220 | 0.057 | 0.301

  * •

Cora, CiteSeer and PubMed [30] are three classic homophilic citation networks.
In these networks, nodes correspond to academic papers, and edges signify the
citation links between papers. The node features are derived from bag-of-word
representations of the papers, and the labels categorize each paper into
specific research topics.

  * •

Computer and Photo [32] are segments of the Amazon co-purchase graph, where
nodes represent goods, edges indicate that two goods are frequently bought
together, node features are bag-of-words encoded product reviews, and class
labels are given by the product category.

  * •

Chameleon and Squirrel [33] are two heterophilic networks derived from
Wikipedia. In these networks, nodes represent Wikipedia web pages, and edges
correspond to hyperlinks between these pages. The features are comprised of
informative nouns extracted from the Wikipedia content, while the labels
reflect the average traffic of each web page.

  * •

Actor [34] is a heterophilic actor co-occurrence network where nodes represent
actors, and edges signify that two actors have appeared together in the same
movie. The features are derived from keywords found on the actors’ Wikipedia
pages, while the labels consist of significant words associated with each
actor.

  * •

Cornell and Texas [34] are heterophilic networks from the WebKB1 project
representing computer science departments at three universities. Nodes are
departmental web pages, edges represent hyperlinks, features are derived using
bag-of-words, and labels categorize page types. These networks illustrate
heterophilic connections where linked pages often differ in type.

###  A.5 Experimental Settings

We conduct the semi-supervised node classification task using a basic GCN
model, where the node set is randomly divided into the train/validation/test
set with 60%/20%/20%. For fairness, we generate 10 random splits using
different seeds and evaluate all approaches on these identical splits,
reporting the average performance for each method. Our method is compared with
five baseline training strategies including the traditional backpropagation
(BP) [22], PEPITA [9], two versions of the forward-forward algorithm (abbr.
FF+LA, FF+VN) [7, 15], two versions of the cascaded forward algorithm (abbr.
CaFo+MSE, CaFo+CE) [11] and the FORWARDGNN Single-Forward algorithm (abbr. SF)
[15] specifically designed for GNNs. To ensure fairness, we train all
approaches using the same GCN architecture, which includes 3 graph
convolutional layers and 64 hidden units—sufficiently representative for all
datasets. We employ Adam as the optimization algorithm, refraining from using
any regularization techniques other than an appropriate L2 penalty specific to
Adam. The evaluation metric used is accuracy (acc), presented with a 95%
confidence interval.

Table 5: Hyper-parameters of proposed method on real-world datasets. | Learning rate | Hidden unit | $\alpha$ | Iteration epoch | $\epsilon$ | Weight decay | Training epoch
---|---|---|---|---|---|---|---
Cora | 0.01 | 64 | 0.1 | 50 | 0.5 | 0.0005 | 1000
CiteSeer | 0.01 | 64 | 0.01 | 200 | 0.5 | 0.0005 | 1000
PubMeb | 0.01 | 64 | 0.1 | 200 | 0.5 | 0.0005 | 1000
Photo | 0.001 | 64 | 0.01 | 50 | 0.5 | 0.0005 | 1000
Computer | 0.001 | 64 | 0.01 | 50 | 0.5 | 0.0005 | 1000
Texas | 0.01 | 64 | 0.9 | 50 | 0.5 | 0.0 | 1000
Cornell | 0.01 | 64 | 0.5 | 50 | 0.5 | 0.0 | 1000
Actor | 0.01 | 64 | 0.5 | 50 | 0.5 | 0.0 | 1000
Chameleon | 0.01 | 64 | 0.5 | 200 | 0.5 | 0.0 | 1000
Squirrel | 0.01 | 64 | 0.5 | 200 | 0.5 | 0.0 | 1000

The codes of DFA-GNN are based on the GNNs in the PyTorch version by Deep
Graph Library (DGL) [36]. To generate pseudo errors, we search the optimal
$\alpha$ in Eq. 8 within {0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9}, the iteration
epoch for Eq. 8 within {5, 10, 30, 50, 100, 150, 200}, $\epsilon$ in Eq. 9
within {0.3, 0.5, 0.7, 0.9}. For the training of our method, we search the
learning rate within {0.001, 0.01, 0.1} and weight decay within {0.0005, 0}.
All the experiments are run on AMD EPYC 7542 32-Core Processor with Nvidia
GeForce RTX 3090. We list the hyper-parameter values used in our model in Tab.
5.

###  A.6 Time Comparison

Table 6: Average running time per epoch (s). For layer-wise training methods like PEPITA, CaFo, FF and SF, the total time taken by each layer per epoch is reported. | BP | PEPETA | CaFo+CE | FF+LA | FF+VN | SF | ours
---|---|---|---|---|---|---|---
Cora | $7.56\text{e}^{-3}$ | $8.73\text{e}^{-3}$ | $7.61\text{e}^{-1}$ | $3.14\text{e}^{-1}$ | $2.83\text{e}^{-1}$ | $5.49\text{e}^{-2}$ | $5.66\text{e}^{-2}$
CiteSeer | $1.06\text{e}^{-2}$ | $1.11\text{e}^{-2}$ | $7.68\text{e}^{-1}$ | $2.59\text{e}^{-1}$ | $2.61\text{e}^{-1}$ | $6.88\text{e}^{-2}$ | $5.68\text{e}^{-2}$
PubMed | $1.07\text{e}^{-2}$ | $1.07\text{e}^{-2}$ | $8.24\text{e}^{-1}$ | $6.94\text{e}^{-1}$ | $7.61\text{e}^{-1}$ | $5.34\text{e}^{-1}$ | $6.76\text{e}^{-2}$
Photo | $8.74\text{e}^{-3}$ | $1.03\text{e}^{-2}$ | $7.98\text{e}^{-1}$ | $2.11$ | $1.91$ | $4.87\text{e}^{-1}$ | $5.81\text{e}^{-2}$
Computer | $1.08\text{e}^{-2}$ | $1.05\text{e}^{-2}$ | $7.80\text{e}^{-1}$ | $4.82$ | $4.14$ | $7.61\text{e}^{-1}$ | $6.29\text{e}^{-2}$
Texas | $6.13\text{e}^{-3}$ | $1.07\text{e}^{-2}$ | $8.05\text{e}^{-1}$ | $1.47\text{e}^{-1}$ | $1.56\text{e}^{-1}$ | $6.88\text{e}^{-2}$ | $5.60\text{e}^{-2}$
Cornell | $5.42\text{e}^{-3}$ | $1.06\text{e}^{-2}$ | $7.46\text{e}^{-1}$ | $1.51\text{e}^{-1}$ | $1.24\text{e}^{-1}$ | $3.59\text{e}^{-2}$ | $5.53\text{e}^{-2}$
Actor | $9.45\text{e}^{-3}$ | $1.03\text{e}^{-2}$ | $7.83\text{e}^{-1}$ | $6.84\text{e}^{-1}$ | $6.71\text{e}^{-1}$ | $2.80\text{e}^{-1}$ | $5.80\text{e}^{-2}$
Chameleon | $6.24\text{e}^{-3}$ | $1.13\text{e}^{-2}$ | $7.97\text{e}^{-1}$ | $2.28\text{e}^{-1}$ | $2.09\text{e}^{-1}$ | $6.88\text{e}^{-2}$ | $5.61\text{e}^{-2}$
Squirrel | $7.77\text{e}^{-3}$ | $1.20\text{e}^{-2}$ | $7.78\text{e}^{-1}$ | $5.82\text{e}^{-1}$ | $5.05\text{e}^{-1}$ | $1.21\text{e}^{-1}$ | $5.79\text{e}^{-2}$

###  A.7 Codes of baseline models

We implemented all baseline models using the official code and leveraged
libraries such as PyTorch Geometric [37], DGL [36], and Deeprobust [24]. The
URLs of all the baseline approaches we used are listed in Tab. 7. Our code
will be made public upon the acceptance of this paper.

Table 7: Codes of baseline approaches. | URL
---|---
FF | https://github.com/mpezeshki/pytorch_forward_forward
FORWARDGNN | https://github.com/facebookresearch/forwardgnn
PEPITA | https://github.com/GiorgiaD
CaFo | https://github.com/Graph-ZKY/CaFo
GCN | https://github.com/tkipf/gcn
SGC | https://github.com/Tiiiger/SGC
APPNP | https://github.com/benedekrozemberczki/APPNP
GraphSage | https://github.com/williamleif/GraphSAGE
GAT | https://github.com/PetarV-/GAT
ChebNet | https://github.com/dsgiitr/graph_nets