# Dynamic Spiking Graph Neural Networks

Nan Yin1, Mengzhu Wang2, Zhenghan Chen3, Giulia De Masi4, Bin Gu5,1*, Huan
Xiong6,1 Corresponding authors.

###### Abstract

The integration of Spiking Neural Networks (SNNs) and Graph Neural Networks
(GNNs) is gradually attracting attention due to the low power consumption and
high efficiency in processing the non-Euclidean data represented by graphs.
However, as a common problem, dynamic graph representation learning faces
challenges such as high complexity and large memory overheads. Current work
often uses SNNs instead of Recurrent Neural Networks (RNNs) by using binary
features instead of continuous ones for efficient training, which would
overlooks graph structure information and leads to the loss of details during
propagation. Additionally, optimizing dynamic spiking models typically
requires propagation of information across time steps, which increases memory
requirements. To address these challenges, we present a framework named
Dynamic Spiking Graph Neural Networks (Dy-SIGN). To mitigate the information
loss problem, Dy-SIGN propagates early-layer information directly to the last
layer for information compensation. To accommodate the memory requirements, we
apply the implicit differentiation on the equilibrium state, which does not
rely on the exact reverse of the forward computation. While traditional
implicit differentiation methods are usually used for static situations, Dy-
SIGN extends it to the dynamic graph setting. Extensive experiments on three
large-scale real-world dynamic graph datasets validate the effectiveness of
Dy-SIGN on dynamic node classification tasks with lower computational costs.

## Introduction

Graph Neural Networks (GNNs) (Scarselli et al. 2008) have been widely applied
in various fields to learn the graph representation by capturing the
dependencies of nodes and edges, such as relation detection (Schlichtkrull et
al. 2018; Chen, Li, and Tang 2020; Mi and Chen 2020; Xu et al. 2019b) and
recommendation (Wu et al. 2022). However, most GNNs are primarily designed for
static or non-temporal graphs, which cannot meet the requirement of dynamic
evolution over time in practice. The established solution is to extend the
dynamic graphs into sequence models directly. Typically, these methods (Yin et
al. 2023b; Shi et al. 2021; Kumar, Zhang, and Leskovec 2019; Yin et al. 2022a)
utilize the Recurrent Neural Networks (RNNs) (Cho et al. 2014) to capture the
dynamic evolution of graphs for numerous downstream tasks such as time series
prediction or graph property prediction (Yin et al. 2022b; Wieder et al. 2020;
Yin et al. 2023a).

Despite the promising performance of dynamic graphs, the majority of these
approaches typically involve complex structures that consume significant
computational resources during training and testing. Inspired by the way the
brain process information, Spiking Neural Networks (SNNs) represent the event
or clock-driven signals as inference for updating the neuron nodes parameters
(Brette et al. 2007). Different from traditional deep learning methods, SNNs
utilize discrete spikes information instead of continuous features, resulting
in significantly lower power consumption during model training. Considering
the inherent characteristics of SNNs (Maass 1997; Pfeiffer and Pfeil 2018;
Schliebs and Kasabov 2013), a few recent works (Zhu et al. 2022; Xu et al.
2021; Li et al. 2023) have attempted to integrate SNNs into the GNNs framework
to tackle the issue of high computational complexity. These methods transform
the node features into a series of spikes with Poisson rate coding, and follow
a graph convolution layer with SNN neurons, which employ a membrane potential
threshold to convert continuous features to spike information (Kim et al.
2020; Bu et al. 2022). Although Spiking Graph Networks (SGNs) are gradually
gaining attention, the use of SNNs in dynamic graphs is still less explored,
which is a more common scenario in life. To address the gap, the work focuses
on the problem of spiking dynamic graph, which applies SNNs for dynamic graph
node classification.

However, the problem is highly challenging due to the following reasons: (1)
Information Loss. The representation of GNNs includes the information on graph
structure and neighboring nodes, which are crucial for downstream tasks.
However, SNNs employ spike signals instead of continuous features, leading to
the loss of details regarding the structure and neighbors. Moreover, with the
evolution of graphs over time, the information loss issue may further
deteriorate the graph representation. (2) Memory Consumption. The RNN-based
dynamic graph methods typically require significant memory resources to store
the temporal node information (Sak, Senior, and Beaufays 2014). Moreover, SNNs
inherently operate with multiple time latencies (i.e., calculate the spike
signals with time latency steps in each SNN layer). If we simply replace the
GNN layer with the SNN layer on each time step, we need to store the temporary
spikes in SNN layer and temporal information at each time step simultaneously,
which further exacerbates memory consumption.

In this paper, we present a novel framework named Dynamic Spiking Graph Neural
Networks (Dy-SIGN) for node classification. The primary insight of proposed
Dy-SIGN is to thoroughly explore how to apply SNNs to dynamic graphs, and
address the challenges of information loss and memory consumption by using the
information compensation mechanism and implicit differentiation on the
equilibrium state. On the one hand, the information compensation mechanism
aims to make up for the information loss during forward propagation. However,
implementing the mechanism in each layer of SNNs would significantly increase
the model complexity. Thus, we propose to establish an information channel
between the shallow and final layers to incorporate the original information
directly into feature representations. This approach not only reduces the
model complexity but also mitigates the impact of information loss. On the
other hand, inspired by recent advances in implicit methods (Bai, Koltun, and
Kolter 2020; Xiao et al. 2021) that view neural networks as solving an
equilibrium equation for fixed points and provide alternative implicit models
defined by the equation, we provide a variation training method that is
suitable for dynamic spiking graph neural networks. Specifically, Dy-SIGN
simplifies the non-differentiable items in backpropagation, thus avoiding the
huge computational overhead of traditional SNNs due to the use of surrogate
learning techniques. In this way, the calculation of the gradient would
significantly reduce memory consumption. We conduct extensive experiments to
demonstrate the effectiveness of proposed Dy-SIGN in comparison to the state-
of-the-art methods across various scenarios.

In summary, our contributions are as follows: (1) Motivation: From the
perspective of practical application and data analysis, we propose the Dy-
SIGN, which is the first attempt to introduce implicit differentiation into
dynamic graph. (2) Methodology: We propose a novel approach called Dy-SIGN
that incorporates SNNs into dynamic graphs to release the information loss and
memory consumption problem. (3) Experiments: Extensive experiments validate
the superiority of the proposed Dy-SIGN over the state-of-the-art methods.

## Related Work

### SNN Training Methods

In the supervised training of SNNs, there are two primary research directions.
One direction focuses on establishing a connection between the spike
representations of SNNs, such as firing rates, and equivalent Artificial
Neural Networks (ANNs) activation mappings. This connection enables the ANN-
to-SNN conversion (Diehl et al. 2015; Hunsberger and Eliasmith 2015; Rueckauer
et al. 2017; Rathi et al. 2020), and the optimization of SNNs using gradients
computed from this equivalent mappings (Thiele, Bichler, and Dupret 2020; Wu
et al. 2021; Zhou et al. 2021; Xiao et al. 2021; Meng et al. 2022). These
methods usually require a relatively large number of time-steps to achieve
performance comparable to ANNs, suffering from high latency and usually more
energy consumption. The other direction is to directly train SNNs with back-
propagation (Bohte, Kok, and La Poutre 2000; Esser et al. 2015; Bellec et al.
2018; Huh and Sejnowski 2018), which typically employs the surrogate gradients
(Shrestha and Orchard 2018) method to overcome the non-differentiable nature
of the binary spiking functions and direct train SNNs from scratch. This
follows the backpropagation through time (BPTT) framework. BPTT with surrogate
gradients can achieve extremely low latency, however, it requires large
training memory to maintain the computational graph unfolded over time.

### Dynamic GNNs

Dynamic GNNs have achieved impressive performance in various tasks. With the
help of RNNs (Cho et al. 2014), static GNNs can be extended to model dynamic
processes by employing RNN architectures (Rossi et al. 2020; Pareja et al.
2020; Shi et al. 2021; Rossi et al. 2020; Xu et al. 2019a). TGN (Rossi et al.
2020) and JODIE (Kumar, Zhang, and Leskovec 2019) update the node hidden state
by RNN units for representation learning. EvolveGCN (Pareja et al. 2020) uses
the RNN to regulate the model parameters on each time step. However, the RNN-
based dynamic graph methods could save the historical information for graph
representation, they typically require massive computational costs and memory
consumption. To effectively model the dynamic evolution of graphs while
minimizing computational and memory requirements, we introduce the implicit
models and SNNs into dynamic GNNs.

### Feedback Models

Implicit models are promising approaches to deep learning that utilize
implicit layers to determine the outputs. In contrast to explicit models,
which typically require storing intermediate activations for backpropagation,
implicit models use the fixed-point solution (Bai, Kolter, and Koltun 2019;
Bai, Koltun, and Kolter 2020) to perform backpropagation without saving these
intermediate activations. This results in constant complexity for the implicit
models, which is a significant advantage for large models. DEQ (Bai, Kolter,
and Koltun 2019) demonstrates the ability of implicit models in sequence
modeling. MDEQ (Bai, Koltun, and Kolter 2020) incorporates multiscale modeling
into implicit deep networks, enabling tasks such as image classification and
semantic segmentation. To further enhance the efficiency of implicit models,
(Gu et al. 2020) extend the concept of implicit fixed-point equilibrium to
graph learning, to address the problem of evaluation and training for
recurrent GNNs. (Liu et al. 2022) propose a multiscale graph neural network
with implicit layers to model multiscale information on graphs at multiple
resolutions. Although implicit models have shown promise in various areas,
their application to dynamic spiking graphs is still relatively unexplored.

## Preliminary

### Spiking Neuron Models

SNNs utilize binary activations in each layer, which limits the representation
capacity. To address the issue, SNNs introduce a temporal dimension, known as
latency $K$. In the forward pass of SNNs, inputs are presented as streams of
events and repeated for $K$ time steps to produce the final result. The leaky-
integrate-and-fire (LIF) model is commonly used to describe the dynamics of
spiking neurons. In LIF, each neuron integrates the received spikes as the
membrane potential $\bm{u}_{\tau,i}$, which can be formulated as a first-order
differential equation,

$\textbf{LIF: }\bar{\lambda}\frac{du_{\tau}}{d\tau}=-(u_{\tau}-u_{rest})R\cdot
I(\tau),\quad u_{\tau}<V_{th}\ ,$ (1)

where $I(\tau)$ is the input current, $V_{th}$ is the spiking threshold, and
$R$ and $\bar{\lambda}$ are resistance and time constant, respectively. When
$u_{\tau}$ reaches $V_{th}$ at time $\tau$, a spike is generated and
$u_{\tau}$ is reset to the resting potential $u_{\tau}=u_{rest}$, which is
usually taken as 0. The spike train is expressed by the Dirac delta function:
$s_{\tau}=\sum_{t^{f}}\delta(\tau-t^{f})$. We consider a simple current model
$I_{\tau,i}=\sum_{j}w_{ij}s_{\tau,j}+b,$ where $w_{ij}$ is the weight from
neuron $j$ to neuron $i$. Then, the general form of LIF is described as:

$\left\\{\begin{array}[]{lr}u_{\tau+1,i}=\lambda(u_{\tau,i}-V_{th}s_{\tau,i})+\sum_{j}w_{ij}s_{\tau,j}+b,\\\
s_{\tau+1,i}=\mathbb{H}(u_{\tau+1,i}-V_{th}),\end{array}\right.$ (2)

where $\mathbb{H}(x)$ is the Heaviside step function, which is the non-
differentiable spiking function. $s_{\tau,i}$ is the binary spike train of
neuron $i$, and $\lambda<1$ is a leaky term related to the constant $\tau_{m}$
and discretization time interval used in the LIF model. The constant $R$,
$\bar{\lambda}$, and time step-size are absorbed into the weights $w_{ij}$ and
bias $b$. The training of SNNs follows the process of BPTT, and the gradients
with $K$ time latency steps are calculated with:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\bm{W}^{l}}$
$\displaystyle=\sum_{\tau=1}^{K}\frac{\partial\mathcal{L}}{\partial\bm{s}_{\tau}^{l+1}}\frac{\partial\bm{s}_{\tau}^{l+1}}{\partial\bm{u}_{\tau}^{l+1}}\Bigg{(}\frac{\partial\bm{u}_{\tau}^{l+1}}{\partial\bm{W}^{l}}$
(3)
$\displaystyle+\sum_{k<\tau}\prod_{i=\tau-1}^{k}\left(\frac{\partial\bm{u}_{i+1}^{l+1}}{\partial\bm{u}_{i}^{l+1}}+\frac{\partial\bm{u}_{i+1}^{l+1}}{\partial\bm{s}_{i}^{l+1}}\frac{\partial\bm{s}_{i}^{l+1}}{\partial\bm{u}_{i}^{l+1}}\right)\frac{\partial\bm{u}_{k}^{l+1}}{\partial\bm{W}^{l}}\Bigg{)},$

where $\bm{W}^{l}$ is the trainable matrix on $l$-th layer and $\mathcal{L}$
is the loss. The terms
$\frac{\partial\bm{s}_{\tau}^{l}}{\partial\bm{u}_{\tau}^{l}}$ are non-
differentiable, and surrogate derivatives are typically used instead.

### Dynamic GNNs

Given the dynamic graph $\mathcal{G}=\\{G_{1},\cdots,G_{t},\cdots,G_{T}\\}$
with $T$ time steps. On each snapshot
$G_{t}=(\bm{A}_{t},\bm{X}_{t},\mathcal{V}_{t},\mathcal{E}_{t})$, where
$\bm{A}_{t}$ is the adjacency matrix, $\bm{X}_{t}\in\mathbb{R}^{N\times d}$ is
$N$ node features with dimension $d$,
$\mathcal{V}_{t}=\\{v_{1}^{t},\cdots,v_{N}^{t}\\}$ and $\mathcal{E}_{t}$ are
the set of nodes and edges on time step $t$. Dynamic graph methods typically
extract the graph features on each time step and then model the evolution over
time, which is formulated as:

$\displaystyle\bm{h}_{v}^{t,l}=\mathcal{C}^{t,l}$
$\displaystyle\left(\bm{h}_{v}^{t,l-1},\mathcal{A}^{t,l}\left(\\{\bm{h}_{u}^{t,l-1}\\}_{u\in\mathcal{N}(v)}\right)\right),$
(4) $\displaystyle\bm{h}_{v}^{t+1}$
$\displaystyle=Evo(\bm{h}_{v}^{1,L},\cdots,\bm{h}_{v}^{t,L}),$

where $\mathcal{N}(v)$ is the neighbors of $v$. $\mathcal{A}^{t,l}$ and
$\mathcal{C}^{t,l}$ denote the aggregation and combination operations at the
$l$-th layer on time step $t$, respectively. $L$ is the number of layers of a
graph and $Evo$ means the evolution operation over time $1$ to $t$, which is
typically implemented with RNN (Cho et al. 2014) or LSTM (Hochreiter and
Schmidhuber 1997).

## The Proposed Dy-SIGN

Figure 1: An overview of the proposed Dy-SIGN. The Spiking Graph Neural
Network (SGN) combines the SNNs and GNNs for node representation learning. The
information compensation mechanism transfers the information from the shallow
layer to the last to mitigate the information loss issue. The variation
training method is applied to calculate the fix-point on each time latency in
SGN and is used for dynamic prediction.

### Overview

This paper introduces a novel approach named Dy-SIGN for semi-supervised
dynamic spiking graph node classification. Recognizing that if the time
latency $\tau$ in SNN tends to infinity, models will retain the details
information of input. However, since the large $\tau$ would cause the
vanishing gradient problem (Hochreiter 1998) and significantly increase the
complexity, we propose the information compensation mechanism that bridges the
feature from the beginning to the last layer and includes the shallow
representation in the final embedding. Additionally, we propose a variation of
the training method in dynamic graph, which is proved to be equivalent to the
implicit differentiation. This method simplifies the calculation of gradient,
which relies on the equation of implicit differentiation rather than the
forward procedure, thereby reducing memory consumption. The detailed
illustration of our Dy-SIGN can be seen in Figure 1, and we will introduce Dy-
SIGN in detail.

### Information Compensation Spiking Graph Neural Network

Spiking Graph Network (SGN) (Zhu et al. 2022; Xu et al. 2021) usually applies
the Bernoulli encoding to transform the node representation to the spike
signals for propagation. Specifically, on time step $t\in[1,\cdots,T]$ with
$T$ denotes the length of dynamic graph time window, we have the graph
$G_{t}=(\bm{A}_{t},\bm{X}_{t})$, where $\bm{A}_{t}$ is the adjacency matrix of
$G_{t}$ and $\bm{X}_{t}$ is the node features. SGN first encodes the initial
features into binary signals
$\\{\tilde{\bm{X}}_{t,1},\cdots,\tilde{\bm{X}}_{t,\tau},\cdots,\tilde{\bm{X}}_{t,K}\\}$,
where $\tau\in[1,\cdots,K]$ means the time latency step in SGN, and $K$ is the
length of SGN time latency window. Then, the layer-wise (update on different
layers) spiking graph propagation is defined as:

$\bm{s}_{t,\tau}^{l}=\Phi\left(Cov(\bm{A},\bm{s}_{t,\tau}^{l-1}),\bm{s}_{t,\tau-1}^{l}\right),$
(5)

where $Cov$ is the graph convolutional layer which is the same as Equation 4,
$S_{t,\tau}^{l}$ denotes the nodes spiking features on time step $t$ and time
latency step $\tau$ with $l$-th layer.
$\bm{S}_{t,1}^{1}=\tilde{\bm{X}}_{t,1}$, $\Phi(\cdot)$ is the non-linear
function that combine historical information $\bm{S}_{t,\tau-1}^{l}$ into
current state. After that, the temporal-wise (update on each time latency in
SGN) membrane potentials and firing rate follows:

$\left\\{\begin{array}[]{lr}\bm{u}_{t,\tau+1}^{l}=\lambda\bm{u}_{t,\tau}^{l}(\bm{1}-\bm{s}_{t,\tau}^{l})+\sum\bm{W}^{l}Con(\bm{A},\bm{s}_{t,\tau+1}^{l-1}),\\\
\bm{s}_{t,\tau+1}^{l}=\mathbb{H}(\bm{u}_{t,\tau+1}^{l}-V_{th}).\end{array}\right.$

For the specific node representation
$\bm{x}_{i}=[\bm{x}_{i1},\cdots,\bm{x}_{id}]$, where $\bm{x}_{i}$ denotes the
$i$-th node features of graph, and $d$ is the feature dimension. The spiking
signals are sampled with Bernoulli distribution with $K$ time latency in SGN,
which is denoted as
$\tilde{\bm{x}}_{i}=\\{\tilde{\bm{x}}_{1,i},\cdots,\tilde{\bm{x}}_{\tau,i},\cdots,\tilde{\bm{x}}_{K,i}\\}$
with
$\tilde{\bm{x}}_{\tau,i}=[\tilde{\bm{x}}_{\tau,i1},\cdots,\tilde{\bm{x}}_{\tau,id}]$.
Then we have $P(\tilde{\bm{x}}_{\tau,ij}=1)=\bm{x}_{ij}$ and
$P(\tilde{\bm{x}}_{\tau,ij}=0)=1-\bm{x}_{ij}$. Assume the parameters of each
spike neuron are $w_{i}=[w_{i1},\cdots,w_{id}]$, the combined spike input
$z_{i}$ for the next SGN layer holds:

$\displaystyle z_{\tau,i}=\sum_{j=1}^{d}$ $\displaystyle
w_{ij}\tilde{\bm{x}}_{\tau,ij}^{\tau},$ (6)
$\displaystyle\mathbb{E}(z_{\tau,i})=\sum_{j=1}^{d}w_{ij}$
$\displaystyle\mathbb{E}(\tilde{\bm{x}}_{\tau,ij})=\sum_{j=1}^{d}w_{ij}\bm{x}_{ij}.$

According to (Chung and Lu 2002), the error bound of SGN holds:

$\displaystyle\lim_{\tau\to\infty}P\left(z_{\tau,i}<\mathbb{E}(z_{\tau,i})-\epsilon\right)\leq
e^{-\epsilon^{2}/2\sigma},$ (7)
$\displaystyle\lim_{\tau\to\infty}P\left(z_{\tau,i}>\mathbb{E}(z_{\tau,i})+\epsilon\right)\leq
e^{-\epsilon^{2}/2(\sigma+\hat{w}_{i}\epsilon/3)},$

where $\hat{w}_{i}=max\\{w_{i1},\cdots,w_{id}\\}$. From Equation 7, we observe
that as $\tau\to\infty$, the difference between SGN and GNN will be with the
probability of $p=e^{-\epsilon^{2}/2(\sigma+\hat{w}_{i}\epsilon/3)}$ to exceed
the upper and lower bounds. This reveals that the spiking signals would
preserve the details information of continuous features when $\tau\to\infty$.
However, with the increase of $\tau$, SGN becomes difficult to train and may
suffer from the vanishing gradient problem due to the coefficient of $\lambda$
in Equation Information Compensation Spiking Graph Neural Network. To address
the issue, we design the information compensation mechanism for SGN that
directly transfers features from the first layer to the last layer for node
embeddings. Formally:

$\left\\{\begin{array}[]{lr}\bm{u}^{1}_{\tau}=\lambda\bm{u}^{1}_{\tau}-1+\bm{W}^{1}s^{N}_{\tau}+\bm{F}^{1}\bm{x}_{\tau}-V_{th}s^{1}_{\tau},\\\
\bm{u}^{l}_{\tau}=\lambda\bm{u}^{l}_{\tau-1}+\bm{F}^{l}s^{l-1}_{\tau}-V_{th}s^{l}_{\tau},\quad
l=2,\cdots,N,\end{array}\right.$

where $\bm{u}^{l}_{\tau}$ denotes the neuronal membrane potential at time
$\tau$ on $l$-th layer. $\bm{F}^{l}$ and $\bm{W}^{l}$ are the trainable
parameters on the $l$-th layer, $\bm{F}^{1}$ is the information compensation
matrix. In this way, the information compensation SGN follows the form of
multi-layer structures of feedback model (Bai, Kolter, and Koltun 2019; Bai,
Koltun, and Kolter 2020). This type of structure has several potential
advantages: (1) The forward and backward procedures are decoupled, avoiding
the problem of non-differentiable spiking functions. (2) Using implicit
differentiation in the equilibrium state, we can compute the gradient without
saving the exact forward procedure, thus reducing memory consumption.

### Variation of training SGN

The traditional training method of SGN follows BPTT, which replaces the non-
differentiable term
$\frac{\partial\bm{s}^{l}_{\tau}}{\partial\bm{u}^{l}_{\tau}}$ with the
surrogate derivatives in Equation 3. However, BPTT relies on multiple
backpropagation paths, with would consume a large amount of memory. Similarly
to (Xiao et al. 2022), we set the gradient of the Heaviside step function to
0, which is formulated as
$\frac{\partial\bm{u}^{l}_{\tau+1}}{\partial\bm{s}^{l+1}_{\tau}}\frac{\partial\bm{s}^{l+1}_{\tau}}{\partial{u}^{l+1}_{\tau}}=0$.
Then, the gradient of parameters $\bm{W}^{l}$ is:

$\displaystyle\frac{\partial L}{\partial\bm{W}^{l}}$
$\displaystyle=\sum_{\tau=1}^{K}\frac{\partial
L}{\partial\bm{s}^{l+1}_{\tau}}\frac{\partial\bm{s}^{l+1}_{\tau}}{\partial\bm{u}^{l+1}_{\tau}}\left(\sum_{k\leq\tau}\lambda^{\tau-k}\frac{\partial\bm{u}^{l+1}_{k}}{\partial\bm{W}^{l}}\right)$
(8) $\displaystyle=\sum_{\tau=1}^{K}\frac{\partial
L}{\partial\bm{s}^{l+1}_{\tau}}\frac{\partial\bm{s}^{l+1}_{\tau}}{\partial\bm{u}^{l+1}_{\tau}}\left(\sum_{k\leq\tau}\lambda^{\tau-k}\bm{s}^{l}_{k}\right).$

During the forward procedure is
$\hat{\bm{a}}^{l}_{\tau+1}=\lambda\hat{\bm{a}}^{l}_{\tau}+\bm{s}^{l}_{\tau+1}$,
where the presynaptic activities can be denoted as
$\hat{\bm{a}}^{l}_{\tau+1}=\sum_{k\leq\tau}\lambda^{\tau-k}\bm{s}^{l}_{k}$. By
calculating the gradient of $\frac{\partial
L}{\partial\bm{s}^{l+1}_{\tau}}\frac{\partial\bm{s}^{l+1}_{\tau}}{\partial\bm{u}^{l+1}_{\tau}}$,
we can directly compute the value of Equation 8 without considering the
backpropagation through
$\frac{\partial\bm{u}^{l+1}_{\tau+1}}{\partial\bm{u}^{l+1}_{\tau}}$, which
would decrease the complexity and memory consumption of the model. On each
time latency $\tau$, the output is denoted as $\hat{\bm{a}}^{l+1}_{\tau}$, and
the output of the SGN is
$\\{\hat{\bm{a}}^{l}_{1},\cdots,\hat{\bm{a}}^{l}_{\tau},\cdots,\hat{\bm{a}}^{l}_{K}\\}$
on $l$-th layer.

### Comparing with Feedback Model

Note that, in the spiking dynamic graph framework, $t\in[1,\cdots,T]$ stands
for the time steps of each graph, and $\tau\in[1,\cdots,K]$ is the time
latency in SGN. At each time step $t$, we apply the information compensation
SGN to extract the graph features as Equation Information Compensation Spiking
Graph Neural Network. In the traditional feedback model, the weighted average
firing rate and inputs are denoted as
$\bm{a}_{t,K}=\frac{\sum_{\tau=1}^{K}\lambda^{K-\tau}\bm{s}_{t,\tau}}{\sum_{\tau=1}^{K}\lambda^{K-\tau}}$
and
$\bar{\bm{x}}_{t,K}=\frac{\sum_{\tau=0}^{K}\lambda^{K-\tau}\bm{x}_{t,\tau}}{\sum_{\tau=0}^{K}\lambda^{K-\tau}}$,
where $\bm{s}_{t,\tau}$ and $\bm{x}_{t,\tau}$ denote the firing rate and input
on time latency $\tau$ in SGN and on time step $t$ of dynamic graph. The LIF
model approximate an equilibrium point $\bm{a}^{\star}_{t}$ that satisfies
$\bm{a}^{\star}_{t}=\sigma\left(\frac{1}{V_{th}}(\bm{Wa^{\star}}_{t}+\bm{Fx^{\star}}_{t})\right)$.
The characteristic is similar to the presynaptic activities
$\hat{\bm{a}}^{l}_{t,\tau+1}=\sum_{k\leq\tau}\lambda^{\tau-k}\bm{s}^{l}_{t,k}$.
Considering the last time latency $K$ on layer $l$, we have
$\hat{\bm{a}}^{l}_{t,K}=\sum_{\tau=1}^{K}\lambda^{K-\tau}\bm{s}^{l}_{t,\tau}$,
which equals to $C\bm{a}_{t,K}^{l}$ with
$C=\sum_{\tau=1}^{K}\lambda^{K-\tau}$. In other words, the fire rate
$\hat{\bm{a}}$ in Variation of training SGN at the last time latency step $K$
on layer $l$ is equivalent to the traditional feedback model. Thus, we prove
that the variation of training SGN is equivalent to the traditional feedback
model, and we can calculate the gradient of parameters with Equation 8
directly.

Algorithm 1 Learning Algorithm of Dy-SIGN

Input: Dynamic graph $\mathcal{G}=\\{G_{1},\cdots,G_{T}\\}$; Label $\bm{y}$;
Network parameters $\bm{\theta}$; Network layers $L$; Time latency of SGN $K$.

Output: Trained model parameters $\bm{\theta}$.

1: Initialize $\bm{\theta}$.

2: // Forward:

3: for $t=1,\cdots,T$ do

4: for $l=1,\cdots,L$ do

5: Calculate the average firing rate $\bm{a}_{t,K}^{l}$ with Equation 10;

6: Collect the fixed point representation $\bm{a}_{t}^{\star}$ on layer $L$
and time step $t$;

7: end for

8: end for

9: Calculate the output of Dy-SIGN $\hat{\bm{y}}$ and the loss $\mathcal{L}$
with Equation 11;

10: // Backward:

11: for $l=L,\cdots,1$ do

12: Calculate the gradient of SGN with Equation 8;

13: Update the parameters $\bm{\theta}$.

14: end for

### Dynamic Spiking Graph Neural Network

The proposed information compensation SGN is designed for a fixed time step
$t$. However, since the graphs may change over time, it remains a challenge to
integrate the temporal dynamics of SGN with dynamic graphs. We propose a novel
method by propagating the medium
$\bm{a}_{t,K}=\\{\bm{a}_{t,K}^{1},\cdots,\bm{a}_{t,K}^{l},\cdots,\bm{a}_{t,K}^{L}\\}$
at different time steps. At time step $t+1$, we set the initial membrane
potential to $\bm{u}_{t+1,1}=\bm{a}_{t,K}$, and the update process of membrane
potentials is:

$\displaystyle\bm{u}_{t+1,K}$
$\displaystyle=\lambda\bm{u}_{t+1,K-1}+\bm{Ws}_{t+1,K-1}$ (9)
$\displaystyle+\bm{Fx}_{t+1,K-1}-V_{th}\bm{s}_{t+1,K}.$

The average firing rates is defined as
$\bm{a}_{t+1,K}=\frac{\sum_{\tau=1}^{K}\lambda^{K-\tau}\bm{s}_{t+1,\tau}}{\sum_{\tau=1}^{K}\lambda^{K-\tau}}$,
the average inputs as
$\bar{\bm{x}}_{t+1,K}=\frac{\sum_{\tau=0}^{K}\lambda^{K-\tau}\bm{x}_{t+1,\tau}}{\sum_{\tau=0}^{K}\lambda^{K-\tau}}$,
and $\bm{u}_{t+1,1}=\bm{a}_{t,K}$, $\bm{s}_{t+1,1}=\bm{0}$. Then, we have:

$\displaystyle\bm{a}_{t+1,K}=$
$\displaystyle\frac{1}{V_{th}}\Bigg{(}\frac{\sum_{i=0}^{K-2}\lambda^{i}}{\sum_{i=0}^{K-1}\lambda^{i}}\bm{W}\bm{a}_{t+1,K-1}+\bm{F}\bar{\bm{x}}_{t+1,K-1}$
(10)
$\displaystyle-\frac{\bm{u}_{t+1,K}}{\sum_{i=0}^{K-1}\lambda^{i}}+\frac{\bm{a}_{t,K}}{\sum_{i=0}^{K-1}\lambda^{i}}\Bigg{)}$
$\displaystyle=$
$\displaystyle\sigma\left(\frac{1}{V_{th}}\left(\frac{\sum_{i=0}^{K-2}\lambda^{i}}{\sum_{i=0}^{K-1}\lambda^{i}}\bm{W}\bm{a}_{t+1,K-1}+\bm{F}\bar{\bm{x}}_{t+1,K-1}\right)\right)$
$\displaystyle-\frac{\bm{u}_{t+1,K}}{V_{th}\sum_{i=0}^{K-1}\lambda^{i}}+\frac{\bm{a}_{t,K}}{V_{th}\sum_{i=0}^{K-1}\lambda^{i}},$

where $\sigma(x)=\left\\{\begin{aligned} &1,&x>1\\\ &x,&0\leq x\leq 1\\\
&0,&x<0\end{aligned}\right.$. For each time step $t$, we will first calculate
the equilibrium point $\bm{a}_{t,K}\to\bm{a}_{t}^{\star}$, which is fixed for
time step $t+1$. Therefore, the LIF model gradually approximates the
equilibrium point $\bm{a}^{\star}_{t}$ that satisfies
$\bm{a}_{t}^{\star}=\sigma\left(\frac{1}{V_{th}}(\bm{W}\bm{a}^{\star}_{t}+\bm{Fx}^{\star}_{t})\right)$
with $\bar{\bm{x}}_{t,K}\to\bm{x}^{\star}_{t}$.

Table 1: Macro-F1 score comparisons on three dynamic graph datasets with different training ratios. The results are averaged over five runs, and the best results are in boldface. - denotes time-consuming. Methods | DBLP | Tmall | Patent
---|---|---|---
40% | 60% | 80% | 40% | 60% | 80% | 40% | 60% | 80%
DeepWalk | 67.08 | 67.17 | 67.12 | 49.09 | 49.29 | 49.53 | 72.32$\pm$0.9 | 72.25$\pm$1.2 | 72.05$\pm$1.1
Node2Vec | 66.07 | 66.81 | 66.93 | 54.37 | 54.55 | 54.58 | 69.01$\pm$0.9 | 69.08$\pm$0.9 | 68.99$\pm$1.0
HTNE | 67.68 | 68.24 | 68.36 | 54.81 | 54.89 | 54.93 | - | - | -
M2DNE | 69.02 | 69.48 | 69.75 | 57.75 | 57.99 | 58.47 | - | - | -
DyTriad | 60.45 | 64.77 | 66.42 | 44.98 | 48.97 | 51.16 | - | - | -
MPNN | 64.19$\pm$0.4 | 63.91$\pm$0.3 | 65.05$\pm$0.5 | 47.71$\pm$0.8 | 47.78$\pm$0.7 | 50.27$\pm$0.5 | - | - | -
JODIE | 66.73$\pm$1.0 | 67.32$\pm$1.1 | 67.53$\pm$1.3 | 52.62$\pm$0.8 | 54.02$\pm$0.6 | 54.17$\pm$0.2 | 77.57$\pm$0.8 | 77.69$\pm$0.6 | 77.67$\pm$0.4
EvolveGCN | 67.22$\pm$0.3 | 69.78$\pm$0.8 | 71.20$\pm$0.7 | 53.02$\pm$0.7 | 54.99$\pm$0.7 | 55.78$\pm$0.6 | 79.67$\pm$0.4 | 79.76$\pm$0.5 | 80.13$\pm$0.4
TGAT | 71.18$\pm$0.4 | 71.74$\pm$0.5 | 72.15$\pm$0.3 | 56.90$\pm$0.6 | 57.61$\pm$0.7 | 58.01$\pm$0.7 | 81.51$\pm$0.4 | 81.56$\pm$0.6 | 81.57$\pm$0.5
SpikeNet | 70.88$\pm$0.4 | 71.98$\pm$0.3 | 74.64$\pm$0.5 | 58.84$\pm$0.4 | 61.13$\pm$0.8 | 62.40$\pm$0.6 | 83.53$\pm$0.6 | 83.85$\pm$0.7 | 83.90$\pm$0.6
Dy-SIGN | 70.94$\pm$0.1 | 72.07$\pm$0.1 | 74.67$\pm$0.5 | 57.48$\pm$0.1 | 60.94$\pm$0.2 | 61.89$\pm$0.1 | 83.57$\pm$0.3 | 83.77$\pm$0.2 | 83.91$\pm$0.2
Table 2: Micro-F1 score comparisons on three dynamic graph datasets with different training ratios. The results are averaged over five runs, and the best results are in boldface. - denotes time-consuming. Methods | DBLP | Tmall | Patent
---|---|---|---
40% | 60% | 80% | 40% | 60% | 80% | 40% | 60% | 80%
DeepWalk | 66.53 | 66.89 | 66.38 | 57.11 | 57.34 | 57.88 | 71.57$\pm$1.3 | 71.53$\pm$1.0 | 71.38$\pm$1.2
Node2Vec | 66.80 | 67.37 | 67.31 | 60.41 | 60.56 | 60.66 | 69.01$\pm$0.9 | 69.08$\pm$0.9 | 68.99$\pm$1.0
HTNE | 67.68 | 68.24 | 68.36 | 54.81 | 54.89 | 54.93 | - | - | -
M2DNE | 69.23 | 69.47 | 69.71 | 64.21 | 64.38 | 64.65 | - | - | -
DyTriad | 65.13 | 66.80 | 66.95 | 53.24 | 56.88 | 60.72 | - | - | -
MPNN | 65.72$\pm$0.4 | 66.79$\pm$0.6 | 67.74$\pm$0.3 | 57.82$\pm$0.7 | 57.66$\pm$0.5 | 58.07$\pm$0.6 | - | - | -
JODIE | 68.44$\pm$0.6 | 68.51$\pm$0.8 | 68.80$\pm$0.9 | 58.36$\pm$0.5 | 60.28$\pm$0.3 | 60.49$\pm$0.3 | 77.64$\pm$0.7 | 77.89$\pm$0.5 | 77.93$\pm$0.4
EvolveGCN | 69.12$\pm$0.8 | 70.43$\pm$0.6 | 71.32$\pm$0.5 | 59.96$\pm$0.7 | 61.19$\pm$0.6 | 61.77$\pm$0.6 | 79.39$\pm$0.5 | 79.75$\pm$0.3 | 80.01$\pm$0.3
TGAT | 71.10$\pm$0.2 | 71.85$\pm$0.4 | 73.12$\pm$0.3 | 62.05$\pm$0.5 | 62.92$\pm$0.4 | 63.32$\pm$0.7 | 80.79$\pm$0.7 | 80.81$\pm$0.6 | 80.93$\pm$0.6
SpikeNet | 71.98$\pm$0.3 | 72.35$\pm$0.8 | 74.86$\pm$0.5 | 63.52$\pm$0.7 | 64.84$\pm$0.4 | 66.10$\pm$0.3 | 83.48$\pm$0.8 | 83.80$\pm$0.7 | 83.88$\pm$0.9
Dy-SIGN | 71.90$\pm$0.1 | 72.61$\pm$0.4 | 74.96$\pm$0.2 | 62.93$\pm$0.3 | 64.10$\pm$0.3 | 65.82$\pm$0.2 | 83.50$\pm$0.2 | 83.47$\pm$0.1 | 83.90$\pm$0.2

Finally, we have fixed point representation over time, i.e.,
$\bm{a}^{\star}=\\{\bm{a}_{1}^{\star},\cdots,\bm{a}_{t}^{\star},\cdots,\bm{a}_{T}^{\star}\\}$.
We concatenate all the embeddings for the final node classification, which is
formulated as:

$\hat{\bm{y}}=FC(||_{t=1}^{T}\bm{a}^{\star}),\quad\mathcal{L}=-\sum_{r\in\bm{y}_{L}}\bm{y}_{r}ln\hat{\bm{y}}_{r},$
(11)

where $||$ denotes the concatenation operation, $FC$ is the fully connect
layer, $\hat{\bm{y}}$ is the nodes prediction, $\bm{y}$ is the ground-truth
labels, $\bm{y}_{L}$ means the set of labeled nodes, and $\mathcal{L}$ is the
cross-entropy loss.

In summary, the proposed dynamic spiking graph neural network has several
advantages. Firstly, $\bm{a}^{\star}_{t}$ is the fixed point of each time step
on different layers, which can be considered as the hidden embeddings of time
step $t$. Compared to traditional RNN-based dynamic graph methods, directly
propagating $\bm{a}^{\star}_{t}$ over time would reduce the computational cost
of calculating the hidden states and lower the model complexity. Secondly,
traditional static feedback models usually set the initial states to $0$,
which cannot meet the requirement of dynamic graphs. By sending the previous
state to the next time step, the model is able to capture the long temporal
dependency for prediction. The detailed algorithm is shown in Algorithm 1.

Figure 2: (a) The memory consumption of SpikeNet and Dy-SIGN on DBLP dataset.
(b) The training time of different methods. (c) Hyperparameter sensitivity
analysis of $\lambda$.

## Experiments

### Experimental Settings

To verify the effectiveness of the proposed Dy-SIGN, we conduct experiments on
three large real-world graph datasets, i.e., DBLP (Lu et al. 2019), Tmall (Lu
et al. 2019) and Patent (Hall, Jaffe, and Trajtenberg 2001). The statistics
and details introduction are presented in Appendix. We compared Dy-SIGN with
various competing methods, including two static graph methods ( i.e., DeepWalk
(Perozzi, Al-Rfou, and Skiena 2014) and Node2Vec (Grover and Leskovec 2016)),
seven dynamic graph methods (i.e., HTNE (Zuo et al. 2018), M2DNE (Lu et al.
2019), DyTriad (Zhou et al. 2018), MPNN (Panagopoulos, Nikolentzos, and
Vazirgiannis 2021), JODIE (Kumar, Zhang, and Leskovec 2019), EvolveGCN (Pareja
et al. 2020) and TGAT (da Xu et al. 2020)), and one spiking method SpikeNet
(Li et al. 2023). The details are introduced in Appendix. As for the
implementation, we follow the same settings with (Li et al. 2023) and report
the Macro-F1 and Micro-F1 results under different training ratios (i.e., 40%,
60%, and 80%). Besides, we use 5% for validation. The hidden dimension of all
the methods is set to 128, and the batch size to 1024. The total training
epochs are 100 and the learning rate is 0.001.

### Performance Comparison

Comparison of Performance. The classification results on different datasets
under various training ratios are presented in Table 1 and 2. From the
results, we find that the proposed Dy-SIGN achieves competitive performance
compared with other methods. From the results, we have the following
observations: (1) The static methods DeepWalk and Node2Vec perform worse than
the others, which indicates that simply applying static graph methods to
dynamic graphs ignores the contribution of historical information for
representation. (2) The methods HTNE, M2DNE, DyTrid, and MPNN fail to learn
meaningful representations on the large-scale Patent dataset. The potential
reason behind this is the high computational complexity of these models,
leading to extensive time consumption for both training and prediction. This
makes it impractical to achieve competitive performance within an acceptable
time frame. (3) Although the spiking methods SpikeNet and Dy-SIGN apply the
binary information for representation learning, the performance is still
better than JODIE, EvolveGCN, and TGAT. This phenomenon indicates that the
spike-based learning method is also competitive in both computational
complexity and performance compared to the traditional methods. (4) The
proposed Dy-SIGN outperforms SpikeNet in half of the settings. However, as
shown in Figure 2 (a), the memory consumption of Dy-SIGN is about only half of
SpikeNet under the same experiment environment, demonstrating the superiority
of Dy-SIGN. We attribute this to the fact that by applying the variation
training method, Dy-SIGN achieves more efficient results.

Comparison of Runtime Complexity. We further compare the runtime complexity
between Dy-SIGN with SpikeNet, JODIE, EvolveGCN, and TGAT, which is shown in
Figure 2 (b). From the results, we find that the SNN-based methods are
significantly more efficient than the ANNs methods. The reason for this is
attributed to the fact that SNN-based methods use binary signals instead of
continuous features, allowing the matrix multiplication operation to be
replaced by an accumulation operation. Additionally, the proposed Dy-SIGN is
slightly more efficient than SpikeNet method, the potential reason is that Dy-
SIGN uses the simple form to calculate the gradient with Equation 8, ignoring
the time-consuming calculation of gradient in BPTT.

### Sensitivity Analysis

In this section, we examine the impact of hyperparameters on the performance
of our proposed Dy-SIGN. Specifically, the parameter $\lambda$ determines the
amount of information retained for the next time latency step representation.
We test the values of $\lambda$ in the range of {0.6, 0.7, 0.8, 0.9, 1, 1.1,
1.2} with other parameters fixed to determine the optimal value. The results
are depicted in Figure 2 (c). From the results, we observe that as the value
of $\lambda$ increases, the performance initially improves and then gradually
declines. We attribute the reason to the fact that the smaller $\lambda$
cannot provide sufficient historical information for effective representation
learning. On the other hand, larger $\lambda$ may lead to worse performance
since the current time latency step representation may be influenced by too
much historical information. Thus, we set the default value of $\lambda$ to 1.

## Conclusion

In this paper, we study the problem of combining SNNs with dynamic graphs
using implicit differentiation for node classification and propose a novel
method named Dy-SIGN. To tackle the issue of information loss on graph
structure and details during SNN propagation, we propose an information
compensation mechanism. This mechanism passes the original structure and
features to the last layer of the network, which then participates in node
representation learning. This structure is very similar to traditional
feedback models. Based on this, we use explicit differentiation and a
variation training method to address the issue of high memory consumption in
the combination of SNNs and dynamic graphs. Extensive experiments on real-
world large-scale datasets validate the superiority of the proposed Dy-SIGN.

Broader Impacts and Limitations. Our research combines SNNs with dynamic
graphs to address the issue of high memory consumption in dynamic graphs, as
well as mitigate the problem of information loss in SNNs. This research is
expected to expand a new branch in the field of GNNs: how to efficiently train
and deploy GNN models in low-power and low-memory environments. The limitation
of this work is that the study of combining SNNs and GNNs is still in the
early stages, and how to effectively merge the two still needs to be explored.
Furthermore, the method of implicit differentiation is merely one of the
means, and exploring efficient solutions remains highly pressing.

## Acknowledgments

This work is part of the research project “ENERGY-BASED PROBING FOR SPIKING
NEURAL NETWORKS”, performed at Mohamed bin Zayed University of Artificial
Intelligence (MBZUAI), funded by Technology Innovation Institute (TII)
(Contract No. TII/ARRC/2073/2021).

## References

  * Bai, Kolter, and Koltun (2019) Bai, S.; Kolter, J. Z.; and Koltun, V. 2019.  Deep equilibrium models.  In _NeurIPS_.
  * Bai, Koltun, and Kolter (2020) Bai, S.; Koltun, V.; and Kolter, J. Z. 2020.  Multiscale deep equilibrium models.  In _NeurIPS_.
  * Bellec et al. (2018) Bellec, G.; Salaj, D.; Subramoney, A.; Legenstein, R.; and Maass, W. 2018.  Long short-term memory and learning-to-learn in networks of spiking neurons.  In _NeurIPS_ , volume 31.
  * Bohte, Kok, and La Poutre (2000) Bohte, S.; Kok, J.; and La Poutre, H. 2000.  SpikeProp: Backpropagation for Networks of Spiking Neurons Error-Backpropagation in a Network of Spik-ing Neurons.  In _ESANN_.
  * Brette et al. (2007) Brette, R.; Rudolph, M.; Carnevale, T.; Hines, M.; Beeman, D.; Bower, J. M.; Diesmann, M.; Morrison, A.; Goodman, P. H.; Harris, F. C.; et al. 2007.  Simulation of networks of spiking neurons: a review of tools and strategies.  _Journal of computational neuroscience_ , 23: 349–398.
  * Bu et al. (2022) Bu, T.; Ding, J.; Yu, Z.; and Huang, T. 2022.  Optimized potential initialization for low-latency spiking neural networks.  In _AAAI_.
  * Chen, Li, and Tang (2020) Chen, S.; Li, Z.; and Tang, Z. 2020.  Relation R-CNN: A Graph Based Relation-Aware Network for Object Detection.  _IEEE Signal Processing Letters_ , 27.
  * Cho et al. (2014) Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Chung and Lu (2002) Chung, F.; and Lu, L. 2002.  Connected components in random graphs with given expected degree sequences.  _Annals of combinatorics_ , 6(2): 125–145.
  * da Xu et al. (2020) da Xu; chuanwei ruan; evren korpeoglu; sushant kumar; and kannan achan. 2020.  Inductive representation learning on temporal graphs.  In _ICLR_.
  * Diehl et al. (2015) Diehl, P. U.; Neil, D.; Binas, J.; Cook, M.; Liu, S.-C.; and Pfeiffer, M. 2015.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.  In _IJCNN_.
  * Esser et al. (2015) Esser, S. K.; Appuswamy, R.; Merolla, P.; Arthur, J. V.; and Modha, D. S. 2015.  Backpropagation for energy-efficient neuromorphic computing.  In _NeurIPS_ , volume 28.
  * Fey and Lenssen (2019) Fey, M.; and Lenssen, J. E. 2019.  Fast graph representation learning with PyTorch Geometric.  _arXiv preprint arXiv:1903.02428_.
  * Grover and Leskovec (2016) Grover, A.; and Leskovec, J. 2016.  node2vec: Scalable feature learning for networks.  In _KDD_.
  * Gu et al. (2020) Gu, F.; Chang, H.; Zhu, W.; Sojoudi, S.; and El Ghaoui, L. 2020.  Implicit graph neural networks.  _NeurIPS_.
  * Hall, Jaffe, and Trajtenberg (2001) Hall, B. H.; Jaffe, A. B.; and Trajtenberg, M. 2001.  The NBER patent citation data file: Lessons, insights and methodological tools.
  * Hochreiter (1998) Hochreiter, S. 1998.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ , 6(02): 107–116.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural computation_ , 9(8): 1735–1780.
  * Huh and Sejnowski (2018) Huh, D.; and Sejnowski, T. J. 2018.  Gradient descent for spiking neural networks.  In _NeurIPS_.
  * Hunsberger and Eliasmith (2015) Hunsberger, E.; and Eliasmith, C. 2015.  Spiking deep networks with LIF neurons.  _arXiv preprint arXiv:1510.08829_.
  * Kim et al. (2020) Kim, S.; Park, S.; Na, B.; and Yoon, S. 2020.  Spiking-YOLO: spiking neural network for energy-efficient object detection.  In _AAAI_.
  * Kumar, Zhang, and Leskovec (2019) Kumar, S.; Zhang, X.; and Leskovec, J. 2019.  Predicting dynamic embedding trajectory in temporal interaction networks.  In _KDD_.
  * Li et al. (2023) Li, J.; Yu, Z.; Zhu, Z.; Chen, L.; Yu, Q.; Zheng, Z.; Tian, S.; Wu, R.; and Meng, C. 2023.  Scaling Up Dynamic Graph Representation Learning via Spiking Neural Networks.  In _AAAI_.
  * Liu et al. (2022) Liu, J.; Hooi, B.; Kawaguchi, K.; and Xiao, X. 2022.  MGNNI: Multiscale Graph Neural Networks with Implicit Layers.  In _NeurIPS_.
  * Lu et al. (2019) Lu, Y.; Wang, X.; Shi, C.; Yu, P. S.; and Ye, Y. 2019.  Temporal network embedding with micro-and macro-dynamics.  In _CIKM_.
  * Maass (1997) Maass, W. 1997.  Networks of spiking neurons: the third generation of neural network models.  _Neural networks_ , 10(9): 1659–1671.
  * Meng et al. (2022) Meng, Q.; Xiao, M.; Yan, S.; Wang, Y.; Lin, Z.; and Luo, Z.-Q. 2022.  Training high-performance low-latency spiking neural networks by differentiation on spike representation.  In _CVPR_.
  * Mi and Chen (2020) Mi, L.; and Chen, Z. 2020.  Hierarchical graph attention network for visual relationship detection.  In _CVPR_.
  * Panagopoulos, Nikolentzos, and Vazirgiannis (2021) Panagopoulos, G.; Nikolentzos, G.; and Vazirgiannis, M. 2021.  Transfer graph neural networks for pandemic forecasting.  In _AAAI_.
  * Pareja et al. (2020) Pareja, A.; Domeniconi, G.; Chen, J.; Ma, T.; Suzumura, T.; Kanezashi, H.; Kaler, T.; Schardl, T.; and Leiserson, C. 2020.  Evolvegcn: Evolving graph convolutional networks for dynamic graphs.  In _AAAI_.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in pytorch.
  * Perozzi, Al-Rfou, and Skiena (2014) Perozzi, B.; Al-Rfou, R.; and Skiena, S. 2014.  Deepwalk: Online learning of social representations.  In _KDD_ , 701–710.
  * Pfeiffer and Pfeil (2018) Pfeiffer, M.; and Pfeil, T. 2018.  Deep learning with spiking neurons: Opportunities and challenges.  _Frontiers in neuroscience_ , 12: 774.
  * Rathi et al. (2020) Rathi, N.; Srinivasan, G.; Panda, P.; and Roy, K. 2020.  Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation.  In _ICML_.
  * Rossi et al. (2020) Rossi, E.; Chamberlain, B.; Frasca, F.; Eynard, D.; Monti, F.; and Bronstein, M. 2020.  Temporal graph networks for deep learning on dynamic graphs.  _arXiv preprint arXiv:2006.10637_.
  * Rueckauer et al. (2017) Rueckauer, B.; Lungu, I.-A.; Hu, Y.; Pfeiffer, M.; and Liu, S.-C. 2017.  Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.  _Frontiers in neuroscience_ , 11: 682.
  * Sak, Senior, and Beaufays (2014) Sak, H.; Senior, A.; and Beaufays, F. 2014.  Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition.  _arXiv preprint arXiv:1402.1128_.
  * Scarselli et al. (2008) Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagenbuchner, M.; and Monfardini, G. 2008.  The graph neural network model.  _IEEE Transactions on Neural Networks_ , 20(1): 61–80.
  * Schlichtkrull et al. (2018) Schlichtkrull, M.; Kipf, T. N.; Bloem, P.; Van Den Berg, R.; Titov, I.; and Welling, M. 2018.  Modeling relational data with graph convolutional networks.  In _European Semantic Web Conference_.
  * Schliebs and Kasabov (2013) Schliebs, S.; and Kasabov, N. 2013.  Evolving spiking neural network—a survey.  _Evolving Systems_ , 4: 87–98.
  * Shi et al. (2021) Shi, M.; Huang, Y.; Zhu, X.; Tang, Y.; Zhuang, Y.; and Liu, J. 2021.  GAEN: graph attention evolving networks.  In _IJCAI_.
  * Shrestha and Orchard (2018) Shrestha, S. B.; and Orchard, G. 2018.  Slayer: Spike layer error reassignment in time.  In _NeurIPS_.
  * Thiele, Bichler, and Dupret (2020) Thiele, J. C.; Bichler, O.; and Dupret, A. 2020.  SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes.  In _ICML_.
  * Wieder et al. (2020) Wieder, O.; Kohlbacher, S.; Kuenemann, M.; Garon, A.; Ducrot, P.; Seidel, T.; and Langer, T. 2020.  A compact review of molecular property prediction with graph neural networks.  _Drug Discovery Today: Technologies_ , 37: 1–12.
  * Wu et al. (2021) Wu, H.; Zhang, Y.; Weng, W.; Zhang, Y.; Xiong, Z.; Zha, Z.-J.; Sun, X.; and Wu, F. 2021.  Training spiking neural networks with accumulated spiking flow.  In _AAAI_.
  * Wu et al. (2022) Wu, S.; Sun, F.; Zhang, W.; Xie, X.; and Cui, B. 2022.  Graph neural networks in recommender systems: a survey.  _ACM Computing Surveys_ , 55(5): 1–37.
  * Xiao et al. (2022) Xiao, M.; Meng, Q.; Zhang, Z.; He, D.; and Lin, Z. 2022.  Online Training Through Time for Spiking Neural Networks.  In _NeurIPS_.
  * Xiao et al. (2021) Xiao, M.; Meng, Q.; Zhang, Z.; Wang, Y.; and Lin, Z. 2021.  Training feedback spiking neural networks by implicit differentiation on the equilibrium state.  In _NeurIPS_.
  * Xu et al. (2019a) Xu, D.; Cheng, W.; Luo, D.; Liu, X.; and Zhang, X. 2019a.  Spatio-Temporal Attentive RNN for Node Classification in Temporal Attributed Graphs.  In _IJCAI_.
  * Xu et al. (2019b) Xu, H.; Jiang, C.; Liang, X.; and Li, Z. 2019b.  Spatial-aware graph relation network for large-scale object detection.  In _CVPR_.
  * Xu et al. (2021) Xu, M.; Wu, Y.; Deng, L.; Liu, F.; Li, G.; and Pei, J. 2021.  Exploiting spiking dynamics with spatial-temporal feature normalization in graph learning.  In _IJCAI_.
  * Yin et al. (2022a) Yin, N.; Feng, F.; Luo, Z.; Zhang, X.; Wang, W.; Luo, X.; Chen, C.; and Hua, X.-S. 2022a.  Dynamic hypergraph convolutional network.  In _ICDE_.
  * Yin et al. (2022b) Yin, N.; Shen, L.; Li, B.; Wang, M.; Luo, X.; Chen, C.; Luo, Z.; and Hua, X.-S. 2022b.  DEAL: An Unsupervised Domain Adaptive Framework for Graph-level Classification.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 3470–3479.
  * Yin et al. (2023a) Yin, N.; Shen, L.; Wang, M.; Lan, L.; Ma, Z.; Chen, C.; Hua, X.-S.; and Luo, X. 2023a.  CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification.  _arXiv preprint arXiv:2306.04979_.
  * Yin et al. (2023b) Yin, N.; Shen, L.; Xiong, H.; Gu, B.; Chen, C.; Hua, X.-S.; Liu, S.; and Luo, X. 2023b.  Messages are never propagated alone: Collaborative hypergraph neural network for time-series forecasting.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_, (01): 1–15.
  * Zhou et al. (2018) Zhou, L.; Yang, Y.; Ren, X.; Wu, F.; and Zhuang, Y. 2018.  Dynamic network embedding by modeling triadic closure process.  In _AAAI_.
  * Zhou et al. (2021) Zhou, S.; Li, X.; Chen, Y.; Chandrasekaran, S. T.; and Sanyal, A. 2021.  Temporal-coded deep spiking neural network with easy training and robust performance.  In _AAAI_.
  * Zhu et al. (2022) Zhu, Z.; Peng, J.; Li, J.; Chen, L.; Yu, Q.; and Luo, S. 2022.  Spiking Graph Convolutional Networks.  In _IJCAI_.
  * Zuo et al. (2018) Zuo, Y.; Liu, G.; Lin, H.; Guo, J.; Hu, X.; and Wu, J. 2018.  Embedding temporal network via neighborhood formation.  In _KDD_.

Table 3: The specific statistics of the experimental datasets. Dataset | #Nodes | #Edges | #Classes | #Time steps
---|---|---|---|---
DBLP | 28,085 | 236,894 | 10 | 27
Tmall | 577,314 | 4,807,545 | 5 | 186
Patent | 2,738,012 | 13,960,811 | 6 | 25

##  Appendix A Datasets

In our work, we conduct our experiments on three public large-scale graph
datasets, i.e., DBLP (Lu et al. 2019), Tmall (Lu et al. 2019) and Patent
(Hall, Jaffe, and Trajtenberg 2001). The statistics of the datasets are
presented in Table 3, and the details introduction are as follows:

  * •

DBLP. DBLP (Lu et al. 2019) is an academic co-author graph that is derived
from the extensive bibliography website. This dataset is widely used in
research to study various aspects of academic collaborations and knowledge
dissemination within the scientific community. In the DBLP graph, each node
represents an author, while edges between nodes indicate collaborative
relationships between pairs of authors who have jointly published papers
together.

  * •

Tmall. Tmall (Lu et al. 2019) is a bipartite graph that is constructed from
sales data gathered in 2014 from the popular e-commerce platform Tmall.com.
This dataset captures the interactions between users and items, where each
node represents either a user or an item, and each edge corresponds to a
purchase transaction along with a timestamp indicating the time of the
purchase. In the Tmall dataset, users are connected to items based on their
purchasing behavior. This rich interaction data provides valuable insights
into user preferences, item popularity, and the dynamics of e-commerce
transactions. The dataset is particularly interesting for studying
recommendation systems, personalized marketing, and understanding user-item
interactions in an online retail environment.

  * •

Patent. The Patent dataset (Hall, Jaffe, and Trajtenberg 2001) is a citation
network consisting of United States patents spanning the years 1963 to 1999.
This dataset provides a valuable resource for studying the interconnections
and relationships between patents in various domains. Each node in the Patent
dataset represents a patent, and the edges between the nodes indicate citation
relationships, where one patent cites another. This citation network captures
the flow of knowledge and innovation within the patent system. The dataset
contains patents from diverse technological domains and covers a wide range of
topics and industries.

##  Appendix B Baselines

In this section, we compared the proposed Dy-SIGN with a wide range of
competing approaches, the details introduction is as follows:

Static Graph Methods. We compare the proposed Dy-SIGN with two widely used
static graph methods, i.e., DeepWalk (Perozzi, Al-Rfou, and Skiena 2014) and
Node2Vec (Grover and Leskovec 2016). For these baselines, we set the drop rate
in the range of {0.1,0.3,0.5,0.7} to select the best performance, the
embedding size of all methods is set to 128 as default for the sake of
fairness.

  * •

DeepWalk. DeepWalk (Perozzi, Al-Rfou, and Skiena 2014) performs random walks
to generate an ordered sequence of nodes from a static graph to create
contexts for each node, then applies a skip-gram model to these sequences to
learn representations.DeepWalk (Perozzi, Al-Rfou, and Skiena 2014) is a graph
embedding method that leverages random walks to capture the structural
information of a static graph. The main idea behind DeepWalk is to generate
sequences of nodes by performing random walks on the graph and then learn node
representations from these sequences.

  * •

Node2Vec. Node2Vec (Grover and Leskovec 2016) is an extension of the DeepWalk
algorithm that incorporates biased random walks to explore the neighborhood of
nodes and learn representations on a static graph. Node2Vec has been widely
used for various graph analysis tasks, such as node classification, link
prediction, and visualization. By incorporating biased random walks, Node2Vec
can capture both the local and global structural characteristics of a graph,
providing more informative and expressive node embeddings.

Dynamic Graph Methods. We also compare the proposed Dy-SIGN with eight dynamic
graph methods, i.e., HTNE (Zuo et al. 2018), M2DNE (Lu et al. 2019), DyTriad
(Zhou et al. 2018), MPNN (Panagopoulos, Nikolentzos, and Vazirgiannis 2021),
JODIE (Kumar, Zhang, and Leskovec 2019), EvolveGCN (Pareja et al. 2020), TGAT
(da Xu et al. 2020) and SpikeNet (Li et al. 2023). We implement all the models
with the codes provided by the corresponding paper. We search the
hyperparameters by ranging the learning rate over {0.01,0.005,0.001, 0.0005,
0.0001} and dropout rate over {0.3,0.5,0.7} and report the best reports. For
fairness, we set the embedding size of all the models to 128 as default.

  * •

HTNE. HTNE (Zuo et al. 2018) is a network embedding method that combines the
Hawkes process and attention mechanism to capture the formation sequences of
neighborhoods in a network. By integrating the Hawkes process and attention
mechanism, HTNE can effectively capture the temporal dynamics and interaction
patterns in network neighborhoods. This enables the learned embeddings to
better represent the evolving nature of network structures and facilitate
various downstream tasks such as link prediction, community detection, and
anomaly detection.

  * •

M2DNE. M2DNE (Lu et al. 2019) is a network embedding method that specifically
focuses on capturing the structural and temporal properties of evolving
graphs. It achieves this by incorporating a temporal attention point process
and a general dynamics equation into the embedding process.

  * •

DyTriad. DyTriad (Zhou et al. 2018) is a dynamic network modeling approach
that combines the modeling of structural information and evolution patterns
based on the triadic closure process. By incorporating both aspects, DyTriad
is able to effectively capture the dynamics of evolving graphs.

  * •

MPNN. MPNN (Panagopoulos, Nikolentzos, and Vazirgiannis 2021) is a time-series
variant of the message-passing neural network (MPNN) that incorporates a two-
layer LSTM (Long Short-Term Memory). MPNN is designed to capture long-range
temporal dependencies in temporal graphs by encoding the dynamics into the
node representations.

  * •

JODIE. JODIE (Kumar, Zhang, and Leskovec 2019) utilizes a dual Recurrent
Neural Network (RNN) architecture to update node embeddings in an evolving
graph based on observed interactions. This approach enables the prediction of
future embedding trajectories.

  * •

EvolveGCN. EvolveGCN (Pareja et al. 2020) is a method that incorporates
Recurrent Neural Networks (RNNs) to dynamically evolve the parameters of Graph
Neural Networks (GNNs) over time. By adapting the GNN parameters at each time
step, EvolveGCN effectively captures the evolving dynamics of a sequence of
graphs.

  * •

TGAT. TGAT (da Xu et al. 2020) is an inductive learning method specifically
designed for dynamic graph data. It leverages self-attention mechanisms as
fundamental building blocks and introduces a functional time encoding scheme
to effectively capture both temporal and topological information in dynamic
graphs. This approach has demonstrated state-of-the-art performance across
various dynamic graph learning tasks.

  * •

SpikeNet. SpikeNet (Li et al. 2023) introduces a scalable framework designed
to effectively capture both the temporal dynamics and structural patterns of
temporal graphs. This framework addresses the challenge of efficiently
analyzing large-scale temporal graph data.

##  Appendix C Our Method

We implement the proposed Dy-SIGN with PyTorch (Paszke et al. 2017) and
PyTorch Geometric library (Fey and Lenssen 2019). For the implementation, we
adopt the identical settings as described in (Li et al. 2023) and present the
evaluation results in terms of Macro-F1 and Micro-F1 scores using various
training ratios, namely 40%, 60%, and 80%. Additionally, we allocate 5% of the
data for validation purposes. In our implementation, we set the hidden
dimension of all methods to 128 to ensure consistency. The batch size is
configured as 1024 to balance computational efficiency and model performance.
To train the models effectively, we conduct a total of 100 training epochs and
employ a learning rate of 0.001. These settings allow for a fair and
consistent evaluation across different methods, enabling us to assess their
performance under comparable conditions. By reporting Macro-F1 and Micro-F1
scores, we can comprehensively evaluate the models’ performance in terms of
both overall and individual class classification accuracy. Overall, our
implementation follows a standardized approach, ensuring reproducibility and
facilitating meaningful comparisons between different methods.