# Graph Learning under Distribution Shifts: A Comprehensive Survey on Domain
Adaptation, Out-of-distribution, and Continual Learning

Man Wu, Xin Zheng, Qin Zhang, Xiao Shen, Xiong Luo, Xingquan Zhu, Fellow,
IEEE, Shirui Pan, Senior Member, IEEE Man Wu and Xiong Luo are with the School
of Computer and Communication Engineering, University of Science and
Technology Beijing, Beijing, China. Email<EMAIL_ADDRESS>Xin
Zheng is with the Department of Software System and Cybersecurity (SSC),
Monash University, Melbourne, Australia. E-mail<EMAIL_ADDRESS>Qin
Zhang is with the College of Computer Science and Software Engineering,
Shenzhen University, Guangdong, China. Email<EMAIL_ADDRESS>Xiao Shen
is with the School of Computer Science and Technology, Hainan University,
Haikou, China. E-mail<EMAIL_ADDRESS>Xingquan Zhu is with the Dept. of
Electrical Engineering & Computer Science, Florida Atlantic University, USA.
Email<EMAIL_ADDRESS>Shirui Pan is with the School of Information and
Communication Technology and Institute for Integrated and Intelligent Systems
(IIIS), Griffith University, Queensland, Australia. Email:
<EMAIL_ADDRESS>
###### Abstract

Graph learning plays a pivotal role and has gained significant attention in
various application scenarios, from social network analysis to recommendation
systems, for its effectiveness in modeling complex data relations represented
by graph structural data. In reality, the real-world graph data typically show
dynamics over time, with changing node attributes and edge structure, leading
to the severe graph data distribution shift issue. This issue is compounded by
the diverse and complex nature of distribution shifts, which can significantly
impact the performance of graph learning methods in degraded generalization
and adaptation capabilities, posing a substantial challenge to their
effectiveness. In this survey, we provide a comprehensive review and summary
of the latest approaches, strategies, and insights that address distribution
shifts within the context of graph learning. Concretely, according to the
observability of distributions in the inference stage and the availability of
sufficient supervision information in the training stage, we categorize
existing graph learning methods into several essential scenarios, including
graph domain adaptation learning, graph out-of-distribution learning, and
graph continual learning. For each scenario, a detailed taxonomy is proposed,
with specific descriptions and discussions of existing progress made in
distribution-shifted graph learning. Additionally, we discuss the potential
applications and future directions for graph learning under distribution
shifts with a systematic analysis of the current state in this field. The
survey is positioned to provide general guidance for the development of
effective graph learning algorithms in handling graph distribution shifts, and
to stimulate future research and advancements in this area.

###### Index Terms:

Graph learning, graph neural network, domain adaptation, out-of-distribution
learning, continual learning

##  1 Introduction

Graph structural data is ubiquitous in various real-world application domains,
including social networks [1, 2, 3, 4, 5], biological networks [6, 7, 8, 9],
road networks [10, 11, 12], and computer networks [13, 14, 15]. In these
diverse domains, complex relationships among nodes, intricately woven through
edges, harbor valuable information within entities, graph structures, and
overarching graph data patterns. In this case, graph learning [16] techniques
have arisen as responses to better analyzing and understanding various graph
types, serving a wide range of graph-related tasks with promising inference
abilities, covering drug discovery [8, 17], knowledge graph exploration [18,
19], social network analysis [20, 21], recommender systems [22, 23], and
physical movement prediction[24], etc.

Despite the success of prevalent graph learning approaches, the presence of
distribution shifts in graph data poses a substantial constraint on the
capabilities of current methods [25]. This is due to the dynamic and evolving
nature of real-world graph data. For example, social networks evolve with new
users and shifting relationships over time, leading to significant variations
on node features and edge connections, so that the performance of graph
learning models would degrade on recommendation systems or trend analysis
[26]. In financial networks, the relationships between entities (like stocks,
commodities, or institutions) change due to market trends, economic policies,
or global events [27, 28]. A graph learning model trained on historical market
data might underperform when the underlying relationships between entities
shift. In biology, networks representing interactions between proteins, genes,
or species in an ecosystem can change due to mutations, environmental
pressures, or disease outbreaks [29]. Models predicting disease spread or gene
interactions must adapt to these shifts. In addition, transportation systems
(like road networks, air traffic, or public transit systems) experience shifts
due to factors like urban development, changes in travel patterns, or
infrastructure modifications. These shifts impact models used for optimizing
routes, predicting congestion, or planning new infrastructure [30]. These
shifts in statistical distributions of graph data observed in nodes, edges,
and different graphs considerably complicate the graph learning process,
posing challenges for effective model deployment and application in real-world
scenarios.

To this end, in this work, our focus is on engaging in graph learning under
distribution shifts, specifically referring to the scenarios where the
disparities of graph data probability distributions might encompass all
aspects of graph components, including node features, graph structures, and
label distributions. Consequently, graph learning models encounter
difficulties in achieving precise generalization to previously unseen graph
data distributions during the testing phase.

In recent years, there has been a growing interest in exploring the paradigms
of graph learning under distribution shifts [31, 3, 4, 32, 25, 33, 34, 35, 36,
37, 38, 39, 40], to enable models to comprehend complex scenes, objects, and
concepts in both static and dynamic scenarios for graph data. Concretely, the
graph data distribution shift scenarios can be categorized based on whether
the test-stage distributions are observable and whether the available
supervision information is sufficient. Therefore, they can be classified into
three main categories, as shown in Fig. 1:

  * •

Observed Shifts: Known test-stage data, variable supervision, where observed
test-stage graph data that may exhibit potential unknown distribution shifts
from the training stage, available supervisions could be either sufficient or
insufficient

  * •

Unobserved Shifts: Unknown test-stage data, limited supervision, where
unobserved test-stage graph data distributions with diverse and multiple shift
types, available supervisions typically tend to be insufficient

  * •

Sequential Temporal Shifts: Time-dependent, unobserved data, where unobserved
test-stage graph data distributions that arrive sequentially with dynamic
temporal shifts over time.

Figure 1: Overview of graph data distribution shifts and corresponding graph
learning methods.

Based on these three types of graph data distribution shift scenarios, in this
work, we conduct a comprehensive review and provide a systematic taxonomy of
existing graph learning methods that deal with the challenges of distribution
shift learning in graph data. Concretely, current graph learning methods can
also be classified into three categories, with each category corresponding to
a particular distribution shift scenario:

  * •

Graph Domain Adaptation Learning, aims to transfer graph learning models from
a training (source) domain to a test (target) domain and require their
proficient performance on the target domain with differing graph data
distributions [31, 41, 42, 3, 43, 4]. $\rightarrow$ Observed Shifts.

  * •

Graph Out-of-distribution Learning, aims to empower the graph learning models
to effectively learn test graph data with different distributions from the
training data and potential new classes unseen in training [44, 25], enabling
models’ well generalization capabilities. $\rightarrow$ Unobserved Shifts.

  * •

Graph Continual Learning, aims to enable graph learning models to assimilate
new information contingent upon changes in the evolving graph data
distributions, while concurrently refining pre-existing knowledge and
addressing nascent and previously unseen tasks [45, 46]. $\rightarrow$
Sequential Temporal Shifts.

Therefore, in this survey, our goal is to provide an in-depth understanding of
graph data distribution shifts and delve into various graph learning models
and approaches that tackle distribution shift challenges. We will examine key
concepts, critical challenges, previous limitations, and evaluation protocols
associated with graph learning under distribution shifts. Moreover, we will
discuss the potential real-world applications and highlight the promising
future directions within this research area. To the best of our knowledge,
this is the first comprehensive survey on graph learning with general
distribution shift scenarios. Nevertheless, it is worth establishing
connections between our work and several prior studies that concentrate on
specific aspects of graph distribution shift: Li et al. [25] reviewed the
graph out-of-distribution generalization methods and Yang et al. [44] reviewed
the graph out-of-distribution detection methods, which are two types of out-
of-distribution tasks under the unobserved graph data shift scenario,
respectively. Yuan et al. [45] and Febrinanto et al. [46] reviewed the graph
continual learning methods and benchmarks under the sequential temporal graph
data shift scenario. In contrast, our emphasis is distinctly placed on a
broader perspective of graph learning under distribution shifts. This
encompasses a more extensive and fresh method review to furnish a more
comprehensive research roadmap.

We envision that this survey will serve as an important resource for
researchers and practitioners interested in graph learning under distribution
shifts, provide insights into the latest developments and future research
efforts, and facilitate informed decision-making in the areas of model
selection, architectural design, and evaluation strategies for the continuous
development of graph learning models, with potential benefits for both
academic and industrial applications.

In summary, the core contributions of this survey work can be presented as
follows:

  * •

Broad-ranging Graph Distribution Shift Scenarios. To the best of our
knowledge, this is the first survey work in the field of graph learning
covering broad-ranging graph distribution shift scenarios, facilitating in-
depth understanding and analysis for dealing with diverse and complex graph
data distribution shift cases.

  * •

Comprehensive Review and Taxonomy. We provide a systematic taxonomy of
existing graph learning advancements with various distribution shifts. This
taxonomy includes three key categories: graph domain adaptation learning for
observable shifts with known test-stage graphs, graph out-of-distribution
learning for unobserved shifts with unknown test-stage graphs, as well as
graph continual learning for sequential temporal shifts with time-evolving
graphs.

  * •

Highlighted Practical Applications and Future Direction. We highlight the
practical utility of graph learning in addressing distribution shifts,
spanning from applications in scientific discovery to personalized daily life
recommendations. Additionally, we identify several promising future research
directions, intending to inspire and propel the progress of this research
field.

##  2 Problem Definition

In this section, we outline the background of graph neural networks, list
commonly used notations, and define graph related concepts.

Given a graph $G=(V,E)$ where $V=\\{v_{1},\cdots,v_{N}\\}$ is the node set
with $N$ nodes and $E=\\{e_{i,j}|i,j=1,\ldots,N\\}\in V\times V$ is the edge
set. The neighbor set of node $v$ is denoted as $\mathcal{N}(v)=\\{u:(v,u)\in
E\\}$. The node features are represented by a feature matrix
$\mathbf{X}\in\mathbb{R}^{N\times d}$, where the $i$-th row
$\mathbf{x}_{i}\in\mathbb{R}^{d}$ is the feature vector of node $v_{i}$ and
$d$ is the number of feature dimensions. The topological structure of $G$ is
represented by the adjacency matrix $\mathbf{A}\in\\{0,1\\}^{n\times n}$,
where $\mathbf{A}_{i,j}=1$ if the nodes $v_{i}$ and $v_{j}$ are connected,
i.e., $\exists e_{i,j}\in E$, otherwise $\mathbf{A}_{i,j}=0$.
$\mathbf{Y}\in\mathbb{R}^{N\times C}$ is the node label matrix of $G$, where
$C$ is the already-known classes used for training with
$\mathcal{Y}=\\{1,\cdots,C\\}$. Therefore, a graph is composed of node
features, topological structure, as well as node labels, denoted as
$G=(\mathbf{X},\mathbf{A},\mathbf{Y})$.

Moreover, given a set of graphs with $M$ graphs, we have
$\mathbb{G}=\\{(G_{1},y_{1}),\ldots,(G_{M},y_{M})\\}$, and
$\mathbb{Y}=\\{y_{i}\in\mathbb{R}^{1\times K}\\}_{i=1}^{M}$ is the label set
of the graph set. Here, each label $y_{i}$ corresponds to one of the $K$
classes to which the graph $G_{i}$ belongs. In this section, we primarily
focus on illustrating the concepts of various graph distribution learning
methodologies through the lens of node-level graph learning tasks. These
principles can be readily extended to encompass edge-level and graph-level
learning scenarios as well.

######  Definition 1 (Conventional Closed-set Graph Learning).

Given a graph $G=(\mathbf{X},\mathbf{A},\mathbf{Y})$, a GNN model
$f_{\boldsymbol{\theta}}(\cdot)$ takes the node attributes $\mathbf{X}$ and
graph structure $\mathbf{A}$ as inputs. With the assumption that test data
distribution $\mathcal{P}_{te}$ and train data distribution $\mathcal{P}_{tr}$
share the same feature space and label space, for typical closed-set graph
learning problem, the feature encoder and the classifier of the GNN model are
optimized to minimize the expected risk [47] as

$f_{\boldsymbol{\theta}^{*}}=arg\min_{\boldsymbol{\theta}}\mathbb{E}_{G\sim\mathcal{P}_{tr}}[\mathcal{L}(f_{\boldsymbol{\theta}}(\mathbf{X},\mathbf{A}),\mathbf{Y})]$
(1)

where $f_{\boldsymbol{\theta}^{*}}$ is the optimal GNN model trained on the
training graph that achieves the minimal loss. And the loss function
$\mathcal{L}(\cdot)$ measures the discrepancy between the output prediction
$\mathbf{\hat{Y}}=f_{\boldsymbol{\theta}}(\mathbf{X},\mathbf{A})$ and the
ground-truth node labels $\mathbf{Y}$. Generally, $\mathcal{L}(\cdot)$ can be
the cross-entropy function to discriminate between known classes.

In this case, on the test graph
$G_{te}=(\mathbf{X}_{te},\mathbf{A}_{te})\sim\mathcal{P}_{te}$, we have
$\mathbf{\hat{Y}}_{te}=f_{\boldsymbol{\theta}^{*}}(\mathbf{X}_{te},\mathbf{A}_{te})$
for making inference of the test node labels.

######  Definition 2 (Graph Domain Adaptation Learning).

Given a graph
$G_{src}=(\mathbf{X}_{src},\mathbf{A}_{src},\mathbf{Y}_{src})\sim\mathcal{P}_{src}$
with sufficient labeled instances that are drawn from the source distribution
$\mathcal{P}_{src}$ called the source domain, and a target graph
$G_{tgt}=(\mathbf{X}_{tgt},\mathbf{A}_{tgt},\mathbf{Y}_{tgt}^{{\dagger}})\sim\mathcal{P}_{tgt}$
with a small number of labels (subscript† denotes partially observed labels)
or no labels $\mathbf{Y}_{tgt}^{{\dagger}}=\emptyset$ that are drawn from the
target distribution $\mathcal{P}_{tgt}$ called the target domain, where
$\mathcal{P}_{src}\neq\mathcal{P}_{tgt}$ but all nodes in ${G}_{src}$ and
${G}_{tgt}$ are constrained in the same $C$-classes as
$\\{\mathbf{Y}_{src},\mathbf{Y}_{tgt}\\}\in\mathcal{Y}=\\{1,\cdots,C\\}$.

The goal of graph domain adaptation is to learn an optimal graph learning
model $f_{\boldsymbol{\theta}^{*}}$ with both the source and target domains,
so that it can achieve the minimal loss on the target domain, and the marginal
distributions satisfy:

$\displaystyle
f_{\boldsymbol{\theta}^{*}}=\operatorname*{argmin}_{{\boldsymbol{\theta}}}\mathbb{E}_{G_{src}\sim\mathcal{P}_{src},G_{tgt}\sim\mathcal{P}_{tgt}}[\mathcal{L}_{GDA}(f_{\boldsymbol{\theta}}(G_{src},G_{tgt})].$
(2)

In graph domain adaptation (GDA), both the source distribution
$\mathcal{P}_{src}$ and the target distribution $\mathcal{P}_{tgt}$ are
available during training. For different application scenarios of GDA
research, the loss function $\mathcal{L}_{GDA}(\cdot)$ would show different
formulations. For instance, in unsupervised GDA, the given target domain is
totally unlabeled with $\mathbf{Y}_{tgt}^{{\dagger}}=\emptyset$,
$\mathcal{L}_{GDA}(\cdot)$ would be an unsupervised learning objective without
using the target domain labels. Moreover, in semi-supervised GDA, the given
target domain contains a small number of labels with
$\mathbf{Y}_{tgt}^{{\dagger}}\neq\emptyset$, $\mathcal{L}_{GDA}(\cdot)$ could
use partial target domain labels to optimize graph machine learning models.

######  Definition 3 (Graph Out-of-distribution Learning).

Given a training graph
$G_{tr}=(\mathbf{X}_{tr},\mathbf{A}_{tr},\mathbf{Y}_{tr})\sim\mathcal{P}_{tr}$
where $\mathbf{Y}_{tr}\in\mathcal{Y}_{tr}=\\{1,\ldots,C\\}$, and a test graph
$G_{te}=(\mathbf{X}_{te},\mathbf{A}_{te},\mathbf{Y}_{te})\sim\mathcal{P}_{te}$
where $\mathbf{Y}_{te}\in\mathcal{Y}_{te}=\\{1,\ldots,C,C+1,\ldots\\}$,
denoting there exists unseen node classes from $\\{C+1,\ldots\\}$ in the test
graph, and $\mathcal{P}_{tr}\neq\mathcal{P}_{te}$, open-world graph learning
aims to learn an optimal graph function $f_{\boldsymbol{\theta}^{*}}$ that can
achieve the minimal loss on the training distribution $P_{tr}$ and also
achieve optimal performance on data from the test distribution $P_{te}$ that
may contain new label space $\overline{\mathcal{Y}}=\\{1,\ldots,C,unknown\\}$,
where $\mathcal{Y}_{te}\supseteq\overline{\mathcal{Y}}\supset\mathcal{Y}_{tr}$
with the minimization of the expected risk [47]:

$f_{\boldsymbol{\theta}^{*}}=arg\min_{\boldsymbol{\theta}}\mathbb{E}_{G_{tr}\sim\mathcal{P}_{tr},G_{te}\sim\mathcal{P}_{te}}[\mathcal{L}_{G-OOD}(f_{\boldsymbol{\theta}}(G_{tr})].$
(3)

Here, in open-world graph learning, only the training graph distribution
$\mathcal{P}_{tr}$ is available during training, and the test distribution
$\mathcal{P}_{te}$ is unavailable. The predicted class
$unknown\in\overline{\mathcal{Y}}$ contains a group of novel categories, which
may contain more than one class.

When only focusing on the detection of the unseen-class nodes, the problem
degenerates into Graph OOD detection problem, i.e., a binary-class problem to
learn whether a test example is from seen classes $\mathcal{Y}_{tr}$ or not.
In addition, When $Y_{te}=Y_{tr}$, i.e. there is no unseen class during test,
the model only concentrates on the feature distribution shift (i.e. the
difference between the $\mathcal{P}_{te}$ and $\mathcal{P}_{tr}$), the problem
will degenerate to graph OOD generalization problem. We refer to graph OOD
generalization, graph OOD detection and open-world graph learning collectively
as graph out-of-distribution (G-OOD) learning. A more detailed taxonomy is
illustrated in section 3.2 and Fig. 2.

Both GDA and G-OOD aim to transfer the learned knowledge from a training set
to a test set with a different data distribution. Differently, graph continual
learning (GCL) aims to adapt to the dynamics in real-world applications by
continually training the graph learning model on a sequence of training sets
with evolving data distributions.

######  Definition 4 (Graph Continual Learning).

Given a sequence of training graphs
$\mathcal{D}_{tr}=\\{G^{(t)}_{tr}\\}_{t=t_{0}}^{t_{k}}$ that are drawn from
the training distribution $\mathcal{P}_{tr}$, where $t$ is the index of the
training task at the $t$-th time step. And when $\forall t\neq t^{\prime}$, we
have $\mathcal{P}^{(t)}_{tr}\neq\mathcal{P}^{(t^{\prime})}_{tr}$. In this
case, the goal of graph continual learning (GCL) is to learn an optimal graph
learning model $f_{\boldsymbol{\theta}^{*}}$ incrementally from the sequential
training graph set. At the learning session $t$, only the training graph
$G^{(t)}_{tr}\sim\mathcal{P}^{(t)}_{tr}$ can be accessed as:

$f_{\boldsymbol{\theta}^{(t)*}}=\operatorname*{argmin}_{\boldsymbol{\theta}^{(t)}}\mathbb{E}_{G^{(t)}_{tr}\sim
P^{(t)}_{tr}}[\mathcal{L}_{GCL}(f_{\boldsymbol{\theta}^{(t)}}(G^{(t)}_{tr}))].$
(4)

$f_{\boldsymbol{\theta}^{(t)*}}$ is expected to achieve the minimal loss on
the data drawn from the test distribution
$G_{te}^{(t^{\prime})}\sim\mathcal{P}^{(t^{\prime})}_{te}$ after any learning
session $t^{\prime}>=t$, as
$f_{\boldsymbol{\theta}^{(t)*}}(G_{te}^{(t^{\prime})})=\mathbf{Y}_{te}^{(t^{\prime})}$

Overall, the summary of the information used by these three research problems
in both the training and inference stages is presented in Table I.
Specifically, graph domain adaptation learning (GDA) and graph out-of-
distribution learning (G-OOD) are more concerned with the distribution shifts
of static data. The main difference between them is that GDA deals with
observed shifts, meaning it handles known test-stage data, whereas G-OOD
tackles unobserved shifts, or unknown test-stage data. In contrast, graph
continual learning (GCL) targets the distribution shifts of dynamic data,
focusing on sequential temporal shifts.

TABLE I: A summary of information used in graph domain adaption learning,
graph out-of-distribution learning, and graph continual learning in both
training and inference stages.

| Training | Inference | Labels
---|---|---|---
Graph Domain Adaption Learning |  $G_{src}(tr)$, $G_{tgt}(te)$ | $G_{tgt}(te)$ | $\mathbf{Y}_{tgt}\subseteq\mathbf{Y}_{src}$
Graph Out-of-distribution Learning | $G_{tr}$ | $G_{te}$ | $\mathbf{Y}_{tr}\subseteq\mathbf{Y}_{te}$
Graph Continual Learning | $G_{tr}^{(t)}|_{t=t_{0}}^{t_{k}}$ | $G_{te}^{(t^{\prime})}|_{t^{\prime}=t_{k+1}}^{t_{m}}$ | $\mathbf{Y}_{tr}\subseteq\mathbf{Y}_{te}$

##  3 Categorization And Frameworks

Figure 2: Overview of the general taxonomy of graph learning under
distribution shifts. Figure 3: Hierarchical classification of graph learning
under distribution shifts.

Fig. 3 summarizes the general taxonomy of graph learning under distribution
shifts and the related methods reviewed in this paper. The taxonomy is
centered on the shifting scenarios and design details of the graph learning
algorithms under distribution shifts. To begin with, according to the shifting
scenarios of the explanations, methods for graph learning under distribution
shifts can be classified into three categories: Graph domain adaptation, out-
of-distribution graph learning and continual graph learning. We further
consider the design details of different methods and classify them in a
hierarchical manner for ease of understanding and comparative analysis.

###  3.1 Taxonomy of Graph Domain Adaptation Learning

Domain adaptation addresses the issue of declining model performance due to
differences in data distributions across different domains, aiming to enhance
the model’s generalization capability on the target domain. Domain adaptation
methods can be further categorized into three types: semi-supervised,
unsupervised, and test-time graph transformation.

Semi-supervised Domain Adaptation. These approaches focus on training the
model using labeled data from the source domain as well as unlabeled data from
both the source and target domains, aiming to ensure a strong performance in
the target domain.

Unsupervised Domain Adaptation. These methods concentrate on model transfer
without the availability of labeled target domain data. The core idea is to
reduce the feature distribution differences between the source and target
domains through feature alignment, thereby enhancing the model’s
generalization ability to the target domain.

Test-time Graph Transformation (Adaptation). These approaches are centered
around adjusting the graph data of the target domain during testing to match
the characteristics of the target domain, enhancing the model’s adaptability
to the target domain.

###  3.2 Taxonomy of Graph Out-of-distribution Learning

Out-of-distribution (OOD) graph learning addresses the challenge of learning
from graphs with different features than those seen during training. OOD graph
learning can be further categorized into three types: Graph Out-of-
Distribution Generalization, Graph Out-of-Distribution Detection, and Open-
world Graph Learning.

Graph Out-of-distribution Generalization. This type of OOD graph learning
focuses on developing models that can generalize well to graphs with different
features than the ones seen in training. The goal is to ensure that the
model’s performance remains satisfactory even when confronted with graphs from
previously unseen distributions.

Graph Out-of-distribution Detection. In this scenario, the focus is on
identifying or detecting graphs that belong to out-of-distribution categories.
The goal is to design models capable of flagging or distinguishing graphs that
significantly deviate from the training data distribution, potentially
indicating new or unfamiliar graph patterns.

Open-world Graph Learning. Open-world graph learning deals with the challenge
of learning in situations where the set of possible graph classes is not known
in advance. This means that the model must not only adapt to new graph classes
but also make decisions about classifying data into known classes or
identifying data as belonging to a novel class.

###  3.3 Taxonomy of Graph Continual Learning

Continual graph learning addresses the challenge of acquiring knowledge from a
streaming of graph data that arrives over time and continuously evolves. The
methods of continual graph learning can be further classified into four
categories: architectural, regularization, rehearsal, and hybrid.

Architectural Approach. These approaches focus on modifying the specific
architecture of networks, activation functions, or layers of algorithms to
address a new task and prevent the forgetting of previous tasks.

Regularization Approach. These approaches consolidate the learned knowledge by
adding a regularization item to the loss function, constraining the neural
weights from updating in a direction that compromises performance on prior
tasks.

Rehearsal Approach. These approaches maintain a memory buffer preserving the
information of prior tasks and replay it when learning new tasks to mitigate
catastrophic forgetting.

Hybrid Approach. These approaches combine more than one continual learning
approach to take advantage of multiple approaches and improve the performance
of models.

##  4 Graph Domain Adaptation Learning

Figure 4: The overview of graph domain adaptation learning.

Domain adaptation is the task of transferring knowledge learned from a source
domain to a target domain where the distribution of the data is different. In
the context of graph learning, domain adaptation refers to the task of
adapting graph-based learning models to new domains where the graph structure,
node features, or edge features may be different. The requirement for domain
adaptation in graph learning arises in many real-world applications where the
graph data may change over time, or where models trained on one dataset need
to be applied to a new dataset with different characteristics. For example, in
social network analysis, models trained on one social network may not
generalize well to a different social network due to differences in the graph
structure and user behavior. In drug discovery, models trained on one set of
molecules may not generalize well to a different set of molecules due to
differences in the chemical properties of the molecules.

The methods for graph domain adaptation can be roughly divided into three
groups: semi-supervised domain adaptation [31, 3, 4, 32, 48], unsupervised
domain adaptation [41, 43, 42, 5, 49, 50], and test-time graph transformation
(adaptation) [15, 8, 7, 6]. The categories of graph domain adaptation are
shown in Fig. 4.

  * •

Semi-supervised Domain Adaptation. These approaches involve adapting from a
source domain to a target domain using a small amount of labeled data and
unlabeled data from the target domain.

  * •

Unsupervised Domain Adaptation. These approaches aim to enhance the model’s
cross-domain generalization performance by reducing the distribution
discrepancy between the source and target domains through feature alignment,
without relying on any labeled target data.

  * •

Test-time Graph Transformation (Adaptation). These approaches refer to
adjusting the graph data of the target domain during testing to accommodate
the characteristics of the target domain. This approach aims to enhance the
performance of a model by fine-tuning its behavior based on the specifics of
the target domain during the testing phase.

###  4.1 Semi-supervised Domain Adaptation

Semi-supervised domain adaptation denotes a circumstance wherein labeled data
is accessible within the source domain, whereas the target domain possesses a
limited fraction of labeled data. The primary aim of this paradigm is to
bolster the model’s efficacy within the target domain through adept
utilization of these labeled data instances from the source domain, along with
any potentially available labeled data within the target domain. At its
essence, this approach amalgamates the principles of supervised and
unsupervised learning. This synergy is directed towards orchestrating a more
potent adaptation to the data distribution inherent within the target domain.
The ultimate aspiration is to engender robust predictive outcomes within this
new domain context.

DA-AGE [31] is a model based on adversarial domain adaptation, designed to
address distribution shift issues. It combines graph-based semi-supervised
learning techniques within the framework of deep learning to leverage
unlabeled data. This model is capable of training deep neural network (DNN)
models using both labeled and unlabeled data, thereby tackling challenges
arising from data distribution differences between the source and target
domains.

DASGA [3] is a method used for graph domain adaptation. This algorithm is
based on scenarios where the source graph has a larger number of annotated
nodes. It achieves this by learning the label function spectrum in the source
graph and transferring it to the target graph, even when the target graph has
fewer labeled nodes. This process relies on transformations between Fourier
bases to address the lack of one-to-one correspondence between Fourier basis
vectors in two independently constructed graph domains, thus enabling flexible
alignment between these two graphs.

Figure 5: AdaGCN incorporates adversarial domain adaptation and graph
convolution methods.

AdaGCN [4] is a graph transfer learning framework that combines adversarial
domain adaptation and graph convolution techniques. It consists of two main
components: a semi-supervised learning part and an adversarial domain
adaptation part. The former aims to utilize the label information available in
both the source and target networks to learn representations capable of
distinguishing nodes of different categories. The role of the latter is to
alleviate distribution differences between the source and target domains,
thereby promoting the effectiveness of knowledge transfer.

DAGNN [32] is an end-to-end framework for cross-domain text classification. It
models documents as graphs, employs domain-adversarial training for feature
optimization, and captures non-consecutive semantics. By leveraging knowledge
from different domains with hierarchical graph neural networks, DAGNN improves
task performance by utilizing relationships and patterns across domains.

Figure 6: PGL is an end-to-end progressive graph learning framework that
integrates graph neural networks and episodic training.

PGL [48] is an end-to-end progressive graph learning framework known for
integrating graph neural networks with occasional training. This design helps
mitigate potential mode collapse and employs adversarial learning to reduce
the disparity between the source data distribution and the target data
distribution. The framework is aimed at addressing more realistic open-set
domain adaptation problems, where the target data may include additional
categories not present in the source data.

Discussion. The success of semi-supervised domain adaption relies not only on
the generalization capabilities of transfer models but also depends on the in-
depth analysis and application of limited labeled data in the target domain.
This integrated approach is key for future research and practice.

###  4.2 Unsupervised Domain Adaptation

Unsupervised Domain Adaptation is centered on the task of transferring a
model’s knowledge when there exists an absence of labeled data within the
target domain. The principal goal of this paradigm is to augment the model’s
efficacy within the target domain despite the unavailability of labeled target
domain data. This is realized through the strategic alignment of the feature
distributions inherent to both the source and target domains. The crux of this
approach rests upon the harmonization of features, with the intention of
mitigating the dissimilarities existing between the source and target domain
features. This alignment is instrumental in empowering the model to adeptly
adapt to the distinctive characteristics present within the target domain.

UDA-HGM [41] is an unsupervised domain adaptation method that focuses on
unlabeled data in the target domain, emphasizing the correspondence between
samples from the source and target domains. The research employs hypergraph
representations of the source and target domains, utilizing first-order,
second-order, and third-order similarities to achieve hypergraph matching with
class regularization. To enhance computational efficiency, the research begins
by selecting subsets of samples to construct graphs and then employs a
customized optimization method for graph matching using conditional gradients
and alternating direction method of multipliers for the matching process.

UDA-GCN [43] is an unsupervised domain adaptation graph convolutional network
that focuses on domain adaptation learning for graph data. In its primary
stage, it constructs a dual graph convolutional network component that
operates on the basis of local and global consistencies to aggregate features.
Further, an attention mechanism is applied to establish consistent
representations for nodes across different graph domains. Additionally, a
domain adaptation learning module is introduced to jointly optimize three
different loss functions, facilitating cross-domain knowledge transfer.

CDNE [51] a cross-network deep network embedding model which incorporates
domain adaptation into deep network embedding to learn label-discriminative
and network-invariant node vector representations. For deep network embedding,
CDNE leverages two stacked autoencoders to reconstruct the network proximity
matrix of the source network and the target network respectively.
Additionally, CDNE designs pairwise constraint to map more strongly connected
nodes within each network closer in the embedding space. For domain
adaptation, the marginal and class-conditional MMD constraints have been
incorporated into CDNE to reduce the distribution shifts between the source
and target networks.

ACDNE [52] is an adversarial cross-network deep network embedding model, which
integrates adversarial domain adaptation with deep network embedding. On one
hand, a deep network embedding module is designed by ACDNE, by utilizing two
feature extractors to learn the embeddings from each node’s own attributes and
its neighbors’ attributes respectively. On the other hand, an adversarial
domain adaptation module is employed by ACDNE to make a domain discriminator
compete against the deep network embedding module, so as to learn network-
invariant node embeddings.

DM-GNN [53] is a domain-adaptive message passing graph neural network which
integrates graph neural network with conditional adversarial domain
adaptation. DM-GNN separates ego-embedding learning from neighbor-embedding
learning via dual feature extractors. DM-GNN devises a label-aware propagation
scheme to promote intra-class propagation while avoiding inter-class
propagation, making source embeddings more label-discriminative. To match the
class-conditional distributions between the source and the target networks,
DM-GNN adopts conditional adversarial domain adaptation.

DGASN [54] is a domain-adaptive graph attention-supervised network designed to
address the cross-network homophilous and heterophilous edge classification
problem. DGASN proposes to apply direct supervision on graph attention
learning guided by the edge labels observed from the source network.
Specifically, DGASN assigns lower (higher) attention weights to heterophilous
(homophilous) edges during neighborhood aggregation, so as to guarantee more
label-discriminative embeddings to separate nodes from different classes.

LGA [42] is a fresh model for enhancing distributional and structural
similarities during the adaptation process. It involves embedding data from
two domains into a latent space, aligning their distributions by minimizing
the maximum mean discrepancy metric. Additionally, the transformed manifold is
represented using graphs to maximize graphical structural similarities within
the embedding space. This is accomplished by minimizing the spectrum distance
between graph Laplacians of the embeddings. The spectrum retains intrinsic
manifold information and remains unchanged under data reordering and
eigenspace unitary transformations.

Figure 7: DMGE is a model that utilizes knowledge from different domains.

DMGE (Deep Multi-Graph Embedding) [5] is a model enhancing task performance by
leveraging knowledge from different domains. It constructs a multi-graph based
on the behaviors of users from different domains and utilizes a multi-graph
neural network to learn cross-domain representation in an unsupervised manner.
Particularly, a multiple gradient descent optimizer is used for efficiently
training the model.

GCAN [49] is an end-to-end graph convolutional adversarial network that
achieves unsupervised domain adaptation by jointly modeling data structures,
domain labels, and class labels within a unified deep framework. Additionally,
the model incorporates three effective alignment mechanisms: structure-aware
alignment, domain alignment, and class centroid alignment. These alignment
mechanisms aid in learning domain-specific features and semantic
representations, thereby reducing disparities between different domains and
enabling effective domain adaptation.

Figure 8: UMTDA enables the learning of domain-invariant representations by
sharing similar feature representations through heterogeneous graph attention
networks.

UMTDA [50] is a method designed for the scenario of multiple unlabeled target
domains and one labeled source domain, aiming to perform deep semantic
information propagation. Through a heterogeneous graph attention network, it
strives to learn a shared unified subspace that allows all domains to share
similar feature representations. In this process, attention mechanisms are
used to optimize relationships between multiple domain samples, facilitating
enhanced semantic propagation. Subsequently, by leveraging the graph attention
network, it predicts pseudo-labels for the target domains. By aligning the
centroids of the labeled source domain and the pseudo-label target domain, it
achieves the learning of domain-invariant representations.

Discussion. A key challenge in unsupervised domain adaption is ensuring that
models can generalize well from the source to the target domain without losing
the critical information encoded in the graph structures. Future research
should focus on enhancing these models to better handle the nuances of graph
data, exploring more adaptive and flexible techniques that can account for
varying degrees of distribution shifts.

###  4.3 Test-time Graph Transformation (Adaptation)

Test-time graph transformation (adaptation) aims to adapt models based on test
samples in the presence of distributional shifts. Test time adaptation can be
further subdivided into test time training and fully test time adaptation
according to whether it can access the source data and alter the training of
the source model. Here, we introduce several typical existing methods.

AdaGraph [15] presents the first deep architecture for Predictive Domain
Adaptation. AdaGraph leverages metadata information to build a graph where
each node represents a domain, while the strength of an edge models the
similarity among two domains according to their metadata. Then in order to
exploit the graph for the purpose of DA and AdaGraph has a novel domain-
alignment layers. This framework yields the new state of the art on standard
PDA benchmarks.

GTRANS [8] which optimizes a contrastive surrogate loss to transform graph
structure and node features, and provide theoretical analysis with deeper
discussion to understand this framework. Experimental results on distribution
shift, abnormal features and adversarial attack have demonstrated the
effectiveness of GTRANS.

GAPGC (Graph Adversarial Pseudo Group Contrast) [7] is a test-time training
method designed for GNNs with a contrastive loss variant as the self-
supervised objective during testing. The framework of GAPGC is shown in Fig.
9. Recently, the effectiveness of test-time training has been validated to
improve the performance on OOD test data, where some self-supervised auxiliary
tasks are proposed. The authors argue that the simple augmentations in self-
supervised training (e.g., randomly dropping nodes or edges) could harm the
label-related critical information in graph representations. Therefore, GAPGC
generates relatively reliable pseudo-labels, avoiding the severe shifts caused
by the incorrect positive samples. The proposed adversarial learnable
augmenter and group pseudo-positive samples can promote the relevance between
the self-supervised task and the main task, to enhance the performance of the
main task. The theoretical evidence is also derived to show that GAPGC can
capture minimal sufficient information for the main task from an information
theory perspective, which benefits the predictions on the OOD testing data.

Figure 9: GAPGC is a test-time training method designed for GNNs with contrast
loss variants as self-supervised targets during testing.

GT3 (Graph Test-Time Training with Constraint) [6] is another test-time
training method on graphs, which proposes a hierarchical self-supervised
learning framework. Specifically, it first introduces the global contrastive
learning strategy to encourage node representations to capture the global
information of the whole graph. Global contrastive learning is based on
maximizing the mutual information between the local node representation and
the global graph representation. Then, it presents the local contrastive
learning for distinguishing different nodes from different augmented views of
a graph, so that the node representation can capture more local information.
Besides, an additional constraint is proposed to encourage that the
representations of testing samples are close to the representations of the
training samples. The model’s generalization capacity for the graph
classification task can be enhanced based on this test time training strategy
with self-supervised learning.

Discussion. Test-time Graph Transformation (Adaptation) is a sophisticated
approach aimed at enhancing the adaptability of graph-based machine learning
models to new, potentially diverse target domains by adjusting the graph data
during the testing phase.

TABLE II: A summary of graph domain adaptation (GDA) learning methods.

Category |  Method
---|---
Semi-supervised GDA Learning |  DA-AGE [31]
DASGA [3]
AdaGCN [4]
DAGNN [32]
PGL [48]
Unsupervised GDA Learning |  UDA-HGM [41]
UDA-GCN [43]
CDNE [51]
ACDNE [52]
DM-GNN [53]
DGASN [54]
LGA [42]
DMGE [5]
GCAN [49]
UMTDA [50]
Test-time Graph Transformation (Adaptation) |  AdaGraph [15]
GTRANS [8]
GAPGC [7]
GT3 [6]

##  5 Graph Out-of-distribution Learning

Out-of-distribution (OOD) graph learning refers to the problem of learning
from graphs that have different characteristics than those seen during
training. Specifically, in graph-based machine learning, a common assumption
is that the training and test data come from the same distribution. However,
in many real-world scenarios, this assumption does not hold, and the test
graphs may have different statistical properties than the training graphs. In
such cases, a model trained on the training graphs may not generalize well to
the test graphs. OOD graph learning addresses this problem by developing
models that can handle the distribution shift between the training and test
graphs. One way to achieve this is by learning representations that are
invariant to the differences between the training and test distributions.
Another approach is to explicitly model the distribution shift between the
training and test graphs and adapt the model accordingly.

OOD graph learning has several applications, including social network
analysis, recommender systems, and drug discovery. In these applications, the
graphs may have different characteristics depending on the population or
context, and it is important to develop models that can handle these
differences to achieve good performance in the target application.

Fig. 10 depicts an overview of the mentioned domains, in which the differences
are shown visually.

Figure 10: The overview of graph out-of-distribution learning.

###  5.1 Graph Out-of-distribution Generalization

To tackle the challenges brought by unknown distribution shifts and solve the
graph OOD generalization problem, considerable efforts have been made in
literature, which can be divided into three categories. We select some typical
examples for each category to introduce.

####  5.1.1 Data Augmentation

This category of methods aims to manipulate the input graph data, i.e., graph
augmentation. By systematically generating more training samples to increase
the quantity and diversity of the training set, graph augmentation techniques
are effective in improving the OOD generalization performance.

GAug (Graph Augmentation) [55] is a method for feature-wise graph
augmentation. It utilizes a differentiable edge predictor to generate
augmented graphs by encoding class-homophilic structure, which promotes intra-
class edges and reduces inter-class edges to enhance prediction accuracy and
generalization ability. GAug modifies the input graph using an edge prediction
module and learns to generate new possible edges.

NCLA (Neighbor Contrastive Learning on Learnable Graph Augmentation) [56] is
an end-to-end automatic graph contrastive learning method, applying neighbor
contrastive learning on learnable graph augmentation. NCLA automatically
generates multiple graph augmented views with adaptive topology by the multi-
head graph attention mechanism, where each attention head corresponds to each
augmented view. The attention-based learnable graph augmentation avoids
improper modification of the original topology and can be compatible with
various graph datasets without prior domain knowledge.

GRAND (Graph Random Neural Network) [57] is a framework for enhancing
generalization in GNNs by combining feature-wise graph augmentation methods.
First, it randomly drops node features and propagates perturbed features. Then
in accordance with the graph homophily assumption[58], stochastic augmentation
is employed to generate diverse augmented representations for individual
nodes.

Figure 11: A framework of GraphCL. GraphCL is a mixed-type graph data
augmentation method, which provides four data augmentations, including node
dropping, edge perturbation, attribute masking, and subgraph sampling.

GraphCL (Graph Contrastive Learning) [10] GraphCL is a mixed-type graph data
augmentation method, which provides four data augmentations, including node
dropping, edge perturbation, attribute masking, and subgraph sampling. As
depicted in Fig. 11, two categories of augmentations are randomly sampled from
a pool of augmentations and implemented on the input graph. A GNN encoder and
a projection head are trained with the objective of maximizing the consensus
between their respective representations[59].

####  5.1.2 Model Develoment

This category of methods aims to propose new graph models for learning OOD
generalized graph representations.

DisenGCN [60] is a method that utilizes the DisenConv, a disentangled
multichannel convolutional layer, to learn disentangled representations of
nodes. Within DisenConv, a dynamic process determines the neighboring nodes
that are influenced by specific latent factors and directs them to the
corresponding channels for information extraction and convolution. The
disentangled nature of these latent factors contributes to improved
generalization performance when dealing with out-of-distribution data.

OOD-GNN [9] is a GNN designed backed by confounder balancing theory[61] in
causality. It is designed for poor generalization on unseen testing graphs
with different distributions. It utilizes a nonlinear graph representation
decorrelation method using random Fourier features to encourage independence
between relevant and irrelevant graph representations, eliminating spurious
correlations. OOD-GNN optimizes sample graph weights and the graph encoder to
enhance discriminative graph representations, resulting in improved
predictions.

####  5.1.3 Learning Strategy

This category of methods focuses on exploiting the training schemes with
tailored optimization objectives and constraints to enhance the OOD
generalization capability.

StableGL [62] is a framework for Graph Invariant Learning in GNNs that
addresses unstable inference in different test environments caused by
distribution shifts. It consists of two essential components: Locally stable
learning captures stable node properties by reweighting the neighborhood
aggregation process, while globally stable learning reduces training losses in
different environments, ensuring the generalization of GNNs.

Pretraining-GNN [63] refers to a self-supervised technique employed in the
pre-training of GNNs. The crux of this approach lies in the pre-training of a
highly expressive GNN on both individual nodes and complete graphs, enabling
the simultaneous acquisition of valuable local and global representations.
Consequently, this method effectively sidesteps negative transfer phenomena
and substantially enhances the model’s generalization capabilities across
various downstream tasks.

Discussion. In the graph OOD generalization, while the Data Augmentation
addresses graph OOD generalization by diversifying the training graph data,
Model Development focuses on building inherently more adaptable models, and
Learning Strategy emphasizes the training schemes with tailored optimization
objectives and constraints that enhance the model’s ability to generalize to
new distributions. Each approach tackles the challenge of Graph OOD
Generalization from a different angle, offering a comprehensive capability for
improving model performance on unseen graph data.

###  5.2 Graph Out-of-distribution Detection

Graph Out-of-Distribution detection (Graph OOD detection) aims to identify
test-times samples that are semantically different from the training data
categories, and therefore should not be predicted into the known classes.
Here, we select some typical examples for the introduction.

Uncertainty-GNN [64] is an innovative framework that integrates multiple
uncertainty sources to enhance the performance of GNNs in semi-supervised node
classification tasks. It effectively improves predictions and accuracy by
leveraging deep learning-based and belief/evidence theory-based uncertainties.
Uncertainty-GNN improves GNN performance in node classification tasks by
effectively addressing misclassification and out-of-distribution detection.

GPN (Graph Posterior Networks) [65] is a model that combines node
classification and OOD detection, using variational inference and a latent
variable to approximate the true posterior distribution and differentiate
between in-distribution and OOD nodes. LMN mitigates cross-distribution
mixing, learns from neighboring nodes without explicit OOD labels, and reduces
overfitting with bi-level optimization. Overall, LMN provides a comprehensive
solution for accurate node classification and reliable OOD detection.

LMN [66] is a model that combines node classification and OOD detection, using
variational inference and a latent variable to approximate the true posterior
distribution and differentiate between in-distribution and OOD nodes. LMN
mitigates cross-distribution mixing, learns from neighboring nodes without
explicit OOD labels, and reduces overfitting with bi-level optimization.
Overall, LMN provides a comprehensive solution for accurate node
classification and reliable OOD detection.

GraphDE [11] is a unified framework for graph OOD detection and debiased
learning. It utilizes a latent environment variable and generative models to
address outliers in the training data. An OOD detector is also included to
distinguish between in-distribution and OOD samples, enhancing the robustness
of graph representation learning. Overall, GraphDE is a promising solution for
reliable graph representation learning in real-world applications.

GOOD-D (Graph Out-Of Distribution Detection) [67] is an unsupervised method
for detecting OOD graphs. It utilizes a hierarchical graph contrastive
learning framework to capture attributive and structural patterns from in-
distribution (ID) training data. By maximizing agreement between feature and
structure graph views at multiple levels, GOOD-D effectively detects OOD
graphs. It incorporates a self-adaptive mechanism to balance learning
objectives and OOD scores.

Discussion. Graph OOD Detection aims to equip models with the ability to
identify and detect graphs that represent new, unknown, or significantly
different patterns from those seen during training. This capability is
essential in numerous fields where understanding and reacting to evolving or
novel graph structures is crucial.

Figure 12: The framework of GOOD-D.

###  5.3 Open-world Graph Learning

Open-world Graph Learning is a task where models are trained in a setting
without prior knowledge of the number of classes or their training samples.
The challenge lies in distinguishing unseen classes without labeled samples
from existing classes. This applies to both graph structures and knowledge
graphs. The literature can be categorized into graph structure and knowledge
graph studies.

####  5.3.1 Graph Structure

In the open-world learning of graph structure, the model needs to better learn
the feature representation of graph data, classify nodes belonging to known
classes into correct groups through threshold judgment or classifier output,
etc., and also classify nodes not belonging to existing classes into invisible
classes [68, 69, 70].

OpenWGL [70] tackles open-world graph learning with two key components: Node
Uncertainty Representation Learning and Open-world Classifier Learning. As
shown in Fig. 13, it captures uncertainty using a graph variational
autoencoder for node embeddings and differentiates between seen and unseen
class nodes through label and class uncertainty loss minimization with an
automatic threshold. This approach provides a comprehensive solution for
accurate classification in open-world scenarios.

Figure 13: The framework of OpenWGL. OpenWGL captures uncertainty using a
graph variational autoencoder for node embeddings and differentiates between
seen and unseen class nodes through label and class uncertainty loss
minimization with an automatic threshold.

FATE (Federated Learning via Adapted Transformer) [71] is a federated learning
framework that handles open-world feature extrapolation using graph
representation and learning. It utilizes a backbone network and a graph neural
network to handle new features without retraining. FATE includes two training
strategies for better extrapolation and reduced overfitting. The framework is
effective and scalable, supported by theoretical analysis and experiments.

Co-CGE [72] is an approach for Compositional Zero-Shot Learning (CZSL) in an
open-world setting that aims to recognize unseen compositions of known state-
object primitives. It utilizes a graph convolutional neural network to model
dependencies between states, objects, and compositions. By incorporating
feasibility scores, Co-CGE enhances CZSL performance by considering the
feasibility of unseen compositions.

####  5.3.2 Knowledge Graph

Knowledge graph completion (KGC) aims to fill in missing relationships in a
Knowledge Graph (KG). Traditional closed-world approaches for KGC are limited
in their ability to address a small number of missing relations. However,
recent advancements in KGC involve incorporating knowledge from open-world
resources, like online encyclopedias and newswire corpus, to enhance
completion capabilities.

ConMask [73] is a model specifically designed for open-world Knowledge Graph
Completion (KGC) tasks. It consists of Relationship-Dependent Content Masking,
Target Fusion, and Target Entity Resolution components. By masking irrelevant
information, extracting entity embeddings, and generating similarity rankings,
ConMask effectively connects new entities to knowledge graphs, improving their
completeness in an open-world setting.

Figure 14: The framework of ConMask. It consists of Relationship-Dependent
Content Masking, Target Fusion, and Target Entity Resolution components.

Prob-CBR (Probabilistic Case-based Reasoning) [74] is an approach for open-
world graph completion. It combines probabilistic and case-based reasoning
principles to handle uncertainty in knowledge graphs. By retrieving similar
cases, Prob-CBR predicts attributes using reasoning paths. Its probabilistic
model estimates query effectiveness with simple path statistics. It
dynamically grows with new entities and outperforms other approaches.

MIA [75] is an open-world KGC model that utilizes a word-level attention
mechanism to simulate interactions between entity descriptions, names,
relationship names, and candidate tail entity descriptions. As shown in Fig.
15, it improves representations by incorporating additional textual features
from head entity descriptions. However, its effectiveness in predicting the
tail entity relies on the availability of relationship-related information in
entity descriptions.

Discussion. Open-world Graph Learning is a dynamic and challenging research
area, requiring models to go beyond traditional classification tasks to
recognize and categorize previously unseen classes. In graph structures, it
involves advanced feature representation learning and the innovative
classification of nodes into both known and unseen classes. In the realm of
knowledge graphs, the shift towards leveraging open-world resources for KGC
marks a significant advancement in making these graphs more complete and
reflective of the ever-changing real world. Both areas highlight the need for
models that are adaptable, flexible, and capable of handling the complexities
and uncertainties inherent in real-world data.

Figure 15: The framework of MIA. MIA improves representations by incorporating
additional textual features from head entity descriptions. TABLE III: A
summary of graph out-of-distribution (OOD) learning methods.

Category | Subcategory | Method
---|---|---
Graph OOD Generalization | Data Augmentation | GAug [55]
NCLA [56]
GRAND [57]
GraphCL [10]
Model Development | DisenGCN [60]
OOD-GNN [9]
Learning Strategy | StableGL [62]
Pretraining-GNN [63]
Graph OOD Detection | - | Uncertainty-GNN [64]
GPN [65]
LMN [66]
GraphDE [11]
GOOD-D [67]
Open-world Graph Learning | Graph Structure | OpenWGL [70]
FATE [71]
Co-CGE [72]
Knowledge Graph | ConMask [73]
Prob-CBR [74]
MIA [75]

##  6 Graph Continual Learning

Graph continual learning refers to the problem of learning from a stream of
graph data that arrives over time and continuously evolves. In this setting,
the goal is to develop models that can learn from new graphs as they arrive,
without forgetting what has been learned from previous graphs. The graph
continual learning problem is particularly challenging because the
distribution of graphs can change over time, and the model must adapt to these
changes without degrading its performance as previously.

Graph continual learning has applications in various fields, such as social
network analysis, recommendation systems, and biological network analysis. In
these fields, the graph data can arrive in a stream, and it is essential to
have models that can adapt to the evolving graph data.

The categorization of graph continual learning methods is based on how these
methods consider graph structure data in the incremental learning process.
Thus, the existing graph continual learning works can be divided into four
categories: architectural, regularization, rehearsal, and hybrid, which are as
shown in Fig. 16 and follows:

  * •

Architectural Approach. These approaches focus on modifying the specific
architecture of networks, activation functions, or layers of algorithms to
address a new task and prevent the forgetting of previous tasks.

  * •

Regularization Approach. These approaches consolidate the learned knowledge by
adding a regularization item to the loss function, constraining the neural
weights from updating in a direction that compromises performance on prior
tasks.

  * •

Rehearsal Approach. These approaches maintain a memory buffer preserving the
information of prior tasks and replay it when learning new tasks to mitigate
catastrophic forgetting.

  * •

Hybrid Approach. These approaches combine more than one continual learning
approach to take advantage of multiple approaches and improve the performance
of models.

We summarize the methods of graph continual learning in Table IV.

TABLE IV: A summary of graph continual learning (GCL) methods.

Category | Method
---|---
Architectural (Arch.) | FGN [76]
HPNs [14]
CoST-GCN [77]
PI-GNN [78]
Regularization (Reg.) | DiCGRL [79]
GPIL [13]
TWP [80]
GraphSAIL [81]
RieGrace [82]
LKGE [83]
Rehearsal (Reh.) | ER-GNN [84]
Incremental-GNNs [85]
RBR and PBR [86]
Inverse Degree Sampling [87]
Hybrid | ContinualGNN (Reg.& Reh.) [1]
LDANE (Arch.& Reg.& Reh.) [2]
TrafficStream (Reg.& Reh.) [88]
CMDGAT (Arch.& Reh.) [89]

Figure 16: Graph evolution and graph continual learning categorization. Graph
lifelong learning aims to accommodate a new task $T_{t}$ based on the graph
$G_{t}=G_{t-1}+\Delta G_{t}$ while maintaining the performance in solving
previous tasks $T_{1},T_{2},...,T_{t-1}$. the existing graph continual
learning works can be divided into four categories: architectural,
regularization, rehearsal, and hybrid.

###  6.1 Architectural Approach

In the general continual learning setting, the architectural approaches mainly
pay attention to altering the specific architecture of networks, activation
functions, or layers of algorithms to address new tasks to prevent
catastrophic forgetting of already seen tasks [38, 39, 40]. In graph continual
learning scenario, the architectural approaches solve problems by techniques
such as graph structure evolution, unit expansion, and compression.

Figure 17: The framework of FGN. FGN allows independent processing of graph
nodes during knowledge acquisition and has implications for graph
relationships.

FGN (Feature Graph Networks) [76] is a framework that converts graph
architectures into a format compatible with learning structures. Illustrated
in Fig. 17, it allows independent processing of graph nodes during knowledge
acquisition and has implications for graph relationships. In feature graphs,
the fixed node count corresponds to unchanging feature dimensions in standard
graphs.

HPNs (Hierarchical Prototype Networks) [14] extract abstract knowledge from
expanding graphs using prototypes. They encode attributes and relationships,
generating prototypes at different levels. These prototypes capture shared
attributes and relationships among nodes. HPNs enable the adaptive selection
and composition of prototypes, facilitating the extraction of hierarchical
knowledge.

CoST-GCN [77] is a variant of ST-GCN [24] designed for graph-based human
action recognition. It introduces a Continual Inference Network that enables
step-by-step predictions without redundant frame processing. CoST-GCN reduces
time complexity, accelerates on-hardware, and decreases memory usage during
online inference while maintaining predictive accuracy.

PI-GNN (Parameter Isolation GNN) [78] is a model for continual learning on
dynamic graphs. The method employs a strategy by using parameter isolation and
expansion. The underlying rationale is rooted in the observation that distinct
parameters play varying roles in learning diverse graph patterns. Building
upon this insight, the approach expands model parameters to enable the
assimilation of emerging graph patterns.

Discussion. Both in general continual learning and graph continual learning
scenarios, architectural approaches are key to managing the balance between
learning new tasks and retaining old information. While the general continual
learning setting often focuses on network-level adaptations, graph continual
learning requires specific techniques tailored to the unique nature of graph-
based data, such as evolving graph structures or managing graph complexity
through expansion and compression. These approaches underscore the necessity
for adaptable and flexible models capable of handling the dynamic nature of
continuous learning environments.

###  6.2 Regularization Approach

This approach implements a single model and has a fixed capacity by leveraging
the loss function using the loss term to help consolidate knowledge in the
learning process for new tasks and retain previous knowledge [90, 91]. Prior
knowledge of graph structures and tasks will be maintained to achieve stable
performance while learning novel knowledge.

(a) The disentangle module of DiCGRL.

(b) The updating module of DiCGRL.

Figure 18: The framework of DiCGRL. DiCGRL takes two approaches to
continuously learn a new graph embedding whilst preventing the forgetting of
old knowledge.

DiCGR [79] is an approach designed to support continual learning on graph
data. It addresses the challenge of preserving previously acquired knowledge
while learning new information. DiCGRL achieves it by breaking down relational
triplets into individual components, refining the embeddings to better capture
the graph structure, and incorporating a regularization mechanism to prevent
forgetting of previously learned knowledge. DiCGRL takes two approaches to
continuously learn a new graph embedding whilst preventing the forgetting of
old knowledge, respectively the disentangle module shown in Fig. 18(a) and the
updating module shown in Fig. 18(b).

GPIL [13] is a framework that aims at acquiring proficiency in different
classes. It utilizes a recurrent strategy, pre-training the encoder on base
classes and maintaining it during pseudo incremental learning. To mitigate
catastrophic forgetting, this approach introduces a Hierarchical Attention
Graph Meta Learning framework with augmented loss function using
regularization techniques.

Figure 19: The framework of TWP. TWP leverages the gradients of task-specific
objectives alongside topological preservation in relation to each parameter.

TWP [80] is a framework designed to bolster continual learning while
mitigating the risk of catastrophic forgetting within GNNs.As shown in Fig.
19, TWP computes the importance score for each network parameter, rooted in
its contribution to both the graph’s topological architecture and its
effectiveness in task execution. TWP leverages the gradients of task-specific
objectives alongside topological preservation in relation to each parameter.

GraphSAIL [81] is an Incremental Learning framework for GNN-based recommender
systems. It enables efficient updates, reducing computation time for more
frequent recommendations. GraphSAIL implements inter-distillation to evaluate
the agreement of high-level node encoding between the teacher and student
models, enhancing the performance of its recommendation system.

RieGrace [82] is a self-supervised Riemannian Graph Continual Learner. It
incorporates an Adaptive Riemannian GCN that shapes the Riemannian space using
learned curvature adaptive to each graph. Additionally, it utilizes a Label-
free Lorentz Distillation approach with teacher-student AdaRGCN models for the
graph sequence. Overall, RieGrace enables graph continual learning in adaptive
Riemannian spaces through self-supervision.

LKGE [83] is a model for lifelong learning in expanding knowledge graphs. It
utilizes a masked KG autoencoder, embedding transfer strategy, and embedding
regularization to efficiently transfer and retain knowledge. It streamlines
the process of incorporating new data while preserving existing knowledge in
KG embeddings.

Discussion. The regularization approach in continual learning, particularly
for graph-based models, is a strategy that emphasizes the importance of
retaining prior knowledge while acquiring new information. By carefully
modifying the loss function, this approach seeks to maintain stable
performance across a series of tasks without expanding the model’s capacity.
This is particularly crucial in graph-based applications where the
relationships and structures in the data are complex and evolving.

###  6.3 Rehearsal Approach

This approach regulates retraining processes for previous tasks to strengthen
the relationship between memory and performance on previously learned tasks
[92]. This graph learning approach enables the selection of appropriate
samples of graph representation, such as nodes and edges, for retraining
purposes. The number of samples is carefully considered to minimize the
computational complexity.

ER-GNN [84] is a framework that implements a graph-based experience replay
mechanism. This framework’s core principle involves preserving historical
knowledge within an experience buffer, drawn from the preceding task learning
phases. ER-GNN prevents catastrophic forgetting by retaining previously
learned knowledge using an experienced buffer. It selects and stores relevant
nodes in the buffer using an experience node strategy.

Incremental-GNNs [85] is a model specifically designed for incremental
training on graph data. The model utilizes prior knowledge to enhance the
learning process across a sequence of tasks. This acquired knowledge can be
explicitly stored as historical data or implicitly encoded within the model’s
parameters. The model then evaluates the impact of this accumulated knowledge,
leveraging it to improve the performance of the network.

RBR and PBR (Random-Based Rehearsal and Priority-Based Rehearsal) [86] are
methods for training GNNs on streaming graph data. They use experience replay
with a node sampling strategy that prioritizes higher loss samples. This
allows for continuous learning and adaptation to changing dynamic graph
streams, aiming for faster training, low latency, and high accuracy.

Inverse Degree Sampling [87] is an approach in graph-based recommender systems
that focuses on selecting interaction subsets using an inverse degree
experience sampling strategy. By redirecting the model’s attention to
interactions involving users with infrequent interactions, this strategy
enhances the applicability of reservoir sampling in continual sampling
contexts.

Discussion. In graph continual learning, the rehearsal approach offers a
viable method for reinforcing the model’s memory of previous tasks while
learning new ones. By judiciously retraining on selected graph samples, this
approach helps maintain a balance between retaining past knowledge and
acquiring new information, ensuring sustained performance across a series of
evolving tasks. This strategy is particularly relevant in dynamic environments
where graph data continuously changes or expands.

###  6.4 Hybrid Approach

The hybrid approach combines more than one continual learning approach to take
advantage of each approach and maximize the performance of models [93, 94].

ContinualGNN [1] is a framework that combines two distinct strategies: a
replaying strategy and a regularization approach, functioning synergistically
to mitigate catastrophic forgetting and maintain the established learned
patterns. It identifies and incorporates novel patterns in the evolving graph
structure while solidifying knowledge to maintain acquired insights throughout
the network.

Figure 20: The framework of LDANE. LDANE incorporates attribute constraints to
align learned embeddings with specific attributes, adapting to changes in node
attributes.

LDANE [2] is a framework that acquires low-dimensional representations for
nodes in an expanding network. Underpinned by an architectural paradigm known
as Dynamic Expandable Networks [95], LDANE advances the field by integrating
attribute constraints. As shown in Fig. 20, it incorporates attribute
constraints to align learned embeddings with specific attributes, adapting to
changes in node attributes.

TrafficStream [88] is a framework that captures patterns for new graph
structures and consolidates knowledge between previous and new ones. It uses a
traffic flow forecasting model (SurModel) to represent intelligent
transportation system data. It efficiently updates current network patterns
through processes that consider network structure changes caused by factors
like the addition of new sensors or stations.

CMDGAT [89] is a model that enhances the descriptive capacity of point clouds
by using a novel graph continual network architecture. The architecture
includes an attention mechanism that leverages knowledge from previous point
cloud pairs to associate and aligns points in the current pair. By utilizing
insights from past associations, the network optimally registers points,
improving the overall representation of the point clouds.

Discussion. The hybrid approach in graph continual learning offers a versatile
and effective way to tackle the challenges of learning in dynamic and evolving
environments. By combining the strengths of various continual learning
methods, it aims to achieve robust performance, adaptability, and efficient
use of computational resources. This approach is particularly relevant in
complex learning scenarios where no single method can adequately address all
challenges.

##  7 Applications

###  7.1 AI-aided Drug Discovery.

AI-aided drug discovery [17] is becoming increasingly prevalent when it offers
the great potential to accelerate the development of new pharmaceuticals,
making the process faster, more cost-effective, and more efficient. It has
wide application scenarios across various domains, including generative
chemistry [96], protein structure prediction [97, 98], virtual screening [99,
100], drug repurposing for emerging diseases [101], and many others. Due to
the distribution shift issues of graph data, the compatibility assumption
between training and test sets does not usually hold in the drug discovery-
related fields, especially when the learned knowledge from the training set
may not smoothly provide accurate predicting signals for test samples [6, 17,
8, 7, 9, 11].

For instance, when performing virtual screening for hit finding [17], the
prediction model is typically trained on known target proteins. However, a
‘black swan’ event like COVID-19 can occur, resulting in a new target with
unseen data distribution. The performance on the new target will significantly
degrade. Moreover, for molecular prediction [9], the GNN models trained on
molecules with one group of scaffolds, i.e., the two-dimensional structural
frameworks of molecules, might learn the spurious correlations between the
scaffolds and labels, so that the GNN models would make incorrect predictions
on real-world out-of-distribution testing molecules with different scaffolds.

Hence, dealing with distribution shifts on graph data is a critical way to
address the performance degradation for practical AI-aided drug discovery. The
potential solutions contain but are not limited to designing robust and
generalizable graph learning models through domain adaption, OOD modeling, and
continuing graph learning; Meanwhile, developing curated distribution shifted
graph datasets and benchmarks to better understand distribution shift cases
and characteristics for facilitating and promoting the development of drug
discovery.

###  7.2 Personalized Intelligent Recommendation Systems.

In the recommender system, the training samples with nodes and relations are
usually observed and collected with a certain selection bias [62], that is,
the logged interactions collected from current undergoing systems may warp
(graph) recommenders biased towards popular items, leading to potential
distribution shifts on the deployed environment with caused unfairness towards
less exposed items. Moreover, considering male users often have fewer logged
interactions on e-commerce platforms, the graph recommenders may be trained
with a bias towards female users and thus deteriorate the experience of male
users.

Hence, explicitly reducing the performance variances across different
deployment environments with distribution shifts is the key to the graph
learning based recommender system, for confronting the inconsistencies between
the training and testing distributions, and achieving fairness for different
groups of users. Moreover, due to the continuous and frequent updating
characteristic of recommender systems, tackling the catastrophic forgetting
problem that occurs when training a model in an incremental pattern would
benefit modeling a user’s long-term preference (or an item’s long-term
property) under the evolving distribution shifts [81, 14, 13, 87].

In this way, various graph learning models dealing with distribution shifts
would benefit real-world recommender systems to make more adaptive and stable
predictions of practical unseen users’ preferences, leading to more accurate
and intelligent recommendation [5, 8, 62].

###  7.3 Open-world Knowledge Exploration and Learning.

Real-world information representation and related applications usually rely on
knowledge-driven approaches that utilize knowledge graphs (KGs) with
comprehensive knowledge exploration and learning. Throughout the whole life-
cycle of KG, new facts, unseen entities, and unseen relations continually
emerge, evolve, and grow alongside the development, leading to inevitable
dynamic distribution shifts [74, 83]. In this case, conducting open-world
knowledge exploration and investigating lifelong KG learning under such
temporal distribution shifts, would facilitate the learning and updating from
growing new knowledge, facts, and relations, meanwhile, preserving the
previously learned useful old knowledge and serving for various KG-related
application scenarios [102], covering KG representation learning, KG question
and answering, KG completion [103].

Moreover, such knowledge exploration in open-world KG under distribution
shifts also provides a great chance to collaborate existing KG learning with
the recent prevalent large language models (LLMs) [104, 105], considering LLMs
and KGs are inherently interconnected and can mutually enhance each other. For
instance, DAGNN [32] conducts cross-domain text classification by modeling
documents as graph data structure, and combining the hierarchical structure
and domain labels into a unified deep model. And ChatRule [106] unleashes the
power of LLMs for mining logical rules over knowledge graphs for enhancing the
reasoning ability. It can be anticipated that open-world knowledge exploration
and learning with KG related temporal shift graph learning would be capable of
sophisticated reasoning and decision-making based on a deep understanding of
real-world knowledge, ultimately driving forward the frontiers of AI research
and its practical applications.

###  7.4 Intelligent Transportation System Forecasting.

Traffic flows typically show high-quantity, long-term streaming evolution, and
gradual expansion characteristics in traffic networks. Accurately forecasting
traffic flows plays an important role in improving the efficiency of
intelligent transportation systems [88]. Considering the traffic networks
would change flows of nodes and expanding structures in a long-term period
indicated by spatial-temporal graphs [107, 108], the distribution shift in the
inconsistency between patterns (context attribute shift and the spatial
structure shift) of new data and historical data would negatively affect the
accuracy of forecasting. Hence, the key to conducting graph learning on
intelligent transportation system forecasting under such distribution shifts
is to jointly incorporate long-term stable patterns and temporarily
disappearing patterns, for efficiently capturing patterns from new data while
consolidating historical knowledge using graph continual learning approaches.

Hence, the ongoing development of graph learning models under distribution
shifts with continual learning holds great promise for the future of
transportation management and optimization. It can be anticipated that we can
enhance the accuracy of traffic flow predictions and ultimately contribute to
the efficiency and effectiveness of intelligent transportation systems.

##  8 Future Directions

###  8.1 Graph Data-centric Learning under Distribution Shifts.

Very recently, there has been a growing interest in data-centric graph machine
learning (DC-GML) for the recognition and prioritization of the importance of
graph data [109]. Modeling graph data shifts in cross-domains and various
distributions [110] is an important aspect of DC-GML, as the domain and
distribution shifts of graph data would significantly affect the
generalization ability and robustness of graph learning algorithms. For one
thing, there is some recent research focusing on exploiting the distinct graph
distribution characteristics, emphasizing invariance, stability, evolution,
etc. [111, 112, 8]. For another thing, a very young research topic graph
learning model evaluation problem [113] is explored to observe and predict the
GNN model performance under various graph data distribution shifts (e.g.,
domain changes) without test-time ground truth labels.

Therefore, emphasizing the investigation of distribution shifts in graph data-
centric characteristics, while also tackling the co-development of
distribution-shifted graph data and models, holds great potential as a future
research avenue. This approach ensures that optimization and customization are
applied to both elements, catering to a wide range of practical scenarios in
graph learning and analysis. Consequently, it facilitates a comprehensive
understanding of graph data distribution shifts and enhances the design and
learning process of graph models, ultimately resulting in improved performance
in data-centric graph machine learning.

###  8.2  Cross-modality Distribution Shift Exploration on Graph Learning.

Data types in the real world exhibit various modalities and scales, e.g.,
images, texts, tabular data, audios, and graphs, which are crucial for the
advancement of artificial intelligence and machine learning communities [114].
Considering the complexity of cross-modal data, learning algorithms are used
to make inferences for test data they have not encountered during training,
leading to inevitable inductive bias. Meanwhile, there are natural and more
general distribution shifts among modalities used during model training and
deployment. In such cases, as an important and commonly used structural data
type, graph data might not be explicitly given for training and inference in
real-world multi-modal model development [115].

Hence, constructing graphs from multi-modal data with different node
representations, relationships, and attributes, can be a promising future
research direction under potentially more complex distribution shifts, such as
scene graph construction from images [116], and knowledge graph construction
from texts [105, 117, 32]. Meanwhile, another crucial future problem lies in
conducting effective graph learning on these constructed graphs under
distribution shifts. This involves thoroughly incorporating cross-modal
dependencies. For example, this could entail learning about structural shifts
using flexible message propagation strategies and adaptively transferring
learned node or graph representations across various distributions and
downstream tasks.

###  8.3 Comprehensive Graph Learning Task-driven Evaluation Protocols.

Given the complex and diverse nature of graph data distribution shifts, there
is still potential to enhance the evaluation protocols used in various
downstream tasks of graph learning for future exploration. These improved
protocols should focus on two key aspects: at the model level, to measure
performance across different tasks, and at the graph data level, to assess
data distribution driven by both the graph learning models and the tasks they
are applied to. For instance, in terms of OOD generalization baselines, graph
learning models should be evaluated on both near and far out-of-distribution
graph datasets, which may originate from different domain sources. An instance
of this would be training models on citation networks and then evaluating
their performance on social networks [118].

Therefore, future research should concentrate on developing effective metrics
to evaluate the distance or discrepancy among various graph data, considering
both node-level analysis in a single graph and collective graph-level analysis
in an entire dataset. Additionally, it is also important to focus on
establishing fair and uniform methods to compare and evaluate graph learning
models under varying degrees of distribution shifts [113]. This should
encompass diverse scenarios including graph domain adaptation, OOD
generalization, detection, prediction, etc.

###  8.4 Trustworthy Graph Learning Under Distribution Shifts.

Graph data distribution shifts impose significant challenges to various
aspects of trustworthiness in graph learning [119], including adversarial
robustness under graph distribution related attacks, model explainability to
diverse and complex graph distributions, as well as privacy and fairness
guarantee under different distribution transfer or domain adaption learning.

From the adversarial robustness point of view [120], when small but
intentional adversarial perturbations are imposed on the input graphs, these
alterations can lead to even more significant and unpredictable shifts in
graph data distributions [121, 122]. As a result, this makes graph learning
models more vulnerable and less resistant to adversarial attacks. Hence,
enhancing the robustness of graph learning models under adversarial attacks on
graph data distributions could be a valuable direction to explore for future
research.

Moreover, considering the explainability [123], future graph learning models
are expected to have the ability to interpret or explain their decisions or
outputs when models are dealing with different graph distributions with
diverse characteristics [124].

Furthermore, in terms of the privacy and fairness [125, 126], when graph data
comes from sensitive personal sources or a certain graph learning model is
trained on one specific distribution domain, how to maintain privacy across
different domains within different graph data distributions is still a
challenging aspect for privacy-guaranteed graph learning. Besides, how to
ensure that the predictions of graph learning models do not systematically
disadvantage any particular group or distribution with fairness is another
promising future research question when deployed across varied graph data
domains.

##  9 Conclusion

In this work, we present an in-depth review and synthesis of the forefront
approaches and methodologies in graph learning under distribution shifts. We
categorized graph learning methods into graph domain adaptation learning,
graph out-of-distribution learning, and graph continual learning, based on
their observability of distributions and the availability of supervision
information, where each with a detailed taxonomy and discussion of the current
advancements. The comprehensive analysis, along with the highlighted
discussions on potential applications and future research directions, not only
provides a clear road map of the state-of-the-art approaches, but also help
shed light on the effective development of graph learning techniques with
diverse and complex distribution shifts.

## References

  * [1] J. Wang, G. Song, Y. Wu, and L. Wang, “Streaming graph neural networks via continual learning,” in _Proceedings of the 29th ACM international conference on information & knowledge management_, 2020, pp. 1515–1524.
  * [2] H. Wei, G. Hu, W. Bai, S. Xia, and Z. Pan, “Lifelong representation learning in dynamic attributed networks,” _Neurocomputing_ , vol. 358, 05 2019.
  * [3] M. Pilanci and E. Vural, “Domain adaptation on graphs by learning aligned graph bases,” _IEEE Transactions on Knowledge and Data Engineering_ , 2020\.
  * [4] Q. Dai, X.-M. Wu, J. Xiao, X. Shen, and D. Wang, “Graph transfer learning via adversarial domain adaptation with graph convolution,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 5, pp. 4908–4922, 2022.
  * [5] Y. Ouyang, B. Guo, X. Tang, X. He, J. Xiong, and Z. Yu, “Learning cross-domain representation with multi-graph neural network,” _CoRR_ , vol. abs/1905.10095, 2019.
  * [6] Y. Wang, C. Li, W. Jin, R. Li, J. Zhao, J. Tang, and X. Xie, “Test-time training for graph neural networks,” _arXiv preprint arXiv:2210.08813_ , 2022\.
  * [7] G. Chen, J. Zhang, X. Xiao, and Y. Li, “Graphtta: Test time adaptation on graph neural networks,” _arXiv preprint arXiv:2208.09126_ , 2022.
  * [8] W. Jin, T. Zhao, J. Ding, Y. Liu, J. Tang, and N. Shah, “Empowering graph representation learning with test-time graph transformation,” _arXiv preprint arXiv:2210.03561_ , 2022.
  * [9] H. Li, X. Wang, Z. Zhang, and W. Zhu, “Ood-gnn: Out-of-distribution generalized graph neural network,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [10] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” _Advances in neural information processing systems_ , vol. 33, pp. 5812–5823, 2020.
  * [11] Z. Li, Q. Wu, F. Nie, and J. Yan, “Graphde: A generative framework for debiased learning and out-of-distribution detection on graphs,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 30 277–30 290, 2022.
  * [12] C. Zheng, X. Fan, S. Pan, H. Jin, Z. Peng, Z. Wu, C. Wang, and S. Y. Philip, “Spatio-temporal joint graph convolutional networks for traffic forecasting,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023\.
  * [13] Z. Tan, K. Ding, R. Guo, and H. Liu, “Graph few-shot class-incremental learning,” in _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , 2022, pp. 987–996.
  * [14] X. Zhang, D. Song, and D. Tao, “Hierarchical prototype networks for continual graph representation learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [15] M. Mancini, S. R. Bulo, B. Caputo, and E. Ricci, “Adagraph: Unifying predictive and continuous domain adaptation through graphs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6568–6577.
  * [16] F. Xia, K. Sun, S. Yu, A. Aziz, L. Wan, S. Pan, and H. Liu, “Graph learning: A survey,” _CoRR_ , vol. abs/2105.00696, 2021.
  * [17] Y. Ji, L. Zhang, J. Wu, B. Wu, L. Li, L.-K. Huang, T. Xu, Y. Rong, J. Ren, D. Xue _et al._ , “Drugood: Out-of-distribution dataset curator and benchmark for ai-aided drug discovery–a focus on affinity prediction problems with noise annotations,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 7, 2023, pp. 8023–8031.
  * [18] J. Qiu, J. Tang, H. Ma, Y. Dong, K. Wang, and J. Tang, “Deepinf: Social influence prediction with deep learning,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, Y. Guo and F. Farooq, Eds. ACM, 2018, pp. 2110–2119.
  * [19] L. Luo, Y.-F. Li, G. Haffari, and S. Pan, “Normalizing flow-based neural process for few-shot knowledge graph completion,” _arXiv preprint arXiv:2304.08183_ , 2023.
  * [20] S. Wu, F. Sun, W. Zhang, X. Xie, and B. Cui, “Graph neural networks in recommender systems: A survey,” _ACM Comput. Surv._ , vol. 55, no. 5, pp. 97:1–97:37, 2023. [Online]. Available: https://doi.org/10.1145/3535101
  * [21] X. Zheng, M. Zhang, C. Chen, C. Li, C. Zhou, and S. Pan, “Multi-relational graph neural architecture search with fine-grained message passing,” in _2022 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2022, pp. 783–792.
  * [22] Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” _IEEE Trans. Knowl. Data Eng._ , vol. 29, no. 12, pp. 2724–2743, 2017. [Online]. Available: https://doi.org/10.1109/TKDE.2017.2754499
  * [23] D. Jin, L. Wang, Y. Zheng, G. Song, F. Jiang, X. Li, W. Lin, and S. Pan, “Dual intent enhanced graph neural network for session-based new item recommendation,” in _Proceedings of the ACM Web Conference 2023_ , 2023, pp. 684–693.
  * [24] B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)_ , 2018\.
  * [25] H. Li, X. Wang, Z. Zhang, and W. Zhu, “Out-of-distribution generalization on graphs: A survey,” _arXiv preprint arXiv:2202.07987_ , 2022.
  * [26] C. Pomeroy, R. M. Bond, P. J. Mucha, and S. J. Cranmer, “Dynamics of social network emergence explain network evolution,” _Scientific Reports_ , vol. 10, no. 1, p. 21876, 2020.
  * [27] M. Bardoscia, P. Barucca, S. Battiston, F. Caccioli, G. Cimini, D. Garlaschelli, F. Saracco, T. Squartini, and G. Caldarelli, “The physics of financial networks,” _Nature Reviews Physics_ , vol. 3, no. 7, pp. 490–507, 2021.
  * [28] D. Acemoglu, A. Ozdaglar, and A. Tahbaz-Salehi, “Systemic risk and stability in financial networks,” _American Economic Review_ , vol. 105, no. 2, pp. 564–608, 2015.
  * [29] H. Wu, C. M. Eckhardt, and A. A. Baccarelli, “Molecular mechanisms of environmental exposures and human disease,” _Nature Reviews Genetics_ , vol. 24, no. 5, pp. 332–344, 2023.
  * [30] T. Ercan, N. C. Onat, O. Tatari, and J.-D. Mathias, “Public transportation adoption requires a paradigm shift in urban development structure,” _Journal of cleaner production_ , vol. 142, pp. 1789–1799, 2017.
  * [31] F. Alam, S. Joty, and M. Imran, “Domain adaptation with adversarial training and graph embeddings,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 1077–1087.
  * [32] M. Wu, S. Pan, X. Zhu, C. Zhou, and L. Pan, “Domain-adversarial graph neural networks for text classification,” in _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2019, pp. 648–657.
  * [33] Y. Chen, Y. Zhang, Y. Bian, H. Yang, M. Kaili, B. Xie, T. Liu, B. Han, and J. Cheng, “Learning causally invariant representations for out-of-distribution generalization on graphs,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 22 131–22 148, 2022.
  * [34] H. Li, Z. Zhang, X. Wang, and W. Zhu, “Learning invariant graph representations for out-of-distribution generalization,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [35] A. Baranwal, K. Fountoulakis, and A. Jagannath, “Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 684–693.
  * [36] D. Maltoni and V. Lomonaco, “Continuous learning in single-incremental-task scenarios,” _Neural Networks_ , vol. 116, pp. 56–73, 2019.
  * [37] M. Biesialska, K. Biesialska, and M. R. Costa-jussà, “Continual lifelong learning in natural language processing: A survey,” in _Proceedings of the 28th International Conference on Computational Linguistics_. Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 6523–6541.
  * [38] P. Ruvolo and E. Eaton, “ELLA: An efficient lifelong learning algorithm,” in _Proceedings of the 30th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, S. Dasgupta and D. McAllester, Eds., vol. 28, no. 1. Atlanta, Georgia, USA: PMLR, 17–19 Jun 2013, pp. 507–515. [Online]. Available: https://proceedings.mlr.press/v28/ruvolo13.html
  * [39] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” _CoRR_ , vol. abs/1606.04671, 2016. [Online]. Available: http://arxiv.org/abs/1606.04671
  * [40] V. Lomonaco and D. Maltoni, “Core50: a new dataset and benchmark for continuous object recognition,” _CoRR_ , vol. abs/1705.03550, 2017. [Online]. Available: http://arxiv.org/abs/1705.03550
  * [41] D. Das and C. G. Lee, “Unsupervised domain adaptation using regularized hyper-graph matching,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 3758–3762.
  * [42] Y. Zhang, S. Miao, and R. Liao, “Structural domain adaptation with latent graph alignment,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 3753–3757.
  * [43] M. Wu, S. Pan, C. Zhou, X. Chang, and X. Zhu, “Unsupervised domain adaptive graph convolutional networks,” in _Proceedings of The Web Conference 2020_ , 2020, pp. 1457–1467.
  * [44] J. Yang, K. Zhou, Y. Li, and Z. Liu, “Generalized out-of-distribution detection: A survey,” _arXiv preprint arXiv:2110.11334_ , 2021.
  * [45] Q. Yuan, S.-U. Guan, P. Ni, T. Luo, K. L. Man, P. Wong, and V. Chang, “Continual graph learning: A survey,” _arXiv preprint arXiv:2301.12230_ , 2023.
  * [46] F. G. Febrinanto, F. Xia, K. Moore, C. Thapa, and C. Aggarwal, “Graph lifelong learning: A survey,” _IEEE Computational Intelligence Magazine_ , vol. 18, no. 1, pp. 32–51, 2023.
  * [47] Y. Yu, W.-Y. Qu, N. Li, and Z. Guo, “Open-category classification by adversarial sample generation,” in _International Joint Conference on Artificial Intelligence_ , 2017, pp. 3357–3363.
  * [48] Y. Luo, Z. Wang, Z. Huang, and M. Baktashmotlagh, “Progressive graph learning for open-set domain adaptation,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 6468–6478.
  * [49] X. Ma, T. Zhang, and C. Xu, “Gcan: Graph convolutional adversarial network for unsupervised domain adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8266–8276.
  * [50] X. Yang, C. Deng, T. Liu, and D. Tao, “Heterogeneous graph attention network for unsupervised multiple-target domain adaptation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 4, pp. 1992–2003, 2020.
  * [51] X. Shen, Q. Dai, S. Mao, F.-l. Chung, and K.-S. Choi, “Network together: Node classification via cross-network deep network embedding,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 5, pp. 1935–1948, 2020.
  * [52] X. Shen, Q. Dai, F.-l. Chung, W. Lu, and K.-S. Choi, “Adversarial deep network embedding for cross-network node classification,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 03, 2020, pp. 2991–2999.
  * [53] X. Shen, S. Pan, K.-S. Choi, and X. Zhou, “Domain-adaptive message passing graph neural network,” _Neural Networks_ , vol. 164, pp. 439–454, 2023.
  * [54] X. Shen, M. Shao, S. Pan, L. T. Yang, and X. Zhou, “Domain-adaptive graph attention-supervised network for cross-network edge classification,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [55] T. Zhao, Y. Liu, L. Neves, O. Woodford, M. Jiang, and N. Shah, “Data augmentation for graph neural networks,” in _Proceedings of the aaai conference on artificial intelligence_ , vol. 35, no. 12, 2021, pp. 11 015–11 023.
  * [56] X. Shen, D. Sun, S. Pan, X. Zhou, and L. T. Yang, “Neighbor contrastive learning on learnable graph augmentation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 8, 2023, pp. 9782–9791.
  * [57] W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, “Graph random neural networks for semi-supervised learning on graphs,” _Advances in neural information processing systems_ , vol. 33, pp. 22 092–22 103, 2020.
  * [58] M. McPherson, L. Smith-Lovin, and J. M. Cook, “Birds of a feather: Homophily in social networks,” _Annual review of sociology_ , vol. 27, no. 1, pp. 415–444, 2001.
  * [59] M. Ding, K. Kong, J. Chen, J. Kirchenbauer, M. Goldblum, D. Wipf, F. Huang, and T. Goldstein, “A closer look at distribution shifts and out-of-distribution generalization on graphs,” in _NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2021.
  * [60] J. Ma, P. Cui, K. Kuang, X. Wang, and W. Zhu, “Disentangled graph convolutional networks,” in _International conference on machine learning_. PMLR, 2019, pp. 4212–4221.
  * [61] K. Kuang, P. Cui, S. Athey, R. Xiong, and B. Li, “Stable prediction across unknown environments,” in _proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1617–1626.
  * [62] S. Zhang, K. Kuang, J. Qiu, J. Yu, Z. Zhao, H. Yang, Z. Zhang, and F. Wu, “Stable prediction on graphs with agnostic distribution shift,” _arXiv preprint arXiv:2110.03865_ , 2021.
  * [63] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for pre-training graph neural networks,” _arXiv preprint arXiv:1905.12265_ , 2019.
  * [64] X. Zhao, F. Chen, S. Hu, and J.-H. Cho, “Uncertainty aware semi-supervised learning on graph data,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 827–12 836, 2020.
  * [65] M. Stadler, B. Charpentier, S. Geisler, D. Zügner, and S. Günnemann, “Graph posterior network: Bayesian predictive uncertainty for node classification,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 18 033–18 048, 2021.
  * [66] T. Huang, D. Wang, Y. Fang, and Z. Chen, “End-to-end open-set semi-supervised node classification with out-of-distribution detection,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , L. D. Raedt, Ed. International Joint Conferences on Artificial Intelligence Organization, 7 2022, pp. 2087–2093.
  * [67] Y. Liu, K. Ding, H. Liu, and S. Pan, “Good-d: On unsupervised graph out-of-distribution detection,” in _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_ , 2023, pp. 339–347.
  * [68] Q. Zhang, Z. Shi, X. Zhang, X. Chen, P. Fournier-Viger, and S. Pan, “G2pxy: generative open-set node classification on graphs with proxy unknowns,” in _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_ , 2023, pp. 4576–4583.
  * [69] Q. Zhang, Q. Li, X. Chen, P. Zhang, S. Pan, P. Fournier-Viger, and J. Z. Huang, “A dynamic variational framework for open-world node classification in structured sequences,” in _2022 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2022, pp. 703–712.
  * [70] M. Wu, S. Pan, and X. Zhu, “Openwgl: Open-world graph learning,” in _2020 IEEE international conference on data mining (icdm)_. IEEE, 2020, pp. 681–690.
  * [71] Q. Wu, C. Yang, and J. Yan, “Towards open-world feature extrapolation: An inductive graph learning approach,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 19 435–19 447, 2021.
  * [72] M. Mancini, M. F. Naeem, Y. Xian, and Z. Akata, “Learning graph embeddings for open world compositional zero-shot learning,” _IEEE Transactions on pattern analysis and machine intelligence_ , 2022.
  * [73] B. Shi and T. Weninger, “Open-world knowledge graph completion,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [74] R. Das, A. Godbole, N. Monath, M. Zaheer, and A. McCallum, “Probabilistic case-based reasoning for open-world knowledge graph completion,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 4752–4765.
  * [75] L. Niu, C. Fu, Q. Yang, Z. Li, Z. Chen, Q. Liu, and K. Zheng, “Open-world knowledge graph completion with multiple interaction attention,” _World Wide Web_ , vol. 24, pp. 419–439, 2021.
  * [76] C. Wang, Y. Qiu, D. Gao, and S. Scherer, “Lifelong graph learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 13 719–13 728.
  * [77] L. Hedegaard, N. Heidari, and A. Iosifidis, “Continual spatio-temporal graph convolutional networks,” _Pattern Recogn._ , vol. 140, no. C, may 2023.
  * [78] P. Zhang, Y. Yan, C. Li, S. Wang, X. Xie, G. Song, and S. Kim, “Continual learning on dynamic graphs via parameter isolation,” in _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , ser. SIGIR ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 601–611.
  * [79] X. Kou, Y. Lin, S. Liu, P. Li, J. Zhou, and Y. Zhang, “Disentangle-based Continual Graph Representation Learning,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 2961–2972.
  * [80] H. Liu, Y. Yang, and X. Wang, “Overcoming catastrophic forgetting in graph neural networks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 35, no. 10, 2021, pp. 8653–8661.
  * [81] Y. Xu, Y. Zhang, W. Guo, H. Guo, R. Tang, and M. Coates, “Graphsail: Graph structure aware incremental learning for recommender systems,” in _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. Association for Computing Machinery, 2020, p. 2861–2868.
  * [82] L. Sun, J. Ye, H. Peng, F. Wang, and P. S. Yu, “Self-supervised continual graph learning in adaptive riemannian spaces,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 4, pp. 4633–4642, 2023.
  * [83] Y. Cui, Y. Wang, Z. Sun, W. Liu, Y. Jiang, K. Han, and W. Hu, “Lifelong embedding learning and transfer for growing knowledge graphs,” in _AAAI_ , 2023.
  * [84] F. Zhou and C. Cao, “Overcoming catastrophic forgetting in graph neural networks with experience replay,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 5, 2021, pp. 4714–4722.
  * [85] L. Galke, I. Vagliano, and A. Scherp, “Incremental training of graph neural networks on temporal graphs under distribution shift,” _CoRR_ , vol. abs/2006.14422, 2020.
  * [86] M. Perini, G. Ramponi, P. Carbone, and V. Kalavri, “Learning on streaming graphs with experience replay,” in _Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing_ , 2022, pp. 470–478.
  * [87] K. Ahrabian, Y. Xu, Y. Zhang, J. Wu, Y. Wang, and M. Coates, “Structure aware experience replay for incremental learning in graph-based recommender systems,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 2832–2836.
  * [88] X. Chen, J. Wang, and K. Xie, “Trafficstream: A streaming traffic flow forecasting framework based on graph neural networks and continual learning,” _CoRR_ , vol. abs/2106.06273, 2021.
  * [89] A. Zaman, F. Yangyu, M. S. Ayub, M. Irfan, L. Guoyun, and L. Shiya, “Cmdgat: Knowledge extraction and retention based continual graph attention network for point cloud registration,” _Expert Syst. Appl._ , vol. 214, no. C, mar 2023.
  * [90] Z. Li and D. Hoiem, “Learning without forgetting,” in _Computer Vision – ECCV 2016_ , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 614–629.
  * [91] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [92] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5533–5542, 2016.
  * [93] D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continuum learning,” _CoRR_ , vol. abs/1706.08840, 2017. [Online]. Available: http://arxiv.org/abs/1706.08840
  * [94] D. Maltoni and V. Lomonaco, “Continuous learning in single-incremental-task scenarios,” _CoRR_ , vol. abs/1806.08568, 2018. [Online]. Available: http://arxiv.org/abs/1806.08568
  * [95] J. Lee, J. Yoon, E. Yang, and S. J. Hwang, “Lifelong learning with dynamically expandable networks,” _CoRR_ , vol. abs/1708.01547, 2017.
  * [96] G. Sliwoski, S. Kothiwale, J. Meiler, and E. W. Lowe, “Computational methods in drug discovery,” _Pharmacological reviews_ , vol. 66, no. 1, pp. 334–395, 2014.
  * [97] W. Gao, S. P. Mahajan, J. Sulam, and J. J. Gray, “Deep learning in protein structural modeling and design,” _Patterns_ , vol. 1, no. 9, 2020.
  * [98] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko _et al._ , “Highly accurate protein structure prediction with alphafold,” _Nature_ , vol. 596, no. 7873, pp. 583–589, 2021.
  * [99] J. Lim, S. Ryu, K. Park, Y. J. Choe, J. Ham, and W. Y. Kim, “Predicting drug–target interaction using a novel graph neural network with 3d structure-embedded graph representation,” _Journal of chemical information and modeling_ , vol. 59, no. 9, pp. 3981–3988, 2019.
  * [100] M. Karimi, D. Wu, Z. Wang, and Y. Shen, “Deepaffinity: interpretable deep learning of compound–protein affinity through unified recurrent and convolutional neural networks,” _Bioinformatics_ , vol. 35, no. 18, pp. 3329–3338, 2019.
  * [101] D. Morselli Gysi, Í. Do Valle, M. Zitnik, A. Ameli, X. Gan, O. Varol, S. D. Ghiassian, J. Patten, R. A. Davey, J. Loscalzo _et al._ , “Network medicine framework for identifying drug-repurposing opportunities for covid-19,” _Proceedings of the National Academy of Sciences_ , vol. 118, no. 19, p. e2025581118, 2021.
  * [102] S. Ji, S. Pan, E. Cambria, P. Marttinen, and S. Y. Philip, “A survey on knowledge graphs: Representation, acquisition, and applications,” _IEEE transactions on neural networks and learning systems_ , vol. 33, no. 2, pp. 494–514, 2021.
  * [103] J. Wang, B. Wang, M. Qiu, S. Pan, B. Xiong, H. Liu, L. Luo, T. Liu, Y. Hu, B. Yin _et al._ , “A survey on temporal knowledge graph completion: Taxonomy, progress, and prospects,” _arXiv preprint arXiv:2308.02457_ , 2023\.
  * [104] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu, “Unifying large language models and knowledge graphs: A roadmap,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.
  * [105] L. Luo, Y.-F. Li, G. Haffari, and S. Pan, “Reasoning on graphs: Faithful and interpretable large language model reasoning,” _arXiv preprint arXiv:2310.01061_ , 2023.
  * [106] L. Luo, J. Ju, B. Xiong, Y.-F. Li, G. Haffari, and S. Pan, “Chatrule: Mining logical rules with large language models for knowledge graph reasoning,” _arXiv preprint arXiv:2309.01538_ , 2023.
  * [107] M. Jin, Y.-F. Li, and S. Pan, “Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 19 874–19 886, 2022\.
  * [108] M. Jin, Y. Zheng, Y.-F. Li, S. Chen, B. Yang, and S. Pan, “Multivariate time series forecasting with dynamic graph neural odes,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [109] X. Zheng, Y. Liu, Z. Bao, M. Fang, X. Hu, A. W.-C. Liew, and S. Pan, “Towards data-centric graph machine learning: Review and outlook,” _arXiv preprint arXiv:2309.10979_ , 2023.
  * [110] Y. Guo, C. Yang, Y. Chen, J. Liu, C. Shi, and J. Du, “A data-centric framework to endow graph neural networks with out-of-distribution detection ability,” 2023\.
  * [111] Q. Wu, H. Zhang, J. Yan, and D. Wipf, “Handling distribution shifts on graphs: An invariance perspective,” in _International Conference on Learning Representations_ , 2021.
  * [112] Y. Sui, Q. Wu, J. Wu, Q. Cui, L. Li, J. Zhou, X. Wang, and X. He, “Unleashing the power of graph data augmentation on covariate distribution shift,” in _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023\.
  * [113] X. Zheng, M. Zhang, C. Chen, S. Molaei, C. Zhou, and S. Pan, “Gnnevaluator: Evaluating gnn performance on unseen graphs without labels,” in _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023\.
  * [114] Y. Ektefaie, G. Dasoulas, A. Noori, M. Farhat, and M. Zitnik, “Multimodal learning with graphs,” _Nature Machine Intelligence_ , vol. 5, no. 4, pp. 340–350, 2023.
  * [115] M. Yoon, J. Y. Koh, B. Hooi, and R. Salakhutdinov, “Multimodal graph learning for generative tasks,” 2023.
  * [116] Y. Guo, F. Yin, X.-h. Li, X. Yan, T. Xue, S. Mei, and C.-L. Liu, “Visual traffic knowledge graph generation from scene images,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 21 604–21 613.
  * [117] Y. Wang, N. Lipka, R. A. Rossi, A. Siu, R. Zhang, and T. Derr, “Knowledge graph prompting for multi-document question answering,” _arXiv preprint arXiv:2308.11730_ , 2023.
  * [118] Q. Huang, H. Ren, P. Chen, G. Kržmanc, D. Zeng, P. Liang, and J. Leskovec, “Prodigy: Enabling in-context learning over graphs,” _arXiv preprint arXiv:2305.12600_ , 2023.
  * [119] H. Zhang, B. Wu, X. Yuan, S. Pan, H. Tong, and J. Pei, “Trustworthy graph neural networks: Aspects, methods and trends,” _arXiv preprint arXiv:2205.07424_ , 2022.
  * [120] B. Wu, H. Zhang, X. Yang, S. Wang, M. Xue, S. Pan, and X. Yuan, “Graphguard: Detecting and counteracting training data misuse in graph neural networks,” _arXiv preprint arXiv:2312.07861_ , 2023.
  * [121] H. Zhang, B. Wu, S. Wang, X. Yang, M. Xue, S. Pan, and X. Yuan, “Demystifying uneven vulnerability of link stealing attacks against graph neural networks,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 41 737–41 752.
  * [122] H. Zhang, X. Yuan, C. Zhou, and S. Pan, “Projective ranking-based gnn evasion attacks,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [123] Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, “Gnnexplainer: Generating explanations for graph neural networks,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [124] Y. Liu, K. Ding, Q. Lu, F. Li, L. Y. Zhang, and S. Pan, “Towards self-interpretable graph-level anomaly detection,” in _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * [125] D. Jin, L. Wang, H. Zhang, Y. Zheng, W. Ding, F. Xia, and S. Pan, “A survey on fairness-aware recommender systems,” _arXiv preprint arXiv:2306.00403_ , 2023\.
  * [126] H. Zhang, X. Yuan, Q. V. H. Nguyen, and S. Pan, “On the interaction between node fairness and edge privacy in graph neural networks,” _arXiv preprint arXiv:2301.12951_ , 2023.