# Deep Prompt Tuning for Graph Transformers

Reza Shirkavand,1 Heng Huang, 1

###### Abstract

Graph transformers have gained popularity in various graph-based tasks by
addressing challenges faced by traditional Graph Neural Networks. However, the
quadratic complexity of self-attention operations and the extensive layering
in graph transformer architectures present challenges when applying them to
graph based prediction tasks. Fine-tuning, a common approach, is resource-
intensive and requires storing multiple copies of large models. We propose a
novel approach called deep graph prompt tuning as an alternative to fine-
tuning for leveraging large graph transformer models in downstream graph based
prediction tasks. Our method introduces trainable feature nodes to the graph
and pre-pends task-specific tokens to the graph transformer, enhancing the
model’s expressive power. By freezing the pre-trained parameters and only
updating the added tokens, our approach reduces the number of free parameters
and eliminates the need for multiple model copies, making it suitable for
small datasets and scalable to large graphs. Through extensive experiments on
various-sized datasets, we demonstrate that deep graph prompt tuning achieves
comparable or even superior performance to fine-tuning, despite utilizing
significantly fewer task-specific parameters. Our contributions include the
introduction of prompt tuning for graph transformers, its application to both
graph transformers and message passing graph neural networks, improved
efficiency and resource utilization, and compelling experimental results. This
work brings attention to a promising approach to leverage pre-trained models
in graph based prediction tasks and offers new opportunities for exploring and
advancing graph representation learning.

## Introduction

The remarkable success of transformers in Natural Language Processing (Vaswani
et al. 2017) and Computer Vision (Dosovitskiy et al. 2021) has led to their
increasing popularity in graph applications. In recent years, graph
transformers have been widely adopted in various graph-based tasks (Dwivedi
and Bresson 2020; Kreuzer et al. 2021; Ying et al. 2021; Chen, O’Bray, and
Borgwardt 2022; Rampásek et al. 2022). Graph transformers address key
challenges faced by traditional Graph Neural Networks (GNNs), such as limited
expressiveness (Wang and Zhang 2022; Feng et al. 2022), over-smoothing (Chen
et al. 2020) and over-squashing (Alon and Yahav 2021), by leveraging highly
expressive global self-attention modules. Instead of introducing graph
structure bias into intermediate layers, graph transformers use encoded
structural and positional information of the graph into the input node
features (Chen, O’Bray, and Borgwardt 2022; Kreuzer et al. 2021; Ying et al.
2021).

The quadratic complexity of the self-attention operation, combined with the
extensive layering in graph transformer architectures, presents a significant
challenge when applying them to graph prediction tasks. This challenge arises
from the potential overfitting to small datasets, and even reduced-parameter
versions may lack the necessary representational richness for complex graph
datasets, such as those involved in molecular property prediction. To address
this issue, a viable solution is to adopt a similar approach as in the
application of large language models to downstream NLP tasks (Radford et al.
2019; Devlin et al. 2019). This involves pre-training large graph transformer
models on extensive datasets and subsequently fine-tuning them on smaller
datasets.

While fine-tuning can yield satisfactory results (Howard and Ruder 2018), it
presents a resource-intensive approach due to the need to update the entire
parameter set of the large graph transformer model. Furthermore, fine-tuning
necessitates storing multiple copies of the same large model for different
downstream tasks, which can be especially challenging and even prohibitive on
smaller devices. In contrast to fine-tuning, prompt tuning, originally
proposed for natural language processing (NLP) tasks, offers an alternative
solution. Prompt tuning involves freezing all parameters of a pre-trained
model and only updating either discrete (Brown et al. 2020; Gao, Fisch, and
Chen 2021) or continuous (Li and Liang 2021; Lester, Al-Rfou, and Constant
2021; Liu et al. 2021b, a) lightweight tokens that are added to the inputs.
Despite the success of prompt tuning in the field of NLP, the application of
this technique to graph transformers has yet to be experimentally explored.

Motivated by the concept of prompt tuning, we introduce a novel approach
called _Deep Graph Prompt Tuning (DeepGPT)_ , which serves as an alternative
to fine-tuning for leveraging large graph transformer models in downstream
graph prediction tasks. Fig. 1 illustrates the setup of a typical downstream
graph prediction task, where an input graph is transformed into a final graph
representation vector as the output. Assuming we have access to a pre-trained
large graph transformer network trained on a different dataset, DeepGPT begins
by adding a continuous task-specific graph _prompt token_ to the feature
vectors of all nodes within the input graph. Subsequently, the modified input
graph is fed into the graph transformer. Although this approach focuses on
modifying the graph input level only, it theoretically allows for
approximating various combinations of node-level, edge-level, and graph-level
transformations within the architectures of existing pre-trained GNN models
((Fang et al. 2022). Additionally, we pre-pend continuous layer-specific task-
specific _prefix tokens_ to the embeddings of each layer in the graph
transformer network. This enables the self-attention module of each
transformer layer to attend to the trainable task-specific tokens as if they
were originally part of the sequence of node embeddings. It also increases the
expressive power of our DeepGPT method. While our approach is specifically
designed for graph transformers with global attention modules, it can be
viewed as introducing a set of new nodes with trainable features to the graph
and connecting them to all pre-existing nodes in the context of Message
Passing Graph Neural Networks (MPGNNs).

In contrast to fine-tuning, our approach involves freezing all parameters of
the pre-trained graph transformer and exclusively tuning the task-specific
tokens that are added. This allows for the practicality of storing a single
copy of the pre-trained transformer architecture alongside the task-specific
continuous prompts for different downstream tasks. Through extensive
experiments conducted on various-sized OGB and Moleculenet datasets, we have
demonstrated that our approach delivers comparable, and in some cases even
superior, performance compared to fine-tuning, despite utilizing fewer than
0.5% of task-specific parameters. This significant reduction in the number of
free parameters proves advantageous for small datasets as it mitigates the
risk of overfitting. Furthermore, our method enables the pre-trained graph
transformer to scale effectively to large graphs, as the required
computational resources are substantially reduced.

Our main contributions can be summarized as follows:

  1. 1.

We propose a novel prompt tuning method specifically designed for graph
transformers, which, to the best of our knowledge, is the first of its kind in
the field.

  2. 2.

Our approach, designed for graph transformers with global attention modules,
is equivalent to introducing additional trainable feature nodes connected to
existing nodes within the context of MPGNNs, making it applicable to both
Graph Transformers and MPGNNs.

  3. 3.

Our approach significantly improves the efficiency and resource utilization of
tuning a pre-trained graph transformer for downstream tasks. It eliminates the
need to store separate copies of a large model for different graph prediction
tasks, making it a more streamlined solution. Additionally, our method is
well-suited for both small datasets and scales effectively to handle datasets
with large graphs, addressing the limitations of graph transformers in dealing
with such scenarios.

  4. 4.

Our extensive experiments demonstrate that our method achieves performance
that is on par with, and at times surpasses, fine-tuning, despite utilizing
significantly fewer task-specific parameters. Moreover, our method
demonstrates superiority over lightweight fine-tuning, where only the
classification head is updated.

## Related Work

### Fine-tuning Strategies for Graph Neural Networks

Several approaches have been explored in fine-tuning graph neural networks.
For instance, ChemBERTa (Chithrananda, Grand, and Ramsundar 2020) adopts an
NLP self-supervised pre-training strategy inspired by RoBERTa (Liu et al.
2019), where a portion of tokens in the SMILES string representation of
PUBCHEM graphs are masked, and the model predicts them from other tokens.
(Zhang et al. 2021) propose a motif-based generative pre-training framework,
training GNNs to make topological and label predictions. Another work by (Rong
et al. 2020) introduces a molecular graph pre-training model with self-
supervised tasks at the node, edge, and graph levels, including contextual
property prediction and graph-level motif prediction. (Li, Zhao, and Zeng
2022) design a generative self-supervised strategy that leverages molecular
descriptors and fingerprints to guide the model in capturing structural and
semantic information from large-scale unlabeled molecular graph datasets.
Graph Contrastive Learning (GCL) objectives have also been utilized in several
studies (You et al. 2020, 2021; Velickovic et al. 2019) to enhance the pre-
training process. Additionally, (Ying et al. 2021) pre-train their proposed
Graphormer model on large-scale OGB datasets to capture rich representations,
followed by fine-tuning for specific target tasks.

### Prompt Tuning for Graph Neural Networks

Although there has been an extensive amount of research into both discrete
(Zhang, Wang, and Liu 2018; Minaee et al. 2021; Shin et al. 2020) and
continuous (Liu et al. 2021a, b; Lester, Al-Rfou, and Constant 2021) prompt
tuning for NLP, the application of prompts to the graph domain is relatively
unexplored. (Sun et al. 2022) propose a graph prompting function to transform
a downstream node prediction task to an edge prediction task similar to the
pretext masked-edge prediction used to pre-train a large GNN. (Fang et al.
2022) propose Graph Prompt (GP) as a universal graph prompt tuning method to
existing pre-trained GNNs. They also introduce Graph Prompt Features (GPF) as
a concrete instance of GP. GPF is a trainable token added to the node features
of the input graph and updated during the tuning of the pre-trained model on
the downstream task.

## Preliminary

In this section, we will cover the fundamentals of graph neural networks
(GNNs) and graph transformers, as well as provide an overview of the “pre-
train, fine-tune” paradigm.

### Graph Neural Networks

Let us consider an undirected graph $G=(V,E)$, where $V=\\{1,\cdots,n\\}$
represents the set of nodes and $E$ represents the set of edges. We assume
that each node $i$ is associated with a d-dimensional feature vector $x_{i}\in
R^{d}$ for $i=1,\cdots,n$. Graph Neural Networks (GNNs) employ a message-
passing strategy (Gilmer et al. 2017) to learn node representations by
iteratively aggregating the representations of neighboring nodes. Formally,
the representation of node $i$ at the $k$-th layer is denoted as $h_{i}^{k}$,
with $h_{i}^{0}=x_{i}$. The aggregation and combination operation is defined
as follows:

$h_{i}^{k}=\text{AGG-COMB}({h_{j}^{k-1}:j\in N(i)\cup i};\theta^{k})$ (1)

Here, $N(i)$ represents the set of adjacent nodes to node $i$, and
$\theta^{k}$ is the parameter set of the $k$-th layer. The AGG-COMB operation
collects the embeddings of neighboring nodes and combines them using sum,
mean, or max functions to generate the embeddings for node $i$. In graph
representation learning tasks, a final READOUT function is employed to combine
the node representations of a $K$-layer GNN into the graph embedding $h_{G}$:

$h_{G}=\text{READOUT}({h_{i}^{K}:i=1,\cdots,n})$ (2)

Graph Neural Networks have demonstrated superior performance in various graph
prediction tasks, including node-level (Gao, Wang, and Ji 2018), edge-level
(Li et al. 2021), and graph-level (Hao et al. 2020) tasks.

### Graph Transformers

Unlike conventional GNNs, Transformers (Vaswani et al. 2017) do not explicitly
utilize the graph structure to learn representations. Instead, they treat the
graph as a set of nodes and infer similarities between nodes by employing
self-attention mechanisms on node features. A typical transformer block
consists of a multi-head attention module followed by a feed-forward network.
The input node feature matrix $X$ is linearly projected into Query (Q), Key
(K), and Value (V) matrices, represented as $Q=XW_{Q}$, $K=XW_{K}$, and
$V=XW_{V}$, respectively. The self-attention activations are computed as
follows:

$Attn(X)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{Q}}}\right)V$ (3)

Here, $d_{Q}$ represents the dimension of $Q$. The output of the transformer
layer is obtained by applying a feed-forward network (FNN) to the sum of the
input node features and the self-attention activations:

$X=\text{FFN}(X+Attn(X))$ (4)

Multiple transformer layers are usually stacked together to form a transformer
network. Since self-attention is a permutation-invariant operator, the
transformer produces the same result regardless of changes in the graph
structure, as long as the node features remain unchanged. To incorporate the
structural information of the graph into the transformer architecture,
effective absolute encodings are designed. These encodings are structural and
positional representations of the graph that are added or concatenated to the
input features to enhance the expressiveness and generalization capability of
Graph Transformers. Examples of absolute encodings include Centrality Encoding
(Ying et al. 2021), RWPE (Dwivedi et al. 2022), and Laplacian Positional
Encoding (Kreuzer et al. 2021).

### Pre-training and Fine-tuning of GNNs

Supervised learning of node and graph representations often requires a large
amount of annotated data, which can be challenging to obtain. Consequently,
the “pre-train, fine-tune” paradigm has gained significant attention. In this
paradigm, neural networks are pre-trained on pretext tasks, such as
Contrastive Learning (Hu et al. 2020c; Qiu et al. 2020; Subramonian 2021), and
then fine-tuned on downstream tasks. The process involves training a graph
neural network $f_{\theta}$ on a pre-training dataset $D_{pt}$ by minimizing
the pre-training loss $L_{pt}$, resulting in parameter set $\theta_{pt}$. The
downstream model parameters are initialized using $\theta_{init}=\theta_{pt}$,
and the prediction head of the GNN is replaced. Fine-tuning is performed by
optimizing the downstream loss $L$ on the downstream dataset $D$ using the
whole parameter set of $f$ or a subset of it:

$\min_{\theta,\psi}L(p_{\psi}(f_{\theta}(x_{i})),y_{i})$ (5)

Here, $p_{\psi}$ denotes the new prediction head of the network, and $y_{i}$
corresponds to the ground truth of $x_{i}$.

## Methodology

The “unsupervised pre-train, fine-tune” framework faces a significant
challenge due to the training objective gap between pretext and downstream
tasks. This issue becomes more pronounced when dealing with large neural
networks like transformers, as it requires storing separate copies of the pre-
trained network for each downstream task, incurring high costs. Moreover,
fine-tuning these massive models on large downstream datasets is time-
consuming. To address these limitations, we draw inspiration from the success
of continuous prompt-based learning in NLP (Liu et al. 2021b; Lester, Al-Rfou,
and Constant 2021) and propose deep prompt tuning for Graph Transformers as an
alternative approach for node/graph classification tasks. The framework,
depicted in Figure 1, leverages continuous prompts to guide the graph
transformer’s learning process, reducing the need for separate network copies
and significantly improving efficiency.

Figure 1: Overview of our proposed deep graph prompt tuning framework. Top:
Fine-tuning requires storing separate copies of the pre-trained graph
transformer for each downstream task and updating all parameters of the model.
This approach is memory-intensive, time-consuming, and cost-inefficient.
Bottom: our proposed graph prompt tuning method. We introduce graph prompt
tokens to the input graph representations and each transformer layer
activation. Additionally, we pre-pend prefix tokens to all transformer layers.
By freezing most of the pre-trained model’s parameters and only updating the
concatenated and added prompt tokens, we can fine-tune a large transformer on
different downstream datasets while only storing the prompt tokens, minimizing
memory requirements.

Based on intuition from prompt tuning, we believe that providing a proper
context in the form of graph embeddings can guide a graph transformer without
altering its parameters. By incorporating relevant graph embeddings as
context, the graph transformer can achieve higher accuracy in graph prediction
tasks. This approach extends the concept of prompting beyond adding single
nodes or edges and aims to find a context that influences the transformer’s
encoding of graphs and the generation of predictions. Rather than discrete
alterations to the graph (Sun et al. 2022), we propose optimizing continuous
graph embeddings as context, which can effectively propagate throughout the
graph transformer’s layers, striking a balance between expressiveness and
computational feasibility.

Our approach entails transforming both the input graph and the pre-trained
graph transformer model. We combine graph prompts with deep prefix prompts.
This involves incorporating graph prompt tokens at the input of the graph
transformer and pre-pending prefix tokens to each transformer layer and to
direct the graph transformer in solving downstream node/graph classification
tasks by fine-tuning these task-specific tokens. We demonstrate the
superiority of this approach over using solely graph prompts or prefix tokens,
despite utilizing the same number of parameters. Through our experiments, we
showcase the acceptable, and at times even superior, performance of our
approach with significantly fewer parameters compared to traditional fine-
tuning, where all network parameters are adjusted, or lightweight fine-tuning,
which involves freezing the pre-trained model’s backbone and training only the
prediction head.

### Graph Prompt Tokens

In our prompt tuning method, the first step involves incorporating graph
prompt tokens into the input graph nodes. For a $d$-dimensional node feature
vector $x_{i}\in R^{d}$, we introduce a trainable $d$-dimensional prompt token
$p\in R^{d}$ specific to the task, which is added to each node of the input
graph. This results in a modified feature matrix $X^{(p)}$ derived from the
original feature matrix $X_{i}$ as follows:

$x_{i}^{(p)}=x_{i}+p$ (6)

Prior research by (Fang et al. 2022) has demonstrated that such prompt tokens
possess the theoretical capability to approximate a wide range of complex
node-level, edge-level, and graph-level transformations on most existing Graph
Neural Networks (GNNs) under two conditions: utilizing a Graph Convolutional
Network (GCN) as the underlying architecture and employing sum/mean as the
READOUT operator. Formally, for an input graph $G(X,A)$, with $X$ representing
the node features and $A$ denoting the adjacency matrix, given an arbitrary
transformation $g$, and a pre-trained GNN $f$, there exists a corresponding
prompt token $p^{*}$ satisfying:

$f(X^{p^{*}},A)=f(g(X,A))$ (7)

Although the theoretical foundation of the graph prompt approach primarily
focuses on Convolutional Graph Neural Networks (GNNs), we empirically
demonstrate that the benefits of graph prompts extend to graph transformers as
well. The shared principles between these two models, such as alignment with
the READOUT operator and the introduction of task-specific information,
contribute to the improved performance of graph transformers when employing
the graph prompt technique.

### Graph Transformer Prefix Tokens

The second component of our approach involves the inclusion of prefix tokens
at the beginning of the embeddings after each layer of the frozen transformer
architecture, resulting in updated embeddings. Let $PS_{idx}$ denote the
sequence of prefix tokens, where $p=|PS_{idx}|$ represents the length of the
sequence. Assuming the dimension of the transformer embeddings is $d$, and
denoting the original transformer layer embeddings as $E$, we utilize a soft
prompt matrix $P\in R^{p\times d}$ to generate the new embeddings $E^{*}$
according to the following rule:

$E^{*}[i,:]=\begin{cases}P[i,:]&i\in PS_{idx}\\\ E[i,:]&otherwise\end{cases}$
(8)

The resulting embedding matrix is then passed as input to the subsequent
transformer layer. Importantly, while the graph prompt tokens vary with each
layer, the tokens in earlier layers still have an impact on the embeddings of
later layers. This is due to the influence of the prompt token on the input
embeddings through the self-attention operation, enabling these tokens to
propagate throughout the entire transformer architecture.

### Deep Graph Transformer Prompt Tuning

The shallow approach of simply appending or adding prompt tokens to the input
graph (Li and Liang 2021) encounters two significant challenges. Firstly, the
limited number of trainable parameters leads to unstable training and
unsatisfactory results. Secondly, adding prompts only to the input graph has
minimal impact on the deeper layers of the transformer. To address these
limitations, we propose the Deep Graph Prompt Tuning (DeepGPT) technique,
which involves adding a prompt token to the input the graph transformer and
pre-pending prefix prompts to each transformer layer embedding.

Assuming a pre-trained graph transformer $f_{\theta}$, DeepGPT transformation
$T_{\phi}$, and a downstream dataset $D={(G_{i},y_{i})}{i=1}^{n}$, we fine-
tune the trainable parameter set $\phi$ while keeping the parameters $\theta$
of the pre-trained graph transformer frozen. This fine-tuning process aims to
minimize the downstream loss $L$, such as the binary cross-entropy loss, given
by the following optimization objective:

$\min{\phi}\sum_{i=1}^{n}L(f_{\theta}(T_{\phi}(G_{i})),y_{i})$ (9)

### Relation to Conventional Graph Neural Networks

Although our work primarily focuses on graph transformer architectures, the
construct of prompt tokens we propose is applicable universally. The graph
prompt tokens can be incorporated into various graph neural network models
without assuming a specific structure for the model $f$. Similarly, in the
context of Message Passing Graph Neural Networks (MPGNNs), the prefix tokens
of the transformer architecture can be replaced by adding a set of new
trainable nodes to the graph and connecting them with edges to all existing
nodes, thereby achieving similar effects.

## Experiments

### Experimental Settings

#### Datasets

The graph transformer models in this study were pre-trained on the OGB-LSC (Hu
et al. 2021) quantum chemistry regression dataset, known as PCQM4Mv2. After
pre-training, we assess the efficacy of our approach using two established
sets of machine learning datasets focused on molecular graphs: the Open Graph
Benchmark (OGB) (Hu et al. 2020a), which encompasses datasets of diverse
sizes, covering a wide range of realistic tasks, and the Moleculenet Benchmark
(Wu et al. 2018), which comprises datasets for predicting molecular
properties. These datasets include regression, single-label binary
classification, and multi-label classification tasks, spanning various domains
(For further details, please refer to the Appendix).

#### Graph Transformer Architectures

In our study, we investigate three graph transformer architectures that
address the key challenge of incorporating structure-awareness into the
attention mechanism when graphs are used as input. Firstly, (Ying et al. 2021)
present Graphormer, which modifies the attention mechanism itself to integrate
structural information. Secondly, (Rampásek et al. 2022) propose General
Powerful and Scalable Graph Transformer (GraphGPS), which employs hybrid
architectures, integrating GNNs. Thirdly, (Li, Zhao, and Zeng 2022) introduce
Line Graph Transformer (LiGhT), which utilizes positional encodings for
graphs. While all three architectures share the self-attention module at their
core, they exhibit fundamental differences in overall design, structure, and
the usage of positional and structural encodings.

We conducted experiments with two different settings of the GraphGPS network:
one using regular self-attention transformers (GraphGPS small), and the other
utilizing Performer (Choromanski et al. 2021) modules (GraphGPS large), which
are Transformer architectures capable of accurately estimating regular
(softmax) full-rank-attention Transformers. The performer modules achieve this
accuracy while employing linear space and time complexity instead of the
quadratic complexity found in traditional methods. This allowed us to
demonstrate the effectiveness of our proposed method on different self-
attention modules. The choice of network is explicitly indicated in the result
tables.

#### Message-Passing GNN Baselines

To further evaluate the effectiveness of our DeepGPT framework, we present the
performance results of popular Message-Passing Graph Neural Networks (MPGNN),
namely GatedGCN (Bresson and Laurent 2017), GINE (Hu et al. 2020b), and PNA
(Corso et al. 2020).

#### Training Details

We employ the AdamW (Loshchilov and Hutter 2019) optimizer across all
architectures and datasets for both pre-training and evaluation. We utilize a
learning rate scheduling with a warm-up stage followed by a cosine decay
regiment. We tune a hyper-parameter set including learning rate, weight decay,
and the Prefix Token size $|PS|$. We perform a 5-fold cross validation for all
graph transformer experiments to have a fair evaluation. All experiments are
are conducted on a Lambda machine with 8 NVIDIA RTX A6000 GPUs. (For all
details, see Appendix).

### Results

Tables 1 and 2 demonstrate the performance of our proposed method on the
classification and regression benchmarks, respectively. The performance of
DeepGPT on the OGB benchmarks is presented in Table 3. We compare DeepGPT to
Fine-tuning as well as MPGNN baselines on molecular graph tasks.

Table 1: Comparison of the performance of DeepGPT, Fine Tuning (FT) and MPGNN
baselins on classification benchmarks. Entries marked with an asterisk
($\ast$) indicate the use of Performer self-attention. Results represent the
mean and standard deviation from 5-fold cross-validation.

Model | #Param. | Classification Dataset ($\uparrow$)
---|---|---
BACE | BBBP∗ | ClinTox∗ | Estrogen∗ | MetStab∗ | SIDER | ToxCast∗ | Tox21
Graphormer (Ying et al. 2021) FT | 48M | 0.704 ± 0.018 | 0.778 ± 0.043 | 0.839 ± 0.114 | 0.673 ± 0.034 | 0.607 ± 0.015 | 0.550 ± 0.007 | 0.665 ± 0.032 | 0.582 ± 0.051
Graphormer DeepGPT | 100K | 0.883 ± 0.022 | 0.914 ± 0.016 | 0.884 ± 0.030 | 0.941 ± 0.006 | 0.871 ± 0.004 | 0.646 ± 0.003 | 0.725 ± 0.011 | 0.813 ± 0.010
GraphGPS (Rampásek et al. 2022) FT | 14M ($\ast$: 103M) | 0.898 ± 0.020 | 0.888 ± 0.015 | 0.910 ± 0.046 | 0.933 ± 0.014 | 0.897 ± 0.025 | 0.610 ± 0.005 | 0.722 ± 0.011 | 0.832 ± 0.009
GraphGPS DeepGPT | 50K ($\ast$: 500K) | 0.892 ± 0.022 | 0.901 ± 0.015 | 0.907 ± 0.044 | 0.944 ± 0.010 | 0.899 ± 0.019 | 0.609 ± 0.020 | 0.735 ± 0.005 | 0.832 ± 0.012
LiGhT (Li, Zhao, and Zeng 2022) FT | 90M | 0.880 ± 0.012 | 0.902 ± 0.021 | 0.857 ± 0.035 | 0.942 ± 0.012 | 0.902 ± 0.012 | 0.670 ± 0.005 | 0.745 ± 0.005 | 0.844 ± 0.004
LiGhT DeepGPT | 370K | 0.873 ± 0.020 | 0.917 ± 0.012 | 0.862 ± 0.056 | 0.950 ± 0.010 | 0.912 ± 0.011 | 0.671 ± 0.011 | 0.757 ± 0.011 | 0.843 ± 0.004
GatedGCN (Bresson and Laurent 2017) | 2.8M | 0.833 ± 0.013 | 0.887 ± 0.025 | 0.893 ± 0.041 | 0.919 ± 0.008 | 0.848 ± 0.018 | 0.599 ± 0.015 | 0.683 ± 0.005 | 0.807 ± 0.011
GINE (Hu et al. 2020b) | 1.2M | 0.599 ± 0.045 | 0.613 ± 0.024 | 0.559 ± 0.044 | 0.492 ± 0.024 | 0.540 ± 0.016 | 0.584 ± 0.025 | 0.629 ± 0.025 | 0.714 ± 0.024
PNA (Corso et al. 2020) | 1.8M | 0.845 ± 0.021 | 0.903 ± 0.018 | 0.867 ± 0.011 | 0.927 ± 0.009 | 0.738 ± 0.024 | 0.583 ± 0.012 | 0.673 ± 0.008 | 0.793 ± 0.015

Table 2: Comparison of the performance of DeepGPT, Fine Tuning (FT) and MPGNN
baselins on Moleculenet regression benchmarks.

Model | Moleculenet Regression Dataset ($\downarrow$)
---|---
FreeSolv | ESOL | Lipo
Graphormer FT | 1.680 ± 0.013 | 0.925 ± 0.061 | 0.909 ± 0.237
Graphormer DeepGPT | 1.668 ± 0.114 | 0.943 ± 0.024 | 0.834 ± 0.046
GraphGPS FT | 1.380 ± 0.234 | 0.772 ± 0.102 | 0.673 ± 0.022
GraphGPS DeepGPT | 1.415 ± 0.254 | 0.685 ± 0.130 | 0.528 ± 0.056
LiGhT FT | 0.955 ± 0.051 | 0.565 ± 0.041 | 0.530 ± 0.019
LiGhT DeepGPT | 0.983 ± 0.079 | 0.579 ± 0.038 | 0.548 ± 0.028
GatedGCN | 2.685 ± 1.766 | 1.373 ± 0.219 | 0.568 ± 0.065
GINE | 6.971 ± 3.892 | 3.075 ± 1.034 | 1.440 ± 0.027
PNA | 2.479 ± 0.556 | 1.463 ± 0.341 | 0.684 ± 0.024

Table 3: Comparison of the performance of DeepGPT, Fine Tuning (FT) and MPGNN
baselins on OGB classification benchmark.

Model | OGB Classification Dataset ($\uparrow$)
---|---
MOLHIV (AUROC) | MOLPCBA (AP)
Graphormer FT | 0.805 ± 0.005 | 0.313 ± 0.003
Graphormer DeepGPT | 0.804 ± 0.021 | 0.289 ± 0.009
GraphGPS FT | 0.806 ± 0.007 | 0.301 ± 0.013
GraphGPS DeepGPT | 0.801 ± 0.015 | 0.297 ± 0.020
LiGhT FT | 0.787 ± 0.008 | 0.295 ± 0.006
LiGhT DeepGPT | 0.799 ± 0.010 | 0.270 ± 0.007
GatedGCN | 0.809 ± 0.016 | 0.264 ± 0.021
GINE | 0.679 ± 0.055 | -
PNA | 0.782 ± 0.013 | 0.257 ± 0.006

#### DeepGPT Across Tasks and Model Scales

Based on the results, we observe that DeepGPT exhibits comparable performance
and, in certain cases, even outperforms fine-tuning across various tasks.
Table 10 (Appendix) displays the sizes of the graph transformer architectures
utilized in our experiments. These results demonstrate that our method is
adaptable and effective for a diverse range of graph transformer
architectures, irrespective of their sizes.

#### DeepGPT Across Dataset Scales

It is important to highlight that the downstream datasets used in this study
consist of varying dataset sizes and graph sizes, as depicted in Figure 7
(Appendix). The results demonstrate the versatility of DeepGPT, as it proves
to be effective for small datasets, while also scaling well to larger
datasets. Another noteworthy aspect of graph transformer prompt tuning is its
ability to address the limitations of many graph transformer models, which are
often unsuitable for processing large graphs. By reducing the overhead
associated with fine-tuning such models on large graphs, we achieve
satisfactory results compared to fine-tuning.

#### Benchmarking DeepGPT

The results obtained from our experiments provide compelling evidence of
DeepGPF’s remarkable performance, surpassing all MPGNN baselines across nearly
every task evaluated. Furthermore, the findings offer a glimpse into the
immense potential of prompt tuning large graph transformer models, suggesting
that they have the capacity gain wider adoption in graph prediction tasks.

#### Convergence Speed

Overall, DeepGPT exhibits faster convergence during model training.
Additionally, each DeepGPT epoch is usually quicker than fine-tuning (Figure
2). The increased average epoch duration of DeepGPT on LiGhT is attributed to
the implementation of the original paper, which requires a loop for adding
prompt tokens, impacting the overall speed.

(a) Average number of epochs

(b) Average epoch duration

Figure 2: Comparison of convergence speed of DeepGPT and FT. DeepGPT converges
faster and decreases training and inference epoch duration.

### Ablation Studies

#### Prompt Tokens Contributions

Table 4 demonstrates the impact of adding graph prompt tokens to deep prefix
tuning. Generally, this addition leads to performance improvements even with
the same number of parameters. The addition of graph prompt tokens alone has
not been experimented with due to the limited number of parameters, which
leads to an insignificant impact during model training. Additionally, DeepGPT
outperforms lightweight tuning, where only the classification head of the pre-
trained model is modified.

Table 4: Impact of adding graph prompt tokens to deep prefix tuning.

Model | #Param. | Dataset ($\uparrow$)
---|---|---
BACE | Clintox ∗ | Metstab ∗ | SIDER
Graphormer Lightweight Tuning | 600K | 0.847 ± 0.013 | 0.766 ± 0.082 | 0.863 ± 0.021 | 0.625 ± 0.017
Graphormer Deep Prefix Tuning | 100K | 0.863 ± 0.013 | 0.860 ± 0.062 | 0.883 ± 0.014 | 0.584 ± 0.009
Graphormer DeepGPT | 100K | 0.883 ± 0.022 | 0.884 ± 0.030 | 0.871 ± 0.004 | 0.646 ± 0.003
GraphGPS Lightweight Tuning | 500K | 0.819 ± 0.010 | 0.692 ± 0.107 | 0.673 ± 0.031 | 0.541 ± 0.015
GraphGPS Deep Prefix Tuning | 50K (∗:500K) | 0.854 ± 0.016 | 0.866 ± 0.059 | 0.841 ± 0.014 | 0.598 ± 0.019
GraphGPS DeepGPT | 50K (∗:500K) | 0.892 ± 0.022 | 0.907 ± 0.044 | 0.899 ± 0.019 | 0.609 ± 0.020

#### Prompt Depth

To assess the precise influence of the depth of injected prompt tokens when a
certain number of k layers are allocated for prompts, we inject them into
different sequences of layers within the graph transformer model. Figure 3
illustrates the results. Notably, injecting prompts into middle layers
generally produces more favorable outcomes, outperforming other scenarios. We
further investigate the effects of injecting prompts into the first or last k
layers of the model. The results are shown in Figures 4 and 5. Overall,
injecting tokens into the middle layers yields better performance, while
injecting prompt tokens into the final layers negatively affects the results.
Generally the number of layers the prompt tokens are inserted into does not
have a significant influence on the results.(For more architectures and
datasets, see Appendix).

(a) 2-layer Steps

(b) 3-layer Steps

Figure 3: Impact of the depth of injected prompt tokens using Graphormer on
BACE and BBBP datasets. Injecting prompts into middle layers works better.
Step size does not have much effect.

(a) 2-layer Steps

(b) 3-layer Steps

Figure 4: Impact of injecting prompt tokens to initial layers of Graphormer on
BACE and BBBP datasets.

(a) 2-layer Steps

(b) 3-layer Steps

Figure 5: Impact of injecting prompt tokens to final layers of Graphormer on
BACE and BBBP datasets.

#### Prompt Length

We also conducted experiments to analyze the effect of prompt length. Figure 8
illustrates the results, and interestingly, no clear trend is observed for
increasing prompt length. However, it is worth noting that, in general, adding
more prompt tokens tends to improve performance (For more architectures and
datasets, see Appendix).

Figure 6: Effect of prompt length using GraphGPS.

## Conclusions

In this work, we introduced Deep Graph Prompt Tuning (DeepGPT) as an efficient
alternative to fine-tuning for graph transformer models in downstream graph
based prediction tasks. Our method involves freezing all pre-trained
parameters and only tuning task-specific tokens added to the input and the
graph transformer layers, reducing computational overhead and storage
requirements. Through experiments on various datasets, we showed that DeepGPT
achieves comparable or better performance to fine-tuning with significantly
fewer task-specific parameters, demonstrating its superiority in resource-
intensive applications. This approach addresses the limitations of graph
transformers and offers a promising solution for leveraging large pre-trained
models in graph-based applications.

## References

  * Alon and Yahav (2021) Alon, U.; and Yahav, E. 2021.  On the Bottleneck of Graph Neural Networks and its Practical Implications.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Bresson and Laurent (2017) Bresson, X.; and Laurent, T. 2017.  Residual Gated Graph ConvNets.  _CoRR_ , abs/1711.07553.
  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2020) Chen, D.; Lin, Y.; Li, W.; Li, P.; Zhou, J.; and Sun, X. 2020.  Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , 3438–3445. AAAI Press.
  * Chen, O’Bray, and Borgwardt (2022) Chen, D.; O’Bray, L.; and Borgwardt, K. M. 2022.  Structure-Aware Transformer for Graph Representation Learning.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvári, C.; Niu, G.; and Sabato, S., eds., _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , 3469–3489. PMLR.
  * Chithrananda, Grand, and Ramsundar (2020) Chithrananda, S.; Grand, G.; and Ramsundar, B. 2020.  ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction.  _CoRR_ , abs/2010.09885.
  * Choromanski et al. (2021) Choromanski, K. M.; Likhosherstov, V.; Dohan, D.; Song, X.; Gane, A.; Sarlós, T.; Hawkins, P.; Davis, J. Q.; Mohiuddin, A.; Kaiser, L.; Belanger, D. B.; Colwell, L. J.; and Weller, A. 2021.  Rethinking Attention with Performers.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Corso et al. (2020) Corso, G.; Cavalleri, L.; Beaini, D.; Liò, P.; and Velickovic, P. 2020.  Principal Neighbourhood Aggregation for Graph Nets.  _CoRR_ , abs/2004.05718.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Burstein, J.; Doran, C.; and Solorio, T., eds., _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 4171–4186. Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Dwivedi and Bresson (2020) Dwivedi, V. P.; and Bresson, X. 2020.  A Generalization of Transformer Networks to Graphs.  _CoRR_ , abs/2012.09699.
  * Dwivedi et al. (2022) Dwivedi, V. P.; Luu, A. T.; Laurent, T.; Bengio, Y.; and Bresson, X. 2022.  Graph Neural Networks with Learnable Structural and Positional Representations.  In _International Conference on Learning Representations_.
  * Fang et al. (2022) Fang, T.; Zhang, Y.; Yang, Y.; and Wang, C. 2022.  Prompt Tuning for Graph Neural Networks.  _CoRR_ , abs/2209.15240.
  * Feng et al. (2022) Feng, J.; Chen, Y.; Li, F.; Sarkar, A.; and Zhang, M. 2022.  How Powerful are K-hop Message Passing Graph Neural Networks.  In _NeurIPS_.
  * Gao, Wang, and Ji (2018) Gao, H.; Wang, Z.; and Ji, S. 2018.  Large-Scale Learnable Graph Convolutional Networks.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’18, 1416–1424. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450355520.
  * Gao, Fisch, and Chen (2021) Gao, T.; Fisch, A.; and Chen, D. 2021.  Making Pre-trained Language Models Better Few-shot Learners.  In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , 3816–3830. Association for Computational Linguistics.
  * Gaulton et al. (2012) Gaulton, A.; Bellis, L. J.; Bento, A. P.; Chambers, J.; Davies, M.; Hersey, A.; Light, Y.; McGlinchey, S.; Michalovich, D.; Al-Lazikani, B.; and Overington, J. P. 2012.  ChEMBL: a large-scale bioactivity database for drug discovery.  _Nucleic Acids Res._ , 40(Database-Issue): 1100–1107.
  * Gilmer et al. (2017) Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; and Dahl, G. E. 2017.  Neural Message Passing for Quantum Chemistry.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, 1263–1272. JMLR.org.
  * Hao et al. (2020) Hao, Z.; Lu, C.; Huang, Z.; Wang, H.; Hu, Z.; Liu, Q.; Chen, E.; and Lee, C. 2020\.  ASGN: An Active Semi-Supervised Graph Neural Network for Molecular Property Prediction.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, 731–752. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450379984.
  * Howard and Ruder (2018) Howard, J.; and Ruder, S. 2018.  Universal Language Model Fine-tuning for Text Classification.  In Gurevych, I.; and Miyao, Y., eds., _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , 328–339. Association for Computational Linguistics.
  * Hu et al. (2021) Hu, W.; Fey, M.; Ren, H.; Nakata, M.; Dong, Y.; and Leskovec, J. 2021.  Ogb-lsc: A large-scale challenge for machine learning on graphs.  _arXiv preprint arXiv:2103.09430_.
  * Hu et al. (2020a) Hu, W.; Fey, M.; Zitnik, M.; Dong, Y.; Ren, H.; Liu, B.; Catasta, M.; and Leskovec, J. 2020a.  Open graph benchmark: Datasets for machine learning on graphs.  _Advances in neural information processing systems_ , 33: 22118–22133.
  * Hu et al. (2020b) Hu, W.; Liu, B.; Gomes, J.; Zitnik, M.; Liang, P.; Pande, V. S.; and Leskovec, J. 2020b.  Strategies for Pre-training Graph Neural Networks.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Hu et al. (2020c) Hu, Z.; Dong, Y.; Wang, K.; Chang, K.-W.; and Sun, Y. 2020c.  GPT-GNN: Generative Pre-Training of Graph Neural Networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, 1857–1867. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450379984.
  * Kreuzer et al. (2021) Kreuzer, D.; Beaini, D.; Hamilton, W. L.; Létourneau, V.; and Tossou, P. 2021\.  Rethinking Graph Transformers with Spectral Attention.  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 21618–21629.
  * Lester, Al-Rfou, and Constant (2021) Lester, B.; Al-Rfou, R.; and Constant, N. 2021.  The Power of Scale for Parameter-Efficient Prompt Tuning.  In Moens, M.; Huang, X.; Specia, L.; and Yih, S. W., eds., _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , 3045–3059. Association for Computational Linguistics.
  * Li, Zhao, and Zeng (2022) Li, H.; Zhao, D.; and Zeng, J. 2022.  KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction.  In Zhang, A.; and Rangwala, H., eds., _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_ , 857–867. ACM.
  * Li et al. (2021) Li, S.; Zhou, J.; Xu, T.; Huang, L.; Wang, F.; Xiong, H.; Huang, W.; Dou, D.; and Xiong, H. 2021.  Structure-Aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD ’21, 975–985. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383325.
  * Li and Liang (2021) Li, X. L.; and Liang, P. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , 4582–4597. Association for Computational Linguistics.
  * Liu et al. (2021a) Liu, X.; Ji, K.; Fu, Y.; Du, Z.; Yang, Z.; and Tang, J. 2021a.  P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.  _CoRR_ , abs/2110.07602.
  * Liu et al. (2021b) Liu, X.; Zheng, Y.; Du, Z.; Ding, M.; Qian, Y.; Yang, Z.; and Tang, J. 2021b.  GPT Understands, Too.  _CoRR_ , abs/2103.10385.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Minaee et al. (2021) Minaee, S.; Kalchbrenner, N.; Cambria, E.; Nikzad, N.; Chenaghlu, M.; and Gao, J. 2021.  Deep learning–based text classification: a comprehensive review.  _ACM computing surveys (CSUR)_ , 54(3): 1–40.
  * Podlewska and Kafel (2018) Podlewska, S.; and Kafel, R. 2018.  MetStabOn—online platform for metabolic stability predictions.  _International journal of molecular sciences_ , 19(4): 1040.
  * Qiu et al. (2020) Qiu, J.; Chen, Q.; Dong, Y.; Zhang, J.; Yang, H.; Ding, M.; Wang, K.; and Tang, J. 2020.  GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, 1150–1160. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450379984.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Rampásek et al. (2022) Rampásek, L.; Galkin, M.; Dwivedi, V. P.; Luu, A. T.; Wolf, G.; and Beaini, D. 2022.  Recipe for a General, Powerful, Scalable Graph Transformer.  In _NeurIPS_.
  * Rong et al. (2020) Rong, Y.; Bian, Y.; Xu, T.; Xie, W.; Wei, Y.; Huang, W.; and Huang, J. 2020.  Self-Supervised Graph Transformer on Large-Scale Molecular Data.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Shin et al. (2020) Shin, T.; Razeghi, Y.; IV, R. L. L.; Wallace, E.; and Singh, S. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , 4222–4235. Association for Computational Linguistics.
  * Subramonian (2021) Subramonian, A. 2021.  Motif-driven contrastive learning of graph representations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 15980–15981.
  * Sun et al. (2022) Sun, M.; Zhou, K.; He, X.; Wang, Y.; and Wang, X. 2022.  GPPT: Graph Pre-Training and Prompt Tuning to Generalize Graph Neural Networks.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , KDD ’22, 1717–1727. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450393850.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In Guyon, I.; von Luxburg, U.; Bengio, S.; Wallach, H. M.; Fergus, R.; Vishwanathan, S. V. N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 5998–6008.
  * Velickovic et al. (2019) Velickovic, P.; Fedus, W.; Hamilton, W. L.; Liò, P.; Bengio, Y.; and Hjelm, R. D. 2019.  Deep Graph Infomax.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang and Zhang (2022) Wang, X.; and Zhang, M. 2022.  How Powerful are Spectral Graph Neural Networks.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvári, C.; Niu, G.; and Sabato, S., eds., _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , 23341–23362. PMLR.
  * Wu et al. (2018) Wu, Z.; Ramsundar, B.; Feinberg, E. N.; Gomes, J.; Geniesse, C.; Pappu, A. S.; Leswing, K.; and Pande, V. 2018.  MoleculeNet: a benchmark for molecular machine learning.  _Chemical science_ , 9(2): 513–530.
  * Ying et al. (2021) Ying, C.; Cai, T.; Luo, S.; Zheng, S.; Ke, G.; He, D.; Shen, Y.; and Liu, T. 2021\.  Do Transformers Really Perform Badly for Graph Representation?  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 28877–28888.
  * You et al. (2021) You, Y.; Chen, T.; Shen, Y.; and Wang, Z. 2021.  Graph Contrastive Learning Automated.  In Meila, M.; and Zhang, T., eds., _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , 12121–12132. PMLR.
  * You et al. (2020) You, Y.; Chen, T.; Sui, Y.; Chen, T.; Wang, Z.; and Shen, Y. 2020.  Graph Contrastive Learning with Augmentations.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Zhang, Wang, and Liu (2018) Zhang, L.; Wang, S.; and Liu, B. 2018.  Deep learning for sentiment analysis: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 8(4): e1253.
  * Zhang et al. (2021) Zhang, Z.; Liu, Q.; Wang, H.; Lu, C.; and Lee, C. 2021.  Motif-based Graph Self-Supervised Learning for Molecular Property Prediction.  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 15870–15882.

##  Appendix A Experimental Details

### Datasets Description

Table 5: Overview of graph-level prediction datasets used in this study.

Dataset | #Graphs | Prediction Task | Metric
---|---|---|---
BACE(Wu et al. 2018) | 1513 | binary classification | AUROC
BBBP(Wu et al. 2018) | 2039 | binary classification | AUROC
ClinTox(Wu et al. 2018) | 1478 | 2-task classification | AUROC
Estrogen(Gaulton et al. 2012) | 3122 | 2-task classification | AUROC
MetStab(Podlewska and Kafel 2018) | 2267 | 2-task classification | AUROC
SIDER(Wu et al. 2018) | 1427 | 27-task classification | AUROC
Tox21(Wu et al. 2018) | 7831 | 12-task classification | AUROC
ToxCast(Wu et al. 2018) | 8575 | 617-task classification | AUROC
ESOL(Wu et al. 2018) | 1128 | regression | RMSE
FreeSolv(Wu et al. 2018) | 642 | regression | RMSE
Lipophilicity(Wu et al. 2018) | 4200 | regression | RMSE
ogbg-molhiv(Hu et al. 2020a) | 41127 | binary classification | AUROC
ogbg-molpcba(Hu et al. 2020a) | 437929 | 128-task classification | Average Precision
PCQM4Mv2(Hu et al. 2021) | 3746620 | regression | Mean Abs. Error

Figure 7: Overview of datasets

The summary of the molecular property datasets utilized in our experiments can
be found in Table 5. Below, you will find a breakdown of the specifics for
each individual dataset:

#### BACE

is a dataset comprising molecules that act as inhibitors of human
$\beta$-secretase 1 (BACE-1) (Wu et al. 2018).

#### BBBP

records whether a molecule can penetrate the blood-brain barrier (Wu et al.
2018).

#### ClinTox

consists of drugs that have been approved by the Food and Drug Administration
(FDA) but were subsequently removed from clinical trials due to their toxic
effects.

#### Estrogen

contains molecules with known activities towards the estrogen receptors,
extracted from the ChEMBL dataset (Gaulton et al. 2012).

#### MetStab

measures the half-lifetime of molecules within an organism (Podlewska and
Kafel 2018).

#### SIDER

records adverse drug reactions of marketed drugs grouped into 27 system organ
classes (Wu et al. 2018).

#### ToxCast

contains multiple toxicity labels for molecules obtained through high-
throughput screening tests (Wu et al. 2018).

#### Tox21

Tox21 is a public database containing the toxicity of compounds (Wu et al.
2018).

#### ESOL

ESOL records the solubility of compounds (Wu et al. 2018).

#### Lipophilicity

Lipophilicity measures the molecular membrane permeability and solubility (Wu
et al. 2018).

#### FreeSolv

FreeSolv contains the hydration free energy of small molecules in water from
both experiments and alchemical free energy calculation (Wu et al. 2018).

#### OGBG-MolHIV and OGBG-MolPCBA

The datasets ogbg-molhiv and ogbg-molpcba (Hu et al. 2020a) (MIT License) are
molecular property prediction datasets utilized by OGB, which were originally
sourced from MoleculeNet. TThe primary objective of ogbg-molhiv is the binary
classification of a molecule’s suitability to inhibit HIV replication. On the
other hand, ogbg-molpcba, derived from PubChem BioAssay, is designed to
predict the outcomes of 128 bioassays in a multi-task binary classification
setting.

#### PCQM4Mv2

is considered one of the largest graph prediction datasets, comprising over
3.7 million molecules. Its primary objective is to utilize the 2-D graphs of
molecules to predict their HOMO-LUMO energy gap in electronvolts (eV).

### Dataset Splits

The main benchmarking results were obtained using 5-fold cross-validation
runs, except for pretraining on PCQM4Mv2, which was conducted in a single run
using the standard train/valid/test split. We ensured consistent data
splitting for cross-validation and used the same seeds for models and tuning
methods in all experiments. Furthermore, for ablations studies we performed
replication experiments from scratch, and the results previously presented in
the main text were not reused.

### Hyperparameters

In our experiments, we conduct hyperparameter tuning for several parameters.
Due to the large total number of hyperparameters, we avoid using a grid-search
scheme for tuning. We adopt the AdamW optimizer with a linear ”warm-up”
strategy, gradually increasing the learning rate at the start of training,
followed by either cosine or linear decay. The specific values for the warm-up
period, base learning rate, and total number of epochs are adjusted
individually for each dataset and are provided in Tables 6, 8, 9, 7 along with
other relevant hyperparameters. The size of modles

Table 6: The pre-training and tuning hyperparameters for Graphormer(Ying et
al. 2021).

Hyperparameter | Pre-training | Tuning
---|---|---
# Layers | 12 | 12
# Attention Heads | 32 | 32
Hidden Dim | 768 | 768
Dropout | 0.1 | {0, 0.1}
Learning Rate | 2e-4 | { 3e-4, 1r-4}
Learning Rate Decay | polynomial | linear
Weight Decay | 1e-6 | {0, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1e1, 2e1}
Gradient Clipping | 5.0 | 5.0
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Prefix Length | - | $\\{10,20,30\dots 110\\}$
Epochs | {100, 200} | 100

Table 7: The pre-training and tuning hyperparameters for LiGhT(Li, Zhao, and
Zeng 2022).

Hyperparameter | Pre-training | Tuning
---|---|---
# Layers | 12 | 12
# Attention Heads | 12 | 12
Hidden Dim | 768 | 768
Dropout | 0.1 | {0, 0.1}
Learning Rate | 2e-4 | { 3e-4, 1r-4}
Learning Rate Decay | polynomial | linear
Weight Decay | 1e-6 | {0, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1e1, 2e1}
Gradient Clipping | 5.0 | 5.0
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Prefix Length | - | $\\{10,20,30\dots 110\\}$
Epochs | {100, 200} | 100

Table 8: The pre-training and tuning hyperparameters for GraphGPS
small(Rampásek et al. 2022).

Hyperparameter | Pre-training | Tuning
---|---|---
# Layers | 16 | 16
# Attention Heads | 8 | 8
Hidden Dim | 256 | 256
Dropout | 0.1 | {0, 0.1}
Learning Rate | 2e-4 | { 1e-3, 3e-4, 1r-4}
Learning Rate Decay | polynomial | cosine with warm-up
Learning Rate Warm-up Epochs | - | 5
Weight Decay | 1e-6 | {0, 1e-6, 1e-5, 1e-4}
Gradient Clipping | 5.0 | 5.0
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Prefix Length | - | $\\{10,20,30\dots 110\\}$
Epochs | {100, 200} | 100

Table 9: The pre-training and tuning hyperparameters for GraphGPS
large(Rampásek et al. 2022).

Hyperparameter | Pre-training | Tuning
---|---|---
# Layers | 12 | 12
# Attention Heads | 16 | 16
Hidden Dim | 768 | 768
Dropout | 0.1 | {0, 0.1}
Learning Rate | 2e-4 | { 1e-3, 3e-4, 1r-4}
Learning Rate Decay | polynomial | cosine with warm-up
Learning Rate Warm-up Epochs | - | 5
Weight Decay | 1e-6 | {0, 1e-6, 1e-5, 1e-4}
Gradient Clipping | 5.0 | 5.0
AdamW $\beta$ | (0.9, 0.999) | (0.9, 0.999)
Prefix Length | - | $\\{10,20,30\dots 110\\}$
Epochs | {100, 200} | 100

Table 10: Graph Transformer architectures and number of parameters. Model | Size
---|---
GraphGPS Small (Rampásek et al. 2022) | 14M
Graphormer (Ying et al. 2021) | 48M
LiGhT (Li, Zhao, and Zeng 2022) | 90M
GraphGPS Large (Rampásek et al. 2022) | 103M

##  Appendix B More Ablation Studies

Table 11: Ablation studies of each component.

Model | Moleculenet Regression Dataset ($\downarrow$)
---|---
ESOL | FreeSolv | Lipo
Graphormer Lightweight Tuning | 1.123 ± 0.044 | 3.085 ± 0.378 | 0.915 ± 0.032
Graphormer Deep Prefix Tuning | 0.910 ± 0.041 | 1.657 ± 0.351 | 0.863 ± 0.026
Graphormer DeepGPT | 0.943 ± 0.024 | 1.668 ± 0.114 | 0.834 ± 0.046
GraphGPS Lightweight Tuning | 2.306 ± 0.419 | 4.793 ± 3.652 | 1.043 ± 0.091
GraphGPS Deep Prefix Tuning | 0.589 ± 0.087 | 2.221 ± 0.177 | 0.459 ± 0.040
GraphGPS DeepGPT | 0.685 ± 0.130 | 1.415 ± 0.254 | 0.528 ± 0.056

Table 12: Ablation studies of each component

Model | #Params | Classification Dataset ($\uparrow$)
---|---|---
BACE | BBBP∗ | ClinTox∗ | Estrogen∗ | MetStab∗ | SIDER | Tox21 | ToxCast∗
Graphormer Lightweight Tuning |  | 0.847 ± 0.013 | 0.882 ± 0.016 | 0.766 ± 0.082 | 0.935 ± 0.015 | 0.863 ± 0.021 | 0.625 ± 0.017 | 0.797 ± 0.008 | 0.705 ± 0.011
Graphormer Deep Prefix Tuning |  | 0.863 ± 0.013 | 0.904 ± 0.013 | 0.860 ± 0.062 | 0.937 ± 0.016 | 0.883 ± 0.014 | 0.584 ± 0.009 | 0.790 ± 0.010 | 0.705 ± 0.015
Graphormer DeepGPT |  | 0.883 ± 0.019 | 0.914 ± 0.016 | 0.884 ± 0.030 | 0.941 ± 0.006 | 0.871 ± 0.004 | 0.646 ± 0.003 | 0.813 ± 0.010 | 0.725 ± 0.011
GraphGPS Lightweight Tuning |  | 0.819 ± 0.010 | 0.751 ± 0.041 | 0.692 ± 0.107 | 0.877 ± 0.012 | 0.673 ± 0.031 | 0.541 ± 0.015 | 0.667 ± 0.008 | 0.519 ± 0.005
GraphGPS Deep Prefix Tuning |  | 0.854 ± 0.016 | 0.835 ± 0.030 | 0.866 ± 0.059 | 0.902 ± 0.014 | 0.841 ± 0.014 | 0.598 ± 0.019 | 0.821 ± 0.014 | 0.690 ± 0.009
GraphGPS DeepGPT |  | 0.892 ± 0.022 | 0.901 ± 0.015 | 0.907 ± 0.044 | 0.944 ± 0.010 | 0.899 ± 0.019 | 0.609 ± 0.020 | 0.832 ± 0.012 | 0.735 ± 0.005

Figure 8: Effect of prompt length using Graphormer.

(a) Fixed 3 layers

(b) First k layers.

(c) Last k layers.

Figure 9: Impact of the depth of injected prompt tokens using GraphGPS on BACE
dataset. The notation ”[x-y]” represents the layer-interval where continuous
prompts are added (e.g., ”3-6” indicates prompts added to transformer layers
from 3 to 6) .

(a) Fixed 2 layers

(b) First k layers

(c) Last k layers

Figure 10: Impact of injecting prompt tokens to using GraphGPS on Clintox.