# Can we Soft Prompt LLMs for Graph Learning Tasks?

Zheyuan Liu  University of Notre Dame<EMAIL_ADDRESS>,  Xiaoxin He  National
University of Singapore<EMAIL_ADDRESS>,  Yijun Tian  University of
Notre Dame<EMAIL_ADDRESS>and  Nitesh V. Chawla  University of Notre Dame
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Graph plays an important role in representing complex relationships in real-
world applications such as social networks, biological data and citation
networks. In recent years, Large Language Models (LLMs) have achieved
tremendous success in various domains, which makes applying LLMs to graphs
particularly appealing. However, directly applying LLMs to graph modalities
presents unique challenges due to the discrepancy and mismatch between the
graph and text modalities. Hence, to further investigate LLMs’ potential for
comprehending graph information, we introduce GraphPrompter, a novel framework
designed to align graph information with LLMs via soft prompts. Specifically,
GraphPrompter consists of two main components: a graph neural network to
encode complex graph information and an LLM that effectively processes textual
information. Comprehensive experiments on various benchmark datasets under
node classification and link prediction tasks demonstrate the effectiveness of
our proposed method. The GraphPrompter framework unveils the substantial
capabilities of LLMs as predictors in graph-related tasks, enabling
researchers to utilize LLMs across a spectrum of real-world graph scenarios
more effectively111Code is available at
https://github.com/franciscoliu/graphprompter..

Large Language Models; Graph Neural Networks; Natural Language Processing;
Graph Representation Learning

††journalyear: 2024††copyright: acmlicensed††conference: Companion Proceedings
of the ACM Web Conference 2024; May 13–17, 2024; Singapore,
Singapore.††booktitle: Companion Proceedings of the ACM Web Conference 2024
(WWW ’24 Companion), May 13–17, 2024, Singapore, Singapore††isbn:
979-8-4007-0172-6/24/05††doi: 10.1145/3589335.3651476††ccs: Computing
methodologies Neural networks††ccs: Information systems Language models††ccs:
Computing methodologies Semantic networks

##  1\. Introduction

Large Language Models (LLMs) have demonstrated significant success across
various domains (Chowdhery et al., 2023; Touvron et al., 2023; Ouyang et al.,
2022; Lewkowycz et al., 2022; Wei et al., 2024; Wang et al., 2024), largely
attributed to their extensive knowledge memorized during the pretraining phase
and their exceptional ability to generalize during the fine-tuning process on
diverse textual datasets (Hoffmann et al., 2022; Liang et al., 2022; Tian et
al., 2024a; Carlini et al., 2022; Tan et al., 2024). This success has spurred
interest in combining graph neural networks (GNNs) with LLMs to enhance their
capabilities in understanding and modeling graphs (Yang et al., 2021; Zhang et
al., 2022; Ostendorff et al., 2022; Tian et al., 2024b; He et al., 2024),
including implementing LLMs as encoders to process features within GNNs (Chai
et al., 2023; Mavromatis et al., 2023; He et al., 2023; Yu et al., 2023), and
employing LLMs as aligners with GNNs to enhance performance (Zhao et al.,
2022; Wen and Fang, 2023; Brannon et al., 2023).

However, directly applying LLMs to graph modalities presents unique challenges
due to the discrepancy and mismatch between the graph and text modalities. For
example, existing works primarily map the graph into text, ignoring the
irrelevant information and noises pertinent to the graphs. This oversight
results in an inadequate understanding of crucial structural knowledge within
the graphs. Furthermore, the challenge is amplified when dealing with large
graphs containing thousands or millions of nodes and edges, as this complexity
hinders LLMs’ ability to grasp intricate structural information.

These limitations motivate our investigation into using a graph as a soft
prompt encoded by GNNs for LLMs in graph learning tasks. The intuition behind
this approach is that GNN are more proficient in aggregating and transforming
information from the neighbourhood information, which could provide
topological information to LLMs. Additionally, it can guide the LLM in
selecting relevant information from textual input and control the generation
process for token generation. Specifically, in this work, we aim to examine
the efficacy of a soft graph prompt in informing the LLM’s predictions. Hence,
a natural yet pivotal research question arise: Can we soft prompt LLMs for
graph leanring tasks?

To answer this question, we introduce GraphPrompter, a novel framework that
combines the strength of GNNs and LLMs to process and comprehend graph-
structured data. In particular, GraphPrompter capitalizes on the frozen LLM as
a robust feature extractor that taps into its vast pretraining knowledge, thus
enabling us to avoid extensive task-specific fine-tuning. In parallel, the GNN
works on the graph to produce node embeddings, which are later concatenated
with a prompt instruction to guide LLMs for graph learning tasks. The LLM, due
to its powerful autoregressive nature, generates a language response based on
the fused graph and text information, effectively turning the LLM into a
powerful tool for graph understanding tasks. See Figure 1 for an illustration.

This hybrid approach is particularly suitable for graphs with textual
attributes (i.e., textual graphs), which requires an understanding of both the
textual content and the graph structure, such as identifying the subcategory
of an academic paper based on its citation network and abstract. We
experimentally demonstrate that our framework is capable of prompting LLMs for
graph learning tasks on five benchmark datasets under node classification and
link prediction tasks. It showcases the potential for significant advancements
in the use of LLMs for complex data structures beyond traditional text,
opening up new avenues for research and application in the realm of AI
assistants capable of intricate graph comprehension. Our main contributions
are as follows:

  * •

To the best of our knowledge, this is the very first work investigating
whether LLMs can understand graph learning tasks via soft prompting.

  * •

We propose GraphPrompter, a novel plug-and-play framework that first employ
GNN to get node representations from the textual graph. Then the obtained
embeddings are concatenated with a prompt instruction to guide LLMs for graph
learning tasks.

  * •

Extensive experiments demonstrate the effectiveness of our proposed framework
under both node classification and link prediction tasks across various graph
benchmarks.

Figure 1. Illustration of the proposed GraphPrompter for node classification
task. The process involves extracting a k-hop subgraph for each node, feeding
it into a GNN followed by a projection layer. Simultaneously, the textual
attributes associated with each node are processed by the text embedder. The
resulting node embedding is then concatenated with text embeddings, serving as
a soft prompt to guide the LLM for graph learning tasks.

##  2\. Method

To enhance the alignment of graph knowledge with LLM, we introduce
GraphPrompter, a plug-and-play pipeline that fuses post-processed node
embeddings with LLMs. This integration is designed to leverage the rich
semantic context LLMs provide, enhancing the interpretability and utility of
graph representations, as illustrated in Figure 1.

###  2.1. Graph Section

A graph can be represented as $G=(V,E)$, where $V$ is a set of vertices, and
$E$ is a set of edges. For each node $v_{i}\in V$, we define the 3-hop
subgraph $G_{s_{i}}$ as the induced subgraph containing all nodes within three
hops from $v_{i}$. Subsequently, a GNN is employed to compute the node
embeddings $X_{i}$ for each of these subgraphs $G_{s_{i}}$, as follows:

(1) $X_{i}=\text{GNN}(G_{s_{i}})\in\mathbb{R}^{d_{g}},$

where $d_{g}$ is the hidden dimension of the GNN. Each embedding $X_{i}$
captures the structural information of the corresponding subgraph, providing a
rich representation of the topological features associated with $v_{i}$.

We employ a projection layer, specifically, a multilayer perceptron (MLP), to
align the node embedding with the vector space of the LLM:

(2) $\hat{X}_{i}=\text{MLP}(X_{i})\in\mathbb{R}^{d_{l}},$

where $d_{l}$ is the hidden dimension of the LLM.

Both the GNN and the projection layer are trained in an end-to-end manner,
ensuring that the embeddings $X_{i}$ are discriminative for the downstream
task. The rationale behind this approach is twofold: first, to leverage the
GNN’s inherent capacity to aggregate and transform local neighborhood
information into meaningful embeddings; and second, to enable the subsequent
integration with the LLM, which interprets these embeddings as a soft prompt
in conjunction with textual data.

###  2.2. LLM Section

After encoding the graph structure, the GraphPrompter proceeds to process the
textual information associated with each node. This is where the power of the
LLM comes into play, as it is kept frozen for efficient fine-tuning while
preserving the LLM’s pre-trained knowledge. For each node $v_{i}$, given its
associated textual attribute $T_{i}$ (e.g., title and abstract of a paper in a
citation graph), we tokenize $T_{i}$ using the frozen tokenizer of the LLM as
follows:

(3) $\displaystyle T_{\text{tokens}}=\text{Tokenizer}(T_{i}).$

The tokenizer converts the text into a sequence of discrete tokens
$T_{\text{tokens}}$ in the LLM’s vocabulary. Subsequently, these tokens are
then embedded into a continuous space:

(4) $\displaystyle
T_{\text{emb}}=\text{Embed}(T_{\text{tokens}})\in\mathbb{R}^{M\times d_{l}},$

where $M$ is the sequence length and $d_{l}$ is the hidden dimension of the
LLM. These embeddings are designed to capture the semantic meaning of the
text, complementing the structural information provided by the GNN.

Next, we concatenate the node embeddings and the text embeddings, denoted as
$[\hat{X}_{i},T_{\text{emb}}]$, which will go through the self-attention
layers in the LLM as usual.

The motivation for this step is to ensure that the LLM can process the rich
semantic content of the text alongside the graph embeddings. By encoding the
text attributes into a compatible format, we aim to capitalize on the LLM’s
advanced understanding of natural language, which is crucial for tasks that
require a combination of structural and textual data interpretation, such as
node classification in citation networks. Here, we point out the key
distinction between node classification and link prediction: node
classification evaluates information from a single node, whereas link
prediction considers the attributes of two nodes, specifically their origin
and destination.

##  3\. Experiments

(a) Dense (Title + Abstract)

(b) Sparse (Title Only)

Figure 2. Node classification experiments comparing the proposed GraphPrompter
with baseline methods (i.e., soft prompt tuning and fine-tuning). The $x$ axis
shows dataset types, and the $y$ axis displays accuracy scores. Figure 2(a)
illustrates a dense semantic setting including both the title and abstract of
a paper within the node embeddings, while Figure 2(b) illustrates a sparse
semantic setting with title only.

In this section, we conduct extensive experiments to validate the
effectiveness of the GraphPrompter. Specifically, our experiments aim to
address the central question: Can we soft prompt LLMs for graph learning
tasks?

###  3.1. Experiment setups

####  3.1.1. Datasets and baseline models.

Our experiments mainly focus on node classification and link prediction using
various graph benchmarks: Cora, Citeseer, Pubmed, Ogbn-arxiv, and Ogbn-
products. For Ogbn-products, given its substantial scale of 2 million nodes
and 61 million edges, we have employed a node sampling strategy to obtain a
subgraph containing 54k nodes and 74k edges. We compare our approach with
three kinds of baseline models: 1) Pure GNN; 2) Frozen LLM, which includes
zero shot and soft prompt tuning; and 3) Tuned LLM with LoRA. We employ the
LLAMA2-7B (Touvron et al., 2023) as the LLM backbone. For the pure GNN method,
we use GAT (Veličković et al., 2017). In the LLM-frozen setting, for zero-
shot, we use a frozen LLM for direct node classification/edge prediction using
textual attributes of the nodes. In soft prompt tuning, we keep the parameters
of the LLM frozen and adjust only the prompt. In LLM-tuned, the original LLM
parameters are updated for downstream tasks by utilizing LoRA.

####  3.1.2. Implementation Details.

We report the mean of five independent runs with different seeds. The
experiments are conducted on 2 A100 GPUs (80GB).

##  4\. Experiments Result

To answer the central question, we conduct experiments across various graph
benchmark to comprehensively evaluate the effectiveness of GraphPrompter. The
performance is reported in Table 1 and Table 2.

###  4.1. Node Classification Task

Method | Cora | Citeseer | Pubmed | Ogbn-arxiv | Ogbn-products
---|---|---|---|---|---
Node Classification Accuracy (%) ${\color[rgb]{0,0,1}\uparrow}$
GAT | 84.69 | 70.78 | 84.09 | 71.82 | 70.52
Zero-Shot | 43.31 | 29.22 | 91.39 | 44.23 | 15.05
Soft Prompt Tuning | 70.31 | 70.97 | 91.45 | 71.99 | 75.14
Fine-tuning + LoRA | 75.97 | 73.45 | 94.68 | 74.58 | 78.99
Subgraph Prompt Tuning | 80.17 | 72.29 | 93.84 | 75.04 | 75.30
GraphPrompter \+ LoRA | 80.26 | 73.61 | 94.80 | 75.61 | 79.54

Table 1. Node classification result of our proposed GraphPrompter, with a
number of baselines under various graph benchmarks: Cora, Citeseer, Pubmed,
Ogbn-arxiv and Ogbn-products. For each model, we present the mean accuracy
from five independent runs, with both the title and abstract of papers
provided as input along with specific instructions. Bold indicates the best
performance and underline indicates the runner-up.

In Table 1, we present the results of node classification experiments
conducted on a variety of benchmark datasets, where both title and abstract
information are provided to LLM alongside detailed instructions. As it can be
seen from the table, the integration of the proposed method (GraphPrompter)
with LoRA consistently surpasses the performance of other baseline methods
across diverse benchmarks. In particular, GraphPrompter with LoRA in the
PubMed and Citeseer datasets with top accuracies of 94.80 % and 73.61 %,
respectively. The naive fine-tuning with LoRA approach typically appears as
the runner-up and has small performance gap with GraphPrompter. Though
subgraph prompt tuning approach has relatively close accuracy score with naive
fine-tuning, it still not as effective as fine-tuning on node classification
task. Furthermore, we observe that zero-shot generally ranks lowest in
performance across all benchmark datasets, with the exception of PubMed
dataset. This reflects the limitation of LLMs in capturing graph knowledge
without the auxiliary processing provided by GNNs. Additionally, two distinct
experimental setups are illustrated in Figure 2, where Figure 2(a) encompasses
both title and abstract within the node embeddings, and Figure 2(b) restricts
the node embeddings to only the title. The comparative analysis illustrated in
the figures further illustrates the superior average performance of the
proposed method (GraphPrompter), which combines subgraph prompt tuning and
GraphPrompter, in comparison to other baseline approaches, specifically soft
prompt tuning and fine-tuning methodologies.

###  4.2. Link Prediction Task

In Table 2, we present the results of link prediction experiments conducted on
a variety of benchmark datasets, where only the title is provided to LLM. As
seen in Table 2, subgraph prompt tuning approach consistently surpasses
baseline models in link prediction tasks on various benchmarks. It shows
remarkable performance especially on the Citeseer dataset, achieving a 93.49 %
accuracy rate. GraphPrompter \+ LoRA is usually the runner-up model for the
majority case. Notably, similar to node classification task, the Zero-Shot
approach generally scores lowest, underscoring the necessity of GNN
integration for LLMs to effectively process and understand graph data. It is
notable that under both node classification and link prediction, GraphPrompter
can continue outperforming traditional GNN and soft prompting techniques. This
observation indicates that GraphPrompter, is particularly adept at capturing
and utilizing the complex intersection between graph components and textual
information, leading to a better performance on various graph related tasks.

Method | Cora | Citeseer | Pubmed | Ogbn-arxiv | Ogbn-products
---|---|---|---|---|---
Link Prediction Accuracy (%) ${\color[rgb]{0,0,1}\uparrow}$
GAT | 90.71 | 87.40 | 86.18 | 72.93 | 65.67
Zero-Shot | 21.94 | 7.69 | 10.23 | 34.41 | 7.47
Soft Prompt Tuning | 86.77 | 88.87 | 83.98 | 69.13 | 62.78
Fine-tuning + LoRA | 87.58 | 87.91 | 81.33 | 70.10 | 67.31
Subgraph Prompt Tuning | 89.15 | 93.49 | 87.20 | 75.28 | 68.18
GraphPrompter \+ LoRA | 90.10 | 91.67 | 86.49 | 73.21 | 69.55

Table 2. Link prediction result of our proposed GraphPrompter, with a number
of baselines under various graph benchmarks: Cora, Citeseer, Pubmed, Ogbn-
arxiv and Ogbn-products. For each model, we present the mean accuracy derived
from five independent runs conducted on the link prediction task, with only
the abstract of papers are provided as input. Bold indicates the best
performance and underline indicates the runner-up.

##  5\. Analysis

The experiments on benchmark datasets for node classification and link
prediction tasks demonstrate LLMs’ adaptability and efficiency in graph
learning, enabled by our proposed soft graph prompting strategies. Here, we
summarize the key findings:

  * •

Performance Benchmark: Overall, our proposed method ranks either as the best
or runner-up across all benchmark datasets for both tasks. This consistently
high performance affirms a positive response to our central question: we can
effectively soft prompt for LLMs in graph learning tasks.

  * •

Methodological Superiority: The fine-tuning approach, while competitive,
displayed a marginal performance gap when compared to GraphPrompter. This
indicates that while traditional fine-tuning remains effective, the
specialized soft graph prompt tuning strategy designed for graph data offers a
more potent solution for enhancing LLM’s graph learning capabilities.

  * •

Challenges with Zero-Shot Learning: The zero-shot approach consistently ranked
the lowest across all benchmarks for both tasks. This reflects the challenges
LLMs face in comprehending graph structure and semantics, highlighting the
necessity for tailored approaches like GraphPrompter in graph learning tasks.

##  6\. Conclusion

In conclusion, our study reveals the significant potential of leveraging LLMs
for interpreting graph structures through a prompt tuning methodology. To
facilitate this study, we presents GraphPrompter, a novel plug-and-play
framework that integrates the capabilities of Large Language Models with the
structural insights of Graph Neural Networks for the task of node
classification and link prediction in graph data. Specifically, GraphPrompter
consists of two sections: the graph section utilizes GNN for structural
insights, while the text section applies LLM to interpret node-related textual
data. Extensive experiments and in-depth studies demonstrate the superiority
of GraphPrompter across multiple benchmark datasets. Future work could focus
on extending our pipeline to other more complex graph level tasks.

## References

  * (1)
  * Brannon et al. (2023) William Brannon, Suyash Fulay, Hang Jiang, Wonjune Kang, Brandon Roy, Jad Kabbara, and Deb Roy. 2023.  ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings.  _arXiv preprint arXiv:2305.14321_ (2023).
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ (2022).
  * Chai et al. (2023) Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. 2023.  Graphllm: Boosting graph reasoning ability of large language model.  _arXiv preprint arXiv:2310.05845_ (2023).
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023\.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ (2023).
  * He et al. (2023) Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. 2023.  Explanations as Features: LLM-Based Features for Text-Attributed Graphs.  _arXiv preprint arXiv:2305.19523_ (2023).
  * He et al. (2024) Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024.  G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering.  _arXiv preprint arXiv:2402.07630_ (2024).
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022\.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ (2022).
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022\.  Solving quantitative reasoning problems with language models. In _NeurIPS_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022\.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ (2022).
  * Mavromatis et al. (2023) Costas Mavromatis, Vassilis N Ioannidis, Shen Wang, Da Zheng, Soji Adeshina, Jun Ma, Han Zhao, Christos Faloutsos, and George Karypis. 2023.  Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs.  _arXiv preprint arXiv:2304.10668_ (2023).
  * Ostendorff et al. (2022) Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. 2022.  Neighborhood contrastive learning for scientific document representations with citation embeddings.  _arXiv preprint arXiv:2202.06671_ (2022).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback. In _NeurIPS_.
  * Tan et al. (2024) Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2024.  Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning.  _arXiv preprint arXiv:2402.04401_ (2024).
  * Tian et al. (2024a) Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh V Chawla. 2024a.  TinyLLM: Learning a Small Student from Multiple Large Language Models.  _arXiv preprint arXiv:2402.04616_ (2024).
  * Tian et al. (2024b) Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V Chawla, and Panpan Xu. 2024b.  Graph neural prompting with large language models. In _AAAI_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Wang et al. (2024) Yu Wang, Nedim Lipka, Ryan A Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. 2024.  Knowledge graph prompting for multi-document question answering. In _AAAI_.
  * Wei et al. (2024) Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024.  Llmrec: Large language models with graph augmentation for recommendation. In _WSDM_. 806–815.
  * Wen and Fang (2023) Zhihao Wen and Yuan Fang. 2023.  Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting.  _arXiv preprint arXiv:2305.03324_ (2023).
  * Yang et al. (2021) Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021.  GraphFormers: GNN-nested transformers for representation learning on textual graph. In _NeurIPS_.
  * Yu et al. (2023) Jianxiang Yu, Yuxiang Ren, Chenghua Gong, Jiaqi Tan, Xiang Li, and Xuecang Zhang. 2023.  Empower text-attributed graphs learning with large language models (llms).  _arXiv preprint arXiv:2310.09872_ (2023).
  * Zhang et al. (2022) Yu Zhang, Zhihong Shen, Chieh-Han Wu, Boya Xie, Junheng Hao, Ye-Yi Wang, Kuansan Wang, and Jiawei Han. 2022.  Metadata-induced contrastive learning for zero-shot multi-label text classification. In _WWW_.
  * Zhao et al. (2022) Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. 2022.  Learning on large-scale text-attributed graphs via variational inference.  _arXiv preprint arXiv:2210.14709_ (2022).