# Towards a Deep(er) Understanding of Interaction through Modeling,
Simulation, and Optimization

Florian Fischer<EMAIL_ADDRESS>0000-0001-7530-6838 ,
Arthur Fleig<EMAIL_ADDRESS>0000-0003-4987-7308 ,  Markus Klar
<EMAIL_ADDRESS>0000-0003-2445-152X ,  Viktorija Paneva
<EMAIL_ADDRESS>0000-0002-5152-3077 and  Jörg Müller
<EMAIL_ADDRESS>0000-0002-4971-9126 University of
BayreuthUniversitätsstr. 30BayreuthGermanyD-95447

(2023)

###### Abstract.

The traditional user-centered design process can hardly keep up with the ever
faster technical development and increasingly diverse user preferences. As a
solution, we propose to augment the tried-and-tested approach of conducting
user studies with simulation and optimization of the entire human-computer
interaction loop. This approach allows to better understand phenomena through
explicit modeling, build virtual prototypes through simulation, and improve
interaction techniques through optimization. Building predictive user models
also supports the creation and validation of HCI theories, and constitutes a
decisive step towards new, intelligent, and adaptive user interfaces. We
report our experience in virtually developing new interaction techniques on
the example of acoustic levitation, and present our optimization-based
framework for HCI. With this, we strive to gain a better understanding of
interaction and at the same time feed the discussion on questions such as
which tools and tutorials are necessary to make virtual prototyping more
accessible to different audiences.

optimization framework, design process, virtual prototyping, interaction
technique, interface dynamics, acoustic levitation, ultrasonic levitation,
path following, volumetric displays, user model, biomechanical model, model
predictive control, reinforcement learning

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs:
Human-centered computing HCI theory, concepts and models††ccs: Human-centered
computing Systems and tools for interaction design††ccs: Computing
methodologies Computational control theory

##  1\. Introduction

As society becomes increasingly technologized, new tracking and display
technologies, such as virtual, augmented, and mixed reality, have enabled
massive growth in the design space of interaction techniques. However,
interaction with technology still often feels lackluster and unnatural
compared to how we interact with objects in nature. To design more natural
interaction techniques, it is crucial to better understand interaction
techniques and user intentions. This becomes increasingly more challenging, as
not only the devices but also user preferences become more diverse.

The traditional user-centered design process in Human-Computer Interaction
(HCI) is focused on creating interfaces that work well for a specific user
group. It tends to rely heavily on user feedback, which can be time-consuming,
costly, and may not always provide actionable insights. As such, it is
struggling to keep up. We propose to add into the mix a principled model-based
design process, where simulation and optimization of the whole human-computer
interaction loop is key. This does not replace the tried-and-tested approach
of conducting user studies, but rather combines model-based simulation and
optimization with user studies that need to be run much less frequently and at
a much later stage.

Simulation provides a virtual environment for testing the behavior and
performance of HCI systems and interfaces. It allows designers to keep the
prototype entirely virtual, and evaluate user experience and assess user
engagement, for example, in Virtual Reality (VR). This helps to identify
potential problems and to make improvements early on in the design process,
before real prototypes are even built. Modeling the underlying interface
dynamics is not only useful for the interaction design and testing, but also
for optimizing the performance of the interface itself.

In addition to modeling an interaction technique, we can build a generative
user model. This helps in identifying potential barriers to usability and
“makes design and engineering more predictable and robust processes” (Murray-
Smith et al., 2022). By simulating the user, we avoid physical, emotional, or
ethical risks, and prevent causing stress to real users in exhausting user
studies. Finally, the ability of a simulation to match real user behavior is a
strong indicator of whether we understand an interactive system, thus,
building user models also supports the creation and validation of new theories
in HCI.

Below we report on our most recent advances in this domain. In Section 2, we
investigate the development of future interactive systems for acoustic
levitation interfaces, and provide examples how modeling and simulation can
help assess user engagement and test interaction parameters with virtual
prototypes, and improve interface performance. In Section 3, we provide a
comprehensive framework that encompasses the (biomechanical) simulation of
users, where we focus on generating user _movement_. In Section 4, we reflect
on limitations and current open questions in the field.

##  2\. Acoustic Levitation

Figure 1. (A) A user performing a pointing task on the real prototype (left),
and in the Levitation Simulator (Paneva et al., 2020a) (middle, right), using
LeviCursor (Bachynskyi et al., 2018) for selecting targets in 3D space (marked
in red). (B) A levitated particle traverses a periodic path at a frequency of
10Hz to reveal volumetric images in mid-air. Using the OptiTrap (Paneva et
al., 2022) algorithm we can specify physically feasible trap trajectories that
render generic shapes in optimal time. Figure adapted from (Paneva et al.,
2020a) and (Paneva et al., 2022).

Some of the greatest visionaries in HCI imagined the interface of the future
as “a room where the computer can control the existence of matter”
(Sutherland, 1965), and as “a dynamic physical material that reflects the
changes in digital states in real time” (Ishii et al., 2012). With acoustic
levitation technology, we see vast potential to get a step closer to these
great visions of the ultimate mixed reality, where the digital and the
physical world are fully merged. Acoustic levitation displays offer a novel
and innovative way of displaying and interacting with digital content in real
physical space, by using sound waves to manipulate physical matter. The
interface typically consists of two opposing phased arrays of transducers.
Each transducer emits ultrasonic waves at a frequency inaudible for humans, of
40kHz. By appropriately setting the phase and amplitude of each transducer, we
can generate acoustic traps at the points where the acoustic forces converge.
In these nodes of low acoustic pressure, it is possible to suspend millimeter-
sized particles in mid-air. By moving the acoustic trap in 3D space, we can
digitally control the position of the levitated physical matter, and in this
manner, generate dynamic visualizations in physical space without the need for
wearables or any other gadgets.

When developing and designing for radically novel interfaces such as acoustic
levitation, in addition to the financial and time costs associated with user
testing mentioned earlier, other challenges, such as availability and
operability occur. Building and maintaining interactive levitation interfaces
requires specific technical expertise, knowledge of acoustics, microsecond
synchronization, and submillimeter calibration of the system components. In
addition, the interface can be difficult to debug. In case of problems, the
only observable effect is the levitating particle shooting out in an
uncontrollable manner. This can pose a barrier for designers, artists, game
developers, researchers, etc. to start experimenting with this novel
technology, and test prototypes of their applications with users. To solve
this problem we propose virtual prototyping, an approach that has proven
successful in other disciplines, e.g., automotive, product design, and
manufacturing (Antonino Gomes de Sá and Gabriel Zachmann, 1999; Berg and
Vance, 2017). We developed the Levitation Simulator (Paneva et al., 2020a) –
an interactive simulation tool in VR that can be used to iteratively develop
and prototype ideas for acoustic levitation interfaces and even conduct user
tests and formal experiments. Only after the development has converged, the
resulting system can be validated using a real apparatus. The simulator
consists of two modules – an interaction and a simulation module. The
interaction module is implemented within the Unity Game engine, and it can
receive user input via a motion capture system or VR controllers. In the
simulation module, we incorporated a model of a levitated particle moving in
an acoustic field, which allowed for the simulation of physically accurate
dynamics of the virtual particle. We validated the tool by performing a
pointing study in the Levitation Simulator and on the real prototype, using
LeviCursor (Bachynskyi et al., 2018), a levitated 3D physical cursor. Figure
1(A) shows a user performing repetitive aimed movements in mid-air between two
three-dimensional spherical targets on the real prototype, and in the
Levitation Simulator. The results showed comparable performance. Further
testing with gaming applications showed that the Levitation Simulator can
provide good predictions, regarding user interaction and engagement with the
real prototype. In future studies, the Levitation Simulator can also be useful
in exploring the multimodality of acoustic levitation displays, e.g., by
augmenting the levitated particles with mid-air haptic feedback to potentially
improve display accessibility (Paneva et al., 2020b; Carter et al., 2013).

Modeling of the underlying interface dynamics is not only useful for the
interaction design and testing, but also for optimizing the performance of the
interface itself. We demonstrated this on an acoustic levitation interface
that uses the persistence of vision effect to render smooth levitated graphics
in real time, by rapidly moving a levitated particle along a periodic path.
OptiTrap (Paneva et al., 2022) is an automated numerical approach that
computes trap trajectories, i.e., position and timings of the acoustic traps,
to generate physically feasible, nearly time-optimal paths that reveal generic
mid-air shapes on the levitator. To achieve this, we derived a multi-
dimensional model of the acoustic forces around a trap, and formulated and
solved a non-linear path following problem. As a result of this trajectory
optimization, we were able to render bigger and more complex shapes (e.g.,
involving sharp edges and sudden changes in curvature), than previously
possible (see Figure 1(B)).

On the example of acoustic levitation interfaces, we have demonstrated in
practice, that modeling, simulation, and optimization methods can be of great
benefit for the efficient and agile development of innovative user interfaces.

##  3\. Modeling Human Movement During Interaction

In addition to developing novel displays and interaction techniques,
simulation and optimization can be used to model human decision-making
processes during interaction with computers and virtual systems. A graphical
representation of our proposed framework is given in Figure 2.

Using the terminology of an Optimal Control Problem (OCP) (Diedrichsen et al.,
2010), we assume that for a given interaction task, humans aim to find a
sequence of valid controls (e.g., neuromuscular control signals) such that the
resulting movement minimizes a given internalized cost function reflecting
both task-specific goals (e.g., pointing at a specific target) and their
individual preferences (e.g., using the right index finger for pointing). This
is consistent with the general assumption of human rationality, which states
that given a set of options, humans will select the option that provides them
with the greatest benefit (Silver et al., 2021). One challenge is thus to
“encode” a given task instruction, or rather, the internal objectives humans
derive from that, into a mathematically precise cost function to be minimized
(or, alternatively, a reward function to be maximized).

The human biomechanical model, input device, interface dynamics all have to be
taken into account in this optimization. These are included in the system
dynamics, capturing all the constraints that stem from the user model (e.g.,
kinematics and perception), input and output devices (e.g., transfer
functions), and interface dynamics.

Solving the resulting OCP leads to a simulation of the complete human-
computer-loop that allows us to infer information such as movement times,
cursor and joint trajectories, or muscle expenditure. Depending on the
complexity of the system dynamics and cost function, the OCP can be solved
using different methods. Three of the most important approaches to obtain
(approximately) optimal movement trajectories for a given task, technique, and
user group are discussed below.

Figure 2.  Our proposed optimization-based framework of human movement during
interaction. The combination of system dynamics and cost/reward function
results in an optimal control problem that can be solved using established
methods. Figure adapted from (Klar et al., 2022).

###  3.1. Linear-Quadratic Gaussian Regulator

In the case of linear system dynamics and a quadratic cost function, the
Linear-Quadratic Regulator (LQR) yields a unique optimal control policy $\pi$,
mapping an arbitrary state $x$ to the control $u^{\star}=\pi(x)$ that is
optimal to apply when being at state $x$. An extension to stochastic (linear)
system dynamics, where controls and states are perturbed by Gaussian noise at
each simulation step, is called the Linear-Quadratic Gaussian Regulator (LQG).

For both LQR and LQG, the control policy is again linear in the (expected)
state, i.e., $\pi(x)=Lx$ holds for some matrix $L$ that can be computed once
in advance (i.e., during the planning stage) (Todorov et al., 1998). In
particular, the generated solution trajectories are closed-loop, i.e., the
control adapts to perturbances that may occur during execution (e.g., due to
model inaccuracies, system and control noise, or unexpected deviations from
predicted states). Since LQR and LQG allow to solve the OCP analytically, the
optimal control policy can be computed very fast (typically within a few
seconds) once before the movement starts, and optimal closed-loop controls are
available in real-time during the movement.

We have introduced the LQR/LQG framework to the HCI community, and have shown
its applicability to mouse pointing (Fischer et al., 2020, 2022). For the LQG
with control and observation noise, we have shown that effort costs applied on
a finite time horizon, together with terminal distance and stability costs,
are sufficient to generate characteristic cursor trajectories (e.g., bell-
shaped velocity profiles, with speed-accuracy trade-off attributed to signal-
dependent noise (Harris and Wolpert, 1998)). Moreover, reciprocal 1D mouse
pointing trajectories were reproduced significantly better than using the
minimum jerk model (Flash and Hogan, 1985) or pure dynamic models (Müller et
al., 2017), also capturing between-trial variability.

However, the assumption of linear dynamics prevents the LQG controller from
dealing with nonlinearities, as they e.g., arise from the biomechanical
constraints of human movements. To overcome these limitations, we proposed the
use of Model Predictive Control (MPC) (Klar et al., 2022).

###  3.2. Model Predictive Control

MPC is a receding horizon approach that handles nonlinearities, provides
optimality and convergence guarantees in many cases, and is inherently robust
to uncertainties due to its closed-loop nature (Grüne and Pannek, 2017). It
has become a standard control method for (non)linear dynamical systems from
both academic and application perspectives (Qin and Badgwell, 2003). Instead
of solving the original OCP that describes the entire interaction movement, a
sequence of shorter OCPs is solved during motion execution. Thus, the movement
time does not have to be fixed in advance and, in addition, the complexity of
the OCP that we need to solve is reduced in time.

External deviations that might occur during movement execution are taken into
account by the receding horizon control principle. After solving an OCP, only
the first part of the optimal control sequence is applied, the updated state
of the system (which may deviate from the expected state) is observed, and a
new OCP is setup starting from this state. This procedure of alternating
planning and execution steps is continued until the interaction task is
completed.

The proposed MPC framework can be used with analytical (nonlinear) models or
even “black-box” implementations. As an application, we investigated mid-air
pointing using a biomechanical model of the upper extremity implemented in the
fast physics engine MuJoCo (Todorov et al., 2012). This model is based on a
state-of-the-art OpenSim model by Seth et al. (Seth et al., 2018), consists of
a torso, right shoulder, and arm, and has seven independent joints that can be
directly actuated via applied torques. We combined this physical model with a
second-order model for aggregated muscles at the individual joints derived
from van der Helm et al. (Van der Helm and Rozendaal, 2000). In order to
assess how well our simulation reflects human movement, we captured motion
data in a mid-air pointing user study featuring different pointing techniques.
By comparing three cost functions, we found that the combination of distance,
control, and joint acceleration costs best explained the experimentally
observed user behaviour in terms of both end-effector trajectories and joint
angle sequences (Klar et al., 2022). We have also demonstrated the ability to
replicate the behavior of a specific user, and to generate models of new users
by adjusting the model and/or control cost parameters.

###  3.3. Model-Free Reinforcement Learning

As an alternative approach to MPC, we have investigated the ability of model-
free reinforcement learning (RL) to simulate human movement during
interaction. While RL methods have a long tradition in robotics and character
animation, in the last few years they have been increasingly used to predict
human behavior and motion in areas such as neuroscience, digital health, and
sports (Bian et al., 2020; Gottesman et al., 2019; Liu et al., 2022). In
contrast to LQG, and similar to MPC, RL can handle complex, nonlinear system
dynamics. In contrast to MPC, the closed-loop policy learned by policy-
gradient RL methods is not only (approximately) optimal for a single initial
state, but generalizes to arbitrary states explored during training.
Practically, an RL policy thus needs to be “trained” only once (which,
however, might take up to hours or even days) and can then be applied in real-
time during execution (similar to LQG). On the downside, there are much fewer
theoretical optimality and stability guarantees for RL policies than for MPC
and LQG, rendering RL approaches more “experimental”.

Using the same state-of-the-art model of the upper extremity as in Section 3.2
to simulate mid-air pointing movements, we have shown that an RL policy
trained to minimize constant time costs only is able to generate movements
that capture well-established characteristics. In particular, the simulation
trajectories follow Fitts’ Law in the case of aimed reaching, and the
$\nicefrac{{2}}{{3}}$ Power Law for ellipse drawing, while generating bell-
shaped velocity and N-shaped acceleration profiles (Fischer et al., 2021).

Building on these results, we have presented User-in-the-Box (Ikkala et al.,
2022), a modular simulation framework that allows to combine a biomechanical
model with one or multiple sensory input channels, an interaction task
instance, and an RL method used to solve the resulting OCP. In this work we
used a muscle-actuated MuJoCo version of the upper extremity model (with 5
shoulder and arm joints actuated via 26 muscles and tendons, and fixed torso
and wrist), as well as RGB-D based vision, proprioceptive and haptic input
channels, and task-specific reward functions. We simulated four movement-based
interaction tasks of increasing difficulty: mid-air pointing, target tracking,
choice reaction, and remote car control (see Figure 3). Our trained models can
successfully complete the respective interaction tasks, while exhibiting
characteristic movement regularities such as Fitts’ Law, showing that we are
able to simulate interactive motion of real users.

Figure 3. Using policies trained via RL, our simulation is capable of
predicting motion in four interactive tasks with differing perceptual-motor
requirements. Figure reprinted from (Ikkala et al., 2022).

##  4\. Discussion

The approach presented here highlights the benefits that simulation and
optimization can offer to the design, evaluation, and improvement of user
interfaces. However, in order to realize the full potential of model-based
simulation, important aspects need to be discussed.

_Model accuracy._ Although we are able to implement models of increasing
complexity, the question is what aspects qualify for a “good enough” model of
the human and the interface, given a specific interaction task and technique.
Furthermore, what metrics are suitable and accepted by researchers and
pracitioners alike to validate a model?

_Generalizability._ Assuming we have created a “good enough” model for a
specific interaction task and technique and went through the (painful) process
of validating it, how easy is it to apply it to (slightly) different tasks or
account for different user-specific characteristics? Do classes of cost/reward
functions that encapsule a more general setting exist, or are we stuck with
tuning these functions for each task, user model, and interaction technique?

_Deployability._ We see a positive change within the HCI community of making
code publicly available. Building on this, what tools and tutorials
specifically created for interface designers are necessary to wisely augment
the user-centered design process with predictions from model-based simulations
and allow for virtual prototyping? We believe that an ongoing, goal-oriented
dialogue between researchers and practitioners is essential to ensure easy
deployability and leverage the benefits of both real and simulated user data.
Moreover, an interdisciplinary approach would help making virtual prototyping
more accessible to different audiences.

## References

  * (1)
  * Antonino Gomes de Sá and Gabriel Zachmann (1999) Antonino Gomes de Sá and Gabriel Zachmann. 1999.  Virtual reality as a tool for verification of assembly and maintenance processes.  _Computers & Graphics_ 23, 3 (1999), 389 – 403.  https://doi.org/10.1016/S0097-8493(99)00047-3
  * Bachynskyi et al. (2018) Myroslav Bachynskyi, Viktorija Paneva, and Jörg Müller. 2018.  LeviCursor: Dexterous Interaction with a Levitating Object. In _Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces_ (Tokyo, Japan) _(ISS ’18)_. ACM, New York, NY, USA, 253–262.  https://doi.org/10.1145/3279778.3279802
  * Berg and Vance (2017) Leif P. Berg and Judy M. Vance. 2017.  Industry use of virtual reality in product design and manufacturing: a survey.  _Virtual Reality_ 21, 1 (01 Mar 2017), 1–17.  https://doi.org/10.1007/s10055-016-0293-9
  * Bian et al. (2020) Tao Bian, Daniel Wolpert, and Zhong-Ping Jiang. 2020.  Model-Free Robust Optimal Feedback Mechanisms of Biological Motor Control.  _Neural Computation_ 32 (01 2020), 1–34.  https://doi.org/10.1162/neco_a_01260
  * Carter et al. (2013) Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013.  UltraHaptics: Multi-point Mid-air Haptic Feedback for Touch Surfaces. In _Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology_ (St. Andrews, Scotland, United Kingdom) _(UIST ’13)_. ACM, New York, NY, USA, 505–514.  https://doi.org/10.1145/2501988.2502018
  * Diedrichsen et al. (2010) Jörn Diedrichsen, Reza Shadmehr, and Richard B Ivry. 2010\.  The coordination of movement: optimal feedback control and beyond.  _Trends in Cognitive Sciences_ 14, 1 (2010), 31–39.  https://doi.org/10.1016/j.tics.2009.11.004
  * Fischer et al. (2021) Florian Fischer, Miroslav Bachinski, Markus Klar, Arthur Fleig, and Jörg Müller. 2021\.  Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity.  _Scientific Reports_ 11, 1 (2021), 1–15.
  * Fischer et al. (2020) Florian Fischer, Arthur Fleig, Markus Klar, Lars Gruene, and Joerg Mueller. 2020.  An Optimal Control Model of Mouse Pointing Using the LQR.  arXiv:2002.11596 [cs.HC]
  * Fischer et al. (2022) Florian Fischer, Arthur Fleig, Markus Klar, and Jörg Müller. 2022.  Optimal Feedback Control for Modeling Human-Computer Interaction.  _ACM Trans. Comput.-Hum. Interact._ 29, 6, Article 51 (april 2022), 70 pages.  https://doi.org/10.1145/3524122 In Press.
  * Flash and Hogan (1985) Tamar Flash and Neville Hogan. 1985.  The coordination of arm movements: an experimentally confirmed mathematical model.  _Journal of neuroscience_ 5, 7 (1985), 1688–1703.
  * Gottesman et al. (2019) Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. 2019\.  Guidelines for reinforcement learning in healthcare.  _Nature medicine_ 25, 1 (2019), 16–18.
  * Grüne and Pannek (2017) Lars Grüne and Jürgen Pannek. 2017.  _Nonlinear Model Predictive Control_.  Springer International Publishing, Cham, 45–69.  https://doi.org/10.1007/978-3-319-46024-6_3
  * Harris and Wolpert (1998) Christopher M. Harris and Daniel M. Wolpert. 1998.  Signal-dependent noise determines motor planning.  _Nature_ 394, 6695 (09 1998), 780–784.  https://doi.org/10.1038/29528
  * Ikkala et al. (2022) Aleksi Ikkala, Florian Fischer, Markus Klar, Miroslav Bachinski, Arthur Fleig, Andrew Howes, Perttu Hämäläinen, Jörg Müller, Roderick Murray-Smith, and Antti Oulasvirta. 2022.  Breathing Life Into Biomechanical User Models. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 90, 14 pages.  https://doi.org/10.1145/3526113.3545689
  * Ishii et al. (2012) Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012.  Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials.  _interactions_ 19, 1 (Jan. 2012), 38–51.  https://doi.org/10.1145/2065327.2065337
  * Klar et al. (2022) Markus Klar, Florian Fischer, Arthur Fleig, Miroslav Bachinski, and Jörg Müller. 2022\.  Simulating Interaction Movements via Model Predictive Control.  _ACM Trans. Comput.-Hum. Interact._ (dec 2022).  https://doi.org/10.1145/3577016 Just Accepted.
  * Liu et al. (2022) Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojciech M. Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. 2022\.  From motor control to team play in simulated humanoid football.  _Science Robotics_ 7, 69 (Aug 2022).  https://doi.org/10.1126/scirobotics.abo0235
  * Müller et al. (2017) Jörg Müller, Antti Oulasvirta, and Roderick Murray-Smith. 2017\.  Control Theoretic Models of Pointing.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 24, 4, Article 27 (Aug. 2017), 36 pages.  https://doi.org/10.1145/3121431
  * Murray-Smith et al. (2022) Roderick Murray-Smith, Antti Oulasvirta, Andrew Howes, Jörg Müller, Aleksi Ikkala, Miroslav Bachinski, Arthur Fleig, Florian Fischer, and Markus Klar. 2022\.  What Simulation Can Do for HCI Research.  _Interactions_ 29, 6 (nov 2022), 48–53.  https://doi.org/10.1145/3564038
  * Paneva et al. (2020a) Viktorija Paneva, Myroslav Bachynskyi, and Jörg Müller. 2020a.  Levitation Simulator: Prototyping for Ultrasonic Levitation Interfaces in Virtual Reality. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM, New York, NY, USA.  https://doi.org/10.1145/3313831.3376409
  * Paneva et al. (2022) Viktorija Paneva, Arthur Fleig, Diego Martínez Plasencia, Timm Faulwasser, and Jörg Müller. 2022.  OptiTrap: Optimal Trap Trajectories for Acoustic Levitation Displays.  _ACM Trans. Graph._ 41, 5, Article 173 (oct 2022), 14 pages.  https://doi.org/10.1145/3517746
  * Paneva et al. (2020b) Viktorija Paneva, Sofia Seinfeld, Michael Kraiczi, and Jörg Müller. 2020b.  _HaptiRead: Reading Braille as Mid-Air Haptic Information_.  Association for Computing Machinery, New York, NY, USA, 13–20.  https://doi.org/10.1145/3357236.3395515
  * Qin and Badgwell (2003) S Joe Qin and Thomas A Badgwell. 2003.  A survey of industrial model predictive control technology.  _Control engineering practice_ 11, 7 (2003), 733–764.
  * Seth et al. (2018) Ajay Seth, Jennifer L Hicks, Thomas K Uchida, Ayman Habib, Christopher L Dembia, James J Dunne, Carmichael F Ong, Matthew S. DeMers, Apoorva Rajagopal, Matthew Millard, et al. 2018\.  OpenSim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement.  _PLoS computational biology_ 14, 7 (2018), e1006223.
  * Silver et al. (2021) David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. 2021.  Reward is enough.  _Artificial Intelligence_ 299 (2021), 103535.  https://doi.org/10.1016/j.artint.2021.103535
  * Sutherland (1965) Ivan E. Sutherland. 1965\.  The Ultimate Display.  _Multimedia: From Wagner to Virtual Reality_ (1965), 506–508.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  Mujoco: A physics engine for model-based control. In _2012 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 5026–5033.
  * Todorov et al. (1998) Emanuel Todorov, Michael, and I. Jordan. 1998.  Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements.  _Journal of Neurophysiology_ 80 (1998), 696–714.
  * Van der Helm and Rozendaal (2000) Frans CT Van der Helm and Leonard A Rozendaal. 2000.  Musculoskeletal systems with intrinsic and proprioceptive feedback.  In _Biomechanics and neural control of posture and movement_. Springer, 164–174.


# Towards Versatile Graph Learning Approach: from the Perspective of Large
Language Models

Lanning Wei1,2 Jun Gao 3 Huan Zhao4 Quanming Yao5 1Institute of Computing
Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Harbin Institute of Technology (Shenzhen) 44Paradigm. Inc
5Department of Electronic Engineering, Tsinghua University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Graph-structured data are the commonly used and have wide application
scenarios in the real world. For these diverse applications, the vast variety
of learning tasks, graph domains, and complex graph learning procedures
present challenges for human experts when designing versatile graph learning
approaches. Facing these challenges, large language models (LLMs) offer a
potential solution due to the extensive knowledge and the human-like
intelligence. This paper proposes a novel conceptual prototype for designing
versatile graph learning methods with LLMs, with a particular focus on the
“where” and “how” perspectives. From the “where” perspective, we summarize
four key graph learning procedures, including task definition, graph data
feature engineering, model selection and optimization, deployment and serving.
We then explore the application scenarios of LLMs in these procedures across a
wider spectrum. In the “how” perspective, we align the abilities of LLMs with
the requirements of each procedure. Finally, we point out the promising
directions that could better leverage the strength of LLMs towards versatile
graph learning methods. The related source can be found at:
https://github.com/wei-ln/versatile-graph-learning-approaches. 111Work are
done when Lanning and Jun are interns in 4Paradigm.

##  1 Introduction

Graph-structured data are commonly used and applied in real-world
applications, e.g., social networks Hamilton et al. (2017), chemistry and
biomedical molecules Gilmer et al. (2017). Existing methods have achieved
great success in understanding and solving the single task, e.g., formulating
the molecular property prediction as graph classification task Gilmer et al.
(2017), conducting graph sampling when facing large-scale graphs Hamilton et
al. (2017), designing expressive graph learning algorithms (GLAs) to extract
graph structural information Min et al. (2022); Wu et al. (2020), and
selecting appropriate hyper-parameters in evaluation stage Zhang et al.
(2022). Despite their success on one single task, there are still challenges
in graph learning that are yet to be addressed. Firstly, the graph domains and
the learning problems are largely different in real world. Subsequently, human
experts need to configure the complex procedures You et al. (2020)
individually, which brings about substantial requirements for professionalism
and domain knowledge on graph-structured data. All these aspects lead to the
challenge for human experts in handling the graph learning problems that
suited for different tasks, graphs in diverse domains, and can efficiently
conduct the pipeline with as few as possible assistance from humans, i.e.,
towards versatile graph learning methods.

Large language models (LLMs) are treated as the key point in designing
versatile graph learning algorithms due to the maintained knowledge and human-
like intelligence. LLMs, refer to the large-sized (billions-level) pre-trained
language models (PLM) in general, have undergone a rapid succession of
breakthroughs in recent years Zhao et al. (2023b). By pre-training with
comprehensive text data and prompt-tuning on downstream tasks, LLMs pose a
vast store of knowledge and show the ability in achieving the human-level
decision-making ability Zhao et al. (2023b); Wang et al. (2023c). For
instance, LLMs can reason on mathematical problems with chain-of-thoughts Wei
et al. (2022), span different tasks and achieve superior performance Mialon et
al. (2023), solve computer vision tasks from task planning to algorithm
selection and execution like human experts Shen et al. (2023). Based on these
abilities, LLMs show promising potential to serve as Artificial General
Intelligence (AGI) Ge et al. (2023) and general research assistant Huang et
al. (2023b). Therefore, it is natural to directly use or draw on the
successful experience of LLMs when constructing versatile graph learning
methods.

Figure 1: The conceptual prototype of versatile graph learning methods joint
with LLMs. LLMs can be used in sequential graph learning procedures in these
columns, with increased requirements for LLMs in different rows for each
procedure. The rows can be further developed along with the exploration of
different abilities of LLMs.

The numerous advantages of LLMs have triggered an increasing interest in using
them for graph learning problems. Li et al. (2023); Guo et al. (2023); Shen et
al. (2023); Zhang et al. (2023a). For instance, through harnessing the
preserved knowledge , different graph learning tasks across various domains
can be unified into textual descriptions, which subsequently employ LLMs, such
as ChatGPT, to acquire predictions Liu et al. (2023a); Fatemi et al. (2023).
LLMs enhance the textural-attributed graphs (TAGs) which subsequently employ
LLMs, such as ChatGPT, to acquire predictions He et al. (2023); Guo et al.
(2023); Chen et al. (2023). These methods have catalyzed impressive progress
in designing effective techniques for different graph learning problems.

With such a development of graph learning methods joint with LLMs, it is
necessary to explore the systematical way towards versatile graph learning
methods using LLMs. In this paper, we conduct a comprehensive review on
existing methods and develop conceptual prototype by aligning the advantages
and strong abilities of LLMs with the different requirements maintained in
versatile graph learning methods. More specific, the key insights are the
identification of:

Where can be used. Different requirements for LLMs are arise when human
experts manage the graph learning pipeline. Towards versatile graph learning
approach, it is a vital to examine the feasibility of LLMs in diverse graph
learning procedures to fully unleash the versatile abilities of LLMs.
Consequently, we summarize four key procedures based on the requirements
according to a standard machine learning pipeline, i.e., task definition,
feature engineering, model selection and optimization, deployment and serving,
as shown in Fig. 1. Based on these procedures, we can explore a wider spectrum
of application scenarios when employing LLMs, thereby enhancing the
versatility of existing graph learning methods.

How to use. It is important to align the capabilities of LLMs with the
requirements of the four procedures. We summarize three levels of increased
requirements for LLMs, depicted as three rows in Fig. 1, and then provide a
detailed analysis of existing methods. The human-designed methods, represented
in the first row, place no requirements on LLMs, The methods situated in the
second row emphasize the fundamental ability of LLMs to understand, encode,
and reason out the natural language queries. The methods in the third row
require a higher level of human-level decision-making ability and the rich
domain-specific knowledge from LLMs, similar to human experts. The varying
capabilities of LLMs encourage us to investigate their potential applications
in each procedure, prompting the development of versatile graph learning
methods.

It is significant to highlight the proposed conceptual prototype in Fig. 1
when designing versatile graph learning methods based on LLMs. The “Where”
perspective is spanned along with four key procedures in graph learning
pipeline, and the “How” perspective is organized based on the abilities of
LLMs in different levels. Then, the graph learning methods joint with LLMs can
be designed by choosing the desired ability in each procedure, which is
detailed shown in Table 1. Based on these two perspectives, we provide
comprehensive overview of the graph learning methods jointed with LLMs, and
emphasize the potential for a broad exploration spectrum and the usability of
LLMs’ various abilities towards versatile graph learning approaches. Finally,
we suggest promising future directions on the basis of under-explored ability
of LLMs and the property of graph-structured data, i.e., the effectiveness in
understanding the graph structure, large graph foundation model, and universal
graph learning agents.

##  2 Overview

###  2.1 Machine Learning on Graphs

The graph-structured data is presented as
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ and $\mathcal{E}$
are nodes and edges.
$\textbf{A}\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}$ is the adjacency
matrix of this graph, and $\textbf{H}\in\mathbb{R}^{|\mathcal{V}|\times d}$ is
the feature matrix. The graph learning procedures generate the results
$\mathcal{R}$ of graphs learning problem $\mathcal{P}$ on graph $\mathcal{G}$.

Considering the considerable efforts expended by human experts to solve these
diverse graphs and applications, we summarize the key points in graph learning
pipeline as shown in Fig. 1.

  * •

Task definition. Experts formulate the problems $\mathcal{P}$ into specific
graph learning tasks $\mathcal{T}$, e.g., the predictions on nodes, edges,
sub-graphs or graphs.

  * •

Feature engineering. Experts select, combine and transform features towards
better-performed performance by themselves Liu et al. (2022). For instance,
different frequency bands are designed to eliminate certain graph Fourier
modes. Besides, it can be achieved automatically with AutoML Zheng et al.
(2023), which involves selecting appropriate feature sets and combining them
in a data-driven manner.

  * •

Model selection and optimization. Experts select and train architectures
$\mathcal{A}(\textbf{W}^{*})$, e.g., Graph Neural Networks Wu et al. (2020) or
Graph Transformers Min et al. (2022), to address the graph learning problem.
Here, $\mathcal{A}$ represents the selected architecture with the trained
parameter $\textbf{W}^{*}$. In addition to designing and tuning architectures
with human experts, other strategies including graph neural architecture
search (NAS) and hyper-parameter tuning methods are explored, which aim to
automated conducting this procedure and obtaining efficient and effective
solutions Zhang et al. (2021, 2022).

  * •

Deployment and serving. Experts deploy models and then generate the results
$\mathcal{R}$ to serve for users and graph learning problem $\mathcal{P}$,
which are achieved by experts themselves.

###  2.2 Conceptual Prototype

In this paper, we propose a conceptual prototype when designing versatile
graph learning methods jointed with LLMs, following the development of graph
learning procedures and the different abilities of LLMs as illustrated in Fig.
1.

“Where can be used” We analyze and summarize the common requirements in each
key graph learning procedure, which are represented as four columns in Fig. 1.
This analysis led us to explore the potential application scenarios of LLMs in
each procedure, which could broaden the spectrum of potential applications of
LLMs in various graph learning methods. The results show that existing methods
are extensively used in the feature engineering and model selection
procedures, attributing to the need for enhancing features and representations
of textual data.

“How to use” Considering the different requirements in each procedure, we
explore the feasibility of using LLMs according to a hierarchy of requirements
for LLMs’ abilities. This hierarchy is visualized as different rows in Fig. 1,
i.e., no requirements for the use of LLMs (with human experts merely);
requiring the fundamental ability in understanding and reasoning the natural
languages; and requisition of advanced human-like intelligence and domain-
specific knowledge as human experts. The majority methods, represented in the
second row, employ LLMs to assist the human experts due to their proficiency
in question answer, understanding and encoding the natural language queries.
Conversely, a minority methods, as illustrated in the final row, exploit the
potential of LLMs in planning, decomposing and completing tasks with
themselves due to the advanced ability in achieving human-like intelligence.
It is necessary to note that the potential abilities of LLMs are still being
explored and the conceptual prototype represented in the figure may be
extended to include more rows as the development of LLMs progresses.

###  2.3 Comparisons with Contemporaneous Surveys

There exists several contemporaneous surveys that explore how to use LLMs in
feature engineering and model selection procedures. To be specific, Li et al.
(2023); Jin et al. (2023a) categorized the methods based on the role that LLM
played, and Zhang et al. (2023b); Liu et al. (2023b) categorized the methods
from the perspective of graph foundation models, ranging from backbone
architecture construction and pre-training to adaption. Compared with these
methods, this paper considers the full graph learning procedures and the
versatile abilities of LLMs. Firstly, this paper takes into account the
complete graph learning processes, e.g., the task definition, deployment and
serving procedures, that provide a wider spectrum of application scenarios for
LLMs. Besides, this paper further considers the ability of LLMs in achieving
human-like intelligence. It indicates that LLMs can serve as core
orchestrators of autonomous agents, representing a new trend of LLMs Wang et
al. (2023c). This suggestion aligns with recent global research topics He et
al. (2023).

##  3 Graph Learning with LLMs

In this section, we overview the existing methods following the sequential
graph learning pipeline, and then the methods used in each procedure are
introduced following the involvement of LLMs.

Table 1: A summary of graph learning methods jointed with LLMs under the proposed conceptual prototype. represents cells that were created by human experts and do not have any relation to the following procedures that use LLMs.  Methods |  Task Definition | Feature Engineering | Model Selection and Optimization |  Deployment & Serving
---|---|---|---|---
Taxonomy | Description | LLMs | Taxonomy | Description | LLMs
Graphtext Zhao et al. (2023a) |  - | $\mathcal{G}^{T}$ |  The features and node labels of k- hop subgraphs |  - |  Predictor |  predicting labels of centric node |  ChatGPT / GPT-4 |  Using predictions as response
NLGraph Wang et al. (2023b) |  Formulating graph reasoning tasks with human experts | $\mathcal{G}^{T}$ |  Describing the graphs, nodes, edges and learning tasks. |  - |  Predictor |  Predicting the statistics of graphs. |  GPT-3.5 / GPT4 |  Using predictions as response
GPT4Graph Guo et al. (2023) |  - | $\mathcal{G}^{T}$ |  self-prompting to describe the graphs. |  InstructGPT- 3 |  Predictor |  Predicting the statistics and Semantic labels of graphs. |  InstructGPT-3 |  Using predictions as response
TLG Fatemi et al. (2023) |  Formulating graph reasoning tasks with human experts | $\mathcal{G}^{T}$ |  Describing the graphs, nodes, edges and learning tasks. |  - |  Predictor |  Predicting the statistics of graphs. |  PaLM 62B |  Using predictions as response
LLM4Mol Qian et al. (2023) |  - | $\mathcal{G}^{T}$ |  Combining SMILES text and explanations of functional groups, chemical properties, and potential pharmaceutical applications |  ChatGPT |  Predictor |  Fine-tune a pre-trained language model on various molecule related downstream tasks |  RoBERTa |  -
OFA Liu et al. (2023a) |  - | $\mathcal{G}^{ST}$ |  Explanations on $\mathcal{V}^{T}$ and $\mathcal{E}^{T}$ |  ChatGPT |  GLA-centric |  co-trained LLMs and R-GCNs |  Sentence-bert |  -
Chen et al. (2023) |  - | $\mathcal{G}^{ST}$ |  Generating and encoding textual information of nodes |  Open-source LLMs Deberta / LLaMA |  GLA-centric / Alignment-based / Predictor |  Combining GCN/GAT/RevGAT with Sentence-BERT/ Deberta; using Deberta as predictor |  Combining GCN/GAT/RevGAT with Sentence-BERT/ Deberta; using Deberta as predictor |  -
GRAD Mavromatis et al. (2023) |  - | $\mathcal{G}^{ST}$ |  Using the raw texts in graphs |  - |  Alignment-based |  co-trained GNN teacher and LLM student |  BERT |  -
GraphGPT Tang et al. (2023) |  - | $\mathcal{G}^{ST}$ |  Using the raw texts in graphs |  - |  Alignment-based |  Applying contrastive alignment objective on the predictions of pre-trained GNNs and LLMs |  Bert |  -
GraphPrompt Liu et al. (2023c) |  - | $\mathcal{G}^{S}$ |  Sampling and unifying data and tasks |  - |  GFMs |  Pre-train and prompt on downstream tasks |  - |  -
All in One Sun et al. (2022) |  Unified with graph-level tasks | $\mathcal{G}^{S}$ |  Constructing prompt graphs |  - |  GFMs |  Pre-trained with multi-task meta-earning and prompted on downstream tasks |  - |  -
Instruction2GL Wei et al. (2023) |  Formulating tasks with ChatGPT-3.5 | $\mathcal{G}^{S}$ |  Slecting engineering strategy following user instructions |  ChatGPT-3.5 |  LLM-assisted |  Configuring AutoML and |  ChatGPT-3.5 |  Configuring codes and generating response with ChatGPT and the agents
GPT4GNAS Wang et al. (2023a) |  - | $\mathcal{G}^{S}$ |  - |  - |  LLM-assisted |  Selecting and evaluating GNNs on AutoML with LLMs. |  GPT-4 |  -

###  3.1 Task Definition

Task definition transforms the learning problems in real-world applications
into possible solvable tasks with current machine learning techniques. It is
the first step to addressing the machine learning problems and can only be
effectively done through human expertise. LLMs have advantages in
understanding the different applications in real-world, and they have the
ability to assist the complex task planning, decomposition and completion
Mialon et al. (2023); Huang et al. (2023b).

Efforts have been made by combing LLMs with other modalities and graph-
structured data. In computer vision, HuggingGPT Shen et al. (2023) used LLMs
to understand the complex computer vision tasks and complete them sequentially
with the help of LLMs, which can be achieved due to the human-like decision-
making capability of LLMs. However, graph-structured data, which stems from
different domains, presents a unique challenge. The learning tasks related to
graph-structured data are in varied formats, necessitating diverse
configurations for individual tasks Liu et al. (2023b). In the literature,
NLGraph Wang et al. (2023b) and GPT4Graph Guo et al. (2023) evaluated LLMs in
understanding and formulating the graph reasoning tasks over different graph
datasets. Instruction2GL Wei et al. (2023) proposed a LLM-based planning agent
and then mapped the users’ instructions into different graph learning tasks.
Given the experiments of these methods over different tasks, the feasibility
of LLMs in planning and understanding graph learning tasks is obvious.

In conclusion, defining the learning tasks is the first step when facing the
real-world applications on graph-structured data. It can be solved by LLMs
with the ability in understanding the natural languages and the maintained
domain knowledge over graphs. With these abilities, LLMs bring more convenient
interaction manner with users and could alleviate the stress over human
experts. It indicates the potential of LLMs to define, planning, decomposing
the complex graph learning tasks.

###  3.2 Feature Engineering

Graph feature engineering is responsible for combing or transforming data to
more effective features which are built based on the original graph-structured
data. LLMs have the ability to generate descriptions of the graph using
natural language, and can also encode text into an embedding space, serving as
a useful tool in this regard. By introducing this new modality to graphs, an
abundance of textual information can be incorporated, which may lead to a
significant improvement in the effectiveness of the algorithm.

Figure 2: Illustrations of graph data feature engineering strategies that
jointed with LLMs.

####  3.2.1 Textual Feature Construction

As shown in Fig. 2, two types of engineering strategies are designed to
combine textual information based on the graph-structured data. In this paper,
for the clear justification, we denote the structured graph as
$\mathcal{G}^{S}=(\mathcal{V},\mathcal{E})$, and the textual graph
$\mathcal{G}^{T}$ as a word sequence $\\{w_{1},\cdots\\}$ which describes the
nodes and edges in graph. The structured-textual graphs $\mathcal{G}^{ST}$ is
defined as the combination of structures $\mathcal{G}^{S}$ and textual
description $\\{w_{1},\cdots\\}$ of nodes $\mathcal{V}^{T}$, edges
$\mathcal{E}^{T}$, or the graph $\mathcal{G}^{T}$.

Firstly, for structured-textual graph $\mathcal{G}^{ST}$, the textual
descriptions and explanations of nodes, edges or sub-graphs can be generated
by LLMs and used in the following procedures to improve the effectiveness Chen
et al. (2023). To be specific, Chen et al. Chen et al. (2023) highlighted the
text attributes for nodes, GraphGPT Tang et al. (2023) described the graph
structures with natural languages, TAPE He et al. (2023) used GPT-3.5 to
generate the node descriptions, prediction as well as the explanations to
enrich the text information; OFA Liu et al. (2023a) further provided the
descriptions of edges and learning tasks. For textural graph
$\mathcal{G}^{T}$, existing methods use the textual descriptions to replace
the general structures. The representative method GraphText Zhao et al.
(2023a) sampled k-hop ego-subgraphs and then describe the labels and features
of nodes in each hop. TLG Fatemi et al. (2023) further provided 11 types of
text description functions. LLM4Mol Qian et al. (2023) describe the molecules
with SMILES text, and the explanations of functional groups, chemical
properties, and potential pharmaceutical applications generated by ChatGPT.

In conclusion, existing methods enhance the textual information with the help
of LLMs, the enhancement include but not limited to the descriptions on nodes
and connection relationship and explanations on the rich domain information
maintained in nodes, edges or graphs. It is achieved with the abilities of
answering the questions of users based on the vast knowledge maintained in
LLMs.

####  3.2.2 Textual Feature Encoding

Existing methods employ open-source LMs or LLMs to obtaining the text
embedding with vanilla Transformer Vaswani et al. (2017), Sentence-BERT
Reimers and Gurevych (2019), RoBERTa Liu et al. (2019) or PaLM Chowdhery et
al. (2023). These models can be frozen to obtaining the embedding directly, or
co-trained with the graph learning algorithms which will be introduced in the
following.

####  3.2.3 Summary and Discussion

In conclusion, existing methods explore the feasibility of LLMs in generating
novel text modality for graphs. As illustrated in Fig. 1, by combing the
expertise of human experts with the versatile explore of LLMs, these methods
construct and encode the textural information, on top of which the performance
improvements of graph learning methods can be expected. Apart from these
methods, AutoKG Chen and Bertozzi (2023) adopted LLMs to construct the
entities of knowledge graphs based on natural languages, thereby empowering
decisions akin to human expertise. Despite the success of these methods, there
is a notable absence of systematic comparisons and discussions about effective
construction of textual components in the graph, especially for text-sparse
graphs such as traffic and power transmission graphs Jin et al. (2023a), graph
with signal or image features that are challenging to describe using natural
languages.

###  3.3 Model Selection and Optimization

The core objective of graph learning procedures is selecting and training
models that will be used for the downstream tasks, i.e., selecting and
optimizing $\mathcal{A}$ based on the given data $\mathcal{G}$ and tasks
$\mathcal{T}$. LLMs are indispensable in encoding the textual information and
have advantages in making decisions due to the maintained knowledge. In the
following, we will introduce how LLMs are utilized given different types of
data mentioned in Section 3.2.

####  3.3.1 LLMs as Predictor for Textual Graph $\mathcal{G}^{T}$

Existing methods employ open or closed source LLMs, denoted as
$\mathcal{A}_{LLMs}$, to generate the prediction results $\mathcal{R}$ of
graph learning problem $\mathcal{P}$. GraphText Zhao et al. (2023a), a
representative method in the field, used closed-source ChatGPT to predict the
node labels with k-hop sub-graph text in a few-shot manner. TLG Fatemi et al.
(2023) employed open-source PaLM 62B Chowdhery et al. (2023) to predict
different graph properties via the text descriptions, which could be conducted
in zero-shot manner.

In conclusion, when dealing with the textual graph, the utilization of LLMs is
indispensable. These methods pose superior capabilities in generating
explainable predictions compared with general graph learning methods such as
GNNs and graph transformer Zhao et al. (2023a). Furthermore, they could used
in zero-shot manner, which could adapt to different tasks and data easily due
to the strong ability in understanding natural language queries. Nonetheless,
the prompt design and the usage of examples still have large influence on the
model performance Fatemi et al. (2023). Moreover, the robustness, reliability,
and data leakage problems of LLMs are the potential issues that need to be
solved Huang et al. (2023a).

####  3.3.2 LLMs as Co-operator for Structured-Textual Graphs
$\mathcal{G}^{ST}$

With textual and structured graph, LLMs $\mathcal{A}_{LLMs}$ and graph
learning algorithms $\mathcal{A}_{GLAs}$, like GNNs and graph Transformers
Kipf and Welling (2016); Min et al. (2022); Wu et al. (2020); Wang et al.
(2023d), are co-operated towards accurate predictions. Existing methods are
organized following three categories as illustrated in Fig. 3. Firstly, the
GLA-centric methods first use LLMs to encode the textual information, and then
employ graph learning algorithms to obtain predictions based on these enriched
features. By encoding the textual information of nodes and edges with
Sentence-BERT firstly, OFA Liu et al. (2023a) adopted R-GCN Schlichtkrull et
al. (2018) to aggregate the features given different types of edges. TAPE He
et al. (2023) adopted Deberta to encode the textual descriptions and
explanations , and then used RevGAT to learning graph representations.
Secondly, the alignment-based methods are co-operated by aligning the textual
and structured embedding space. For instance, G2P2 Wen and Fang (2023) adopted
three levels contrastive learning when pre-trained the GCN Kipf and Welling
(2016) and Transformer Vaswani et al. (2017). Patton Jin et al. (2023b)
trained GNNs and BERT in a nested manner, and GRAD Mavromatis et al. (2023)
distilled the structured information from GNNs to enhance BERT towards better
understanding on graphs. Finally, the LLM-centric methods use graph learning
algorithms to extract the graph structure information, and then mapped into
the text space of LLMs. These methods could leverage the advantages of graph
learning algorithms in understanding the inherent structural characteristics
and LLMs in feature transformation Li et al. (2023). The representative
GraphLLM Chai et al. (2023) first learned graph representations with Graph
Transformer, and then mapped into prefix and tuned with LLaMA.

Figure 3: Illustrations of LLM-related graph learning algorithms on
structured-textual graphs.

In conclusion, LLMs and graph learning algorithms are co-operated when facing
the structured-textual graphs. Three categories of combinations manners are
explored in the literature, i.e., GLA-centric, alignment-based and LLM-centric
methods, as shown in Fig. 3. In these methods, LLMs are treated as an pure
large models or as language models to enhancing the information extraction,
which can be frozen or trained with GNNs. In the future work, the improvements
can be developed towards larger graph learning models given the model size of
LLMs.

####  3.3.3 LLMs as Advisor for Structured graph $\mathcal{G}^{S}$

Based on the structured graphs without textual information, the LLM-assisted
graph learning methods, i.e., LLMs serve as the graph learning research assist
in selecting models, are proposed considering its strong ability in reasoning
and human-like decision making ability. GPT4GNAS Wang et al. (2023a) used
GPT-4 to guide the designing of GNNs with AutoML. It used a fixed search space
and then let GPT-4 select and revise GNNs from this space towards better-
performed architectures. Instruction2GL Wei et al. (2023) employs GPT-3.5 to
configure the search space and search algorithms to automated conducting graph
learning procedures based on the AutoML technique.

Figure 4: Illustrations of LLM-empowered graph learning algorithms on graph-
structured data.

Moreover, motivated by the advantages of LLMs in handling different tasks by
training on large-scale data from different tasks, the development of graph
foundation models (GFMs) are popular in recent years. To be specific, the
constructions of foundation models following the topics of architecture
backbone, pre-training and adaption to downstream tasks Liu et al. (2023b);
Zhang et al. (2023b); Jin et al. (2023a). Motivated by the “pre-train and
prompt” large model train paradigm, existing methods construct pre-training
data based on different tasks and then propose graph prompt strategy towards
versatile graph learning models. The representative methods GraphPrompt Liu et
al. (2023c) and GPPT Sun et al. (2022) unified the graph learning tasks on
node, edge and graph levels with link prediction task and then pre-train the
GNNs on different tasks. In the prompt tuning stage, GraphPrompt learned the
prompt vector and GPPT designed prompt function on tasks and structures when
facing the different downstream tasks.

In conclusion, when facing the structured graph, LLMs could serve as advisor
to support the graph learning as shown in Fig. 4, i.e., serve as an research
assistant with the human-like decision-making capabilities, and motivate the
construction and training of graph foundation models. Compared with the
latter, the LLM-assisted methods necessitate a higher level of LLM proficiency
in human-like intelligence. All these methods underscore the potential of LLMs
to accomplish the complex graph learning tasks, marking a new trend and a
critical technique in the development of user-friendly, accessible graph
learning methods Wang et al. (2023c); Wei et al. (2023).

####  3.3.4 Summary and Discussions

In graph learning model selection and optimization procedures, LLMs play
different roles depending on the type of graph data encountered. They are
treated as the predictor when facing the textual graph, and serve as the co-
operator towards better combination from two modalities in structured-textual
graphs. When facing these graph graphs, LLMs are explored due to its ability
in understanding the text modality and transforming features. When facing the
structured graph, LLMs are treated as the advisor to assist the graph learning
by leveraging their human-level intelligence and utilizing innovative model
training strategies. Expected for the benefits by incorporating new
modalities, LLMs have difficulties in understanding the graph structures
Fatemi et al. (2023); Guo et al. (2023). Such deficiencies accumulate over
time, contributing to weak robustness and a lack of theoretical justifications
in graph learning tasks. A crucial challenge that needs to be addressed
involves harnessing the power of LLMs by integrating them with existing graph
learning methods, which have a superior understanding of graph structural
information.

###  3.4 Deployment and Serving

Given the user request, deploying solutions and then generating response for
users are the final step of the graph learning pipeline, and they are tightly
connected with users, which indicating that more human engagements are
required to improve the usability and accessibility Yao et al. (2018). LLMs
could generate codes based on the users requirements and call proper APIs
related to the data Wang et al. (2023c); Huang et al. (2023b). Following this
paradigm, LLMs are used as general AI research assistant in writing files,
executing code and inspecting outputs Huang et al. (2023b); Mialon et al.
(2023); Zhang et al. (2023a), displaying high usability and interpretability
in conducting the graph learning procedures. On graph-structured data, the
deployment and serving are related to the tools and packages they used. Graph
Tool-former Zhang (2023) used external API tools to load graph data, and
generate prediction as well as explanations on graphs under different tasks.
Instruction2GL Wei et al. (2023) learned to call APIs in Pytorch Geometric
package to load data, execute graph learning code, and generate the response
based on the users’ requests and code execution logs. As a conclusion, LLMs
serve as development assistants, with human-like ability to use various tools.
They also act as a new interface due to their ability to generate natural
language responses. In the future, LLMs could play even more vital role after
familiar with the external graph developments tools. This advancement will
make graph-based machine learning more accessible and easy to use, thereby
lowering the barriers to entry.

###  3.5 Summary and Discussions

In this section, we have provided detailed analysis on how LLMs are involved
in different graph learning procedures. Given the versatile abilities of LLMs,
it is feasible to integrate them towards versatile graph learning algorithms
as shown in Fig. 1 and Table 1. Considering the rapid progression and yet
under-explored potential of LLMs, significant research opportunities still
exist to explore how to effectively integrate LLMs into graph learning
problems.

##  4 Future Directions

The developments of LLMs and the potential applications are fast growing in
recent years. Beyond existing methods, there are still several promising
directions that could be explored on the graph-structured data.

The ability quantification of LLMs in understanding graphs-structured data.
There is an absence of systematic evaluations regarding the ability of LLMs to
comprehend graph-structured data. To be specific, LLMs are trained with
natural languages, while the graph structure, the key in graph learning, are
represented as the adjacency matrix in general and are new modality for LLMs.
Existing methods, e.g., GPT4Graph Guo et al. (2023) and TLG Fatemi et al.
(2023), attempt to describe graph structures with natural languages and then
evaluate the effectiveness on graph structure reasoning tasks, e.g.,
recognizing the node degree, graph diameter or clustering coefficient.
However, these measures are not adequate to entirely capture the capacity of
LLMs to understand graph structures. Firstly, the descriptions can greatly
impact performance, which may result in insufficient robustness and
reliability. Moreover, Weisfeiler-Lehman test, which is seminal in assessing
the comprehension of graph structures, should be considered to theoretically
validate the effectiveness. Consequently, both from the theoretical and
empirical points of view, there is a need for a benchmark and detailed
analysis to evaluate the competence of LLMs in understanding graph structures.

Large graph foundation models. Excepted for employing LLMs in solving graph
learning tasks, designing graph foundation models and scaling up the model
parameters is a promising direction in handling the different tasks and data
from different domains. To be specific, existing methods have explored the
GNNs and Graph Transformer methods on the given single task Wu et al. (2020);
Min et al. (2022). Compared with foundation models in natural language,
challenges in designing large graph models are the scarcity of extensive pre-
training data and the lack of large-scale backbone architectures Liu et al.
(2023b); Zhang et al. (2023b). Furthermore, designing the graph foundation
models for specific graph domains is more effective and feasible considering
the application scenarios Zhang et al. (2023b), e.g., recommendation systems,
molecules and finance.

Universal Graph learning agents. LLM-based autonomous agents are the prominent
research topic in both academic and industry due to its potential emulating
human-like decision-making processes Wang et al. (2023c). In graph learning,
LLMs could serve as the research assistant when facing the different learning
problems. For example, decomposing the graph learning tasks, writing and
executing graph learning procedures step by step Mialon et al. (2023); Wei et
al. (2023); Huang et al. (2023b), or arranging the graph learning tools when
facing the tasks. Consequently, such an approach holds immense potential to
facilitate advancements in current graph learning methods.

##  5 Conclusion

Exploring the feasibility of employing LLMs in graph learning problems is a
promising direction considering the applications of graphs in the real world.
In this paper, we propose a novel conceptual prototype for versatile graph
learning approaches, and provide a comprehensive overview of existing methods
jointed with LLMs following the graph learning procedures and the different
ability requirements for LLMs. To be specific, we first explore the
feasibility of LLMs used in the graph learning pipeline, including task
definition, graph feature engineering, model selection and optimization,
deployment and serving procedures. In each procedure, we categorize methods
following the involvements of LLMs and provide detailed discussions on these
methods. Finally, we suggest future directions for LLM-based graph learning
approaches, potentially expanding the methodology towards universal graph
learning methods.

## References

  * Chai et al. [2023] Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang.  Graphllm: Boosting graph reasoning ability of large language model.  arXiv preprint arXiv:2310.05845, 2023.
  * Chen and Bertozzi [2023] Bohan Chen and Andrea L Bertozzi.  Autokg: Efficient automated knowledge graph generation for language models.  arXiv preprint arXiv:2311.14740, 2023.
  * Chen et al. [2023] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al.  Exploring the potential of large language models (llms) in learning on graphs.  arXiv preprint arXiv:2307.03393, 2023.
  * Chowdhery et al. [2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  Journal of Machine Learning Research, 24(240):1–113, 2023.
  * Fatemi et al. [2023] Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi.  Talk like a graph: Encoding graphs for large language models.  arXiv preprint arXiv:2310.04560, 2023.
  * Ge et al. [2023] Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, and Yongfeng Zhang.  Openagi: When llm meets domain experts.  arXiv preprint arXiv:2304.04370, 2023.
  * Gilmer et al. [2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In ICML, pages 1263–1272, 2017.
  * Guo et al. [2023] Jiayan Guo, Lun Du, and Hengyu Liu.  Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking.  arXiv preprint arXiv:2305.15066, 2023.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In NeurIPS, pages 1024–1034, 2017.
  * He et al. [2023] Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi.  Explanations as features: Llm-based features for text-attributed graphs.  arXiv preprint arXiv:2305.19523, 2023.
  * Huang et al. [2023a] Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma.  Can llms effectively leverage graph structural information: when and why.  arXiv preprint arXiv:2309.16595, 2023.
  * Huang et al. [2023b] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec.  Benchmarking large language models as ai research agents.  arXiv preprint arXiv:2310.03302, 2023.
  * Jin et al. [2023a] Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han.  Large language models on graphs: A comprehensive survey.  arXiv preprint arXiv:2312.02783, 2023.
  * Jin et al. [2023b] Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han.  Patton: Language model pretraining on text-rich networks.  arXiv preprint arXiv:2305.12268, 2023.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  ICLR, 2016.
  * Li et al. [2023] Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu.  A survey of graph meets large language model: Progress and future directions.  arXiv preprint arXiv:2311.12399, 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * Liu et al. [2022] Renming Liu, Semih Cantürk, Frederik Wenkel, Sarah McGuire, Xinyi Wang, Anna Little, Leslie O’Bray, Michael Perlmutter, Bastian Rieck, Matthew Hirn, et al.  Taxonomy of benchmarks in graph representation learning.  In Learning on Graphs Conference, pages 6–1. PMLR, 2022.
  * Liu et al. [2023a] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang.  One for all: Towards training one graph model for all classification tasks.  arXiv preprint arXiv:2310.00149, 2023.
  * Liu et al. [2023b] Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al.  Towards graph foundation models: A survey and beyond.  arXiv preprint arXiv:2310.11829, 2023.
  * Liu et al. [2023c] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang.  Graphprompt: Unifying pre-training and downstream tasks for graph neural networks.  In Proceedings of the ACM Web Conference 2023, pages 417–428, 2023\.
  * Mavromatis et al. [2023] Costas Mavromatis, Vassilis N Ioannidis, Shen Wang, Da Zheng, Soji Adeshina, Jun Ma, Han Zhao, Christos Faloutsos, and George Karypis.  Train your own gnn teacher: Graph-aware distillation on textual graphs.  arXiv preprint arXiv:2304.10668, 2023.
  * Mialon et al. [2023] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom.  Gaia: a benchmark for general ai assistants.  arXiv preprint arXiv:2311.12983, 2023.
  * Min et al. [2022] Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong.  Transformer for graphs: An overview from architecture perspective.  arXiv preprint arXiv:2202.08455, 2022.
  * Qian et al. [2023] Chen Qian, Huayi Tang, Zhirui Yang, Hong Liang, and Yong Liu.  Can large language models empower molecular property prediction?  arXiv preprint arXiv:2307.07443, 2023.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * Schlichtkrull et al. [2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15, pages 593–607. Springer, 2018.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580, 2023.
  * Sun et al. [2022] Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang.  Gppt: Graph pre-training and prompt tuning to generalize graph neural networks.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1717–1727, 2022.
  * Tang et al. [2023] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang.  Graphgpt: Graph instruction tuning for large language models.  arXiv preprint arXiv:2310.13023, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wang et al. [2023a] Haishuai Wang, Yang Gao, Xin Zheng, Peng Zhang, Hongyang Chen, and Jiajun Bu.  Graph neural architecture search with gpt-4.  arXiv preprint arXiv:2310.01436, 2023.
  * Wang et al. [2023b] Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov.  Can language models solve graph problems in natural language?  arXiv preprint arXiv:2305.10037, 2023.
  * Wang et al. [2023c] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.  A survey on large language model based autonomous agents.  arXiv preprint arXiv:2308.11432, 2023.
  * Wang et al. [2023d] Xu Wang, Huan Zhao, Wei-wei Tu, and Quanming Yao.  Automated 3d pre-training for molecular property prediction.  In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2419–2430, 2023.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * Wei et al. [2023] Lanning Wei, Zhiqiang He, Huan Zhao, and Quanming Yao.  Unleashing the power of graph learning through llm-based autonomous agents.  arXiv preprint arXiv:2309.04565, 2023.
  * Wen and Fang [2023] Zhihao Wen and Yuan Fang.  Prompt tuning on graph-augmented low-resource text classification.  arXiv preprint arXiv:2307.10230, 2023.
  * Wu et al. [2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2020.
  * Yao et al. [2018] Quanming Yao, Mengshuo Wang, Yuqiang Chen, Wenyuan Dai, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu.  Taking human out of learning applications: A survey on automated machine learning.  arXiv preprint arXiv:1810.13306, 2018.
  * You et al. [2020] Jiaxuan You, Zhitao Ying, and Jure Leskovec.  Design space for graph neural networks.  Advances in Neural Information Processing Systems, 33:17009–17021, 2020.
  * Zhang et al. [2021] Ziwei Zhang, Xin Wang, and Wenwu Zhu.  Automated machine learning on graphs: A survey.  2021\.  Survey track.
  * Zhang et al. [2022] Yongqi Zhang, Zhanke Zhou, Quanming Yao, and Yong Li.  Efficient hyper-parameter search for knowledge graph embedding.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2715–2735, 2022.
  * Zhang et al. [2023a] Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou.  Automl-gpt: Automatic machine learning with gpt.  arXiv preprint arXiv:2305.02499, 2023.
  * Zhang et al. [2023b] Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang, and Wenwu Zhu.  Large graph models: A perspective.  arXiv preprint arXiv:2308.14522, 2023.
  * Zhang [2023] Jiawei Zhang.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  arXiv preprint arXiv:2304.11116, 2023.
  * Zhao et al. [2023a] Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang.  Graphtext: Graph reasoning in text space.  arXiv preprint arXiv:2310.01089, 2023.
  * Zhao et al. [2023b] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * Zheng et al. [2023] Ruiqi Zheng, Liang Qu, Bin Cui, Yuhui Shi, and Hongzhi Yin.  Automl for deep recommender systems: A survey.  ACM Transactions on Information Systems, 41(4):1–38, 2023.