# Heterogeneous Graph Neural Architecture Search with GPT-4

Haoyuan Dong1, Yang Gao2, Haishuai Wang2, Hong Yang1, Peng Zhang1
Corresponding author.

###### Abstract

Heterogeneous graph neural architecture search (HGNAS) represents a powerful
tool for automatically designing effective heterogeneous graph neural
networks. However, existing HGNAS algorithms suffer from inefficient searches
and unstable results. In this paper, we present a new large language model
based HGNAS model to improve the search efficiency and search accuracy of
HGNAS. Specifically, we present a new GPT-4 enhanced Heterogeneous Graph
Neural Architecture Search (GHGNAS for short). The basic idea of GHGNAS is to
design a set of prompts that can guide GPT-4 towards the task of generating
new heterogeneous graph neural architectures. By iteratively asking GPT-4 with
the prompts, GHGNAS continually validates the accuracy of the generated HGNNs
and uses the feedback to further optimize the prompts. Experimental results
show that GHGNAS can design new HGNNs by leveraging the powerful
generalization capability of GPT-4. Moreover, GHGNAS runs more effectively and
stably than previous HGNAS models based on reinforcement learning and
differentiable search algorithms.

## Introduction

Heterogeneous graphs are often used to describe complicated relationships
between different types of objects, such as social networks (Jacob, Denoyer,
and Gallinari 2014) and recommendation networks (Dong et al. 2012). Recently,
a new type of Heterogeneous Graph Neural Network models (HGNNs) are proposed
to capture rich semantic information in heterogeneous graphs (Schlichtkrull et
al. 2018). The main idea of HGNNs is to iteratively update node
representations by aggregating information from either neighboring nodes or
high-order nodes generated from meta-paths (Sun et al. 2011). Typical HGNNs
include RGCN (Schlichtkrull et al. 2018), HAN (Wang et al. 2019), and MAGNN
(Fu et al. 2020).

Recently, Neural Architecture Search (NAS) algorithms (Zoph and Le 2016;
Elsken, Metzen, and Hutter 2019) are introduced into HGNNs which can
automatically design neural architectures for HGNNs without human manual
intervention. For example, an early work HGNAS (Gao et al. 2021) uses
reinforcement learning to find the best neural architectures for HGNNs. Based
on HGNAS, a number of new neural architecture search methods are proposed for
heterogeneous graphs. For example, AutoGEL (Wang, Di, and Chen 2021), MR-GNAS
(Zheng et al. 2022) and DHGNA (Zhang et al. 2023b) optimize a supernet with
differentiable search algorithms (Liu, Simonyan, and Yang 2018; Xie et al.
2019), and Space4HGNN (Zhao et al. 2022) introduces random search based on a
modular search space. Moreover, a number of models focus on designing meta-
structures (Huang et al. 2016) which can be taken as generalized forms of
meta-paths to describe relationship patterns between nodes in heterogeneous
graphs. For instance, GEMS (Han et al. 2020) and DiffMG (Ding et al. 2021) use
evolutionary algorithms (Črepinšek, Liu, and Mernik 2013) and differentiable
algorithms respectively to find the best meta-structures for downstream graph
learning tasks.

While current HGNAS methods can discover more superior HGNN models than manual
design, they still necessitate a considerable amount of expertise to construct
the search space and optimize search strategies. Notably, the aforementioned
methods are influenced to some extent by random seeds (Kazimipour, Li, and Qin
2014; Henderson et al. 2018). For instance, the differentiable search method
DiffMG (Ding et al. 2021) typically demonstrates unstable search results
because the performance of differentiable search algorithms depends greatly on
the selection of the search seed. Choosing inappropriate seeds may lead to
search results inferior to manually designed HGNN models. Therefore, while
these methods have provided effective approaches for designing heterogeneous
graph neural networks, reducing reliance on expertise and mitigating the
impact of random seeds remains an open issue that needs to be addressed before
they can be more robustly and widely applied.

Large Language Models (LLMs) (Zhao et al. 2023) such as GPT-4 have shown
powerful generation capability and have been used in neural architecture
search in recent years. For example, AutoML-GPT (Zhang et al. 2023a) uses GPT
to handle data processing, hyper-parameter tuning, and model training. GENIUS
(Zheng et al. 2023) uses GPT-4 to find CNN architectures for downstream tasks.
GPT4GNAS (Wang et al. 2023) uses GPT-4 to generate new homogeneous graph
architectures. However, these methods have not considered heterogeneous graphs
which require a more complicated search space. In order to enable GPT-4 for
heterogeneous graphs, the key issue is to design a new class of prompts that
can guide GPT-4 to iteratively generate better HGNN architectures.

In this paper, we present a new Heterogeneous Graph Neural Architecture Search
method based on the large language model GPT-4, namely GHGNAS. Our method
designs a new class of prompts for GPT-4 which can help generate new
architectures for HGNNs that are suitable for downstream graph learning tasks.
Specifically, the prompts consist of descriptions of the learning task and
input heterogeneous graphs, and descriptions of the search space, search
strategy, and search feedback of the HGNAS method. These prompts can guide
GPT-4 to adapt to the task of generating new neural architectures for HGNNs.
Then, the evaluation result of each new HGNN architecture is taken as feedback
which will be added to the prompt for generating better HGNN architectures. By
repeatedly updating the prompts, GHGNAS can obtain the best HGNN
architectures. We summarize the contribution of the paper as follows:

  * •

This research represents the first work on using GPT-4 to generate new
heterogeneous graph neural architectures, where a new heterogeneous graph
neural architecture search method GHGNAS is presented.

  * •

This work proposes a new class of prompts to enable GPT-4 for heterogeneous
graph neural network architecture search. These prompts can guide GPT-4 to
iteratively generate better HGNN architectures for downstream learning tasks.

  * •

Experiments on benchmark datasets show that the proposed GHGNAS method can
obtain more accurate and stable results than existing methods on heterogeneous
graph neural architecture search

## RELATED WORK

In this part, we survey related work on heterogeneous graph neural
architecture search and large language models for neural architecture search.

### Heterogeneous graph neural architectures search

Existing methods for heterogeneous graph neural architecture search can be
broadly categorized into three classes, i.e., reinforcement learning
algorithms, differentiable search algorithms, and evolutionary search
algorithms.

HGNAS (Gao et al. 2021) represents the first work of using reinforcement
learning to heterogeneous graph neural architecture search. Based on HGNAS,
HGNAS++ (Gao et al. 2023) is proposed to accelerate the search process. These
two methods use reinforcement learning and RNN to first represent a neural
architecture as a sequence and then allow the RNN to act as controller to
sample architectures, where the testing results of the sampled architectures
are taken as rewards to update the weights of the RNN controller. As a result,
the RNN controller iteratively generates better heterogeneous graph neural
architectures.

The second type of method is based on differentiable search algorithms, such
as DARTS (Liu, Simonyan, and Yang 2018). First, a super network containing all
possible operations is constructed, and then the weights of these operations
are updated through gradient descent. The final neural network structure is
composed by selecting the operation with the largest weight. To solve the
problem of graph data heterogeneity, DHGNA (Zhang et al. 2023b) adds
relationships and timestamps that need to be focused on into the search space
to construct a super network, thereby solving the architecture search problem
of dynamic heterogeneous graphs, MR-GNAS (Zheng et al. 2022) sets up a fine-
grained search space for multi-relationship networks, AutoGEL (Wang, Di, and
Chen 2021) incorporates additional operators into the search space to handle
edge features. Different from directly searching for architectures, the goal
of DiffMG (Ding et al. 2021) and PMMM (Li, Xu, and He 2023) is to explore and
find better meta-structures.

The third type of method is based on evolutionary algorithms (Črepinšek, Liu,
and Mernik 2013). This method first initializes a batch of architectures as a
population, with each architecture as an individual. This process is similar
to the evolution process of organisms, using selection, recombination, and
mutation operations to generate a new population, retaining good architectures
and eliminating poorly performing ones. A representative work GEMS (Han et al.
2020) retains valuable meta-structures by using genetic algorithms.

However, the above works haven’t considered using large language models for
heterogeneous graph neural architecture search, which is the focus of this
paper.

### GPT-4 for neural architecture search

Recently, there have been several works proposed to use the GPT-4 for neural
architecture search. Thanks to the powerful generation capability of GPT-4, a
pioneer work GENIUS (Zheng et al. 2023) uses GPT-4 to design neural
architectures for CNNs. The key idea is to allow GPT-4 to learn from the
feedback of generated neural architectures and iteratively generate better
ones. Experimental results on benchmarks demonstrate that GPT-4 can find top-
ranked architectures after several repeats of prompts. Subsequently, AutoML-
GPT (Zhang et al. 2023a) designs prompts for GPT to automatically complete
tasks of data processing, model architecture design, and hyper-parameter
tuning. There are also articles exploring the mutual improvement between LLMs
and AutoML (Tornede et al. 2023).

Despite the above achievements, existing works have not explored GPT-4 for
heterogeneous graph neural architecture search. The search space for
heterogeneous graph neural architecture search is vast which requires
considering different ways of message propagation through different graph
relationships. This paper designs a new set of prompts for a large language
model GPT-4 to enable heterogeneous graph neural architecture search, which
represents the first effort at the combination of GPT-4 and heterogeneous
graph neural architecture search.

Figure 1: An overview of the proposed GHGNAS method, which includes ① design a
prompt containing task description, information about heterogeneous graph,
search space, and search strategy; ② based on the prompt, GPT-4 generates a
batch of candidate architectures; ③ convert the architectures into HGNNs; ④
verify the performance of the architectures in a downstream graph learning
task; ⑤ generate feedback prompts based on the architectures and accuracy, and
⑥ GPT-4 continues to generate new neural architectures based on the feedback.

## PRELIMINARIES

Heterogeneous graph
$G=\\{\mathcal{V},\mathcal{E},\mathcal{T},\mathcal{R},f_{\mathcal{T}},f_{\mathcal{R}}\\}$,
where $\mathcal{V}$ represents a set of nodes,$\mathcal{E}$ represents a set
of edges, $\mathcal{T}$ denotes a set of node types, and $\mathcal{R}$ denotes
a set of edge types. The mapping functions
$f_{\mathcal{T}}(\mathcal{V})\to\mathcal{T}$ and
$f_{\mathcal{R}}(\mathcal{E})\to\mathcal{R}$ map nodes and edges to their
respective types. Here, $|\mathcal{T}|+|\mathcal{R}|>2$.
Meta-paths refer to paths from one node to another, which can be represented
as $m=\begin{aligned}
t_{1}&\overset{r_{1}}{\to}t_{2}\cdots\overset{r_{k}}{\to}t_{k+1}\end{aligned}$,
where $t.\in\mathcal{T}$ and $r.\in\mathcal{R}$. Meta-paths are used to
represent specific relationship patterns.
Heterogeneous graph neural architecture search. Given a heterogeneous graph
$\mathcal{G}$, the objective is to find the best architecture from a given
architecture space $\mathcal{A}$ that maximizes a performance evaluation
metric $c$ as follows:

$\alpha^{*}=\underset{\alpha\in\mathcal{A}}{\operatorname*{argmax}}\
c(\alpha(\mathcal{G})),$ (1)

where $\alpha^{*}\in\mathcal{A}$ represents the discovered optimal
architecture. The performance evaluation metric $c$ can be either AUC or
macro-F1, depending on the choice of downstream learning tasks.

Table 1: Search space of HGNAS. Search space | Operations
---|---
$R^{l}$ | $R^{l}\subseteq R\cup R_{I},\quad 0\leq l<n$
HAggr | sum, mean, max, lstm, att
GNN |  | Encd | Aggr
$gcn$ | $\frac{X_{s}W_{C}}{\sqrt{|N(s,C)||N(t,C)|}}$ | sum
$gat$ | $X_{s}{W}_{C}$ | att
$edge$ | $(X_{s}-X_{t})W_{C}$ | max
$sage$ | $X_{s}W_{C}$ | max
$zero$ | 0 | sum

Search space. We use the default search space of HGNAS, as shown in Table 1.
The search space is designed based on message passing in heterogeneous graphs.
Specifically, message passing can be represented as the following parts:

$M_{r}^{l+1}\leftarrow GNN(X^{l},A^{r})$ (2) $X^{l+1}\leftarrow
HAggr(\\{M_{r}^{l+1}\mid{\forall}r\in R^{l}\\}),$ (3)

where $X^{l}$ represents the node feature of nodes at the $l-th$ layer,
$M_{r}$ represents the aggregated message vector on relation $r$, $A^{r}$
represents the adjacency matrix of relation $r$, and $R^{l}$ represents the
set of relationships at layer $l$ that participate in message passing. $GNN$
is used to calculate message vectors from certain types of relationships, and
$HAggr$ is employed for aggregating messages from different $r$. As shown in
Table 1, candidate $GNN$ operations include ”$gcn$”(Kipf and Welling 2016),
”$gat$”(Velickovic et al. 2017), ”$edge$”(Xu et al. 2018), ”$sage$”(Hamilton,
Ying, and Leskovec 2017), ”$zero$”, and candidate $HAggr$ includes $sum$,
$mean$, $max$, $lstm$, $att$, where $att$ includes learnable parameters. Note
that our approach can also be combined with other search spaces, such as the
search space from DiffMG (Ding et al. 2021).

## Method

Table 2: The prompts of GHGNAS. Prompt |  Template
---|---
Task description $P_{t}$ |  Our task is heterogeneous graph neural architecture, searching for a HGNN architecture that can achieve the best performance on a downstream task
Heterogeneous graph $P_{d}$ |  The data set is [Dataset] ,[Node type and number], [Edge type and number] . The downstream task is [NC\LP] and the target node\link is [Target]
Search space $P_{s}$ |  The architecture of a n-layer HGNN is expressed as [ArchSeq], for each edge, you need to select one from [GNN Operations], then choose an aggregate function from [HAggr Operations] per layer
Search strategy $P_{e}\&P_{o}$ |  Exploration Strategy: Explore as many different architectures in the search space as possible Optimization Strategy: Analyze how to get a better architecture based on existing results
Feedback $P_{f}$ |  The performance of [Architecture] is [Accuracy]

In this section, we introduce the proposed model GHGNAS in detail. First, we
introduce the new prompts. Then, we introduce the search process, where the
GPT-4 model is taken as the controller and generates new architectures.

### GHGNAS Prompts

In order to solve Eq1, we need to use natural language to describe the task of
neural architecture search. In the following, we discuss two key issues.

First, how to let GPT-4 understand the search space and output the
architecture? According to the search space discussed in the previous section,
we need to choose an HGNN for each relationship to calculate $M_{r}$, and
choose an aggregation method for all the messages within the layer. An
intuitive approach is to represent all relationship types involved in message
passing and the aggregation function of each layer as a sequence. Then, the
above process is equivalent to asking GPT-4 to choose a specific value for
each element on the sequence. For each candidate HGNN, we do not specifically
explain their calculation method but emphasize the meaning of ”$zero$”, to
help GPT-4 filter the edges involved in message passing. In the end, we just
need GPT-4 to output the sequence representation of the architecture.

Second, how to avoid exploring redundant architectures and improve search
efficiency? After giving the search space and the output, GPT-4 already can
generate new architectures. However, if GPT-4 is only given a full view of the
search space, they will generate in a random, which leads to random search and
probably repeat the same architecture in the worst case. Therefore, we need to
specify a search strategy to avoid random search.

Our idea is the same as reinforcement learning (Sutton and Barto 2018),
considering two strategies of exploration and exploitation. We divide the
search into two stages. The first stage is the exploration stage, which
requires GPT-4 to explore different architectures in the search space as much
as possible, avoid the output of the same architecture, and avoid falling into
a local optimum. We transform the sequences output by GPT-4 into actual HGNNs
and train them on the input dataset to evaluate their performance in a
downstream task and add the generated architectures and their accuracy results
into the prompt, asking GPT-4 to generate new architectures.

After accumulating a part of the samples, we enter the second stage, i.e., the
optimization stage. At this stage, we no longer need to explore the search
space but require GPT-4 to analyze how to obtain better architectures based on
the already evaluated architectures explored in the previous iterations. The
specific optimization method is chosen by GPT-4, we only state in the prompt
that it needs to be analyzed based on existing architectures before the output
of new sequences of architectures.

Algorithm 1 GHGNAS search algorithm.

0: Heterogeneous graph $\mathcal{G}$; Search space $\mathcal{A}$; Evaluation
metric $c$; # of iterations in exploration $T_{e}$ ; # of iterations in
optimization $T_{o}$ ;

0: The best architecture $\alpha^{*}$

1: Model set $M\leftarrow[]$, model performance set $C\leftarrow[]$

2: Generate prompt $P_{s}$, $P_{d}$based on $\mathcal{A}$ and $\mathcal{G}$

3: $P_{f}=\emptyset$ //Exploration Stage

4: for i=1 to $T_{e}$ do

5: $M_{i}=\textbf{GPT-4}(P_{t}+P_{d}+P_{s}+P_{e}+P_{f}),C_{i}=c(M_{i})$

6: $M=M\cup M_{i},C=C\cup C_{i}$

7: Generate feedback prompt $P_{f}$ based on $M,C$

8: end for

9: $M^{\prime},C^{\prime}\leftarrow TopK(M,C)$

10: Generate prompt $P_{f}$ based on $M^{\prime}$ and $C^{\prime}$
//Optimization Stage

11: for i=1 to $T_{o}$ do

12: $M_{i}=\textbf{GPT-4}(P_{t}+P_{d}+P_{s}+P_{o}+P_{f}),C_{i}=c(M_{i})$

13: $M^{\prime}=M^{\prime}\cup M_{i},C^{\prime}=C^{\prime}\cup C_{i}$

14: Generate feedback $P_{f}$ prompt based on $M^{\prime},C^{\prime}$

15: end for

16: $M^{*}\leftarrow TopK(M\cup M^{\prime})$

17: Re-train $M^{*}$ and select the architecture with the best performance on
the validation set as $\alpha^{*}$

18: return $\alpha^{*}$

### Algorithms

Algorithm 1 outlines the detailed process of GHGNAS. Initially, we construct
the descriptions $P_{s}$ and $P_{d}$ of the search space and the input
heterogeneous graph dataset. Lines 3 to 8 represent the exploration stage,
where the algorithm guides GPT-4 to explore possible architectures in the
search space by combining the prompts given in Table 2 and taking the existing
architectures and their accuracy results as feedback $P_{f}$. At this point,
the main purpose of adding feedback is to avoid generating repeated
architectures. Line 9 shows the optimization stage, adds the current best-
performing $n$ architectures to $P_{f}$, and modifies the strategy of this
stage to $P_{o}$. We ask GPT-4 to analyze how to obtain better architectures
based on these architectures and add newly generated architectures and their
accuracy results to feedback $P_{f}$. At this point, the purpose of adding
feedback is to verify whether the sampling preference for GPT-4 and the
generated architectures are correct. If GPT-4 find that the sampling results
are weaker than existing architectures, they will avoid using the same
sampling preference and redo the analysis.

Due to the token limit of GPT-4, when it is unable to continue generating new
architectures, we will select the currently top-k architectures as knowledge
and add them to the prompt, and then continue to interact with GPT-4. In
addition, the best result may not appear in the last iteration. Thus, after
the search ends, we will read the interaction history to find the top
architectures with the best performance, verifying their accuracy results. In
the last step, we take the architecture with the highest accuracy result as
the ultimate output.

## Experiments

In the experiment, the large language model we use is GPT-4 version 2023-3-14.
The experiments are designed to answer the following three questions:

  * •

Can GPT-4 be used for heterogeneous graph neural architecture search tasks?

  * •

If GPT-4 works, compared with previous manually designed architectures and
existing neural architecture search algorithms, how does the architecture
designed by GHGNAS perform?

  * •

How about the performance of the designed GHGNAS prompts, and how to optimize
the prompts?

According to the above questions, we set up three experiments:

  * •

First, we traverse a smaller search space as benchmark and compare the
accuracy and ranking of GHGNAS with existing neural architecture search
methods.

  * •

Second, in a real-world architecture search environment, we compare GHGNAS
with popular GNNs, HGNNs, and popular graph NAS methods.

  * •

Third, we design ablation study and compare the performance of the variants of
the proposed prompts on downstream graph learning tasks to prove the utility
of our prompts.

### Experimental Setup

Datesets. We evaluate our method on two tasks: node classification and
recommendation. In the node classification task, we use two widely used
academic network datasets, i.e., ACM and DBLP. The target node of the ACM
dataset is Paper (P), and the target node of the DBLP dataset is Author (A).
In the recommendation task, we use two datasets, i.e., Amazon and Yelp. The
target edge of Amazon is User-Item, and the target edge of Yelp is User-
Business. The details of dataset division are the same as HGNAS.
Baselines. First, we compare the most commonly used GCN (Kipf and Welling
2016) and GAT (Velickovic et al. 2017) on homogeneous graphs, as well as the
homogeneous graph architecture search method GraphNAS (Gao et al. 2022). These
methods require the conversion of heterogeneous graphs into homogeneous graphs
as input. In addition, we also compare manually designed HGNNs, such as RGCN
(Schlichtkrull et al. 2018), HAN (Wang et al. 2019), HGT (Hu et al. 2020), and
MAGNN (Fu et al. 2020). We also consider two NAS methods, HGNAS (Gao et al.
2021) and DiffMG (Ding et al. 2021). However, since the dataset and search
space of DiffMG are different from ours, we supplemented the search space
experiment of DiffMG in the appendix.
Hyper-parameters. For GHGNAS, the temperature of the GPT-4 is set to 1 during
the exploration stage, and the temperature is set to 0 during the optimization
stage. And we let GPT-4 generate 20 architectures in each iteration. Finally,
we retrained the top 10 architectures based on their performance scores on the
validation set. This retraining is conducted ten times to select the best-
performing architecture. We use HGNAS to sample 200 architectures for
exploration and to sample 100 results for exploitation. For Random, we
randomly sample 300 HGNNs from the search space. For HGNNs, the learning rate
is set to 0.005, the dropout is set to 0.6, and the hidden dimension is set to
256.

### Experimental Results

Table 3: Experimental results (%) on ACM and DBLP for node classification
w.r.t. macro-F1 on benchmark.

| ACM | DBLP
---|---|---
| macro-F1 | rank | macro-F1 | rank
meta-path | 89.75 | 1725 | 91.72 | 167
All relations | 91.18 | 245 | 90.38 | 240
Random | 91.60 | 38 | 93.51 | 4
HGNAS | 91.52 | 64 | 93.14 | 25
GHGNAS | 91.70 | 20 | 93.52 | 3
Random (avg) | 91.34 | 66.2 | 93.09 | 31.4
HGNAS (avg) | 91.27 | 256.4 | 92.41 | 101.2
GHGNAS (avg) | 91.58 | 48.6 | 93.23 | 20.4

Table 4: Experimental results (%) on ACM and DBLP for node classification
w.r.t. macro-F1 and Experimental results (%) on Amazon and Yelp for link
prediction w.r.t. AUC. ‘avg’ represents the average performance of
architectures obtained from five different searches.

| ACM | DBLP | Amazon | Yelp
---|---|---|---|---
| Val | Test | Val | Test | Val | Test | Val | Test
GCN | 92.47±0.45 | 90.33±0.59 | 59.62±1.31 | 83.03±0.93 | 78.96±0.06 | 78.12±0.06 | 90.43±0.12 | 91.62±0.11
GAT | 92.59±0.19 | 91.25±0.27 | 60.36±0.77 | 84.34±0.94 | 75.00±0.77 | 75.43±0.07 | 85.46±0.06 | 85.51±0.20
RGCN | 92.12±0.49 | 91.25±0.50 | 86.77±0.59 | 93.66±0.76 | 74.77±0.10 | 73.85±0.12 | 88.93±0.19 | 88.63±0.17
HAN | 93.20±0.42 | 91.16±0.27 | 85.01±0.29 | 92.12±0.20 | 74.37±0.37 | 75.96±0.23 | 89.35±0.23 | 89.64±0.20
HGT | 89.12±0.66 | 88.93±0.38 | 84.92±0.43 | 92.97±0.20 | 75.69±0.76 | 74.30±0.02 | 91.22±0.15 | 91.73±0.20
MAGNN | 92.13±0.39 | 91.30±0.38 | 87.13±0.32 | 93.57±0.20 | 75.24±0.38 | 75.90±0.33 | 91.34±0.20 | 90.50±0.30
Random | 92.96±0.31 | 91.70±0.46 | 86.05±0.72 | 93.50±0.44 | 78.70±0.12 | 77.84±0.11 | 91.51±0.69 | 92.03±0.79
Graphnas | 91.73±0.47 | 90.89±0.54 | 79.07±0.20 | 75.49±0.34 | 77.78±0.16 | 78.21±0.21 | 91.44±0.22 | 91.34±0.10
HGNAS | 93.56±0.28 | 91.45±0.10 | 86.15±0.53 | 93.57±0.36 | 79.30±0.22 | 78.58±0.19 | 92.39±0.29 | 92.12±0.14
GHGNAS | 93.69±0.50 | 92.37±0.19 | 86.72±0.47 | 93.84±0.21 | 79.90±0.18 | 79.16±0.15 | 92.78±0.16 | 92.80±0.15
Random (avg) | 92.29±0.56 | 90.89±0.74 | 85.46±1.20 | 93.24±2.13 | 78.28±1.29 | 77.52±1.37 | 90.57±2.25 | 91.03±1.74
Graphnas (avg) | 91.27±1.09 | 90.71±1.02 | 76.63±1.71 | 73.30±1.82 | 77.72±0.35 | 77.64±0.32 | 90.70±0.44 | 91.22±0.29
HGNAS (avg) | 92.57±0.34 | 91.21±0.63 | 85.53±1.69 | 91.35±1.36 | 79.19±0.40 | 78.50±0.39 | 92.20±0.35 | 91.96±0.21
GHGNAS (avg) | 93.11±0.45 | 91.96±0.44 | 85.93±0.28 | 93.54±0.33 | 79.55±0.51 | 78.76±0.36 | 92.33±0.45 | 92.29±0.53

#### Results on the benchmark.

We test GPT-4’s generation capability on heterogeneous graph neural
architecture search within a very small search space. In this search space,
GNN operations only include $"zero"$ and $"gcn"$, and the aggregation function
$HAggr$ only includes $"sum"$. Therefore, the goal of this experiment can be
taken as finding the best meta-structures for a given heterogeneous graph.
When the layer number of HGNNs is set to 2, the benchmark contains 8,192
possible HGNN architectures for ACM and 1,024 for DBLP. We traversed all
possible HGNNs and calculated the average accuracy five times for each
architecture. In this experiment, we ask GPT-4 to explore three iterations and
then optimize two iterations, searching a total of 100 possible architectures.
We use the Random method to search 100 HGNNs. The hyperparameters of HGNAS
remain unchanged because it needs to train the controller.

The results are shown in Table 3. Specifically, ”All relations” represents the
architecture using $"gcn"$ for all relations in Equation 4, and the ”meta-
path” represents the architecture uses $"gcn"$ only for the relations that
constitute the meta-path while setting all other relations to $"zero"$. The
table reports the performance of the best architecture designed based on the
meta-path.

(a) Designed HGNN on ACM. Authon(A), paper(P), subject(S)

(b)  Designed HGNN on DBLP. Authon(A), paper(P), term(T), venue(V)

Figure 2: HGNN designed by GHGNAS on the benchmark.

Firstly, the architecture designed by the meta-path $PAP$ performs the best,
but it still falls short of ”All relations” on ACM. However, on DBLP, the
performance of the architecture designed by the meta-path APV exceeds ‘All
relations’ by 1.34%. Additionally, the NAS method can consider a more flexible
message-passing method, which performs better than the architecture manually
designed based on meta-path and the architecture using all relations.
Secondly, on both datasets, the architecture designed by GHGNAS outperforms
other NAS methods. Furthermore, it exhibits the best average performance
across five distinct searches. Notably, GHGNAS identified an architecture that
ranks third on the DBLP dataset.

Figure 2 illustrates the HGNNs designed by GHGNAS. The left picture shows the
best architecture found by GHGNAS on ACM, which includes the best meta-path
PAP. Compared with the full connection, it discards A-P and S-P in the first
layer. It also contains the best meta-path APV on DBLP and discards messages
from T nodes. This demonstrates that discarding part of messages within
heterogeneous graphs can improve performances.

#### Results on HGNAS search space.

Figure 3:  The performance of the best HGNN model designed by GHGNAS and other
baselines  w.r.t. iterations. The HGNN model designed by GHGNAS outperforms
others during the exploration stage. At the optimization stage, GHGNAS further
improves the HGNN model. Figure 4: The best HGNN architecture designed by
GHGNAS on the ACM (left) and DBLP (right).

According to the first part of the experiment, we confirmed that GPT-4 has the
ability to complete the task of architecture search, so we considered
continuing testing with complete HGNAS search space. In this part, we let
GHGNAS iterate ten times using the exploration strategy, and then iterate five
times using the optimization strategy. The left part of Table 4 shows the
results of the node classification task on the ACM and DBLP datasets w.r.t.
macro-F1 and the right part is the results of the recommendation task on the
Amazon and Yelp datasets w.r.t. AUC. Table 4 provides the results of running
five times on the test set. In addition, Figure 3 shows the accuracy of the
best architecture during iteration.

First, as shown in Table 4, manually designed HGNNs did not achieve good
results, and GAT and GCN sometimes even surpassed some HGNNs. This shows that
manually designed algorithms sometimes convey limited information or convey
information that has a negative impact on the results. Second, although the
performance of HGNNs designed by GHGNAS is slightly weaker than MAGNN on the
DBLP validation set, it is always better than other baselines on the test set.
In addition, since the search process of GHGNAS is insensitive to seeds, it is
always able to design excellent architectures in different search iterations,
thus maintaining superior average performance. In Figure 4, the HGNN
architecture newly designed for ACM still includes the optimal meta-path PAP,
and the architecture for DBLP also includes the optimal meta-path APV. By
employing a more extensive set of GNN operations, they also exhibit improved
performance compared to the HGNNs designed in Figure 2. This indicates that
NAS, which selects GNNs operations for different relations, is advantageous
for downstream tasks, and it also highlights that GHGNAS can design excellent
architecture in complicated search spaces.

#### Ablation study on different prompts.

Table 5: Results of different prompts.

| ACM | DBLP | Amazon | Yelp
---|---|---|---|---
| Macro-f1 | AUC
GHGNAS | 92.46±0.06 | 93.91±0.21 | 79.14±0.17 | 92.81±0.15
no dataset | 92.21±0.36 | 93.76±0.33 | 78.03±0.13 | 92.66±0.08
no operation | 90.75±0.40 | 93.31±0.64 | 72.38±0.14 | 91.72±0.25
no strategy | 92.11±0.41 | 93.62±0.38 | 77.80±0.08 | 92.62±0.21

Figure 5: The performance of HGNN designed by different prompts w.r.t.
iterations. At the end of the search, GHGNAS outperforms its variants.

In this part, we test the impact of different prompts. The prompts include
task description, information about heterogeneous graphs, search space, search
strategy, and feedback after verifying the architecture. We compare the impact
of removing a part of the prompt on the results. The task description and
feedback results need to be retained; otherwise, it will affect the generated
content. When there is no information about the heterogeneous graph, after
generating a sequence representing the architecture, replace the specific
relationship with $edge_{i}$. When there is no search space operation
description, we ask GPT-4 to choose a number for each element in a sequence of
a specified length but do not tell what the numbers specifically mean. A total
of five iterations of exploration and five iterations of optimization are
conducted.

Table 5 shows the best results obtained by different prompts. The results show
that when a part of the description is removed, the outputs of GPT-4 suffer
from an accuracy drop. Among them, removing operations from the search space
has the most serious impact, with a decrease of 6.75% on the Amazon dataset
and a decrease of 1.71% on the ACM dataset. Figure 5 shows the mean accuracy
and variance of the designed HGNNs during the iteration process. Without the
description of the search strategy, even if a batch of better architectures
has been obtained in previous iterations, it is impossible to use these
architectures for prompt optimization. Thus, the average performance always
fluctuates. When there is no description of the operations in the search
space, the average performance on Amazon lags behind other variants. In
addition, even in the optimization stage, the improvement is insignificant on
ACM and DBLP. When there is no information about the input heterogeneous
graphs, although the average performance is not as good as the original
prompt, GHGNAS can still optimize architectures effectively, with the average
results of the generated architectures improving steadily. These observations
show that GPT-4 has good generalization capability and is capable of running
neural architecture search tasks effectively even if it does not know the
specific information of input datasets.

### Discussions

In this section, we discuss based on the above experimental results the
advantages and limitations of the models, the prompts, and interpretability.

Firstly, our method is capable of solving the heterogeneous graph neural
architecture search task and performs better than existing HGNNs and HGNAS
models. The experimental results show the HGNNs designed by GHGNAS are stably
better than HGNAS based on reinforcement learning, which shows that our method
has a better ability to perform neural architecture search. Compared with
DiffMG, which is largely affected by the selected seeds, GHGNAS can search for
accurate architectures in different conversations, and a batch of results with
excellent performance can be obtained during the iterative process. This shows
that using GPT-4 to complete architecture search tasks greatly reduces the
requirements for professional knowledge and solves the pain points of previous
neural architecture search methods. As users, we only need to accurately
describe the task.

Secondly, the results generated by GHGNAS are interpretable. In our task,
GPT-4 is used as a black box optimizer to complete the search and optimization
of the architecture and has achieved better performance than the traditional
interpretable neural architecture search solution. This results in that we
only know that the architecture of GHGNAS search is available, but cannot
explain the decision-making process. However, we can ask GPT-4 to give its own
analysis results and sampling preference in the prompt. An example of a
response from GPT-4 is as follows:

  * •

The operation $"gcn"$ frequently appears in high-performance architectures.
Thus, this operation is likely to have a strong correlation with model
performance, particularly when the edge is (A-P), (S-P), and (P-P).

  * •

The operation $"gat"$ also performs very well, appearing many times in the
best architectures, which implies that $"gat"$ works very well for diverse
layers.

  * •

The operation $"sage"$ rarely appears, particularly at the last entry of the
action list, which indicates that the operator may not perform as expected in
the benchmark datasets.

  * •

The $"edge"$ and $"zero"$, while appearing less frequently, are often found in
high-performing actions as well, indicating that they have a positive impact
on the architecture in specific combinations.

  * •

The models with accuracy above 0.923 use a diversified search strategy,
showing that different types of convolutions are beneficial for the
classifier.

  * •

The operation $"gcn"$ and various convolutions can improve model performance.
It is suggested to avoid using $"sage"$ frequently. However, it might have a
potential advantage when combining $"sage"$ with other operations.

Figure 6 shows an optimization example of GHGNAS. The left part shows an
architecture designed in the exploration stage. GPT-4 based on the analysis
results samples a new architecture that uses more $gcn$ operations and
achieves a 0.93% improvement on the validation set. This means that GPT-4 has
the ability to optimize the architecture.

Third, whether GPT-4 already knows the best architecture for the data set.
This is a concern raised by GENIUS (Zheng et al. 2023). If the pre-training
process of GPT has already included the optimal architecture of a specific
dataset, then using GPT for neural architecture search is a process of
directly finding answers based on pre-training knowledge. The experiments in
this paper can address this concern. First, HGNAS (Gao et al. 2021) was
introduced in 2021, at a time when GPT did not possess knowledge of HGNAS.
Secondly, when we do not provide information on heterogeneous graphs, GPT-4
can still design excellent architectures for the four data sets. It can be
seen that GPT-4 observes and optimizes the performance of different
architectures, rather than directly generating excellent architectures based
on existing knowledge in pre-training.

Figure 6: An example of the GHGNAS optimization stage on ACM. The left part
shows the HGNN designed by GHGNAS after ten iterations using the exploration
strategy. After entering the optimization stage, in the 11th iteration, the
newly generated HGNN achieved an improvement of 0.93%.

Limitations. First, one of the pain points of using GPT-4 for heterogeneous
graph neural architecture search is that the usage of OpenAI’s API is charged.
The understanding and generalization capabilities of GPT-3.5 or other LLMs are
still far behind GPT-4, they prefer to provide the program code that generates
the neural architecture rather than providing the architecture itself. In the
future, we will add other powerful yet free LLMs to the tasks.

Second, the reproducibility of the optimization process is limited. Although
the process has interpretability, the optimization is not the same. Even with
the same prompt, there is some randomness in the content generated by GPT-4
each time. This is also one of the reasons why in the later stages, it is
possible to generate an architecture with poor performance.

Third, GHGNAS still relies on a manually designed search space. We hope that
in the future, GHGNAS can be used to complete the tasks of search space
formulation and exploration at the same time, to the greatest extent reducing
the dependence of heterogeneous graph neural architecture search on knowledge
and experience.

## CONCLUSIONS

In this paper, we introduced a new Heterogeneous Graph Neural Architecture
Search model GHGNAS, which uses the large language model GPT-4 to complete the
task of heterogeneous graph neural architecture search. Experimental results
show that GHGNAS can use the generation and decision-making capabilities of
GPT-4 to find new HGNN architectures that are better than existing models
based on reinforcement learning and differentiable search algorithms. Just by
modifying the prompts, GHGNAS can be migrated to new search spaces and perform
well. GHGNAS alleviates the dependence on professional knowledge and the
unstable search results of existing heterogeneous graph search algorithms. In
future work, we hope to use the capabilities of large language models to solve
a more challenging problem of generating new search spaces for heterogeneous
graph neural architecture search.

## References

  * Črepinšek, Liu, and Mernik (2013) Črepinšek, M.; Liu, S.-H.; and Mernik, M. 2013.  Exploration and exploitation in evolutionary algorithms: A survey.  _ACM computing surveys (CSUR)_ , 45(3): 1–33.
  * Ding et al. (2021) Ding, Y.; Yao, Q.; Zhao, H.; and Zhang, T. 2021.  Diffmg: Differentiable meta graph search for heterogeneous graph neural networks.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 279–288.
  * Dong et al. (2012) Dong, Y.; Tang, J.; Wu, S.; Tian, J.; Chawla, N. V.; Rao, J.; and Cao, H. 2012.  Link prediction and recommendation across heterogeneous social networks.  In _2012 IEEE 12th International conference on data mining_ , 181–190. IEEE.
  * Elsken, Metzen, and Hutter (2019) Elsken, T.; Metzen, J. H.; and Hutter, F. 2019.  Neural architecture search: A survey.  _The Journal of Machine Learning Research_ , 20(1): 1997–2017.
  * Fu et al. (2020) Fu, X.; Zhang, J.; Meng, Z.; and King, I. 2020.  Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding.  In _Proceedings of The Web Conference 2020_ , 2331–2341.
  * Gao et al. (2021) Gao, Y.; Zhang, P.; Li, Z.; Zhou, C.; Liu, Y.; and Hu, Y. 2021.  Heterogeneous Graph Neural Architecture Search.  In _2021 IEEE International Conference on Data Mining (ICDM)_ , 1066–1071. IEEE.
  * Gao et al. (2022) Gao, Y.; Zhang, P.; Yang, H.; Zhou, C.; Tian, Z.; Hu, Y.; Li, Z.; and Zhou, J. 2022\.  Graphnas++: Distributed architecture search for graph neural networks.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Gao et al. (2023) Gao, Y.; Zhang, P.; Zhou, C.; Yang, H.; Li, Z.; Hu, Y.; and Philip, S. Y. 2023.  HGNAS++: efficient architecture search for heterogeneous graph neural networks.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Hamilton, Ying, and Leskovec (2017) Hamilton, W.; Ying, Z.; and Leskovec, J. 2017.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30.
  * Han et al. (2020) Han, Z.; Xu, F.; Shi, J.; Shang, Y.; Ma, H.; Hui, P.; and Li, Y. 2020.  Genetic meta-structure search for recommendation on heterogeneous information network.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, 455–464.
  * Henderson et al. (2018) Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; and Meger, D. 2018\.  Deep reinforcement learning that matters.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Hu et al. (2020) Hu, Z.; Dong, Y.; Wang, K.; and Sun, Y. 2020.  Heterogeneous graph transformer.  In _Proceedings of the web conference 2020_ , 2704–2710.
  * Huang et al. (2016) Huang, Z.; Zheng, Y.; Cheng, R.; Sun, Y.; Mamoulis, N.; and Li, X. 2016.  Meta structure: Computing relevance in large heterogeneous information networks.  In _Proceedings of the 22nd ACM SIGKDD International conference on knowledge discovery and data mining_ , 1595–1604.
  * Jacob, Denoyer, and Gallinari (2014) Jacob, Y.; Denoyer, L.; and Gallinari, P. 2014.  Learning latent representations of nodes for classifying in heterogeneous social networks.  In _Proceedings of the 7th ACM international conference on Web search and data mining_ , 373–382.
  * Kazimipour, Li, and Qin (2014) Kazimipour, B.; Li, X.; and Qin, A. K. 2014.  A review of population initialization techniques for evolutionary algorithms.  In _2014 IEEE congress on evolutionary computation (CEC)_ , 2585–2592. IEEE.
  * Kipf and Welling (2016) Kipf, T. N.; and Welling, M. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Li, Xu, and He (2023) Li, C.; Xu, H.; and He, K. 2023.  Differentiable meta multigraph search with partial message propagation on heterogeneous information networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, 8518–8526.
  * Liu, Simonyan, and Yang (2018) Liu, H.; Simonyan, K.; and Yang, Y. 2018.  Darts: Differentiable architecture search.  _arXiv preprint arXiv:1806.09055_.
  * Schlichtkrull et al. (2018) Schlichtkrull, M.; Kipf, T. N.; Bloem, P.; Van Den Berg, R.; Titov, I.; and Welling, M. 2018.  Modeling relational data with graph convolutional networks.  In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_ , 593–607. Springer.
  * Sun et al. (2011) Sun, Y.; Han, J.; Yan, X.; Yu, P. S.; and Wu, T. 2011.  Pathsim: Meta path-based top-k similarity search in heterogeneous information networks.  _Proceedings of the VLDB Endowment_ , 4(11): 992–1003.
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Tornede et al. (2023) Tornede, A.; Deng, D.; Eimer, T.; Giovanelli, J.; Mohan, A.; Ruhkopf, T.; Segel, S.; Theodorakopoulos, D.; Tornede, T.; Wachsmuth, H.; et al. 2023.  AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks.  _arXiv preprint arXiv:2306.08107_.
  * Velickovic et al. (2017) Velickovic, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; Bengio, Y.; et al. 2017.  Graph attention networks.  _stat_ , 1050(20): 10–48550.
  * Wang et al. (2023) Wang, H.; Gao, Y.; Zheng, X.; Zhang, P.; Chen, H.; and Bu, J. 2023.  Graph Neural Architecture Search with GPT-4.  _arXiv preprint arXiv:2310.01436_.
  * Wang et al. (2019) Wang, X.; Ji, H.; Shi, C.; Wang, B.; Ye, Y.; Cui, P.; and Yu, P. S. 2019.  Heterogeneous graph attention network.  In _The world wide web conference_ , 2022–2032.
  * Wang, Di, and Chen (2021) Wang, Z.; Di, S.; and Chen, L. 2021.  Autogel: An automated graph neural network with explicit link information.  _Advances in Neural Information Processing Systems_ , 34: 24509–24522.
  * Xie et al. (2019) Xie, S.; Zheng, H.; Liu, C.; and Lin, L. 2019.  SNAS: stochastic neural architecture search.  _ICLR_.
  * Xu et al. (2018) Xu, K.; Li, C.; Tian, Y.; Sonobe, T.; Kawarabayashi, K.-i.; and Jegelka, S. 2018\.  Representation learning on graphs with jumping knowledge networks.  In _International conference on machine learning_ , 5453–5462. PMLR.
  * Zhang et al. (2023a) Zhang, S.; Gong, C.; Wu, L.; Liu, X.; and Zhou, M. 2023a.  AutoML-GPT: Automatic Machine Learning with GPT.  _arXiv preprint arXiv:2305.02499_.
  * Zhang et al. (2023b) Zhang, Z.; Zhang, Z.; Wang, X.; Qin, Y.; Qin, Z.; and Zhu, W. 2023b.  Dynamic Heterogeneous Graph Attention Neural Architecture Search.
  * Zhao et al. (2022) Zhao, T.; Yang, C.; Li, Y.; Gan, Q.; Wang, Z.; Liang, F.; Zhao, H.; Shao, Y.; Wang, X.; and Shi, C. 2022.  Space4hgnn: a novel, modularized and reproducible platform to evaluate heterogeneous graph neural network.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2776–2789.
  * Zhao et al. (2023) Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zheng et al. (2023) Zheng, M.; Su, X.; You, S.; Wang, F.; Qian, C.; Xu, C.; and Albanie, S. 2023.  Can GPT-4 Perform Neural Architecture Search?  _arXiv preprint arXiv:2304.10970_.
  * Zheng et al. (2022) Zheng, X.; Zhang, M.; Chen, C.; Li, C.; Zhou, C.; and Pan, S. 2022.  Multi-relational graph neural architecture search with fine-grained message passing.  In _2022 IEEE International Conference on Data Mining (ICDM)_ , 783–792. IEEE.
  * Zoph and Le (2016) Zoph, B.; and Le, Q. V. 2016.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_.

##  Appendix A Appendix

### Comparison with DiffMG

DiffMG (Ding et al. 2021) employs a differentiable search algorithm with the
aim of automatically searching for the meta-structures that are applicable to
heterogeneous graphs and downstream tasks, thereby capturing semantic
information that is more complex than meta-paths. In experiment 5.2.1, we only
considered the meta-structures found by DiffMG and did not evaluate the
performance of GHGNAS in the DiffMG search space. We redesigned the prompts
based on DiffMG’s tasks and search space, and further compare them in the
tasks of ACM and DBLP node classification.

#### Search space

DiffMG defines the search space as a Directed Acyclic Graph (DAG), where nodes
on the DAG represent the intermediate states of the message-passing process,
and edges represent candidate edge types. Furthermore, DiffMG constrains the
search space based on the target node, which can be specifically expressed in
the following form:

$\mathcal{A}_{j,i}=\begin{cases}\mathcal{A}\cup\\{I\\}&j<N\text{and}i=j-1\\\
\mathcal{A}\cup\\{I\\}\cup\\{O\\}&j<N\text{and}i<j-1\\\
\overline{\mathcal{A}}&j=N\text{and}i=N-1\\\
\overline{\mathcal{A}}\cup\\{I\\}\cup\\{O\\}&j=N\text{and}i<K-1\end{cases}.$
(4)

where $\mathcal{A}$ represents the adjacency matrix of the candidate edge
type, $I$ and $O$ represent the identity matrix and the all-zero matrix
respectively. When it reaches the last node of the DAG, it can only choose the
candidate edges that are currently related to the task. This method reduces
the search space and filters out candidate edge types that have no impact on
the final node representation, thereby improving search efficiency. For more
specific information, you can refer to the original paper.

#### Prompts for the DiffMG search space

Taking the ACM data set as an example, we modify the prompt and use natural
language to describe the search space and constraints. Similarly, GHGNAS can
also guide GPT-4 in the prompt to choose candidate edges that are relevant to
the task target. Specifically, the design of the prompt is also similar to
that in the HGNAS search space, we use sequences to represent meta-structures
and require GPT-4 to select a candidate edge type for each element on the
sequence. The specific details are as follows:

Task description $P_{t}$: Our task is to find a meta-structure of
heterogeneous graphs that maximizes accuracy on downstream tasks. Search space
$P_{s}$: For a given meta-structure {H0-H1, H1-H2, H2-H3, H3-H4} {H0-H2,
H0-H3, H1-H3, H0-H4, H1-H4, H2-H4}, H0-H1, H1-H2, H2-H3 should be selected
from [PA, AP, PS, SP, I], H3-H4 should be selected from [PA, PS, I] and H0-H4,
H1-H4, H2-H4 should be selected from [PA, PS, I, O] because the target is P.
For H0-H2, H0-H3, and H1-H3 you can choose from [PA, AP, PS, SP, I, O]

The rest of the part is consistent with the template given in the text. By
simply modifying the template, GHGNAS can be migrated to heterogeneous graph
meta-structure search tasks without the need to design additional search
algorithms.

Table 6: Results of node classification in DiffMG’ search space w.r.t.
macro-F1.

|  | DBLP | ACM
---|---|---|---
DiffMG | seed | Val | Test | Val | Test
0 | 95.30±0.29 | 94.16±0.10 | 92.26±0.43 | 91.30±0.27
1 | 94.70±0.73 | 92.58±0.20 | 91.90±0.45 | 91.42±0.15
2 | 82.63±0.10 | 81.48±0.30 | 93.34±0.63 | 92.07±0.10
3 | 81.00±0.67 | 82.95±0.44 | 90.51±0.85 | 89.55±0.11
4 | 83.37±0.48 | 81.93±0.21 | 92.63±0.47 | 91.30±0.21
GHGNAS | 96.00±0.36 | 93.92±0.16 | 93.55±0.58 | 92.21±0.21
DiffMG (avg) | 87.40±6.26 | 86.62±5.56 | 92.13±0.94 | 91.13±0.93
GHGNAS (avg) | 95.42±0.32 | 93.37±0.25 | 93.05±0.38 | 91.80±0.52

#### Experimental results.

We follow all the default settings of DiffMG, and GHGNAS uses the exploration
strategy and optimization strategy for five iterations each. The results
indicate that GHGNAS surpasses DiffMG in terms of node classification task,
with the exception of a 0.24% lag on the DBLP test set. Moreover, in line with
the experimental findings in the main text, DiffMG can only yield satisfactory
results when utilizing specific seeds, whereas GHGNAS is capable of
discovering superior architectures following ten iterations across various
search processes.

### STATISTICS OF DATASETS

The experiments in this paper use the same datasets and the partitioning
method as in HGNAS (Gao et al. 2021), and the dataset information is shown in
Table 7. For the ACM and DBLP datasets, we use 400 nodes for training and 400
nodes for validation. For Yelp and Amazon, we consider ratings higher than
three as positive pairs, and the rest as negative pairs. Then, we randomly
divide 50% of the positive pairs into training, validation, and test sets at a
ratio of 3:1:1. For the comparison with DiffMG (Ding et al. 2021), the
experimental settings used can be referred to in the original paper.

Table 7: Dataset statistics.

Dataset | Relations(A-B) | #A | #B | #A-B
---|---|---|---|---
ACM | Author-Paper(A-P) | 17,351 | 4,025 | 13,407
| Paper-Subject(P-S) | 4,025 | 72 | 4,025
DBLP | Author-Paper(A-P) | 4,057 | 14,325 | 19,645
| Paper-Term(P-T) | 14,325 | 7,723 | 85,810
| Paper-Venue(P-V) | 14,325 | 20 | 14,328
Amazon | User-item(U-I) | 6,170 | 2,753 | 195,791
| Item-View(I-V) | 2,753 | 3,857 | 5,694
| Item-Category(I-C) | 2,753 | 22 | 5,508
| Item-Brand(I-B) | 2,753 | 334 | 2,753
Yelp | User-Business(U-B) | 16,239 | 14,284 | 198,397
| Business-City(B-Ci) | 14,285 | 47 | 14,267
| User-User(U-U) | 16,239 | 13269 | 158,590
| User-Compliment(U-Co) | 16,239 | 11 | 76,875
| Business-Category(B-Ca) | 14,284 | 511 | 40,009

(a) ACM

(b) DBLP

Figure 7: Meta-structures designed by GHGNAS.


# Integrating Holistic and Local Information to Estimate Emotional Reaction
Intensity

Yini Fang∗ Hong Kong University of Science and Technology  Liang Wu∗ Hong Kong
University of Science and Technology  Frederic Jumelle Bright Nation Limited
Bertram Shi Hong Kong University of Science and Technology

###### Abstract

Video-based Emotional Reaction Intensity (ERI) estimation measures the
intensity of subjects’ reactions to stimuli along several emotional dimensions
from videos of the subject as they view the stimuli. We propose a multi-modal
architecture for video-based ERI combining video and audio information. Video
input is encoded spatially first, frame-by-frame, combining features encoding
holistic aspects of the subjects’ facial expressions and features encoding
spatially localized aspects of their expressions. Input is then combined
across time: from frame-to-frame using gated recurrent units (GRUs), then
globally by a transformer. We handle variable video length with a regression
token that accumulates information from all frames into a fixed-dimensional
vector independent of video length. Audio information is handled similarly:
spectral information extracted within each frame is integrated across time by
a cascade of GRUs and a transformer with regression token. The video and audio
regression tokens’ outputs are merged by concatenation, then input to a final
fully connected layer producing intensity estimates. Our architecture achieved
excellent performance on the Hume-Reaction dataset in the ERI Esimation
Challenge of the Fifth Competition on Affective Behavior Analysis in-the-Wild
(ABAW5). The Pearson Correlation Coefficients between estimated and subject
self-reported scores, averaged across all emotions, were 0.455 on the
validation dataset and 0.4547 on the test dataset, well above the baselines.
The transformer’s self-attention mechanism enables our architecture to focus
on the most critical video frames regardless of length. Ablation experiments
establish the advantages of combining holistic/local features and of multi-
modal integration. Code available at https://github.com/HKUST-NISL/ABAW5.

11footnotetext: The first two authors contributed equally in the paper.

##  1 Introduction

Attention to human affective behaviour analysis has increased in recent years,
due to the multitude of its applications to fields such as robotics, human
computer interaction (HCI) and psychology. The fifth Competition on Affective
Behaviour Analysis in-the-wild (ABAW5) focuses on affective behaviour analysis
in-the-wild. Solutions to its proposed challenges can be used to create
systems that can understand people’s feelings, emotions and behaviours, as
well as machines and robots that can serve as ‘human-centered’ digital
assistants.

The Emotional Reaction Intensity (ERI) Estimation Challenge in ABAW5 addresses
one of the contemporary affective computing problems using the Hume-Reaction
dataset [5], where subjects react to a wide range of emotional video stimuli
while being observed by a webcam in their homes. After viewing each video,
subjects self-annotate the intensity (over a range from 1 to 100) of their
emotional reactions to it along seven dimensions (Adoration, Amusement,
Anxiety, Disgust, Empathetic Pain, Fear, and Surprise).

Many previous approaches to video-based ERI estimation follow a similar
pattern for representing video information, using features taken from among
the final layers of a deep network that has been pre-trained on a large
dataset, such as AffectNet [12]. AffectNet is annotated for holistic
judgements of emotional facial expressions: categorization into one of eight
emotion classes (neutral, happy, angry, sad, fear, surprise, disgust,
contempt) and estimates of the intensity of valence and arousal.Thus, features
extracted from later layers encode holistic judgements made by integrating
information across the entire face. We hypothesize that such features may not
encode more subtle, spatially localized changes in facial geometry, that may
be important for estimating reaction intensity when subjects are simply
viewing stimuli by themselves,

One way of labelling spatially localized changes is the Facial Action Coding
System [7], which defines a set of action units (AUs) that taxonomizes the
configuration of small groups of human facial muscles by their appearance on
the face. Action units are spatially localized. For example, facial units
include “inner brow raiser”, “chin raiser”, “lip corner puller”, etc. There
are 28 main AUs, but most databases are annotated for only a subset (e.g., 12
or 17) of these. Annotations can either be binary (indicating occurrence) or
continuous (indicating intensity). As AUs are spatially localized, we
hypothesize that they may encode information that is complementary to that
encoded by later layers of a deep net trained on AffectNet, yet that is
critical for ERI.

There are two main issues we address here in estimating a single emotional
intensity along each emotional dimension in response to an entire video.
First, videos can be of varying length. For example, in the Hume-Reaction
dataset, videos range in length from 9.9 seconds to 14.98 seconds. Second,
many of the frames within the video are images of the subject with a neutral
expression, as it is often that only isolated moments in the video evoke
changes in expression. Unfortunately, these moments are not known in advance.
The unpredictability and sparsity of these events suggest that some common
methods for extracting fixed length representations of a variable length video
for input to a final classifier, such as randomly or uniformly sampling a
fixed number of frames, or by average pooling across all frames in the video,
are not appropriate, as they may miss or dilute the impact of critical frames.
We address this problem through the use of a transformer architecture, where
each frame corresponds to an input token, but we also add a regression token
similar to the class token used in the Visual Transformer [6], which gathers
information from all frames with weighting by an attention value. Using the
output of the regression token as input to the final classification stage
ensures a fixed dimensional representation that can be computed by combining
information from all frames of a variable length video. The attention
weighting avoids the problem of dilution by many frames containing neutral
expressions, which might be problematic, especially in longer videos.

This paper proposes a multimodal architecture for ERI estimation that includes
the two approaches outlined above. Our architecture uses both video and audio
features, extracted from pre-trained networks (for video) or handcrafted
algorithms (for audio). To the best of our knowledge, this is the first time
that separate representations of holistic (AffectNet based) and local (AU-
based) visual features have been combined for ERI estimation. Temporal
information is extracted in two stages, first by a GRU, which operates
sequentially from video or audio frame-to-frame, and second by a transformer
with regression token, which combines information globally and in parallel.
Our network adopts a late fusion architecture, where video and audio
information is combined after separate temporal aggregation stages, just
before the final classification layer. However, our architecture can be easily
modified for early fusion, a promising direction for future development. Our
architecture achieved excellent performance compared to baseline,
outperforming the multimodal baseline by 91.02% on validation set and by
124.1% on test set.

##  2 Related Work

Emotional Reaction Intensity Estimation with the same dataset has also been
presented in a Hume-Reaction MuSe 2022 [5] sub-challenge. The FaceRNET [10]
was the best-performing model. It uses a CNN-RNN model to capture spatial-
temporal correlations of all the frames in the video, with a routing mechanism
on top. Wang et al. [14] proposed a spatiotemporal transformer architecture
for dynamic facial representation learning and a multi-label graph
convolutional network for emotion dependency modelling. ViPER [13] leverage
the video’s multimodal nature, and propose a transformer-based model to
combine video frames, audio recordings, and generated textual annotations.
These work neglect the importance of Action Unit feature in ERI estimation,
and can be further improved by incorporating AU occurrence and intensity. The
Hume-Reaction MuSe 2022 challenge also reported results from a baseline system
that used Py-Feat [9] to detect the occurrence of 20 different AUs in each
frame, and used these 20 binary values as the input to a Long Short-Term
Memory LSTM-RNN. However, they did not investigate the addition of other
visual features besides AUs, as we describe here.

Figure 1: Our architecture consists of two steams: a video stream and an audio
stream. Visual feature includes holistic feature from a pretrained ResNet18
and local feature from AU detection/intensity estimation. Audio feature is
extracted by MFCC. Each stream comprises GRU and transformer encoder to
integrate information temporally. A learnable regression token is added to the
input to the transformer encoder. The regression tokens from two streams are
concatenated and fed into fully connected layer and Sigmoid for the final
prediction. “FC” stands for Fully Connected Layer. “Concat.” stands for
concatenation.

##  3 Problem Formulation

We denote by $\mathbb{V}$ and $\mathbb{A}$ the visual and audio stream of a
reaction video, and by $\mathbb{Y}=\\{y_{1},...,y_{7}\\}$ seven emotional
reaction intensities, representing Adoration, Amusement, Anxiety, Disgust,
Empathic Pain, Fear, and Surprise, respectively.

Our objective is to predict 7 emotional reaction intensities
$\mathbb{X}=\\{x_{1},...,x_{7}\\}$ given $\mathbb{V}$ and $\mathbb{A}$ using
our proposed architecture $\mathcal{M}$, i.e.,

$\mathbb{X}=\mathcal{M}(\mathbb{V},\mathbb{A}),$ (1)

so that the loss $\mathcal{L}(\mathbb{X},\mathbb{Y})$ is minimal.

In the next section, we will explain $\mathcal{M}$ and $\mathcal{L}$ in
detail.

##  4 Methodology

Figure 1 shows our system’s architecture. Given a video, our system outputs 7
emotional reaction intensities. Our architecture is dual stream: including a
video stream and an audio stream. The video-stream is further separated into
dual streams: one for holistic feature extraction (a ResNet18 network trained
on AffectNet) and one for local feature extraction (AU detection/intensity
estimation by OpenFace). Dual stream features are combined by concatenation
before being input to the next stage. In the video stream, we extract
information spatially for each frame first, then integrate across time by a
cascade of a Gated Recurrent Unit (GRU) and a transformer encoder. In the
audio stream, we extract spectral information from frames of length
$\sim$30ms, then follow a similar temporal integration architecture as used in
the video stream. Finally, we fuse visual and audio features by concatenation.
A fully connected layer merges multimodal information to output seven
emotional reaction intensities. We will elaborate these steps in details in
the following.

###  4.1 Visual Feature Extraction

#### Data Preprocessing

To eliminate variability due to head motion, which interferes with facial
expression recognition, we apply face alignment using the OpenFace Toolkit
[3], which detects 68 facial landmarks. We use them to align the face by
linear warping followed by cropping of the face region. Then we resize the
cropped face images to $224\times 224$ pixel resolution.

#### Holistic Spatial Features

We use a ResNet18 network [8] pretrained on AffectNet [12] to extract holistic
information about the general overall impression of the face. AffectNet is a
large facial expression dataset consisting of 0.4 million images, designed for
supervised facial analysis tasks. We use the 512-dimensional feature vector
immediately before the final classification layer, which contains holistic
information about the facial expression.

#### Local Spatial Features

We use the OpenFace AU detection model [2] to extract both the occurrence and
intensity of 17 AUs (e.g., AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU14, AU15,
AU17, AU23, AU25). This results in a 34-dimensional AU feature vector for each
frame. This model is based on appearance (Histograms of Oriented Gradients)
and geometry features (shape parameters and landmark locations). They propose
a person-specific normalization approach, which allows the model to generalize
well on other datasets.

#### Visual Feature Fusion

We fuse holistic and local spatial features by concatenation, resulting in a
546-dimensional visual feature vector.

###  4.2 Audio Feature Extraction

Mel-frequency Cepstral Coefficients (MFCC), the overall shape of the spectral
envelope, have been widely used in speech recognition tasks. We extract them
from $\sim$30ms frames sampled at 16kHz and with a stride of $\sim$16ms with
the Python Librosa toolkit and use them as the basis of our audio feature
representation. From a set of 32 MFCCs per frame, we obtain a sequence of
1024-dimensional audio feature vectors by combining features from 32 adjacent
frames.

###  4.3 Temporal Integration

Each branch cascades a GRU and a transformer encoder to integrate information
from a variable number of frames per video into a final fixed-dimensional
feature vector, which is used for the final prediction.

#### GRU

The GRU operates sequentially, frame to frame, to capture short- and long-term
temporal correlations in the video.

#### Transformer Encoder

A Transformer encoder stacks several blocks. Each block has the same
architecture, containing a multi-head attention mechanism followed by a fully-
connected feed-forward network. Each frame corresponds to one token. We also
add a regression token, with a set of learned embedding parameters. Through
the self-attention mechanism, the regression token gathers information from
the frames in the video. Only the regression token output after the last stage
of the transformer encoder is passed to the next stage.

#### Readout function

We input the concatenation of the video and audio regression token outputs
into the final readout layer, consisting of a fully connected linear layer
followed by a logistic sigmoid output layer, which restricts the output into
[0, 1]. For each video sample $n$, our model outputs 7 scores
$\mathbb{X}^{n}=\\{x_{1}^{n},...,x_{7}^{n}\\}$.

###  4.4 Training

We freeze the weights in ResNet18 and AU detection model for the sake of
training speed. We train the rest of the model using L2 loss:

$\mathcal{L}(\mathbb{X},\mathbb{Y})=\frac{1}{7\times
N}\sum_{i=1}^{7}\sum_{n=1}^{N}(x_{i}^{n}-y_{i}^{n})^{2},$ (2)

where $N$ is the total number of video samples.

##  5 Experimental Results

Figure 2: Attention from the regression token to each frame in the video
(averaged across all heads).

###  5.1 Dataset

The Hume-Reaction dataset consists of 25,067 videos from 2,222 subjects.
Subjects are from two cultures, 1,084 from South Africa and 1,138 from the
United States, aged from 18.5 to 49.0 years old. The reaction videos vary in
terms of resolution, FPS, background, and lightning noise conditions. The
average video length is 11.62 seconds, and the average number of frames per
video is 248.8 frames.

The dataset is partitioned into three sets: 15,806 for training set, 4,657 for
validation set, and 4,604 for testing set.

###  5.2 Evaluation Metric

Pearson’s correlations coefficient (PCC) is used for the evaluation metric. We
calculate the average PCC across the 7 emotional reactions intensities:

$\mathcal{P}_{ERI}=\frac{\sum_{i=1}^{7}\rho_{i}}{7}.$ (3)

And for each emotion, $\rho_{i}$ is defined as:

$\rho_{i}=\frac{\sum_{n=1}^{N}(x_{i}^{n}-\bar{x}_{i})(y_{i}^{n}-\bar{y}_{i})}{\sqrt{\sum_{n=1}^{N}(x_{i}^{n}-\bar{x}_{i})^{2}}\sqrt{\sum_{n=1}^{N}(y_{i}^{n}-\bar{y}_{i})^{2}}},$
(4)

where $\bar{x}$ and $\bar{y}$ are the mean of predictions and labels.

###  5.3 Baselines

We compare our model with the work that we have mentioned in Sec. 2 and the
provided audio/visual baselines. The audio baseline uses DEEPSPECTRUM [1] to
extract features from the audio signal. The visual baseline uses the feature
vector from the last layer of a ResNet50 [8] trained on VGGface2 [4]. Then
LSTM-RNN is used to process these features and output the prediction.

###  5.4 Implementation Settings

We use a 2-layer GRU with the size of a hidden layer being 256. The number of
transformer encoder blocks and the heads in the multi-head attention layer are
both set to 4. A dropout probability of 0.2 was adopted for the transformer
encoder. All our models are implemented using PyTorch and trained on a single
GeForce 3070 GPU.

For the training parameters, we set an initial learning rate as 1e-4 which
decays every 10 epochs by a factor of 0.5. AdamW optimizer with a weight decay
of 0.5 is applied as the optimizer.

Due to the varying quality and noise of the video, there are 1% frames where
no face is detected. We discard these frames. We also discard videos that have
less than 50 valid images in the training. In the testing, the prediction of
such an invalid video is set to be the average seven scores from other valid
videos, in order to remove its impact on the PCC metrics.

###  5.5 Results

Table 1 shows performance comparison on the validation and test sets. Our
model achieved 0.455 on the validation dataset and 0.4547 on the test data
set, surpassing all the existing methods in the table on both validation and
test sets. Compared to multimodal baseline, our model outperforms it by 91.02%
on validation set and by 124.1% on test set, proving the effectiveness of
adding AU features.

Figure 2 shows the attention from the regression token to each frame of one
long video sample and one short video sample in the validation set. The peaks
in the attention curves show that the attention mechanism can identify frames
that have the most informative expression changes. The peaks are quite sparse
in the video, indicating the importance of using all frames for the
estimation, and the ability of the network to avoid dilution of the
information in key frames by neutral expressions present in most of the
frames.

Table 1: PCC Comparison Methods | Val | Test
---|---|---
| Audio baseline
---
(DEEPSPECTRUM) [5]
0.1087 | 0.0741
| Visual baseline
---
(VGGFACE2) [5]
0.2488 | 0.1830
Visual baseline (FAU) [5] | 0.2840 | 0.2801
Multimodal baseline [5] | 0.2382 | 0.2029
Former-DFER+MLGCN [14] | 0.3454 | N/A
ViPER [13] | 0.3025 | 0.2970
FaceRNet [10] | 0.3590 | 0.3607
ResNet18 [11] | 0.3893 | N/A
ResNet18+DEEPSPECTRUM [11] | 0.3968 | N/A
Ours | 0.4550 | 0.4547

##  6 Ablation Study

To study the effect of different types of features (e.g., holistic ResNet18
feature, local AU feature, and audio feature), we conducted experiments with
different combinations on the validation set, shown in Table 2. Using all the
features achieved the best performance. We can also see that incorporating AU
features lead to a boost in performance.

Table 2: Ablation study on different feature types Combination | Validation
---|---
Only audio | 0.299
Only AU | 0.342
Only ResNet18 | 0.362
ResNet18 + AU | 0.377
ResNet18 + audio | 0.438
ResNet18 + AU + audio | 0.455

We also compared the effect of different types of AUs in Table 3. We can see
that using only AU intensity is slightly better than using only occurrence,
while combining both of them yields the best outcome.

Table 3: Ablation study on AU features Combination | Validation
---|---
AU occurrence | 0.452
AU intensity | 0.454
AU occurrence + intensity | 0.455

##  7 Conclusion

This paper describes a multimodal architecture for Emotional Reaction
Intensity estimation. Our results demonstrate the efficacy of including both
holistic and spatially localized information from face images for this task.
Our use of a transformer encoder with a regression token enables the network
to handle videos with varying length in a consistent manner and to handle the
sparsity of frames indicating emotional reactions within both long and short
videos. Our experimental results indicate that the performance of our
architecture significantly outperforms the baselines. We also conducted
ablation experiments to verify the importance of multiple visual cues and
multiple sensory modalities. Our code implementation is available at
https://github.com/HKUST-NISL/ABAW5.

## References

  * [1] Shahin Amiriparian, Maurice Gerczuk, Sandra Ottl, Nicholas Cummins, Michael Freitag, Sergey Pugachevskiy, Alice Baird, and Björn Schuller.  Snore sound classification using image-based deep spectrum features.  2017\.
  * [2] Tadas Baltrušaitis, Marwa Mahmoud, and Peter Robinson.  Cross-dataset learning and person-specific normalisation for automatic action unit detection.  In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 6, pages 1–6. IEEE, 2015\.
  * [3] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency.  Openface 2.0: Facial behavior analysis toolkit.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 59–66. IEEE, 2018.
  * [4] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018.
  * [5] Lukas Christ, Shahin Amiriparian, Alice Baird, Panagiotis Tzirakis, Alexander Kathan, Niklas Müller, Lukas Stappen, Eva-Maria Meßner, Andreas König, Alan Cowen, et al.  The muse 2022 multimodal sentiment analysis challenge: humor, emotional reactions, and stress.  In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pages 5–14, 2022.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [7] Paul Ekman and Wallace V Friesen.  Facial action coding system.  Environmental Psychology & Nonverbal Behavior, 1978.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] Eshin Jolly, Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Matthew Kenny, and Luke J Chang.  Py-feat: Python facial expression analysis toolbox.  arXiv preprint arXiv:2104.03509, 2021.
  * [10] Dimitrios Kollias, Andreas Psaroudakis, Anastasios Arsenos, and Paraskeui Theofilou.  Facernet: a facial expression intensity estimation network.  arXiv preprint arXiv:2303.00180, 2023.
  * [11] Jia Li, Ziyang Zhang, Junjie Lang, Yueqi Jiang, Liuwei An, Peng Zou, Yangyang Xu, Sheng Gao, Jie Lin, Chunxiao Fan, et al.  Hybrid multimodal feature extraction, mining and fusion for sentiment analysis.  In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pages 81–88, 2022.
  * [12] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * [13] Lorenzo Vaiani, Moreno La Quatra, Luca Cagliero, and Paolo Garza.  Viper: Video-based perceiver for emotion recognition.  In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pages 67–73, 2022.
  * [14] Kexin Wang, Zheng Lian, Licai Sun, Bin Liu, Jianhua Tao, and Yin Fan.  Emotional reaction analysis based on multi-label graph convolutional networks and dynamic facial expression recognition transformer.  In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pages 75–80, 2022.


# Electronic and magnetic properties of epitaxial thin film of Nd0.5Ba0.5MnO3

Siddharth Kumar  Department of Physics, Indian Institute of Science, Bengaluru
560012, India  Shashank Kumar Ojha  Department of Physics, Indian Institute of
Science, Bengaluru 560012, India  Ranjan Kumar Patel  Department of Physics,
Indian Institute of Science, Bengaluru 560012, India  Prithwijit Mandal
Department of Physics, Indian Institute of Science, Bengaluru 560012, India
Nandana Bhattacharya  Department of Physics, Indian Institute of Science,
Bengaluru 560012, India  S. Middey<EMAIL_ADDRESS>Department of Physics,
Indian Institute of Science, Bengaluru 560012, India

###### Abstract

Contradictory reports about the electronic and magnetic behavior of bulk
Nd0.5Ba0.5MnO3 with uniform disorder exist in literature. In this work, we
investigated single crystalline, A-site disordered thin films of
Nd0.5Ba0.5MnO3, grown on SrTiO3 substrate by pulsed laser deposition. The
epitaxial growth of these films in the layer-by-layer fashion has been
confirmed by in-situ reflection high energy electron diffraction, atomic force
microscopy, X-ray reflectivity, and X-ray diffraction measurements.
Interestingly, these films are found to be electrically insulating and exhibit
spin-glass behavior at low temperatures. This offer a distinct approach to
study Ba based half-doped manganites and opportunity to further manipulate
competing charge/orbital ordering and ferromagnetism in such systems, through
heterostructure engineering.

Hole doped manganites ($RE_{1-x}AE_{x}$MnO3 where $RE$ and $AE$ correspond to
a rare-earth and alkaline-earth metal, respectively) exhibit several
interesting phenomena like colossal magnetoresistance (CMR), metal-insulator
transition (MIT), charge/orbital ordering (CO/OO), several types of spin
ordering, etc., as a function of carrier doping and electronic bandwidth [1,
2, 3]. The behavior of these phases are highly tunable under the application
of external perturbations such as magnetic field, electric field, pressure,
strain, light, etc. [2, 3, 1, 4, 5, 6, 7, 8, 9, 10]. The parent member
($RE$MnO3) of the series contains Jahn-Teller active Mn3+ ($d^{4}$:
$t_{2g}^{3},e_{g}^{1}$) ions and the Mn3+-O- Mn3+ superexchange interaction is
antiferromagnetic according to Goodenough-Kanamori rule[11]. The partial
replacement of $RE^{3+}$ by $x$ amount of $AE^{2+}$ results in $x$ amount of
Jahn-Teller inactive Mn4+ ions, together with ferromagnetic Mn3+-O-Mn4+ double
exchange interaction. Thus, the chemical doping introduces randomness in the
potential energy, Mn valency, orbital configuration, magnetic interaction,
etc. Hence, the disorder is expected to play significant role in the physics
of CMR manganite. This issue has been examined by synthesizing ordered and
disordered structure [see Fig. 1(a)] of half doped manganite
$RE_{0.5}$Ba0.5MnO3 [12, 13, 14, 15].

Figure 1: (a) Disordered and ordered structure of $RE_{0.5}$Ba0.5MnO3. (b)
Phase diagram of $RE_{0.5}$Ba0.5MnO3 as a function of $RE$. Reproduced with
permission from Phys. Rev. Lett. 90, 177203 (2003). Copyright 2003 American
Physical Society. SG and AF(A) represent spin-glass states, and $A$-type
antiferromagnetic phase, respectively.

The phase diagram of ordered and disordered samples of $RE_{0.5}$Ba0.5MnO3 are
drastically different, as shown in Fig. 1 and summarized in Ref. 12. Within
this series, Nd0.5Ba0.5MnO3 (NBMO) is a very special case as it is located
extremely near the boundary between the competing CO/OO and ferromagnetic (FM)
phases. Interestingly, observed properties of NBMO are strongly dependent on
the preparation methods. Polycrystalline NBMO sample with Nd/Ba ordering along
the $c$ direction undergoes an A-type antiferromagnetic transition around 270
K with a steep change in resistance. Both phases across the transition are
insulating ($d\rho/dT<$0) [12]. Whereas, the single crystalline, ordered NBMO
sample undergoes a metal to insulator transition around 290 K and an
antiferromagnetic transition at 235 K [13, 16]. While the single crystal of
disordered NBMO shows an MIT and ferromagnetic transition at 150 K [12], the
disordered, polycrystalline sample remains insulating below room temperature
and undergoes a spin glass transition around 50 K [14, 15].

Epitaxial stabilization is another route to obtain single crystalline sample,
which further offers several ways (e.g. strain engineering, interface
engineering, geometrical lattice engineering, etc.) to tune the ground state
of complex oxides [17, 18, 19, 20]. Surprisingly, thin film work on Ba-based
half-doped manganite is very limited [21] and the electronic and magnetic
behavior of single crystalline NBMO film has not been reported so far. A
systematic investigation about the fabrication of NBMO film will also provide
crucial guidelines for the epitaxial growth of other members of the
$RE_{0.5}$Ba0.5MnO3 series.

In this work, we report on the layer by layer growth of disordered NBMO film
on SrTiO3 (STO) (0 0 1) substrate by the pulsed laser deposition technique
(PLD). The epitaxial strain is expected to be small ($\epsilon\sim$ +0.23%) as
the lattice constant of disordered NBMO (3.896Å [14]) is close to that of STO
(3.905 Å). A combined characterization using in-situ reflection high energy
electron diffraction (RHEED), atomic force microscopy (AFM), X-ray diffraction
(XRD), X-ray reflectivity (XRR) established the epitaxial growth of the film
with very smooth surface morphology, excellent crystallinity and sharp
film/substrate interface structure. We have found that the epitaxial NBMO
films are insulating below room temperature and the application of an external
magnetic field (up to 9T) does not result in any transition to a metallic
phase. DC magnetic measurements have found that the film undergoes a spin-
glass transition around 24 K.

Epitaxial thin films of NBMO were grown on single terminated STO (001)
substrates by a PLD system. To prepare a single terminated surface, as
received STO single crystals (5$\times$5$\times$0.5 mm3) from Shinkosa, Japan
were cleaned ultrasonically, with acetone and methanol and heated at 950∘ C
for 1 hour within a box furnace. Afterward, the substrates were washed at room
temperature with DI water, acetone and methanol, consecutively using an
ultrasonicator. For the film deposition, a KrF excimer laser ($\lambda$ = 248
nm) at a repetition rate of 2 Hz was used to ablate a ceramic target of
disordered NBMO. NBMO films, 25 uc in thickness, were grown at 870∘ C under a
dynamic oxygen pressure of 150 mTorr with in-situ RHEED monitoring. After
that, the films were capped with few layers of STO at the same deposition
condition to avoid any possible over oxidation of the film once exposed to the
atmosphere. We also grew a 100 uc thick NBMO film and results of that sample
have been shown in supplementary material. The surface morphology of the bare
substrates and films were investigated using a Park systems AFM. A lab-based
Rigaku Smartlab diffractometer was used for XRD and XRR measurements. DC
magnetic measurements were carried out using a Quantum Design SQUID
magnetometer. Electrical resistance measurements were carried out by standard
four-probe van der Pauw technique in an Oxford Integra LLD system using a
Keithley 2450 source measure unit. Ohmic contacts were made using ultrasonic
bonding of aluminum wire. The upper limit of resistivity measurement in our
setup is $\sim$ 5 $\Omega$-cm.

First, we discuss the morphological and structural characterization of the
treated substrate and the grown film. Fig. 2(a) shows an AFM image of the
treated substrate, which clearly shows the presence of large atomically flat
terraces with width about 200 nm and one unit cell height. The surface
roughness within a terrace is around 150 pm. The sharp specular (0 0) and off-
specular (0 $\pm$1) RHEED spots (Fig. 2(b)) also confirms the smooth surface
morphology of the substrate. The intensity of the (0 0) reflection was
recorded to monitor the progress of the growth. The desired layer-by-layer
growth of the NBMO film is confirmed by the observation of a prominent
oscillation of the RHEED intensity, plotted in Fig. 2(c). Streaky RHEED
patterns (Fig. 2(d)-(e)) establish a very smooth surface morphology of the
film (both before and after the deposition of STO capping layers). The
retention of atomically flat terraces with unit cell height in AFM imaging of
the NBMO/STO film (Fig. 2(f)), similar to the STO substrate, further
demonstrates the excellent surface morphology of the sample.

Figure 2:  (a) AFM surface morphology and (b) RHEED image of single terminated
STO (0 0 1) substrate. The oscillation in the intensity of the RHEED specular
spot during the deposition of NBMO layers signifies layer by layer growth.
RHEED pattern after completion of the deposition of (d) NBMO and (e) STO
capping layers. (f) AFM surface morphology of STO capped NBMO film.  Figure 3:
Long range $\omega$-$2\theta$ scan along [00l] direction of NBMO film; (left
inset) shows the XRR of the same film along with the fit; (right inset) zoomed
view of [002] peak, here the film peak (*) and the Kiessig fringes can be seen
clearly.

More information about the film thickness and interface roughness have been
obtained by XRR. Left inset of Fig. 3 shows an experimentally observed
reflectivity curve along with a simulated pattern using GenX program [22] for
a 25 uc NBMO film, capped with STO. The observation of the intensity
oscillations over the whole scan range indicates a highly crystalline nature
and smooth interfaces. Our XRR fitting finds the NBMO layer thickness is
around 98 Å and the interfacial roughness is less than 3 Å. The epitaxial
nature of the NBMO layers on STO can be seen from $\omega$-$2\theta$ XRD scans
[Fig. 3], which consists of sharp substrate peaks, broad film peaks (denoted
by $\star$) (right inset of Fig. 3), and a set of Kiessig fringes. The out of
plane lattice constant ($c$ = 3.885 Å) is smaller than the bulk lattice
constant, as expected for a tetragonal distortion under tensile strain. The
film thickness obtained from XRD (25$\times c$) for this NBMO film is very
close to the value found from XRR analysis.

Figure 4: (a) Resistivity vs temperature plots for various applied magnetic
fields (b) left and right panels show the activated behaviour and 2D VRH
nature of resistivity respectively (c) MR at various temperatures up to $\pm$9
T field.

Having confirmed the excellent structural quality of the films, we
investigated the electrical and magnetic properties of these films. As shown
in the Fig. 4(a), the 25 uc film is insulating within the temperature range of
our measurement (150 K-300 K). Under zero-magnetic field, $\rho$ shows an
activated behaviour within $T$= 300 K to 240 K as shown in left panel of Fig.
4(b). We have found the activated gap to be $\sim$123 meV, which is very close
to the value (131 meV) obtained for a polycrystalline, disordered NBMO sample
[15]. Below 240 K, the resistivity follows two-dimensional Mott variable range
hopping (VRH) (right panel of Fig. 4(b)). The 100 uc film also exhibits
similar electrical transport (shown in supplementary materials).

The application of a magnetic field ($H$) results in a lower $\rho$ but does
not induce a metallic phase (Fig. 4(a)). We have also measured $\rho$ as a
function of $H$ at several fixed temperatures. The variation of
magnetoresistance (MR) [defined as
$\mathrm{MR}(H)\%=\frac{\rho[H]-\rho[H=0]}{\rho[H=0]}\times 100\%$] as a
function of $H$ have been plotted in Fig. 4(c). The negative MR is a very
common feature of $RE_{1-x}AE_{x}$MnO3 [3], and ranges from few to multiple
orders of magnitude, depending on whether it is related to double-exchange
mechanism (increase of electronic bandwidth due to alignment of Mn spins in an
external magnetic field) [23] or field induced melting of charge-ordering (CO)
[24]. The second scenario is discarded for disordered $RE_{0.5}$Ba0.5MnO3
compounds with lower bandwidth as the uniform disorder breaks the CO/OO
correlation down to nanometer scale [25].

Disordered $RE_{0.5}$Ba0.5MnO3 with short-range CO/OO exhibits spin-glass
behavior [25]. Fig 5 (a) shows the temperature dependence of zero-field cooled
(ZFC) and field cooled (FC) susceptibility ($\chi$). For the ZFC (FC)
measurement, the sample was cooled from 300 K to 5 K in the absence of an
external magnetic field (in presence of $H$ = 5000 Oe) and the magnetization
was recorded during the heating run under the application of $H$ = 5000 Oe in
both cases. A constant diamagnetic background arising from the STO substrate
is subtracted for evaluating $\chi$ of NBMO film. We observe a bifurcation of
ZFC-FC curves near 45 K and a cusp in ZFC curve at 24 K. These are
characteristic of a spin glass transition, which were also observed in
previous studies on the polycrystalline NBMO samples [14, 15]. Fig 5(b) shows
the $M$ vs. $H$ curve of the film after subtracting the diamagnetic
contribution of the STO substrate. A small hysteresis is observed, with a
coercive field of 400 Oe. Magnetic hysteresis is observed in ferromagnets,
spin glasses, superparamagnets, etc., and Arrott plot is a conventional way to
distinguish ferromagnetism from other phases [26]. To obtain these we have
plotted $M^{2}$ as a function of $H/M$ in Fig 5(c). The intercept of the
extrapolated $M^{2}$ (denoted by the red line in Fig 5(c)) is on the positive
$H/M$ axis, which establishes the absence of any ferromagnetism in our NBMO
sample.

The spin glass behavior below 24 K is further confirmed by isothermal remanent
magnetization (IRM) measurement. For this, the film was cooled to 10 K from
room temperature in zero field and a constant magnetic field of 1000 Oe was
applied for 10 minutes. DC moment was recorded as a function of time ($t$)
just after switching off the field. The $M_{IRM}$ exhibits a slow decay [Fig 5
(d)], indicative of glassy behaviour. Moreover, $M_{IRM}$ can be fitted using
a logarithmic function $M_{IRM}=M_{0}-Sln(1+t/t_{0})$ ( red curve in Fig 5(d))
[27] similar to other spin glasses [28, 27]. The viscosity coefficient $S$ is
0.079 emu/cm3.

Figure 5: (a) Magnetic susceptibility vs temperature (b) $M-H$ curve at 5 K
(c) Arrrot plot corresponding to the $M-H$, the red line is a guide to the
intercept of the curve to the x-axis (d) isothermal remnant magnetisation at
10 K along with the logarithmic relaxation fit as discussed in the text.

In summary, we have grown and investigated epitaxial thin film of disordered
NBMO on STO (001) substrate. A combination of RHEED, AFM, XRR and XRD has
confirmed the excellent morphological and structural quality of these films.
In sharp contrast to a MIT and ferromagnetism, observed in single crystal of
A-site disordered NBMO, our films are insulating and become spin-glass at low
temperature. These differences might be related with several factors such as
electronic reconstruction due to interfacial polarity mismatch, finite
thickness effect etc. [29, 30, 31] and need to be explored further. This
present work also opens the prospect of studying phase competition in Ba-based
half-doped manganites through variation of epitaxial strain, interface
engineering and geometrical lattice engineering.

See the supplementary material for the XRD and transport data of a 100 uc
thick NBMO film.

The authors acknowledge AFM, XRD and SQUID facilities at the Department of
Physics, IISc Bangalore. SM acknowledges DST Nanomission grant
(DST/NM/NS/2018/246) and Infosys Foundation, Bangalore for financial support.

## References

  * Imada, Fujimori, and Tokura [1998] M. Imada, A. Fujimori, and Y. Tokura, Rev. Mod. Phys. 70, 1039 (1998).
  * Dagotto, Hotta, and Moreo [2001] E. Dagotto, T. Hotta, and A. Moreo, Physics Reports 344, 1 (2001).
  * Tokura [2006] Y. Tokura, Reports on Progress in Physics 69, 797 (2006).
  * Kuwahara  _et al._ [1995] H. Kuwahara, Y. Tomioka, A. Asamitsu, Y. Moritomo, and Y. Tokura, Science 270, 961 (1995).
  * Hwang  _et al._ [1995] H. Y. Hwang, S.-W. Cheong, P. G. Radaelli, M. Marezio, and B. Batlogg, Phys. Rev. Lett. 75, 914 (1995).
  * Miyano  _et al._ [1997] K. Miyano, T. Tanaka, Y. Tomioka, and Y. Tokura, Phys. Rev. Lett. 78, 4257 (1997).
  * Asamitsu  _et al._ [1997] A. Asamitsu, Y. Tomioka, H. Kuwahara, and Y. Tokura, Nature 388, 50 (1997).
  * Ward  _et al._ [2009] T. Z. Ward, J. D. Budai, Z. Gai, J. Z. Tischler, L. Yin, and J. Shen, Nature Physics 5, 885 (2009).
  * Beaud  _et al._ [2014] P. Beaud, A. Caviezel, S. Mariager, L. Rettig, G. Ingold, C. Dornes, S. Huang, J. Johnson, M. Radovic, T. Huber, _et al._ , Nature materials 13, 923 (2014).
  * Zhang  _et al._ [2016] J. Zhang, X. Tan, M. Liu, S. W. Teitelbaum, K. W. Post, F. Jin, K. A. Nelson, D. N. Basov, W. Wu, and R. D. Averitt, Nature materials 15, 956 (2016).
  * Goodenough [1955] J. B. Goodenough, Phys. Rev. 100, 564 (1955).
  * Akahoshi  _et al._ [2003] D. Akahoshi, M. Uchida, Y. Tomioka, T. Arima, Y. Matsui, and Y. Tokura, Phys. Rev. Lett. 90, 177203 (2003).
  * Yamada  _et al._ [2017] S. Yamada, H. Sagayama, K. Higuchi, T. Sasaki, K. Sugimoto, and T. Arima, Phys. Rev. B 95, 035101 (2017).
  * Trukhanov  _et al._ [2002] S. V. Trukhanov, I. O. Troyanchuk, M. Hervieu, H. Szymczak, and K. Bärner, Phys. Rev. B 66, 184424 (2002).
  * Nakajima, Yoshizawa, and Ueda [2004] T. Nakajima, H. Yoshizawa, and Y. Ueda, Journal of the Physical Society of Japan 73, 2283 (2004), https://doi.org/10.1143/JPSJ.73.2283 .
  * Yamada  _et al._ [2019] S. Yamada, N. Abe, H. Sagayama, K. Ogawa, T. Yamagami, and T. Arima, Phys. Rev. Lett. 123, 126602 (2019).
  * Schlom  _et al._ [2008] D. G. Schlom, L.-Q. Chen, X. Pan, A. Schmehl, and M. A. Zurbuchen, Journal of the American Ceramic Society 91, 2429 (2008).
  * Hwang  _et al._ [2012] H. Y. Hwang, Y. Iwasa, M. Kawasaki, B. Keimer, N. Nagaosa, and Y. Tokura, Nature Mater. 11, 103 (2012).
  * Chakhalian  _et al._ [2014] J. Chakhalian, J. W. Freeland, A. J. Millis, C. Panagopoulos, and J. M. Rondinelli, Rev. Mod. Phys. 86, 1189 (2014).
  * Middey  _et al._ [2016] S. Middey, J. Chakhalian, P. Mahadevan, J. W. Freeland, A. J. Millis, and D. D. Sarma, Annual Review of Materials Research 46, 305 (2016).
  * Nakajima  _et al._ [2007] T. Nakajima, T. Tsuchiya, K. Daoudi, M. Ichihara, Y. Ueda, and T. Kumagai, Chemistry of Materials 19, 5355 (2007), https://doi.org/10.1021/cm071118i .
  * Bjorck and Andersson [2007] M. Bjorck and G. Andersson, Journal of Applied Crystallography 40, 1174 (2007).
  * Urushibara  _et al._ [1995] A. Urushibara, Y. Moritomo, T. Arima, A. Asamitsu, G. Kido, and Y. Tokura, Phys. Rev. B 51, 14103 (1995).
  * Tomioka  _et al._ [1996] Y. Tomioka, A. Asamitsu, H. Kuwahara, Y. Moritomo, and Y. Tokura, Phys. Rev. B 53, R1689 (1996).
  * Mathieu  _et al._ [2004] R. Mathieu, D. Akahoshi, A. Asamitsu, Y. Tomioka, and Y. Tokura, Phys. Rev. Lett. 93, 227202 (2004).
  * Yeung, Roshko, and Williams [1986] I. Yeung, R. M. Roshko, and G. Williams, Phys. Rev. B 34, 3456 (1986).
  * Middey  _et al._ [2011] S. Middey, S. Ray, K. Mukherjee, P. L. Paulose, E. V. Sampathkumaran, C. Meneghini, S. D. Kaushik, V. Siruguri, K. Kovnir, and D. D. Sarma, Phys. Rev. B 83, 144419 (2011).
  * Majumdar  _et al._ [1999] S. Majumdar, E. Sampathkumaran, D. Eckert, A. Handstein, K. Müller, S. Saha, H. Sugawara, and H. Sato, Journal of Physics: Condensed Matter 11, L329 (1999).
  * Garcia-Barriocanal  _et al._ [2010] J. Garcia-Barriocanal, J. Cezar, F. Y. Bruno, P. Thakur, N. Brookes, C. Utfeld, A. Rivera-Calzada, S. Giblin, J. Taylor, J. Duffy, _et al._ , Nature communications 1, 1038 (2010).
  * Lee  _et al._ [2010] J.-S. Lee, D. A. Arena, P. Yu, C. S. Nelson, R. Fan, C. J. Kinane, S. Langridge, M. D. Rossell, R. Ramesh, and C.-C. Kao, Phys. Rev. Lett. 105, 257204 (2010).
  * Bruno  _et al._ [2011] F. Y. Bruno, J. Garcia-Barriocanal, M. Varela, N. M. Nemes, P. Thakur, J. C. Cezar, N. B. Brookes, A. Rivera-Calzada, M. Garcia-Hernandez, C. Leon, S. Okamoto, S. J. Pennycook, and J. Santamaria, Phys. Rev. Lett. 106, 147205 (2011).