# Hybrid Graph: A Unified Graph Representation with Datasets and Benchmarks
for Complex Graphs

Zehui Li 1, Xiangyu Zhao11footnotemark: 1 2, Mingzhu Shen2, Guy-Bart Stan1,
Pietro Liò3, Yiren Zhao2
1Department of Bioengineering, Imperial College London
2Department of Electrical and Electronic Engineering, Imperial College London
3Department of Computer Science and Technology, University of Cambridge
Equal contribution. Correspondence to<EMAIL_ADDRESS>
###### Abstract

Graphs are widely used to encapsulate a variety of data formats, but real-
world networks often involve complex node relations beyond only being
pairwise. While hypergraphs and hierarchical graphs have been developed and
employed to account for the complex node relations, they cannot fully
represent these complexities in practice. Additionally, though many Graph
Neural Networks (GNNs) have been proposed for representation learning on
higher-order graphs, they are usually only evaluated on simple graph datasets.
Therefore, there is a need for a unified modelling of higher-order graphs, and
a collection of comprehensive datasets with an accessible evaluation framework
to fully understand the performance of these algorithms on complex graphs. In
this paper, we introduce the concept of _hybrid graphs_ , a unified definition
for higher-order graphs, and present the _Hybrid Graph Benchmark (HGB)_. HGB
contains 23 real-world hybrid graph datasets across various domains such as
biology, social media, and e-commerce. Furthermore, we provide an extensible
evaluation framework and a supporting codebase to facilitate the training and
evaluation of GNNs on HGB. Our empirical study of existing GNNs on HGB reveals
various research opportunities and gaps, including (1) evaluating the actual
performance improvement of hypergraph GNNs over simple graph GNNs; (2)
comparing the impact of different sampling strategies on hybrid graph learning
methods; and (3) exploring ways to integrate simple graph and hypergraph
information. We make our source code and full datasets publicly available at
https://zehui127.github.io/hybrid-graph-benchmark/.

##  1 Introduction

Graphs are powerful tools for capturing relationships between objects, ranging
from social networks, biology (Rozemberczki et al.,, 2021) to e-commerce
(McAuley et al.,, 2015; He and McAuley,, 2016; Ni et al.,, 2019). However,
real-world large-scale networks often involve complex node relations beyond
only being pairwise. While hypergraphs and hierarchical graphs (McCallum et
al.,, 2000; Getoor,, 2005; Sen et al.,, 2008; Chien et al.,, 2022) have been
introduced to capture these complex node relations, they still cannot fully
represent the real-world scenarios. Although hypergraphs can describe
clustering relations between multiple nodes, they are still limited in their
ability to represent hierarchical relations. On the other hand, while
hierarchical graphs can make up for the limitation of hypergraphs, they are
not flexible enough to represent multi-node relations.

Existing Graph Neural Networks (GNNs) have been proposed for representation
learning on simple graphs (Kipf and Welling,, 2017; Hamilton et al.,, 2017;
Veličković et al.,, 2018; Li et al.,, 2019; Brody et al.,, 2022) and higher-
order graphs (Feng et al.,, 2019; Yadati et al.,, 2019; Chan and Liang,, 2020;
Dong et al.,, 2020; Zeng et al.,, 2020; Bai et al.,, 2021; Brody et al.,,
2022). However, they are usually only evaluated on simple graph datasets,
which do not fully reflect the complexity of real-world graph structures.
Besides, without a unified modeling of higher-order graphs, nor a collection
of comprehensive datasets with an accessible evaluation framework, we cannot
fully uncover the underlying performance of these GNNs on complex graphs.

To address these limitations, we endeavour to construct the _Hybrid Graph
Benchmark (HGB)_ , with a unified view of complex graphs including both
datasets and an evaluation framework. Firstly, we introduce the concept of
_hybrid graphs_ , which provides a unified definition for higher-order graphs.
Hybrid graphs extend the notion of simple graphs by allowing nodes to be
connected to sets of other nodes, rather than just individual nodes. This
enables hybrid graphs to capture more complex node interactions, including
simple and higher-order interactions. Based on this unified definition, we
build 23 real-world hybrid graph datasets across various domains such as
biology, social media, and e-commerce. Compared with other existing hypergraph
datasets (McCallum et al.,, 2000; Getoor,, 2005; Sen et al.,, 2008; Chien et
al.,, 2022) that are on a relatively small scale and do not facilitate
hierarchical node relations, we believe our HGB represents a significant step
forward in the development of comprehensive datasets for evaluating higher-
order graph learning algorithms.

Furthermore, to facilitate fair evaluation of GNNs on our proposed hybrid
graph datasets, we provide an extensible evaluation framework and a supporting
codebase. Our evaluation framework includes several common graph prediction
tasks using their corresponding evaluation metrics, such as node
classification and regression tasks. We benchmark seven widely-used GNN models
(Kipf and Welling,, 2017; Hamilton et al.,, 2017; Veličković et al.,, 2018;
Brody et al.,, 2022; Zeng et al.,, 2020; Bai et al.,, 2021; Brody et al.,,
2022) on HGB, and introduce three novel baselines that incorporate simple and
higher-order graph information, thereby enabling the researchers to
conveniently evaluate their own models and compare the results. In addition,
our empirical study of existing GNNs on HGB reveals various research
opportunities and gaps, which are elaborated in later sections.

To summarise, we provide a comprehensive and unified framework for modelling
and evaluating hybrid graph methods, in the hope of stimulating further
research in this field. The source code and complete datasets of HGB are
publicly available. Our main contributions in this paper are as follows:

  * •

We introduce the _hybrid graphs_ , a unified view of high-order graphs for
fostering further study in representation learning on complex graphs.

  * •

Inspired by this unified mathematical framework, we construct the _Hybrid
Graph Benchmark (HGB)_ consisting of 23 datasets covering a wide range of
real-world applications. We then extend HGB with an easy-to-use and extensible
evaluation framework.

  * •

Through extensive experimentation, we have verified both the necessity and
superiority of our proposed datasets and benchmarking tool. We also draw
insights for the graph representation learning community, such as (1) existing
hypergraph GNNs may not keep outperforming simple graph GNNs on large-scale
networks; (2) appropriate sampling strategies improve the performance of GNNs
on higher-order graphs; and (3) integrating simple and higher-order graph
information can significantly enhance the prediction performance on complex
graphs.

##  2 Related Work

#### Graph Neural Networks on Simple Graphs

Graph Neural Networks (GNNs) on simple graphs encode the nodes through neural
networks, and learn the representations of the nodes through message-passing
within the graph structure. GCNs (Kipf and Welling,, 2017) incorporate the
convolution operation into GNNs. GAT (Veličković et al.,, 2018) and GATv2
(Brody et al.,, 2022) are another family of GNN variants that improves the
expressive power of GNNs through attention mechanisms. GraphSAGE (Hamilton et
al.,, 2017) is a general inductive framework that leverages node information
to efficiently generate node embeddings for previously unseen data. GraphSAINT
(Zeng et al.,, 2020) emphasises the importance of graph sampling-based
inductive learning method to improve training efficiency, especially for large
graphs. While these models succeed in simple graph datasets, it is also of
great research interest to test their performance on higher-order graphs.
However, there lacks a systematic evaluation of these models on higher-order
graph datasets.

#### Graph Neural Networks for Higher-Order Graphs

Higher-order graphs are designed to capture more complex node relations, with
hypergraphs being the dominant practice in deep graph representation learning.
Hypergraphs refer to graphs in which an edge can connect two or more nodes. In
general, GNNs for hypergraphs optimise the node representation through a two-
step process. Initially, the node embeddings within each hyperedge are
aggregated to form a hidden embedding of each hyperedge. Subsequently, the
hidden embeddings of hyperedges with common nodes are aggregated to update the
representations of their common nodes. Both HGNN (Feng et al.,, 2019) and
HyperConv (Bai et al.,, 2021) precisely follow this process. The
expressiveness of hypergraph GNNs could be enhanced by modifying this
procedure. For instance, HyperGCN (Yadati et al.,, 2019) refines the node
aggregation within hyperedges using mediators (Chan and Liang,, 2020);
HyperAtten (Bai et al.,, 2021) proposes to use real value to measure the
degree to which a vertex belongs to a hyperedge; HNHN (Dong et al.,, 2020)
applies nonlinear functions to both node and edge aggregation processes.

Hierarchical graphs refer to graphs where nodes are organised into multiple
levels, and exhibit hierarchical relations. Compared to hypergraphs,
hierarchical graphs are less explored in higher-order graph representation
learning. To integrate node information across different levels in the
hierarchy, HACT-Net (Pati et al.,, 2022) treats nodes in different levels as
separate subgraphs. In HACT-Net, the input features of the lower-level nodes
are first passed through a GNN to produce the lower-level node embeddings.
Then, the computed lower-level node embeddings are concatenated with the input
features of the higer-level nodes, and are passed through another GNN to
produce the higher-level node embeddings. In comparison, HGN (Fang et al.,,
2020) treats the entire hierarchical graph as a whole, by concatenating the
precomputed encodings of all levels into the node features, and passing them
into a single GNN. There is also a group of simple graph methods (Ying et
al.,, 2018; Lee et al.,, 2019; Gao and Ji,, 2019) that propose to learn
hierarchical relations in their GNN layers through graph pooling, but they do
not use hierarchical graph modelling in their input graph structures.

#### Existing Higher-Order Graph Datasets

All above-mentioned hypergraph learning methods are evaluated on hypergraphs
constructed from citation networks, such as Cora (McCallum et al.,, 2000),
CiteSeer (Getoor,, 2005), PubMed (Sen et al.,, 2008) and
DBLP111https://dblp.org/xml/release/. Hyperedges are constructed using either
of the two ways: co-citation (i.e., articles are grouped in the same hyperedge
if they cite the same article) and co-authorship (i.e., articles are grouped
in the same hyperedge if they share the same author). However, hypergraphs
constructed from citation networks suffer from two severe disadvantages,
making them inadequate evaluation metrics: (1) citation networks are too small
in size and are prone to overfitting; and (2) the way hyperedges are
constructed leads to substantial overlaps between hyperedges, thereby limiting
their effectiveness in capturing multi-node relations. Chien et al., (2022)
propose several more hypergraph datasets including adaptations of
Yelp222https://www.yelp.com/dataset, Walmart (Amburg et al.,, 2020) and House
(Chodrow et al.,, 2021), but these datasets are still relatively small-scale
and have not been widely adopted by the hypergraph learning community.

##  3 Hybrid Graph Datasets

###  3.1 Definition of Hybrid Graphs

Formally, a simple graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is a
collection of nodes $\mathcal{V}$ and edges
$\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ between pairs of nodes. And
this graph abstraction assumes that each edge only connects two nodes.
However, as discussed in the previous sections, many real-world networks have
more complex node relations than just pairwise relations. Attempts made to
capture such complex relations include hypergraphs (McCallum et al.,, 2000;
Getoor,, 2005; Sen et al.,, 2008; Chien et al.,, 2022), where a hyperedge can
connect more than two nodes, and hierarchical graphs (Lemons et al.,, 2011),
where nodes are organised into multiple levels. A hypergraph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$ is defined by a set of nodes
$\mathcal{V}$ and a set of hyperedges $\mathcal{E}$, where edges
$e\in\mathcal{E}$ are arbitrary subsets of $\mathcal{V}$. A hierarchical graph
is a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with an acyclic parent
function $f_{p}:\mathcal{V}\to\mathcal{V}$, which defines the node hierarchy.
Both hypergraph and hierarchical graph capture more complex node relations
than a simple graph, but they still cannot fully capture the relations in
real-world applications. Therefore, there is a need for a more general complex
graph abstraction that combines simple graphs, hypergraphs and hierarchical
graphs.

Figure 1: A hybrid graph (left) can (1) contain multiple node levels, where
nodes in adjacent levels can have hierarchical relations, and every node at a
lower hierarchy must belong to a node at a higher hierarchy; and (2) contain
both simple edges (edges that connect exactly two nodes) and hyperedges (edges
can connect two or more nodes). It can be transformed into a hierarchical
graph (upper right), a hypergraph (lower right) or even a simple graph, by
tightening relevant constraints.

We define a _hybrid graph_ as $\mathcal{G}=(\mathcal{V},\mathcal{E},f_{p})$,
where $\mathcal{V}$ is the set of nodes, $\mathcal{E}$ is the set of
hyperedges, and $f_{p}:\mathcal{V}\to\mathcal{V}$ is an acyclic parent
function. Each hyperedge $e\in\mathcal{E}$ is a non-empty subset of
$\mathcal{V}$, which may contain two or more nodes, and is assigned a positive
weight $w(e)$. The parent node $f_{p}(v)$ of a node $v\in\mathcal{V}$ is at
the next level up in the node hierarchy. This unified hybrid graph definition
encloses the traditional definitions of simple graphs, hypergraphs and
hierarchical graphs, making each of them a special type of hybrid graph with
some extra constraints:

  * •

Simple graph: a hybrid graph is a simple graph if and only if

    1. 1.

$\forall e\in\mathcal{E}.\>|e|=2$, i.e., a simple graph can only contain
simple edges;

    2. 2.

$f_{p}$ is a null function, i.e., a simple graph can only contain a single
node level. Therefore, $f_{p}$ can also be omitted in the simple graph
notation.

  * •

Hypergraph: a hybrid graph is a hypergraph if and only if

    1. 1.

$\exists e\in\mathcal{E}.\>|e|\geq 3$, i.e., a hypergraph contains at least
one hyperedge;

    2. 2.

$f_{p}$ is a null function, i.e., a hypergraph can only contain a single node
level. Therefore, $f_{p}$ can also be omitted in the hypergraph notation.

  * •

Hierarchical graph: a hybrid graph is a hierarchical graph if and only if

    1. 1.

$\forall e\in\mathcal{E}.\>|e|=2$, i.e., a hierarchical graph can only contain
simple edges;

    2. 2.

$\exists v\in\mathcal{V}.\>f_{p}(v)\neq v$, i.e., a hierarchical graph
contains at least two node levels;

    3. 3.

In a hierarchical graph, except for the nodes at the highest level, every node
must be connected to another node in the next level up in the node hierarchy.

Figure 1 illustrates the properties of a hybrid graph. In graph representation
learning, a hybrid graph can be represented as a tuple of features
$(\mathbf{X},\mathbf{E},\mathbf{H},\mathbf{W},\mathbf{R})$.
$\mathbf{X}\in\mathbb{R}^{|\mathcal{V}|\times d_{v}}$ is the node feature
matrix of the hybrid graph, with each row
$\mathbf{x}_{v}\in\mathbb{R}^{d_{v}}$ being the $d_{v}$-dimensional features
of node $v$. $\mathbf{E}\in\mathbb{R}^{|\mathcal{E}|\times d_{e}}$ is the
hyperedge feature matrix of the hybrid graph, with each row
$\mathbf{e}_{e}\in\mathbb{R}^{d_{e}}$ being the $d_{e}$-dimensional features
of hyperedge $e$. $\mathbf{H}\in\\{0,1\\}^{|\mathcal{V}|\times|\mathcal{E}|}$
is the node-hyperedge incidence matrix of the hybrid graph, with each entry
$H_{ve}=1$ if $v\in e$ and 0 otherwise. We can optionally separate out the
simple edges from $\mathbf{H}$ and form an adjacency matrix
$\mathbf{A}\in\\{0,1\\}^{|\mathcal{V}|\times|\mathcal{V}|}$, where $A_{uv}=1$
if $\\{u,v\\}\in\mathcal{E}$. Note that simple and hierarchical graphs only
contain adjacency matrices.
$\mathbf{W}\in\mathbb{R}^{|\mathcal{E}|\times|\mathcal{E}|}$ is a diagonal
matrix containing the weights of the hyperedges, with each on-diagonal entry
$W_{ee}=w(e)$. $\mathbf{R}\in\\{0,1\\}^{|\mathcal{V}|\times|\mathcal{V}|}$ is
the parent relation matrix of the hybrid graph, with each entry $R_{uv}=1$ if
$f_{p}(u)=v$ and 0 otherwise. If the nodes are arranged in ascending order of
their levels in the node hierarchy, then $\mathbf{R}$ becomes an upper-
triangular matrix. Note that $\mathbf{W}=\mathbf{I}_{|\mathcal{E}|}$ and
$\mathbf{R}=\mathbf{I}_{|\mathcal{V}|}$ for simple graphs and hypergraphs,
therefore they can be omitted in their representations.

###  3.2 Dataset Construction

Figure 2: Construction of the HGB datasets. The MUSAE datasets are social and
knowledge networks, where the hyperedges are constructed from friend circles
or mutually linked page groups. The GRAND datasets are gene regulatory
networks, where the hyperedges are formed using the positions of genomic
elements on the chromosome. The Amazon datasets are product co-review
networks, where the hyperedges are built from the clusters of the image
embeddings of the products.

The hybrid graph, as a versatile data structure, presents a more comprehensive
view of graph data. It not only captures simple pairwise relationships as a
simple graph does but also retains complex multi-node interactions similar to
hypergraphs and hierarchical graphs. Therefore, it outperforms traditional
graph structures in capturing nuanced data relationships. In light of this, we
introduce the _Hybrid Graph Benchmark (HGB)_ , a novel collection of hybrid
graph datasets consisting of 23 hybrid graphs derived from real-world networks
across varied domains, including biology, social media, and e-commerce. Care
has been taken to ensure that the datasets do not contain any personally
identifiable information. Most of these datasets are one-level hybrid graphs
containing both simple edges and hyperedges. Meanwhile, if we treat the
hyperedges as higher-level virtual nodes representing collections of nodes,
then our constructed datasets can also be interpreted as hybrid graphs with a
two-level node hierarchy. Figure 2 provides an overview of these graphs and
their construction mechanisms. Table 1 reports the key hybrid graph statistics
for each dataset group. The HGB datasets can be splitted into three groups
according to their construction processes:

#### MUSAE

We build eight social networks derived from the Facebook pages, GitHub
developers and Twitch gamers, plus three English Wikipedia page-page networks
on specific topics (chameleons, crocodiles and squirrels) based on the MUSAE
(Rozemberczki et al.,, 2021) datasets. Nodes represent users or articles, and
edges are mutual followers relationships between the users, or mutual links
between the articles. In addition to the original MUSAE datasets, we construct
the hyperedges to be mutually connected sub-groups that contain at least three
nodes (i.e., maximal cliques with sizes of at least 3). We also enable each
dataset to have an option to use either the raw node features, or the
preprocessed node embeddings as introduced in MUSAE.

#### GRAND

We select and build ten gene regulatory networks in different tissues and
diseases from GRAND (Ben Guebila et al.,, 2022), a public database for gene
regulation. Nodes represent gene regulatory elements (Maston et al.,, 2006)
with three distinct types: protein-encoding gene, lncRNA gene (Long et al.,,
2017), and other regulatory elements. Edges are regulatory effects between
genes. The task is a multi-class classification of gene regulatory elements.
We train a CNN (Eraslan et al.,, 2019) and use it to take the gene sequence as
input and create a 4,651-dimensional embedding for each node. The hyperedges
are constructed by grouping nearby genomic elements on the chromosomes, i.e.,
the genomic elements within 200k base pair distance are grouped as hyperedges.

#### Amazon

Following existing works on graph representation learning on e-commerce
networks (Shchur et al.,, 2018; Zeng et al.,, 2020), we further build two
e-commerce hypergraph datasets based on the Amazon Product Reviews dataset
(McAuley et al.,, 2015; He and McAuley,, 2016; Ni et al.,, 2019). Nodes
represent products, and an edge between two products is established if a user
buys these two products or writes reviews for both. However, unlike those
existing datasets, we introduce the image modality into the construction of
hyperedge. To be specific, the raw images are fed into a CLIP (Radford et
al.,, 2021) classifier, and a 512-dimensional feature embedding for each image
is returned to assist the clustering. The hyperedges are then constructed by
grouping products whose image embeddings’ pairwise distances are within a
certain threshold.

Table 1: Aggregated dataset statistics of HGB.

Name | #Graphs |  Avg. #Nodes |  Avg. #Edges |  Avg. #Hyperedges |  Avg. Node Degree |  Avg. Hyperedge Degree |  Avg. Clustering Coef.
---|---|---|---|---|---|---|---
MUSAE-GitHub | 1 | 37,700 | 578,006 | 223,672 | 30.7 | 4.6 | 0.168
MUSAE-Facebook | 1 | 22,470 | 342,004 | 236,663 | 30.4 | 9.9 | 0.360
MUSAE-Twitch | 6 | 5,686 | 143,038 | 110,142 | 50.6 | 6.0 | 0.210
MUSAE-Wiki | 3 | 6,370 | 266,998 | 118,920 | 88.8 | 14.4 | 0.413
GRAND-Tissues | 6 | 5,931 | 5,926 | 11,472 | 2.0 | 1.3 | 0.000
GRAND-Diseases | 4 | 4,596 | 6,252 | 7,743 | 2.7 | 1.3 | 0.000
Amazon-Computers | 1 | 10,226 | 55,324 | 10,226 | 10.8 | 4.0 | 0.249
Amazon-Photos | 1 | 6,777 | 45,306 | 6,777 | 13.4 | 4.8 | 0.290

##  4 Evaluation Framework

###  4.1 Overview

Figure 3: An overview of the HGB framework. (a) The HGB datasets include 23
real-world hybrid graphs, which preserve both simple and higher-order graph
structures. (b) We also build a lightweight training-test framework using
PyTorch Lightning and PyTorch Geometric.

We create an extensible evaluation framework in HGB, simplifying the process
of training and assessing GNNs for both simple graphs and hypergraphs. Figure
3 illustrates the key components of HGB. 23 hybrid graphs are used to train
and evaluate the seven GNNs, including four GNNs for simple graphs: GCN (Kipf
and Welling,, 2017), GraphSAGE (Hamilton et al.,, 2017), GAT (Veličković et
al.,, 2018), GATv2 (Brody et al.,, 2022); two hypergraph GNNs: HyperConv,
HyperAtten (Bai et al.,, 2021); and one sampling-based training strategy:
GraphSAINT (Zeng et al.,, 2020). To fairly compare these models, we evaluate
them under the same hyperparameter settings, which are listed in Appendix B.
We repeat each experiment 5 times with different random seeds, and report
their means and standard deviations in Appendix C. The experiments are
evaluated with the accuracy for the node classification task and the mean
square error (MSE) for node regression tasks. Our results highlight the
challenges and research gaps in developing effective GNNs for real-world
hybrid graphs:

  * •

Existing hypergraph GNNs may not outperform simple graph GNNs, even when the
hyperedges provide meaningful information to the task. (Section 4.2)

  * •

The performance of conventional hypergraph GNNs can be improved with graph
samplers that samples the node mainly based on the simple edge information.
(Section 4.3)

  * •

Combining both simple and higher-order graph information can be substantially
beneficial for node classification and regression tasks in higher-order
graphs. (Section 4.4)

###  4.2 Assessing Performance: Hypergraph GNNs vs. Simple Graph GNNs on HGB

We report the mean accuracies of four simple graph GNNs (GCN, GraphSAGE, GAT,
GATv2) and two hypergraph GNNs (HyperConv, HyperAtten) on HGB in Appendix C.
For MUSAE and GRAND, the information embedded in the hypergraph space and
simple graph space do not contribute to the task objective. For the Amazon
datasets, since the hyperedges are constructed using the embedding of actual
product images, nodes within a hyperedge can provide meaningful information to
the task. Therefore, we expect hypergraph GNNs to outperform simple graph GNNs
on Amazon. However, our experiments show that the performance gain from
hypergraph GNNs on Amazon is only marginal.

Figure 4(a) shows a pairwise comparison of two types of GNNs. In both MUSAE
and GRAND, hypergraph GNNs perform equally or less well than the simple graph
methods, while they perform better than simple graph GNNs on two Amazon Review
graphs. However, Figure 4(b), the aggregated mean accuracy across all graphs
with the standard deviation, indicates that the performance gain of using
hypergraph GNNs may not be significant.

(a)

(b)

Figure 4: Evaluating the accuracy of simple-graph and hypergraph GNNs on HGB:
(a) The scatter plot of the accuracies of hypergraph GNNs with respect to
simple graph GNNs on each dataset. The black dashed line at $y=x$ serves as a
reference line where both GNN types perform equally well, while a dot above
this reference line means hypergraph GNNs perform better than simple graph
GNNs on a given node classification task, and vice versa. The plot indicates
that hypergraph GNNs match simple graph GNNs’ accuracies on GRAND,
underperform on MUSAE, and outperform on Amazon. (b) The bar chart of the
accuracies of both GNN types aggregated within MUSAE, GRAND, and Amazon, shows
that the performance improvement of hypergraph GNNs on Amazon compared to
simple graph GNNs may not be significant, after taking the standard deviations
into consideration.

###  4.3 Optimised Sampling in Hybrid Graph

Various hypergraph sampling strategies are proposed for sampling subgraphs
with the purpose of preserving the graph statistics (Choe et al.,, 2022; Dyer
et al.,, 2021). However, there is a lack of practical implementations of
hypergraph samplers and evaluations of their efficacy. Following the work by
GraphSAINT (Zeng et al.,, 2020) on simple graph sampling, we propose
_HybridGraphSAINT_ , a class of hybrid graph samplers employing GraphSAINT’s
graph sampling approaches. In HybridGraphSAINT, we adopt the same sampling
strategies in GraphSAINT for sampling the simple graph components in a hybrid
graph, making three different types of samplers: node sampler
(HybridGraphSAINT-Node), edge sampler (HybirdGraphSAINT-Edge), and random walk
sampler (HybridGraphSAINT-RW). As for the hyperedges, we use an intuitive
procedure that any hyperedges containing at least one node in the sampled
subgraph are retained, but all nodes not in the subgraph are masked out from
those hyperedges. By using this method, we preserve the original hybrid graph
characteristics in the sampled subgraphs to the maximum extent. We also
construct two naïve random samplers as baselines for evaluation: random node
sampler and random hyperedge sampler, which randomly sample a subset of
nodes/hyperedges from the original hybrid graph according to a uniform
sampling distribution. However, subgraphs sampled using the random node
sampler can be very sparse, while subgraphs sampled using the random hyperedge
sampler can be very dense.

We evaluate these samplers on MUSAE-GitHub and MUSAE-Facebook, the two largest
graphs in HGB. Firstly, we sample subgraphs with various samplers to examine
how they preserve the structure of the original graphs. This is measured by
three graph statistics: average node/hyperedge degree, and clustering
coefficient. We sample subgraphs multiple times and report the average graph
statistics in Table 11 of Appendix C. Among all constructed samplers,
HybirdGraphSAINT-Edge and HybirdGraphSAINT-RW perform the best in preserving
the graph-level statistics.

(a)

(b)

Figure 5: Node classification accuracy of different sampling techniques on (a)
MUSAE-GitHub and (b) MUSAE-Facebook. The charts compare the accuracy of
HyperConv model with three variants of HybirdGraphSAINT samplers at different
subgraph sizes. The size of the subgraph is measured by the number of nodes.
The black dashed line indicates the performance of HyperConv trained on the
whole graph without any sampling. The accuracy increases as subgraph sizes
increased, with both HybridGraphSAINT-RW and HybridGraphSAINT-Node
outperforming standard HyperConv. HybirdGraphSAINT-RW consistently shows
superior performance across different sampling sizes.

We also evaluate HyperConv performance when paired with three different
HybridGraphSAINT samplers. Figure 5 shows distinct patterns regarding how the
accuracy varied with different subgraph sizes. Evaluations were carried out on
two distinct datasets, yielding the following key observations (1) There
exists a positive correlation between the size of sampled subgraphs and model
accuracy across both datasets. (2) At subgraph sizes of 5000 or larger, both
HybridGraphSAINT-RW and HybridGraphSAINT-Node outperform vanilla HyperConv
(trained on the whole graph without sampling). This highlights the potential
for improved performance of hypergraph GNNs when integrating our
HybridGraphSAINT samplers. This performance gain could be explained by the
reduction of over-smoothing brought by sampling subgraphs.

Overall, the HybridGraphSAINT samplers, which mostly use simple edge
information, can both effectively preserve hypergraph statistics and improve
the performance of the hypergraph learning algorithms. This again underpins
the advantage of hybrid graphs over conventional hypergraphs and simple
graphs, emphasising the need to preserve multi-level information.

###  4.4 Integrating Simple Graph and Hypergraph Information

To test whether combining simple and higher-order graph information could
improve the performance of GNNs, we propose a simple algorithm called _Linear
Probe Graph Neural Networks (LP-GNNs)_. LP-GNN consists of two GNNs
$f_{\text{GNN}_{1}}$, $f_{\text{GNN}_{2}}$, plus a linear layer
$f(\mathbf{x})=\bm{\theta}\mathbf{x}+\mathbf{b}$. For a given hybrid graph
$\mathcal{G}=(\mathbf{X},\mathbf{E},\mathbf{H},\mathbf{W},\mathbf{R})$, where
$\mathbf{x}$ denotes the input features of a node, LP-GNN is defined as

$\text{LP-
GNN}(\mathbf{x})=\bm{\theta}\cdot\mathrm{CONCATENATE}(f_{\text{GNN}_{1}}(\mathbf{x}),f_{\text{GNN}_{2}}(\mathbf{x}))+\mathbf{b}$
(1)

For node regression tasks, $\text{LP-GNN}(\mathbf{x})$ is directly used as the
final output. For node classification tasks, a log softmax function is applied
to $\text{LP-GNN}(\mathbf{x})$ to produce the final output.

Table 2 shows the performance of three LP-GNNs with the GNN pair being
GCN+GAT, GCN+HyperConv, and GAT+HyperConv. These models are benchmarked across
five different HGB datasets, and their performances are contrasted with those
of simple graph GNNs. Our focus lies on the performance of LP-GCN+HyperConv
and LP-GAT+HyperConv, which combine simple edge and hyperedge information. In
three of the datasets (Wiki-Chameleon, Breast, and Leukemia), LP-GAT+HyperConv
and LP-GCN+HyperConv demonstrate the highest efficacy. In GitHub and
Computers, those two models also tend to match other GNNs’ performances. LP-
GNNs employ a very simple strategy to combine information from two levels and
show its advantage. This evidents the potential advantages of integrating
information across multiple levels.

Table 2: Accuracies of LP-GNNs, GCN, GAT, and HyperConv. The reciprocals of
mean square errors (1/MSE) are reported for the GNNs’ node regression task
performances on the Chameleon dataset.

Method | GitHub | Chameleon | Breast | Leukemia | Computers
---|---|---|---|---|---
RandomGuess | 0.250 | — | 0.333 | 0.333 | 0.100
GCN | 0.872 $\pm$ 0.000 | 0.137 $\pm$ 0.000 | 0.639 $\pm$ 0.010 | 0.582 $\pm$ 0.001 | 0.756 $\pm$ 0.041
GAT | 0.864 $\pm$ 0.001 | 0.152 $\pm$ 0.004 | 0.643 $\pm$ 0.001 | 0.587 $\pm$ 0.005 | 0.742 $\pm$ 0.043
HyperConv | 0.808 $\pm$ 0.001 | 0.138 $\pm$ 0.000 | 0.645 $\pm$ 0.001 | 0.586 $\pm$ 0.003 | 0.842 $\pm$ 0.020
LP-GCN+GAT | 0.867 $\pm$ 0.001 | 0.181 $\pm$ 0.009 | 0.626 $\pm$ 0.001 | 0.590 $\pm$ 0.002 | 0.930 $\pm$ 0.000
LP-GCN+HyperConv | 0.872 $\pm$ 0.000 | 0.181 $\pm$ 0.001 | 0.652 $\pm$ 0.006 | 0.604 $\pm$ 0.004 | 0.913 $\pm$ 0.001
LP-GAT+HyperConv | 0.860 $\pm$ 0.002 | 0.205 $\pm$ 0.002 | 0.657 $\pm$ 0.001 | 0.601 $\pm$ 0.002 | 0.930 $\pm$ 0.007

##  5 Conclusion and Discussion

We introduce the concept of _hybrid graphs_ , a unified representation of
various complex graph structures. Additionally, we present the _Hybrid Graph
Benchmark (HGB)_ , a collection of real-world hybrid graph datasets
accompanied with an extensible GNN evaluation framework. Through extensive
experiments, we demonstrate that existing hypergraph GNNs are not guaranteed
to outperform simple graph GNNs on large-scale complex networks. In light of
this, we propose a simple model called _Linear Probe Graph Neural Networks
(LP-GNNs)_ , which integrates simple graph and hypergraph information, and
leverages the node prediction performance on the hybrid graphs. We believe
that HGB can significantly aid the research in complex graph representation
learning, and provide valuable insights to future research directions for the
graph representation learning community.

#### Limitations

We define hybrid graphs as graphs with multiple levels. However, the current
collection of datasets in HGB primarily consists of shallow hybrid graphs with
a two-level node hierarchy, and networks with deeper node hierarchies are
desired. In MUSAE, hyperedges are constructed using the maximal cliques of a
graph, which might lead to an overlap of information between the simple edges
and hyperedges. In GRAND, the majority of gene regulatory graphs exhibit
bipartite-like characteristics, which may be limited in connectivity and
clustering for hyperedge constructions. In Amazon, hyperedges are created
based on a pre-defined threshold. Our preliminary experiments show that the
selection of this threshold value can influence the structure of the resultant
hybrid graphs, and a more thorough investigation is required to find the range
of the most appropriate threshold values.

#### Future Work

HGB will be updated on a regular basis, and we welcome inputs from the
community. In the future versions of HGB, we plan to to enable multiple
versions of Amazon datasets by adjusting various pairwise distance thresholds,
and include more real-world large-scale networks featuring multi-level node
hierarchies. Additionally, as simply concatenating simple and hypergraph
information followed by a linear transformation can already enhance the node
prediction performance on hybrid graphs, we will also seek to further improve
the performance following this idea, by incorporating more fine-grained
operations to combine simple graph and higher-order graph information.

## Acknowledgments and Disclosure of Funding

This work was performed using the Sulis Tier 2 HPC platform hosted by the
Scientific Computing Research Technology Platform at the University of
Warwick, and the JADE Tier 2 HPC facility. Sulis is funded by EPSRC Grant
EP/T022108/1 and the HPC Midlands+ consortium. JADE is funded by EPSRC Grant
EP/T022205/1. Zehui Li acknowledges the funding from the UKRI 21EBTA: EB-AI
Consortium for Bioengineered Cells & Systems (AI-4-EB) award, Grant
BB/W013770/1. Xiangyu Zhao acknowledges the funding from the Imperial College
London Electrical and Electronic Engineering PhD Scholarship. Mingzhu Shen
acknowledges the funding from the Imperial College London President’s PhD
Scholarship. For the purpose of open access, the authors have applied a
Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript
version arising.

## References

  * Amburg et al., (2020) Amburg, I., Veldt, N., and Benson, A. (2020).  Clustering in graphs and hypergraphs with categorical edge labels.  In Proceedings of The Web Conference 2020, pages 706–717. ACM.
  * Bai et al., (2021) Bai, S., Zhang, F., and Torr, P. H. (2021).  Hypergraph convolution and hypergraph attention.  Pattern Recognition, 110:107637.
  * Ben Guebila et al., (2022) Ben Guebila, M., Lopes-Ramos, C. M., Weighill, D., Sonawane, A. R., Burkholz, R., Shamsaei, B., Platig, J., Glass, K., Kuijjer, M. L., and Quackenbush, J. (2022).  GRAND: a database of gene regulatory network models across human conditions.  Nucleic Acids Research, 50(D1):D610–D621.
  * Brody et al., (2022) Brody, S., Alon, U., and Yahav, E. (2022).  How attentive are graph attention networks?  In The 10th International Conference on Learning Representations. OpenReview.net.
  * Chan and Liang, (2020) Chan, T.-H. H. and Liang, Z. (2020).  Generalizing the hypergraph laplacian via a diffusion process with mediators.  Theoretical Computer Science, 806:416–428.
  * Chien et al., (2022) Chien, E., Pan, C., Peng, J., and Milenkovic, O. (2022).  You are AllSet: A multiset function framework for hypergraph neural networks.  In The 10th International Conference on Learning Representations,. OpenReview.net.
  * Chodrow et al., (2021) Chodrow, P. S., Veldt, N., and Benson, A. R. (2021).  Generative hypergraph clustering: From blockmodels to modularity.  Science Advances, 7(28):eabh1303.
  * Choe et al., (2022) Choe, M., Yoo, J., Lee, G., Baek, W., Kang, U., and Shin, K. (2022).  MiDaS: Representative sampling from real-world hypergraphs.  In Proceedings of the ACM Web Conference 2022, pages 1080–1092.
  * Dong et al., (2020) Dong, Y., Sawin, W., and Bengio, Y. (2020).  HNHN: Hypergraph networks with hyperedge neurons.  ICML 2020 Graph Representation Learning and Beyond Workshop.
  * Dyer et al., (2021) Dyer, M., Greenhill, C., Kleer, P., Ross, J., and Stougie, L. (2021).  Sampling hypergraphs with given degrees.  Discrete Mathematics, 344(11):112566.
  * Eraslan et al., (2019) Eraslan, G., Avsec, Ž., Gagneur, J., and Theis, F. J. (2019).  Deep learning: new computational modelling techniques for genomics.  Nature Reviews Genetics, 20(7):389–403.
  * Fang et al., (2020) Fang, Y., Sun, S., Gan, Z., Pillai, R., Wang, S., and Liu, J. (2020).  Hierarchical graph network for multi-hop question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8823–8838. ACL.
  * Feng et al., (2019) Feng, Y., You, H., Zhang, Z., Ji, R., and Gao, Y. (2019).  Hypergraph neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3558–3565.
  * Gao and Ji, (2019) Gao, H. and Ji, S. (2019).  Graph u-nets.  In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 2083–2092. PMLR.
  * Getoor, (2005) Getoor, L. (2005).  Link-based classification.  In Advanced Methods for Knowledge Discovery from Complex Data, pages 189–207. Springer.
  * Gotmare et al., (2019) Gotmare, A., Keskar, N. S., Xiong, C., and Socher, R. (2019).  A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation.  In The 7th International Conference on Learning Representations. OpenReview.net.
  * Hamilton et al., (2017) Hamilton, W., Ying, Z., and Leskovec, J. (2017).  Inductive representation learning on large graphs.  In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  * He and McAuley, (2016) He, R. and McAuley, J. (2016).  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In Proceedings of the 25th International Conference on World Wide Web, pages 507–517.
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. (2015).  Adam: A method for stochastic optimization.  In The 3rd International Conference on Learning Representations, Conference Track Proceedings.
  * Kipf and Welling, (2017) Kipf, T. N. and Welling, M. (2017).  Semi-supervised classification with graph convolutional networks.  In The 5th International Conference on Learning Representations, Conference Track Proceedings. OpenReview.net.
  * Lee et al., (2019) Lee, J., Lee, I., and Kang, J. (2019).  Self-attention graph pooling.  In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 3734–3743. PMLR.
  * Lemons et al., (2011) Lemons, N. W., Hu, B., and Hlavacek, W. S. (2011).  Hierarchical graphs for rule-based modeling of biochemical systems.  BMC Bioinformatics, 12:45.
  * Li et al., (2019) Li, G., Muller, M., Thabet, A., and Ghanem, B. (2019).  DeepGCNs: Can GCNs go as deep as CNNs?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9267–9276.
  * Long et al., (2017) Long, Y., Wang, X., Youmans, D. T., and Cech, T. R. (2017).  How do lncRNAs regulate transcription?  Science Advances, 3(9):eaao2110.
  * Maston et al., (2006) Maston, G. A., Evans, S. K., and Green, M. R. (2006).  Transcriptional regulatory elements in the human genome.  Annual Review of Genomics and Human Genetics, 7:29–59.
  * McAuley et al., (2015) McAuley, J., Targett, C., Shi, Q., and Van Den Hengel, A. (2015).  Image-based recommendations on styles and substitutes.  In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43–52.
  * McCallum et al., (2000) McCallum, A. K., Nigam, K., Rennie, J., and Seymore, K. (2000).  Automating the construction of Internet portals with machine learning.  Information Retrieval, 3(2):127–163.
  * Ni et al., (2019) Ni, J., Li, J., and McAuley, J. (2019).  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 188–197.
  * Pati et al., (2022) Pati, P., Jaume, G., Foncubierta-Rodríguez, A., Feroce, F., Anniciello, A. M., Scognamiglio, G., Brancati, N., Fiche, M., Dubruc, E., Riccio, D., Di Bonito, M., De Pietro, G., Botti, G., Thiran, J.-P., Frucci, M., Goksel, O., and Gabrani, M. (2022).  Hierarchical graph representations in digital pathology.  Medical Image Analysis, 75:102264.
  * Radford et al., (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021).  Learning transferable visual models from natural language supervision.  In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 8748–8763. PMLR.
  * Rozemberczki et al., (2021) Rozemberczki, B., Allen, C., and Sarkar, R. (2021).  Multi-scale attributed node embedding.  Journal of Complex Networks, 9(2).
  * Sen et al., (2008) Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T. (2008).  Collective classification in network data.  AI Magazine, 29(3):93.
  * Shchur et al., (2018) Shchur, O., Mumme, M., Bojchevski, A., and Günnemann, S. (2018).  Pitfalls of graph neural network evaluation.  NeurIPS 2018 Relational Representation Learning Workshop.
  * Veličković et al., (2018) Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018).  Graph attention networks.  In The 6th International Conference on Learning Representations, Conference Track Proceedings. OpenReview.net.
  * Yadati et al., (2019) Yadati, N., Nimishakavi, M., Yadav, P., Nitin, V., Louis, A., and Talukdar, P. (2019).  HyperGCN: A new method for training graph convolutional networks on hypergraphs.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  * Ying et al., (2018) Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018).  Hierarchical graph representation learning with differentiable pooling.  In Advances in Neural Information Processing Systems, volume 31, pages 4800–4810. Curran Associates, Inc.
  * Zeng et al., (2020) Zeng, H., Zhou, H., Srivastava, A., Kannan, R., and Prasanna, V. (2020).  GraphSAINT: Graph sampling based inductive learning method.  In The 8th International Conference on Learning Representations. OpenReview.net.

##  Appendix A Full Statistics of HGB Datasets

Table 3: Statistics of all 23 hybrid graph datasets in HGB.

Name |  #Nodes |  #Edges |  #Hyperedges |  Avg. Node Degree |  Avg. Hyperedge Degree |  #Node Features |  #Classes
---|---|---|---|---|---|---|---
MUSAE-Github | 37,700 | 578,006 | 223,672 | 30.66 | 4.591 | 4,005 or 128 | 4
MUSAE-Facebook | 22,470 | 342,004 | 236,663 | 30.44 | 9.905 | 4,714 or 128 | 4
MUSAE-Twitch-DE | 9,498 | 306,276 | 297,315 | 64.49 | 7.661 | 3,170 or 128 | 2
MUSAE-Twitch-EN | 7,126 | 70,648 | 13,248 | 19.83 | 3.666 | 3,170 or 128 | 2
MUSAE-Twitch-ES | 4,648 | 118,764 | 77,135 | 51.10 | 5.826 | 3,170 or 128 | 2
MUSAE-Twitch-FR | 6,549 | 225,332 | 172,653 | 68.81 | 5.920 | 3,170 or 128 | 2
MUSAE-Twitch-PT | 1,912 | 62,598 | 74,830 | 65.48 | 7.933 | 3,170 or 128 | 2
MUSAE-Twitch-RU | 4,385 | 74,608 | 25,673 | 34.03 | 4.813 | 3,170 or 128 | 2
MUSAE-Wiki-Chameleon | 2,277 | 62,742 | 14,650 | 55.11 | 7.744 | 3,132 or 128 | Regression
MUSAE-Wiki-Crocodile | 11,631 | 341,546 | 121,431 | 58.73 | 4.761 | 13,183 or 128 | Regression
MUSAE-Wiki-Squirrel | 5,201 | 396,706 | 220,678 | 152.55 | 30.735 | 3,148 or 128 | Regression
GRAND-ArteryAorta | 5,848 | 5,823 | 11,368 | 1.991 | 1.277 | 4,651 | 3
GRAND-ArteryCoronary | 5,755 | 5,722 | 11,222 | 1.989 | 1.273 | 4,651 | 3
GRAND-Breast | 5,921 | 5,910 | 11,400 | 1.996 | 1.281 | 4,651 | 3
GRAND-Brain | 6,196 | 6,245 | 11,878 | 2.016 | 1.296 | 4,651 | 3
GRAND-Lung | 6,119 | 6,160 | 11,760 | 2.013 | 1.291 | 4,651 | 3
GRAND-Stomach | 5,745 | 5,694 | 11,201 | 1.982 | 1.274 | 4,651 | 3
GRAND-Leukemia | 4,651 | 6,362 | 7,812 | 2.736 | 1.324 | 4,651 | 3
GRAND-Lungcancer | 4,896 | 6,995 | 8,179 | 2.857 | 1.334 | 4,651 | 3
GRAND-Stomachcancer | 4,518 | 6,051 | 7,611 | 2.679 | 1.312 | 4,651 | 3
GRAND-KidneyCancer | 4,319 | 5,599 | 7,369 | 2.593 | 1.297 | 4,651 | 3
Amazon-Computers | 10,226 | 55,324 | 10,226 | 10.82 | 3.000 | 1,000 | 10
Amazon-Photos | 6,777 | 45,306 | 6,777 | 13.37 | 4.800 | 1,000 | 10

##  Appendix B Experimental Details

###  B.1 Training Details

We run all the experiments on NVIDIA A100 PCIe GPU with 40GB RAM (Sulis) and
NVIDIA V100 NVLink GPU with 32GB RAM (JADE), with each experiment taking less
than 2 minutes. Adam (Kingma and Ba,, 2015) is used as the optimiser, and
CosineAnnealingLR (Gotmare et al.,, 2019) is used as the learning rate
scheduler for all training. All models are trained for 50 epochs. For each
experiment, the nodes of the used hybrid graph are split into the train,
validation, and test sets with a split ratio of 6:2:2. For node classification
tasks, BCEWithLogitsLoss is used as the loss function, which is defined as:

$\mathcal{L}_{\text{BCEWithLogits}}(\mathbf{y},\hat{\mathbf{y}})=-\frac{1}{n}\sum_{i=1}^{n}\big{[}y_{i}\cdot\log(\sigma(\hat{y}_{i}))+(1-y_{i})\cdot\log(1-\sigma(\hat{y}_{i}))\big{]}$
(2)

where $n$ is the total number of elements in $\mathbf{y}$ and
$\hat{\mathbf{y}}$, $y_{i}$ is the $i$-th element of $\mathbf{y}$, the batch
of true values, and $\hat{y}_{i}$ is the $i$-th element of $\hat{\mathbf{y}}$,
the batch of raw (i.e., non-sigmoid-transformed) predicted values. $\sigma$
denotes the sigmoid function, which transforms the raw predictions into the
range (0, 1). For node regression tasks, MSELoss is used as the loss function,
which is defined as:

$\mathcal{L}_{\text{MSE}}(\mathbf{y},\hat{\mathbf{y}})=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$
(3)

where $n$ is the total number of elements in $\mathbf{y}$ and
$\hat{\mathbf{y}}$, $y_{i}$ is the $i$-th element of $\mathbf{y}$, the batch
of true values, and $\hat{y}_{i}$ is the $i$-th element of $\hat{\mathbf{y}}$,
the batch of predicted values.

###  B.2 Hyperparameter Settings

We perform a hyperparameter search for the learning rate and keep the hidden
layer dimension the same for different models, the hyperparameters used for
training each architecture are listed in Table 4. All seven GNNs (GCN,
GraphSAGE, GAT, GATv2, HyperConv, HyperAtten, and GraphSAINT) share the same
learning rate, hidden dimension, and dropout rate. HyperAtten has an
additional hyperparameter, which is the hyperedge aggregation function. This
function determines how the hyperedge is constructed from the nodes within it.
The possible options for this function are ‘sum’ and ‘concatenate’. In this
work, we have selected ‘sum’ as the hyperedge aggregation function. For
GraphSAINT, the additional hyperparameters are subgraph size, measured by the
number of nodes in the subgraph, and the batch size, which is the number of
subgraphs to sample in each epoch. Different subgraph sizes are applied
according to the sizes of the original hybrid graphs.

Table 4: Hyperparameter selections for the experiments. Method | Hyperparameter | Value
---|---|---
All | Learning rate | 0.01
Hidden dimension | 32
Dropout rate | 0.5
HyperAtten | Hyperedge aggregation function | Sum
GraphSAINT | Subgraph size (MUSAE-GitHub, MUSAE-Facebook) | 5000
Subgraph size (MUSAE-Twitch-PT) | 1000
Subgraph size (others) | 3000
Batch size | 5

##  Appendix C Results

We evaluate the performance of seven GNNs on all 23 HGB datasets. Each
experiment is repeated five times with different random seeds, and the results
are summarised in Tables 5, 6, 7, 8, 9 and 10. In the node classification
tasks of MUSAE, GCN and GraphSAGE perform the best in the on GitHub and
Facebook, as shown in Table 5, while all GNNs perform roughly the same on
Twitch, as shown in Table 6. In the three node regression tasks on MUSAE-Wiki,
GraphSAINT stands out among other GNNs, as shown in Table 10. Table 7 shows
that HyperConv and HyperAtten outperform other simple graph GNNs on five of
the six hybrid graphs in GRAND-Tissues. For hybrid graphs in GRAND-Diseases as
shown in Table 8, GraphSAGE exhibits a superior performance. For the two
Amazon hybrid graph datasets, as shown in Table 9, GraphSAINT consistently
achieves the best performance. Overall, hypergraph GNNs tend to outperform
simple graph GNNs on GRAND-Tissues and Amazon, perform equally as simple graph
GNNs on MUSAE-Twitch, MUSAE-Wiki and GRAND-Diseases, and underperform simple
graph GNNs on MUSAE-GitHub and MUSAE-Facebook.

Table 11 show the statistics of the subgraphs obtained by five different
samplers. We sample subgraphs from both MUSAE-GitHub and MUSAE-Facebook, and
then compute the average statistics of the sampled subgraphs.
HybridGraphSAINT-Edge and HybridGraphSAINT-RW perform the best in preserving
the graph-size agnostic statistics. The two random samplers tend to sample
subgraphs with distinct structures from the original hybrid graphs.

Table 5: Accuracies of the selected GNNs on the MUSAE-Facebook and MUSAE-GitHub datasets. GCN and GraphSAGE perform the best on these datasets, and the hypergraph GNNs tend to perform worse than the simple graph GNNs. Method | Facebook | GitHub
---|---|---
RandomGuess | 0.250 | 0.250
GCN | 0.886 $\pm$ 0.001 | 0.872 $\pm$ 0.000
GraphSAGE | 0.902 $\pm$ 0.002 | 0.871 $\pm$ 0.002
GAT | 0.876 $\pm$ 0.001 | 0.864 $\pm$ 0.001
GATv2 | 0.901 $\pm$ 0.001 | 0.866 $\pm$ 0.001
HyperConv | 0.792 $\pm$ 0.001 | 0.808 $\pm$ 0.001
HyperAtten | 0.523 $\pm$ 0.002 | 0.775 $\pm$ 0.001
GraphSAINT | 0.896 $\pm$ 0.001 | 0.871 $\pm$ 0.001
Table 6: Accuracies of the selected GNNs on the MUSAE-Twitch datasets. All
selected GNNs achieve similar performance on this group of datasets.

Method | TwitchES | TwitchFR | TwitchDE | TwitchEN | TwitchPT | TwitchRU
---|---|---|---|---|---|---
RandomGuess | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500
GCN | 0.721 $\pm$ 0.004 | 0.624 $\pm$ 0.001 | 0.655 $\pm$ 0.002 | 0.620 $\pm$ 0.003 | 0.689 $\pm$ 0.006 | 0.745 $\pm$ 0.000
GraphSAGE | 0.690 $\pm$ 0.002 | 0.616 $\pm$ 0.003 | 0.657 $\pm$ 0.001 | 0.605 $\pm$ 0.000 | 0.672 $\pm$ 0.013 | 0.745 $\pm$ 0.001
GAT | 0.694 $\pm$ 0.002 | 0.623 $\pm$ 0.000 | 0.645 $\pm$ 0.004 | 0.594 $\pm$ 0.006 | 0.664 $\pm$ 0.007 | 0.743 $\pm$ 0.002
GATv2 | 0.710 $\pm$ 0.003 | 0.625 $\pm$ 0.001 | 0.651 $\pm$ 0.003 | 0.618 $\pm$ 0.005 | 0.687 $\pm$ 0.009 | 0.745 $\pm$ 0.000
HyperConv | 0.715 $\pm$ 0.001 | 0.624 $\pm$ 0.002 | 0.654 $\pm$ 0.002 | 0.587 $\pm$ 0.007 | 0.701 $\pm$ 0.005 | 0.741 $\pm$ 0.001
HyperAtten | 0.695 $\pm$ 0.000 | 0.623 $\pm$ 0.001 | 0.610 $\pm$ 0.003 | 0.553 $\pm$ 0.003 | 0.641 $\pm$ 0.000 | 0.743 $\pm$ 0.000
GraphSAINT | 0.713 $\pm$ 0.008 | 0.622 $\pm$ 0.003 | 0.653 $\pm$ 0.004 | 0.610 $\pm$ 0.011 | 0.677 $\pm$ 0.006 | 0.746 $\pm$ 0.002

Table 7: Accuracies of the selected GNNs on the GRAND-Tissues datasets.
Hypergraph GNNs tend to outperform simple graph GNNs on these datasets, with
HyperAtten being the best-performing method.

Method | ArteryAorta | ArteryCoronary | Breast | Brain | Lung | Stomach
---|---|---|---|---|---|---
RandomGuess | 0.333 | 0.333 | 0.333 | 0.333 | 0.333 | 0.333
GCN | 0.627 $\pm$ 0.007 | 0.662 $\pm$ 0.001 | 0.639 $\pm$ 0.010 | 0.625 $\pm$ 0.000 | 0.650 $\pm$ 0.000 | 0.643 $\pm$ 0.000
GraphSAGE | 0.628 $\pm$ 0.002 | 0.663 $\pm$ 0.001 | 0.644 $\pm$ 0.000 | 0.618 $\pm$ 0.002 | 0.646 $\pm$ 0.005 | 0.630 $\pm$ 0.010
GAT | 0.628 $\pm$ 0.004 | 0.663 $\pm$ 0.000 | 0.643 $\pm$ 0.001 | 0.625 $\pm$ 0.001 | 0.648 $\pm$ 0.004 | 0.643 $\pm$ 0.000
GATv2 | 0.630 $\pm$ 0.000 | 0.663 $\pm$ 0.000 | 0.644 $\pm$ 0.000 | 0.624 $\pm$ 0.001 | 0.650 $\pm$ 0.001 | 0.642 $\pm$ 0.000
HyperConv | 0.626 $\pm$ 0.008 | 0.662 $\pm$ 0.000 | 0.645 $\pm$ 0.001 | 0.625 $\pm$ 0.000 | 0.650 $\pm$ 0.000 | 0.643 $\pm$ 0.000
HyperAtten | 0.647 $\pm$ 0.003 | 0.670 $\pm$ 0.003 | 0.633 $\pm$ 0.003 | 0.632 $\pm$ 0.003 | 0.661 $\pm$ 0.004 | 0.636 $\pm$ 0.004
GraphSAINT | 0.630 $\pm$ 0.000 | 0.663 $\pm$ 0.000 | 0.644 $\pm$ 0.000 | 0.625 $\pm$ 0.000 | 0.650 $\pm$ 0.000 | 0.643 $\pm$ 0.000

Table 8: Accuracies of the selected GNNs on the GRAND-Diseases datasets. GraphSAGE achieves the best performance on these datasets. Hypergraph GNNs and simple graph GNNs exhibit similar performance on these datasets. Method | Leukemia | LungCancer | StomachCancer | KidneyCancer
---|---|---|---|---
RandomGuess | 0.333 | 0.333 | 0.333 | 0.333
GCN | 0.582 $\pm$ 0.001 | 0.596 $\pm$ 0.001 | 0.602 $\pm$ 0.007 | 0.581 $\pm$ 0.002
GraphSAGE | 0.604 $\pm$ 0.016 | 0.615 $\pm$ 0.015 | 0.602 $\pm$ 0.016 | 0.596 $\pm$ 0.014
GAT | 0.587 $\pm$ 0.005 | 0.596 $\pm$ 0.000 | 0.600 $\pm$ 0.007 | 0.581 $\pm$ 0.003
GATv2 | 0.583 $\pm$ 0.000 | 0.591 $\pm$ 0.005 | 0.596 $\pm$ 0.002 | 0.579 $\pm$ 0.006
HyperConv | 0.586 $\pm$ 0.003 | 0.593 $\pm$ 0.003 | 0.596 $\pm$ 0.004 | 0.577 $\pm$ 0.006
HyperAtten | 0.593 $\pm$ 0.008 | 0.608 $\pm$ 0.008 | 0.604 $\pm$ 0.022 | 0.578 $\pm$ 0.012
GraphSAINT | 0.583 $\pm$ 0.000 | 0.595 $\pm$ 0.000 | 0.596 $\pm$ 0.000 | 0.582 $\pm$ 0.000
Table 9: Accuracies of the selected GNNs the Amazon datasets. GraphSAINT performs the best among all GNNs on these datasets, and hypergraph GNNs tend to outperform simple graph GNNs. Method | Computers | Photos
---|---|---
RandomGuess | 0.100 | 0.100
GCN | 0.756 $\pm$ 0.041 | 0.295 $\pm$ 0.017
GraphSAGE | 0.582 $\pm$ 0.108 | 0.366 $\pm$ 0.061
GAT | 0.742 $\pm$ 0.043 | 0.434 $\pm$ 0.074
GATv2 | 0.566 $\pm$ 0.046 | 0.420 $\pm$ 0.075
HyperConv | 0.842 $\pm$ 0.020 | 0.337 $\pm$ 0.059
HyperAtten | 0.663 $\pm$ 0.005 | 0.465 $\pm$ 0.033
GraphSAINT | 0.875 $\pm$ 0.020 | 0.512 $\pm$ 0.141
Table 10: MSEs ($\downarrow$) of the selected GNNs the MUSAE-Wiki datasets. GraphSAINT performs the best on these datasets, and hypergraph GNNs tend to perform at the same level as simple graph GNNs. Method | Chameleon | Squirrel | Crocodile
---|---|---|---
GCN | 7.319 $\pm$ 0.000 | 8.761 $\pm$ 0.001 | 6.779 $\pm$ 0.005
GraphSAGE | 6.945 $\pm$ 0.005 | 8.310 $\pm$ 0.003 | 6.380 $\pm$ 0.005
GAT | 6.557 $\pm$ 0.154 | 8.093 $\pm$ 0.054 | 6.249 $\pm$ 0.261
GATv2 | 7.290 $\pm$ 0.019 | 8.600 $\pm$ 0.011 | 6.717 $\pm$ 0.005
HyperConv | 7.230 $\pm$ 0.002 | 8.706 $\pm$ 0.000 | 6.712 $\pm$ 0.001
HyperAtten | 7.451 $\pm$ 0.000 | 8.782 $\pm$ 0.000 | 6.942 $\pm$ 0.000
GraphSAINT | 5.165 $\pm$ 0.027 | 7.541 $\pm$ 0.023 | 4.898 $\pm$ 0.035
Table 11: Statistics of the subgraphs obtained by various samplers.
HybridGraphSAINT-Edge and HybridGraphSAINT-RW perform the best in preserving
the original hybrid graph characteristics.

|  MUSAE-GitHub |  MUSAE-Facebook
---|---|---
Samplers |  Avg. Node Degree |  Avg. Hyperedge Degree |  Clustering Coef. |  Avg. Node Degree |  Avg. Hyperedge Degree |  Clustering Coef.
Original | 30.66 | 0.168 | 4.590 | 30.44 | 0.360 | 9.905
HybridGraphSAINT-Node | 42.59 | 0.268 | 13.20 | 42.72 | 0.268 | 13.11
HybridGraphSAINT-Edge | 38.63 | 0.366 | 11.42 | 38.66 | 0.366 | 11.50
HybridGraphSAINT-RW | 35.35 | 0.203 | 6.48 | 35.03 | 0.206 | 6.54
RandomNode | 2.25 | 0.020 | 19.16 | 2.58 | 0.037 | 21.73
RandomHyperedge | 72.38 | 0.282 | 7.48 | 72.74 | 0.279 | 7.44

##  Appendix D Data Accessibility

The source code and full datasets of HGB is publicly available at
https://zehui127.github.io/hybrid-graph-benchmark/. While the raw dataset in
JSON format is hosted at https://zenodo.org/record/7982540, we recommend the
users to access the datasets through our Python library hybrid-graph, which is
installable via pip. This would allow the users to read the hybrid graphs in
the format of PyTorch Geometric Data objects.

##  Appendix E Licence

The raw data for the MUSAE datasets are licenced under the the GNU General
Public Licence, version 3 (GPLv3)333https://www.gnu.org/licenses/gpl-3.0.html.
The raw data for the GRAND datasets are licenced under the Creative Commons
Attribution-ShareAlike 4.0 International Public Licence (CC BY-SA
4.0)444https://creativecommons.org/licenses/by-sa/4.0/. The raw data for the
Amazon datasets are licenced under the Amazon Service
licence555https://s3.amazonaws.com/amazon-reviews-pds/LICENSE.txt. Having
carefully observed the licence requirements of all data sources and code
dependencies, we apply the following licence to our source code and datasets:

  * •

The source code of HGB is licenced under the MIT
licence666https://opensource.org/license/mit/;

  * •

The MUSAE and GRAND datasets are licenced under the GPLv3 licence3;

  * •

The Amazon datasets are licenced under the Amazon Service licence5.

##  Appendix F Ethics Statement

All datasets constructed in HGB are generated from public open-source
datasets, and the original raw data downloaded from the data sources do not
contain any personally identifiable information or other sensitive contents.
The node prediction tasks for the HGB datasets are designed to ensure that
they do not, by any means, lead to discriminations against any social groups.
Therefore, we are not aware of any social or ethical concern of HGB. Since HGB
is a general benchmarking tool for representation learning on complex graphs,
we also do not forsee any direct application of HGB to malicious purposes.
However, the users of HGB should be aware of any potential negative social and
ethical impacts that may arise from their chosen downstream datasets or tasks
outside of HGB, if they intend to use the HGB datasets as pre-training
datasets to perform trasnfer learning.