# Finite-N corrections to dynamical Lorentz symmetry breaking

Y. M. P. Gomes<EMAIL_ADDRESS>Departamento de Física Teórica,
Universidade do Estado do Rio de Janeiro, 20550-013, Rio de Janeiro, RJ,
Brazil  M. J. Neves<EMAIL_ADDRESS>Departamento de Física, Universidade
Federal Rural do Rio de Janeiro, BR 465-07, 23890-971, Seropédica, RJ, Brazil

###### Abstract

A model of $N$ 4-component massless fermions in a quartic self-interaction
based on the ref. [18] is investigated in the presence of chemical potential
and temperature via optimized perturbation theory that accesses finite-N
contributions. We use the generating functional approach to calculate the
corrections at one-loop to the effective potential of the model. The model
introduces an auxiliary vector field that breaks the Lorentz symmetry in the
presence of a temperature $(T)$, and of a chemical potential $(\mu)$. These
thermodynamics quantities are introduced through the Matsubara formalism.
Thereby, the one-loop integrals are modified and via the principle of minimum
sensitivity, we obtain the gap equations of the model. The correspondent
solutions of these equations define the vacuum states of the model associated
with the background vector field. We discuss the solutions of the four cases
in which $(T=0,\mu=0)$, $(T\neq 0,\mu\neq 0)$, $(T\neq 0,\mu=0)$ and
$(T=0,\mu\neq 0)$, where the effective potential is so obtained as a function
of the background vector field, the chemical potential, and the temperature.
The work demonstrates the persistence of second-order phase transitions in the
finite-N improvement made on the finite-$N$ self-interacting fermions.

##  I introduction

Neutrino physics is one of the keys to understanding the origin of elementary
particles beyond the Standard Model (SM). It is known that neutrinos do not
have mass in the SM approach, however, experimental results point out mass
measurements through the transition probabilities in oscillations phenomena
[1, 2, 3]. The most famous approach to generate masses for the neutrinos is
known as see-saw mechanisms [4, 5, 6, 7, 8, 9], that also motivate the models
with new particle contents [10, 11, 12, 13].

The Lorentz symmetry violation (LSV) also provides another framework beyond
the SM in that background $4$-vectors are introduced and constrained via decay
widths, scattering processes, or astrophysical tests [14, 15]. This is known
as the Standard Model Extension (SME) [16, 17]. In connection with neutrino
physics, an LSV scenario also emerges on effective models with quartic self-
interacting couplings involving fermions in which background fields are
introduced as auxiliaries fields that play a fundamental role in the
understanding of the vacuum via effective potential formalism [18, 31, 32,
33]. The measurements of the neutrino’s oscillations impose interesting
constraints on the LSV parameters associated with the vacuum of the fermion
quartic models [18]. In this sense, fermionic models with quartic self-
interactions also are motivated by the study of physics in low dimensions, as
the Thirring model [19], the Gross-Neveu model [20], and the Nambu-Jona-
Lanisio (NJL) model for quantum chromodynamics (QCD) [21]. In all these
models, the dynamical symmetry breaking shows the vacuum properties and the
possible phase transitions in the presence of a temperature and chemical
potential.

Recently, the study of finite-N effects on symmetry breaking and its effects
on many systems have been analyzed, as graphene-like systems [22, 23, 24],
bidimensional semimetals [27] and nuclear and quark matter as well [25, 26].
The nonperturbative calculations have shown new features that pass unseen in
the large-N approximations, such as first-order transitions. Thus, inspired by
those intriguing non-perturbative results, in this paper we investigate the
vacuum phase transitions of $N$ massless fermions in the presence of quartic
self-interactions based on the ref. [18]. This self-interaction between
pseudo-current $j^{5}_{\mu}=\bar{\psi}\gamma_{\mu}\gamma_{5}\psi$ generates a
fermionic condensate that breaks parity (P) and (CPT) symmetries, along with
the Lorentz symmetry and therefore could be a source to matter anti-matter
asymmetry.

The generating functional approach from quantum field theory is used to obtain
the effective potential at one loop in terms of one auxiliary $4$-vector field
plus the optimized perturbation theory (OPT) method that brings the non-
perturbative results to the model. The temperature $(T)$ and the chemical
potential $(\mu)$ are introduced via Matsubara formalism directly through the
time-momentum integral. Consequently, the Lorentz symmetry is broken in the
model. We choose the time-like $4$-vector for our analysis such that, the
principle of minimal sensitivity (PMS) and the gap equations are derivative
from the effective potential. Therefore, we obtain the solutions of these
equations for the four cases : (i) $(T=0,\mu=0)$, (ii) $(T\neq 0,\mu\neq 0)$,
(iii) $(T\neq 0,\mu=0)$, and (iv) $(T=0,\mu\neq 0)$. The solutions for the
effective potential are discussed, and the correspondent vacuum states are
shown numerically in cases (i) and (iii), and analytically in the second case
(ii).

The paper is organized as follows: In the section (II), the $N$ 4-component
fermions model in the presence of a quartic self-interaction is presented, and
the effective potential at one-loop is obtained in terms of one-loop momentum
integrals. In the section (III), the PMS and GAP equations are evaluated in
the vacuum states, and the effective potential is studied for the case
$(T=0,\mu=0)$ with the subsection (III.1). We introduce the temperature and
chemical potential through the momentum integrals in the section (IV). This
section is subdivided into three subsections: The first subsection (IV.1) is
for $(T\neq 0,\mu=0)$, and the second one (IV.2) is for $(T=0,\mu\neq 0)$. We
discuss the phase diagrams in the third subsection (IV.3). Finally, the
conclusions are considered in the section (V). The useful integrals in the
paper are shown in the appendix (VI).

We use the natural units $\hbar=c=k_{B}=1$, and the Minkowski metric is
$\eta^{\mu\nu}=\mbox{diag}(+1,-1,-1,-1)$ for throughout this work.

##  II The $N$ fermions model in the presence of a quartic self-interaction

The model is governed by the lagrangian [18]:

$\mathcal{L}=\overline{\psi}_{k}\,i\,\not{\partial}\,\psi_{k}-\frac{g}{2N}\left(\,\overline{\psi}_{k}\gamma_{\mu}\gamma_{5}\psi_{k}\,\right)^{2}\;,$
(1)

where $\psi_{k}$ is a massless four-component fermion,
$k=\left\\{\,1,...,N\,\right\\}$ is the index that specifies the flavor, the
repeated $k$-index means an implicit sum over the flavors, $g$ is a coupling
constant with length dimension to the square in four dimensions, in which
$g<0$ for attractive interaction, $\not{\partial}=\gamma^{\mu}\partial_{\mu}$
is the slashed partial operator, and $\gamma^{\mu}$ are the usual Dirac
matrices that satisfy the relation
$\gamma^{\mu}\,\gamma^{\nu}=\eta^{\mu\nu}\,\mathds{1}-2\,i\,\Sigma^{\mu\nu}$.
The auxiliary vector field $A_{\mu}$ is introduced as follows :

$\mathcal{L}=\overline{\psi}_{k}\left(i\not{\partial}+\not{A}\gamma_{5}\right)\psi_{k}+\frac{N}{2g}A_{\mu}A^{\mu}\;,$
(2)

where
$A_{\mu}=-\frac{g}{N}\,\overline{\psi}_{k}\gamma_{\mu}\gamma_{5}\psi_{k}$.
Therefore, the effective Lagrangian (2) sets the model that we investigate
with the beyond large-N approach. In previous works [31, 32], the dynamical
breaking of Lorentz symmetry on a massless fermionic system has been studied
through the model represented by equation (2) by integrating out the fermionic
fields and assuming the large-N limit to compute the effective potential.

In order, the introduction of corrections beyond the large-N limit to the
model are implemented via the optimized perturbation theory (OPT) deformation
based on [22, 23, 28, 29], through the fictitious $\delta$-parameter :

$\displaystyle\mathcal{L}_{\delta}$ $\displaystyle=$
$\displaystyle(1-\delta)\mathcal{L}_{0}(\mathtt{a})+\delta\mathcal{L}=$
$\displaystyle=\overline{\psi}_{k}\left[\,i\not{\partial}+\left(\not{\mathtt{a}}-\left(\not{\mathtt{a}}-\not{A}\right)\delta\right)\gamma_{5}\,\right]\psi_{k}+\frac{\delta
N}{2g}A_{\mu}A^{\mu}\;,$

where $\mathtt{a}_{\mu}$ is a regulator vector field containing the finite-N
corrections information and $\mathcal{L}_{0}$ is the kinetic term of the
fermionic Lagrangian coupled to $\mathtt{a}_{\mu}$. The correspondent
classical equations of motion for the auxiliary fields $A^{\mu}$ and $a^{\mu}$
are, respectively, read as

$\displaystyle\frac{\delta
N}{g}\,A_{\mu}+\delta\left(\overline{\psi}_{k}\gamma_{\mu}\gamma_{5}\psi_{k}\right)=0\;,$
(4a)
$\displaystyle(\delta-1)\,\overline{\psi}_{k}\gamma_{\mu}\gamma_{5}\psi_{k}=0\;,$
(4b)

in which the limit $\delta\rightarrow 1$ recovers the original lagrangian of
(2), and $\mathtt{a}_{\mu}$ disappears naturally in (II). As usual in quantum
field theory, one can integrate out the fermionic degree of freedom, resulting
in an effective action for the bosonic field $A_{\mu}$. Specifically, the
generating functional associated with (II) yields the effective potential

$\displaystyle
N^{-1}\,V_{eff}(A_{\mu},\mathtt{a})=\frac{\delta}{2g}A_{\mu}A^{\mu}$
$\displaystyle+\int\frac{d^{4}p}{(2\pi)^{4}}\,\mbox{Tr}\ln\left(\not{p}+{\not{\mathtt{a}}}\gamma_{5}\right)$
$\displaystyle+\delta\int\frac{d^{4}p}{(2\pi)^{4}}\,\mbox{Tr}\left[\frac{(\not{\mathtt{a}}-\not{A})\gamma_{5}}{\not{p}+\not{\mathtt{a}}\gamma_{5}}\right]$
$\displaystyle+\frac{i}{2}\int\frac{d^{4}p}{(2\pi)^{4}}\,\mbox{Tr}\Bigg{[}\,\frac{\Sigma(\mathtt{a})}{\not{p}+\not{\mathtt{a}}\gamma_{5}}\,\Bigg{]}+O(\delta^{2})\;,\hskip
28.45274pt$ (5)

here one applies an expansion on $\delta$-parameter up to $\delta^{2}$ with
$\delta\ll 1$. The numerator of the third trace in (II) is

$\Sigma(\mathtt{a})\approx\frac{\delta
g}{N}\int\frac{d^{4}q}{(2\pi)^{4}}\frac{i}{\not{q}+\not{\mathtt{a}}\gamma_{5}}\;+O(\delta^{2}),$
(6)

that is expanded up to the first order of the $\delta$-parameter, and the
trace (Tr) acts on the spinor space. The correspondent diagrams of these
contributions are illustrated in the figure (1).

Figure 1: The Feynman diagrams contributing up to $\delta^{1}$ to the
effective potential. The first sets the effective potential in the large-N
limit. The black dot represents a $\delta$-insertion. The last diagram brings
the first correction to the effective potential due to the self-energy (figure
based on ref. [22]).

The common fraction of the integrals in (II) can be written as

$\frac{1}{\not{p}+\not{\mathtt{a}}\gamma_{5}}=\frac{N_{0}(p)}{D(p)}\;,$ (7)

in which the numerator is
$N_{0}(p)=(\not{p}+\not{\mathtt{a}}\gamma_{5})(p^{2}-\mathtt{a}^{2}-[\,\not{p}\,,\,\not{\mathtt{a}}]\gamma_{5})$,
and the denominator is $D(p)=(p+\mathtt{a})^{2}(p-\mathtt{a})^{2}$. Thereby,
the effective potential can be written as

$\displaystyle N^{-1}\,V_{eff}(A^{\mu},\mathtt{a}^{\mu})$ $\displaystyle=$
$\displaystyle\frac{\delta}{2g}A_{\mu}A^{\mu}+\,\Omega(\mathtt{a})+\delta\left(\mathtt{a}^{\mu}-A^{\mu}\right)\,\Pi_{\mu}(\mathtt{a})-\frac{\delta
g}{2N}\,\Xi(\mathtt{a})\;,$ (8)

where $\Omega(\mathtt{a})$, $\Pi_{\mu}(\mathtt{a})$ and $\Xi(\mathtt{a})$ are
defined by the integrals

$\displaystyle\Omega(\mathtt{a})\\!$ $\displaystyle=$
$\displaystyle\\!\sum_{\lambda=-1}^{+1}\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\,\ln(p+\lambda\mathtt{a})^{2}\;,$
(9a) $\displaystyle\Pi^{\mu}(\mathtt{a})\\!$ $\displaystyle=$
$\displaystyle\\!\sum_{\lambda=-1}^{+1}\lambda\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\,\frac{p^{\mu}+\lambda\mathtt{a}^{\mu}}{(p+\lambda\mathtt{a})^{2}}\;,$
(9b) $\displaystyle\Xi(\mathtt{a})\\!$ $\displaystyle=$
$\displaystyle\\!\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\frac{d^{4}q}{(2\pi)^{4}}\frac{\mbox{Tr}\left[\,N_{0}(p)\,N_{0}(q)\,\right]}{D(p)\,D(q)}\,,\hskip
19.91684pt$ (9c)

and we have used the trace properties $\mbox{tr}\ln(A)=\ln\det(A)$, and
$\det(A^{-1})=(\det A)^{-1}$. We also have introduced the cutoff parameter
$(\Lambda)$ (that has energy dimension) to regularize the momentum integrals
from (II) and (6). Thereby, the modified integrals are finite and naturally
diverge in the ultraviolet regime when $\Lambda\rightarrow\infty$.
Consequently, the effective potential is also regularized by the UV cutoff
$(\Lambda)$. In the next section, we investigate the contributions of the
finite parts for the effective potential in the case of the time-like
background fields.

##  III PMS and gap equation beyond the large-N limit

The principle of minimal sensitivity (PMS) applied to the effective potential
is defined by the following identity:

$\frac{\partial
V_{eff}}{\partial\mathtt{a}_{\mu}}\Bigg{|}_{\bar{\mathtt{a}},\delta=1}=0\;.$
(10)

The PMS equation which defines

$\displaystyle\left[\,\,\frac{\partial\Omega}{\partial\mathtt{a}_{\mu}}+\Pi_{\mu}(\mathtt{a})+\left(\mathtt{a}-A\right)^{\nu}\,\frac{\partial\Pi_{\nu}}{\partial\mathtt{a}_{\mu}}-\frac{g}{2N}\frac{\partial\Xi}{\partial\mathtt{a}_{\mu}}\,\right]\Bigg{|}_{\bar{\mathtt{a}}^{\mu},\delta=1}\\!\\!\\!=0\;.$
(11)

The solutions of equation (11) given by $\bar{\mathtt{a}}$ depend on various
original parameters including the couplings and the pseudo-vector field
$A_{\mu}$. This results in nonperturbative dependencies in the model’s
coupling and other parameters. The gap equation defined by

$\frac{\partial V_{eff}}{\partial
A_{\mu}}\Bigg{|}_{A^{\mu}=\bar{A}^{\mu},\mathtt{a}^{\mu}=\bar{\mathtt{a}}^{\mu},\delta=1}=0\;,$
(12)

that yields the relation

$\bar{A}^{\mu}-g\,\Pi^{\mu}(\bar{\mathtt{a}})=0\;.$ (13)

The equations (11) and (13) are evaluated at the states that we denote as
$\bar{A}^{\mu}$ and $\bar{\mathtt{a}}^{\mu}$, respectively. The
$\bar{A}^{\mu}$ means the vacuum state of the model, whereas
$\bar{\mathtt{a}}^{\mu}$ is an auxiliary state that is function of
$\bar{A}^{\mu}$ by the equation (13).

###  III.1 The case $T=0$ and $\mu=0$

For our analysis of the phase transitions, we base our analysis on the
cosmological principle of space isotropy to consider time-like background
fields such that $A^{\mu}=(A^{0},{\bf 0})$ and
$\mathtt{a}^{\mu}=(\mathtt{a}^{0},{\bf 0})$ in the integrals (9a)-(9c), and in
the effective potential (15). Under these assumptions, the physical results of
the integrals (9a)-(9c) are reduced to

$\displaystyle\Omega(\mathtt{a}_{0})\\!$ $\displaystyle=$
$\displaystyle\\!\frac{\Lambda^{4}}{8\pi^{2}}+\frac{\mathtt{a}_{0}^{4}}{12\pi^{2}}\;,$
(14a) $\displaystyle\Pi_{\mu}(\mathtt{a}_{0})\\!$ $\displaystyle=$
$\displaystyle\\!\delta_{\mu 0}\,\frac{\mathtt{a}_{0}^{3}}{3\pi^{2}}\;,$ (14b)
$\displaystyle\Xi(\mathtt{a}_{0})\\!$ $\displaystyle=$
$\displaystyle\\!-\frac{\mathtt{a}_{0}^{6}}{9\pi^{4}}\;,\;\;\;\;\;\;$ (14c)

where the regulator parameter $\Lambda$ remains in the
$\Omega(\mathtt{a}_{0})$, which can be interpreted as an energy scale of the
model. Therefore, the effective potential in this case is

$\displaystyle N^{-1}\,V_{eff}(A_{0},\mathtt{a}_{0})\\!$ $\displaystyle=$
$\displaystyle\\!\frac{\mathtt{a}_{0}^{3}\,A_{0}}{3\pi^{2}}+\frac{\mathtt{a}_{0}^{6}\,g}{18\pi^{4}N}-\frac{\mathtt{a}_{0}^{4}}{4\pi^{2}}+$
(15)
$\displaystyle+\frac{A_{0}^{2}}{2g}+\frac{\Lambda^{4}}{4\pi^{2}}\;.\;\;\;\;\;$

The PMS and GAP equations are, respectively, given by

$\displaystyle\frac{\bar{\mathtt{a}}_{0}^{3}\,g}{3\pi^{4}N}-\frac{\bar{\mathtt{a}}_{0}}{\pi^{2}}+\frac{\bar{A}_{0}}{\pi^{2}}=0\;,$
(16a)
$\displaystyle\bar{A}_{0}=-g\,\frac{\bar{\mathtt{a}}_{0}^{3}}{3\pi^{2}}\;,$
(16b)

in which the vacuum state is set by the time-like vector
$\bar{A}^{\mu}=(\bar{A}^{0},{\bf 0})$, and
$(\bar{\mathtt{a}}^{\mu}=\bar{\mathtt{a}}^{0},{\bf 0})$ is the correspondent
auxiliary state.

The asymptotic behavior of the PMS solution is the N-large limit (LN), when
$N\rightarrow\infty$, is reduced to :

$\displaystyle\mbox{LN}\;\mathrel{\mathop{\ordinarycolon}}\;\bar{\mathtt{a}}_{0}=\bar{A}_{0}\hskip
8.5359pt,\hskip 8.5359pt\bar{A}_{0}^{2}+\frac{3\pi^{2}}{g}=0\;.$ (17)

Thereby, the large-$N$ limit of the Lorentz symmetry is broken, if $g=-|g|$,
and the gap solution is given by $\bar{A}_{0}=\frac{\sqrt{3}\pi}{\sqrt{|g|}}$,
that confirms the result obtained in the ref. [31]. In the limit
$N\rightarrow\infty$, the effective potential is given by

$V_{eff}^{LN}(A_{0})=-\frac{A_{0}^{2}}{2g}+\frac{A_{0}^{4}}{12\pi^{2}}+\frac{1}{4\pi^{2}}\;.$
(18)

Clearly, the non-trivial minimum emerges when $g<0$. Although the analytic
form of the finite-N contributions to the effective potential has an intricate
non-linear character, one can plot it via numerical implementation. The
finite-N effective potential normalized by the factor $V_{0}=\Lambda^{4}$ is
plotted in Fig. (2) as a function of dimensionless variable $A_{0}/\Lambda$.
We choose $N=3$ in both plots. The Gap equation can be solved and is given by
:

$\bar{A}_{0}=\frac{\sqrt{3}\pi}{\sqrt{|g|}}\frac{1}{(1-1/N)^{3/2}}\;,$ (19)

and for $N=3$, the extra factor ${(1-1/N)^{-3/2}}$ is given a multiplicative
factor of approximately $1.837$. It is important to highlight that this extra
factor reflects the non-perturbative nature of the OPT approach.

Figure 2: The plot of the effective potential over $V_{0}=\Lambda^{4}$ as
function of the dimensionless variable $A_{0}/\Lambda$. In this figure, we
choose $N=3$, and $g\Lambda^{2}=-20$ (blue line). The black line sets the
limit $N\rightarrow\infty$.

Next, we introduce the temperature $(T)$ and chemical potential $(\mu)$ via
Matsubara formalism modifying the $p^{0}$-component by $p^{0}\rightarrow
i\omega_{n}-\mu$, and summing over all the Matsubara frequencies $\omega_{n}$.
This modification is clarified in the appendix VI. Thus, the formalism opens
the three possible cases $(T\neq 0,\mu\neq 0)$, $(T\neq 0,\mu=0)$, and
$(T=0,\mu\neq 0)$ that will be discussed in the next subsections.

##  IV Introducing the temperature and chemical potential

Whenever the thermodynamic quantities $T$ and $\mu$ are present, the
translational symmetry in time is broken, but it is kept for the space
coordinates. Thereby, we consider the $\mathtt{a}^{\mu}$ as a time-like
$4$-vector $\mathtt{a}^{\mu}=(\mathtt{a}^{0},{\bf 0})$, without any loss of
generality. This choice also is justified here in the case of a non-null
temperature in which the analysis of the phase transition emerges of a time-
like background vector, see [31]. The $\Omega$-integral (9a) in the presence
of a temperature $(T)$, and of a chemical potential $(\mu)$ is so written as :

$\displaystyle\Omega(\mathtt{a}_{0})=\int^{\Lambda}\frac{d^{3}p}{(2\pi)^{3}}\,\times$
$\displaystyle
T\sum_{\lambda=-1}^{+1}\sum_{n=-\infty}^{\infty}\ln\left[\left(i\omega_{n}-\mu+\lambda\mathtt{a}_{0}\right)^{2}-|{\bf
p}|^{2}\right]\;,$ (20)

that yields the result

$\displaystyle\Omega(\mathtt{a}_{0})\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!-\frac{T^{4}}{\pi^{2}}\sum_{\lambda=-1}^{+1}\left[\text{Li}_{4}\left(-e^{-\beta\mu_{\lambda}}\right)+\text{Li}_{4}\left(-e^{\beta\mu_{\lambda}}\right)\right]$
(21) $\displaystyle+\frac{\Lambda^{4}}{4\pi^{2}}\;,$

where

$\beta=T^{-1}$ is the inverse of the temperature,
$\mu_{\lambda}=\mu-\lambda\,\mathtt{a}_{0}\,(\lambda=\pm 1)$ is the chemical
potential summed to the time-component $\mathtt{a}_{0}$, and $\text{Li}_{4}$
is poly-logarithm function. In this result, $\Lambda$ is a cutoff parameter
with energy dimension. The integral (9b) for $T\neq 0$ is

$\displaystyle\Pi_{\mu}(\mathtt{a}_{0})=2\,\delta_{\mu
0}\\!\int^{\Lambda}\frac{d^{3}p}{(2\pi)^{3}}\\!\sum_{\lambda=-1}^{+1}\lambda$
$\displaystyle\times\sum_{n=-\infty}^{\infty}\frac{(i\omega_{n}-\mu+\lambda\mathtt{a}_{0})}{(i\omega_{n}-\mu+\lambda\mathtt{a}_{0})^{2}-|{\bf
p}|^{2}}\;,$ (22)

that yields

$\displaystyle\Pi_{\mu}(\mathtt{a}_{0})=\frac{T^{3}}{\pi^{2}}\,\delta_{\mu
0}\\!\sum_{\lambda=-1}^{+1}\lambda\left[\,\text{Li}_{3}\left(-e^{\beta\mu_{\lambda}}\right)\right.$
$\displaystyle\left.-\text{Li}_{3}\left(-e^{-\beta\mu_{\lambda}}\right)\,\right]\;.\;\;\;$
(23)

The $\Xi$-integral (9c), for $T\neq 0$, is read

$\displaystyle\Xi(\mathtt{a}_{0})\\!$ $\displaystyle=$
$\displaystyle\\!4\mathtt{a}_{0}\left[f_{1}(\mu_{+})f_{2}(\mu_{-})-f_{1}(\mu_{-})f_{2}(\mu_{+})\right]+$
(24)
$\displaystyle+4f_{2}(\mu_{-})f_{2}(\mu_{+})-4\mathtt{a}_{0}^{2}f_{1}(\mu_{-})f_{1}(\mu_{+})\;,\;$

where

$\displaystyle f_{1}(\mu_{\pm})$ $\displaystyle=$
$\displaystyle\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\frac{1}{(p\pm\mathtt{a}_{0})^{2}}=$
(25)
$\displaystyle=\frac{T^{2}}{4\pi^{2}}\left[\,\text{Li}_{2}\left(-e^{-\beta\mu_{\pm}}\right)+\text{Li}_{2}\left(-e^{\beta\mu_{\pm}}\right)\,\right]\;,$
$\displaystyle+\frac{\Lambda^{2}}{8\pi^{2}}\leavevmode\nobreak\
\leavevmode\nobreak\ ,\hskip 14.22636pt$

and

$\displaystyle f_{2}(\mu_{\pm})$ $\displaystyle=$
$\displaystyle\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\frac{p_{0}}{(p\pm\mathtt{a}_{0})^{2}}=$
(26)
$\displaystyle=\frac{T^{3}}{2\pi^{2}}\left[\,\text{Li}_{3}\left(-e^{\beta\mu_{\pm}}\right)-\text{Li}_{3}\left(-e^{-\beta\mu_{\pm}}\right)\,\right]+$
$\displaystyle\mp\,\mathtt{a}_{0}\,f_{1}(\mu_{\pm})\;,$

where $\text{Li}_{2,3}$ is a Poly-logarithm function of second/three degree.
Using these results we can discuss the particular cases of $(T\neq 0,\mu=0)$
and $(T=0,\mu\neq 0)$.

###  IV.1 The case $T\neq 0$ and $\mu=0$

Taking the limit $\mu\rightarrow 0$, the gap equation reads :

$\bar{A}_{0}=-\frac{2gT^{3}}{\pi^{2}}\left[\text{Li}_{3}\left(-e^{-\bar{\mathtt{a}}_{0}/T}\right)-\text{Li}_{3}\left(-e^{\bar{\mathtt{a}}_{0}/T}\right)\right]\;.$
(27)

The PMS equation in the limit $\mu\rightarrow 0$ is

$\displaystyle\left[\text{Li}_{2}\left(-e^{\bar{\mathtt{a}}_{0}/T}\right)+\text{Li}_{2}\left(-e^{-\bar{\mathtt{a}}_{0}/T}\right)\right]\times$
$\displaystyle\left[\bar{\mathtt{a}}_{0}-\bar{A}_{0}+\frac{2gT^{3}}{\pi^{2}N}\text{Li}_{3}\left(-e^{\bar{\mathtt{a}}_{0}/T}\right)+\right.$
$\displaystyle\left.-\frac{2gT^{3}}{\pi^{2}N}\,\text{Li}_{3}\left(-e^{-\bar{\mathtt{a}}_{0}/T}\right)\right]=0\;,$
(28)

in which we can check that $\bar{\mathtt{a}}_{0}=0$ implies into
$\bar{A}_{0}=0$. One can find the numerical solution for $\bar{A}_{0}(T)$ by
combining both equations. The solution is illustrated in Fig. 3 for
representative values of $N$ and $g$. Substituting the gap equation in the PMS
equation, and taking the limit $\bar{\mathtt{a}}_{0}\to 0$, we obtain the
solutions for the critical temperature

$T_{c}=\frac{\sqrt{3}}{\sqrt{|g|}}\frac{1}{\sqrt{1-1/N}}\;.$ (29)

The $N$-large limit fixes the critical temperature at $T_{c}=\sqrt{3/|g|}$.

Figure 3: The plot of the gap solution as a function of the dimensionless
variable $T/\Lambda$ at $\mu=0$. In this figure, we choose $N=3$, and
$g\Lambda^{2}=-1$ (blue line). The black line sets $g\Lambda^{2}=-1$ and the
limit $N\rightarrow\infty$.

###  IV.2 The case $T=0$ and $\mu\neq 0$

In the case of $T\rightarrow 0$, we need to use the limits

$\displaystyle\lim_{T\rightarrow 0}\left[\frac{1}{1+e^{\frac{|{\bf
p}|\pm\mu_{\lambda}}{T}}}\right]\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\Theta(-|{\bf p}|\mp\mu_{\lambda})\;,$ (30a)
$\displaystyle\lim_{T\rightarrow 0}T\ln\left[1+e^{-\frac{|{\bf
p}|+\mu_{\lambda}}{T}}\right]\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!-\left(|{\bf p}|+\mu_{\lambda}\right)\Theta(-|{\bf
p}|-\mu_{\lambda})\;,$ $\displaystyle\lim_{T\rightarrow
0}T\ln\left[1+e^{-\frac{|{\bf p}|-\mu_{\lambda}}{T}}\right]\\!\\!$
$\displaystyle=$ $\displaystyle\\!\\!\left(-|{\bf
p}|+\mu_{\lambda}\right)\Theta(-|{\bf p}|+\mu_{\lambda})\;,$

in which the $\Theta$-Heaviside function satisfies the conditions
$\Theta(x)=0$ for $x<0$, and $\Theta(x)=1$ for $x>0$. Using these properties,
the $\Omega$-integral is

$\lim_{T\rightarrow
0}\Omega(\bar{\mathtt{a}}_{0})=\frac{1}{12\pi^{2}}(6\,\bar{\mathtt{a}}_{0}^{2}\,\mu^{2}+\bar{\mathtt{a}}_{0}^{4}+3\pi^{4}\Lambda^{4}+\mu^{4})\;.$
(31)

Analogously, the result for $\Pi^{\mu}$ is

$\displaystyle\Pi_{\mu}(\bar{\mathtt{a}}_{0})=-\delta_{\mu
0}\;\frac{\bar{\mathtt{a}}_{0}}{3\pi^{2}}\,(\bar{\mathtt{a}}_{0}^{2}+3\mu^{2})\;.$
(32)

The limits (30a)-(30), when used in the functions $f_{1}$ and $f_{2}$ from
(25) and (26) are :

$\displaystyle\lim_{T\rightarrow 0}f_{1}(\mu_{\pm})\\!$ $\displaystyle=$
$\displaystyle\\!-\frac{(\mu_{\pm})^{2}-\Lambda^{2}}{8\pi^{2}}\;,$ (33a)
$\displaystyle\lim_{T\rightarrow 0}f_{2}(\mu_{\pm})\\!$ $\displaystyle=$
$\displaystyle\\!-\frac{\mu_{\pm}^{3}}{6\pi^{2}}\pm\mathtt{a}_{0}\frac{\mu_{\pm}^{2}-\Lambda^{2}}{8\pi^{2}}\;,$
(33b)

where the $\Xi$-integral is reduced to

$\displaystyle\Xi(\bar{\mathtt{a}}_{0})=-\frac{(\bar{\mathtt{a}}_{0}^{2}-\mu^{2})^{3}}{9\pi^{4}}\;.$
(34)

Thereby, the PMS and GAP equations are, respectively, given by

$\displaystyle\bar{\mathtt{a}}_{0}^{2}\,\bar{A}_{0}-\bar{\mathtt{a}}_{0}^{3}\left(1+\frac{2g\mu^{2}}{3\pi^{2}N}\right)+\frac{\bar{\mathtt{a}}_{0}^{5}\,g}{3\pi^{2}N}+$
$\displaystyle+\mu^{2}\,\bar{\mathtt{a}}_{0}\left(\frac{3\,g\,\mu^{2}}{\pi^{2}N}-1\right)+\bar{A}_{0}\,\mu^{2}=0\;,\hskip
28.45274pt$ (35a)
$\displaystyle\frac{\bar{A}_{0}}{g}+\frac{\bar{\mathtt{a}}_{0}}{3\pi^{2}}\,(\bar{\mathtt{a}}_{0}^{2}+3\mu^{2})=0\;.\hskip
28.45274pt$ (35b)

By combining both equations one can numerically find the solution for
$\bar{A}_{0}(\mu)$ and it is shown in Fig. 4 for some representative values of
$N$ and $g$. Once again, we can check that $\bar{\mathtt{a}}_{0}=0$ implies
into $\bar{A}_{0}=0$. Using the analysis of the PMS equation in the limit
$\bar{\mathtt{a}}_{0}\to 0$, after the application of the gap equation, the
critical value of chemical potential $(\mu_{c})$ satisfies the equation

$g\,\mu_{c}^{4}\left(\frac{1}{N}-3\right)-3\pi^{2}\mu_{c}^{2}=0\;,$ (36)

where the symmetry is restored as the solution of (36). The critical chemical
potential is constrained by the conditions $g<0$ and $N>1$. The solutions are
given by

$\mu_{c}=\frac{\pi}{\sqrt{|g|}}\frac{1}{\sqrt{1-\frac{1}{3N}}}\;,$ (37)

where the N-large limit has the result $\mu_{c}\approx\frac{\pi}{\sqrt{|g|}}$.

Figure 4: The plot of the gap solution as a function of the dimensionless
variable $\mu/\Lambda$ at $T=0$. In this figure, we choose $N=3$, and
$g\Lambda^{2}=-1$ (blue line). The black line sets $g\Lambda^{2}=-1$ and the
limit $N\rightarrow\infty$.

For last, the density of the symmetric phase at $T=0$ can be written as:

$n(T=0,\mu)=\frac{\mu^{3}}{3\pi^{2}}\left(1+\frac{|g|\mu^{2}}{\pi^{2}N}\right)\leavevmode\nobreak\
\leavevmode\nobreak\ .$ (38)

From this result, one can write the density at $\mu_{c}$ where the symmetry
breaks, and is given by:

$n(T=0,\mu_{c})=\frac{\sqrt{3}\pi(3N-4)}{(3N-1)^{5/2}\left(\frac{|g|}{N}\right)^{3/2}}\leavevmode\nobreak\
\leavevmode\nobreak\ .$ (39)

and in the large-N limit one finds
$n(T=0,\mu_{c})\rightarrow\frac{\pi}{3}|g|^{-3/2}$. For $N=3$ one has
$n(T=0,\mu_{c})\approx 1.64\times\frac{\pi}{3}|g|^{-3/2}$, which gives a
correction of $64\%$ on the critical density.

###  IV.3 Phase portrait

The phase diagram is shown in the fig. (6), and one can see that the critical
curve is modified in the finite-$N$ approach. Besides the quantitative
modification, there is no qualitative difference, i.e., the phase transition
still has a continuous behavior, and thus is a second-order phase transition.
In order to access the information about the density one need to start from
the thermodynamic potential defined as
$\Phi=V_{eff}(\bar{A}_{0},\bar{\mathtt{a}}_{0})$. Now, the density can be
properly and consistently defined as

$-\frac{\partial\Phi}{\partial\mu}=n-\frac{\partial\bar{A}_{0}}{\partial\mu}\frac{\partial\Phi}{\partial\bar{A}_{0}}-\frac{\partial\bar{\mathtt{a}}_{0}}{\partial\mu}\frac{\partial\Phi}{\partial\bar{\mathtt{a}}_{0}}\;,$
(40)

and by use of the PMS and gap equations, one shows that
$n=-\frac{\partial\Phi}{\partial\mu}$. This expression is used to construct
the phase portrait in Fig. 6.

Figure 5: The phase portrait of critical temperature versus the critical
chemical potential, for $g\Lambda^{2}=-1\leavevmode\nobreak\ $. The blue and
black lines correspond to the second-order phase transition. The black line
sets the transition that occurs in the limit $N\rightarrow\infty$.  Figure 6:
Plot of the phase diagram normalized density $n/\Lambda^{3}$, for $N=3$ and
$g\Lambda^{2}=-1\leavevmode\nobreak\ $.The blue line corresponds to the
second-order phase transition for $N=3$. A black dashed line represents the
second-order transition from the large-$N$ assumption.

##  V Conclusions

In this paper, we propose the study of $N$ massless fermions based on the ref.
[18] in the presence of temperature and chemical potential. The model has a
fermionic quartic self-interaction that introduces an auxiliary background
$4$-vector $A^{\mu}$, and another background $4$-vector $a^{\mu}$, and that,
in the presence of a temperature and chemical potential, breaks the Lorentz
symmetry. Using the path integral approach of quantum field theory, we obtain
the effective potential of the model at the one-loop, in which the temperature
is introduced via Matsubara formalism directly in the momentum integrals.
Thereby, we obtain the principle of minimal sensitivity (PMS) and the GAP
equations in terms of the $4$-vector fields $A^{\mu}$ and $a^{\mu}$.

The vacuum solution for the gap equation given by
$\bar{A}_{0}=\frac{\sqrt{3}\pi}{\sqrt{|g|}}\frac{1}{(1-1/N)^{3/2}}$ is
representative in terms to show the non-perturbative properties of the OPT
approach, since the non-polynomial character of the correction in comparison
with the large-$N$ results. The non-perturbative results show up also on the
critical temperature at zero chemical potential
$T_{c}=\frac{\sqrt{3}}{\sqrt{|g|}}\left(\sqrt{1-1/N}\right)^{-1/2}$ and the
critical chemical potential at zero temperature given by
$\mu_{c}=\frac{\pi}{\sqrt{|g|}}\left(1-1/3N\right)^{-1/2}$. These non-
perturbative results are one of our main results.

Going further, the phase portrait is shown in Fig. 6 and one can see that the
finite-$N$ corrections generate a region on the phase space
$T\times\text{density}$ which is larger than the large-$N$ counterpart.
Furthermore, one does not find any first-order phase transition, and the
finite-$N$ correction maintains the second-order phase transition between the
symmetric and LSV phases. In this work, the mass of the particles was
neglected in order to extract analytical results. The Lagrangian (1) may
exhibit interesting effects if a mass term is present. In particular, this
could lead to the occurrence of a first-order phase transition, which warrants
further investigation. This type of transition could be important and generate
LSV bubbles in the metastable region. Effects from $\delta^{2}$ corrections
can also bring new information about the phase transition and will be a target
of forthcoming work.

##  VI Appendix

The fermion propagator can be rewritten as follows:

$S(p)=\frac{1}{\not{p}+\not{\mathtt{a}}\gamma_{5}}=\frac{(\not{p}+\not{\mathtt{a}}\gamma_{5})(p^{2}-\mathtt{a}^{2}-[\,\not{p}\,,\,\not{\mathtt{a}}\,]\,\gamma_{5})}{(p^{2}-\mathtt{a}^{2})^{2}+4[\,p^{2}\,\mathtt{a}^{2}-(p\cdot\mathtt{a})^{2}\,]}\;,$
(41)

and through reorganization of the terms the denominator can be rewritten as
$D(p)=(p+\mathtt{a})^{2}(p-\mathtt{a})^{2}$. For the study of the
thermodynamic properties, one uses the Matsubara formalism, in which the
$p^{0}$-momentum integration is replaced by the sum :

$\int^{\Lambda}\frac{d^{4}p}{(2\pi)^{4}}\rightarrow
T\sum_{n=-\infty}^{\infty}\int^{\Lambda}\frac{d^{3}p}{(2\pi)^{3}}\;,$ (42)

where $p_{0}=i\omega_{n}-\mu=(2n-1)\,i\,\pi\,T-\mu$. Therefore, one has some
useful identities

$\displaystyle I_{0}(\mu,\omega_{p})$ $\displaystyle=$ $\displaystyle
T\\!\sum_{n=-\infty}^{\infty}\ln\left[\left(\omega_{n}+i\mu\right)^{2}+\omega_{p}^{2}\right]$
(43a) $\displaystyle=$
$\displaystyle\omega_{p}+T\ln\left(1+e^{-\frac{\omega_{p}+\mu}{T}}\right)$
$\displaystyle+\,T\ln\left(1+e^{-\frac{\omega_{p}-\mu}{T}}\right)\;,$
$\displaystyle I_{1}(\mu,\omega_{p})$ $\displaystyle=$ $\displaystyle
T\sum_{n=-\infty}^{\infty}\frac{1}{\left(\omega_{n}+i\mu\right)^{2}+\omega_{p}^{2}}$
(43b)
$\displaystyle=\frac{1}{2\omega_{p}}\left[1-\frac{1}{1+e^{(\omega_{p}+\mu)/T}}-\frac{1}{1+e^{(\omega_{p}-\mu)/T}}\right]\,,\hskip
22.76228pt$ $\displaystyle I_{2}(\mu,\omega_{p})$ $\displaystyle=$
$\displaystyle
T\sum_{n=-\infty}^{\infty}\frac{\omega_{n}+i\mu}{\left(\omega_{n}+i\mu\right)^{2}+\omega_{p}^{2}}$
(43c)
$\displaystyle=\frac{i}{2}\left[\frac{1}{1+e^{(\omega_{p}+\mu)/T}}-\frac{1}{1+e^{(\omega_{p}-\mu)/T}}\right]\;,$

where $\omega_{p}=|{\bf p}|$ for a massless fermion.

###### Acknowledgements.

Y.M.P.G would like to thank R.O.R., M.B.P., and G.P.B. for the insightful
discussions. Y.M.P.G. is supported by a postdoctoral grant from Fundação
Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ),
grant No. E26/201.937/2020.

## Bibliography

## References

  * [1] T. Araki et al. (KamLAND Collaboration), Measurement of Neutrino Oscillation with KamLAND: Evidence of Spectral Distortion, Phys. Rev. Lett. 94, 081801 (2005).
  * [2] C. Athanassopoulos et al, Phys. Rev. Lett. 77, 3082 (1996).
  * [3] C. Athanassopoulos et al, Phys. Rev. Lett. 81, 1774 (1998).
  * [4] P. Minkowski, $\mu\rightarrow e\,\gamma$ at a rate of one out of $10^{9}$ muon decays?, Phys. Lett. B 67, 421 (1977).
  * [5] R. N. Mohapatra and G. Senjanović, Neutrino Mass and Spontaneous Parity Nonconservation, Phys. Rev. Lett. 44, 912 (1980).
  * [6] R. N. Mohapatra and G. Senjanovic, Neutrino Masses and Mixings in Gauge Models with Spontaneous Parity Violation, Phys. Rev. D 23, 165 (1981).
  * [7] T. Yanagida, Horizontal gauge symmetry and masses of neutrinos, Conf. Proc. C 7902131, 95 (1979).
  * [8] M. Gell-Mann, P. Ramond and R. Slansky, Complex Spinors and Unified Theories, Conf. Proc. C 790927, 315 (1979) [arXiv/hep-th:1306.4669].
  * [9] S. L. Glashow, The Future of Elementary Particle Physics, NATO Sci. Ser. B 61, 687 (1980).
  * [10] J. C. Pati and A. Salam,, Lepton number as the fourth ”color”, Phys. Rev. D 10, 275 (1974).
  * [11] R. N. Mohapatra and J. C. Pati, ”Natural” left-right symmetry , Phys. Rev. D 11, 2558 (1975).
  * [12] G. Senjanovic and R. N. Mohapatra, Exact left-right symmetry and spontaneous violation of parity, Phys. Rev. D 12, 1502 (1975).
  * [13] Y. Zhang, H. An, X. Ji and R. N. Mohapatra, General CP Violation in Minimal Left-Right Symmetric Model and Constraints on the Right-Handed Scale, Nucl. Phys. B 802, 247 (2008).
  * [14] Y. M. P. Gomes, P. C. Malta, and M. J. Neves, Testing Lorentz-symmetry violation via electroweak decays, Phys. Rev. D 101, 075001 (2020).
  * [15] V. A. Kostelecký and M. Mewes, Astrophysical tests of Lorentz and CPT violation with photons, The Astrophysical Journal Letters 689, L1 (2008).
  * [16] D. Colladay, V. A. Kostelecký, CPT violation and the Standard Model, Phys. Rev. D 55, 6760-6774 (1997).
  * [17] D. Colladay, V. A. Kostelecký, Lorentz-violating extension of the Standard Model, Phys. Rev. D 58, 116002 (1998).
  * [18] Y. M. P. Gomes and M. J. Neves, Reconciling LSND and Super-Kamiokande data through the dynamical Lorentz symmetry breaking in a four-Majorana fermion model, Phys. Rev. D 106, 015013 (2022).
  * [19] W. Thirring, A Soluble Relativistic Field Theory, Annals of Physics 3 (1958) 91–112.
  * [20] David J. Gross and André Neveu , Dynamical symmetry breaking in asymptotically free field theories, Phys. Rev. D 10, 3235 (1974).
  * [21] Y. Nambu and G. Jona-Lasinio, Dynamical Model of Elementary Particles Based on an Analogy with Superconductivity, Phys. Rev. 122, 345 (1961).
  * [22] J. L. Kneur, M. B. Pinto, R. O. Ramos and E. Staudt, Updating the phase diagram of the Gross–Neveu model in $2+1$ dimensions, Phys. Lett. B, 657 (1-3), 136-142, (2007).
  * [23] Jean-Loic Kneur, et al. Emergence of tricritical point and liquid-gas phase in the massless $2+1$ dimensional Gross-Neveu model, Phys. Rev. D 76, 045020 (2007).
  * [24] Kneur, J. L., Pinto, M. B., and Ramos, R. O. Phase diagram of the magnetized planar Gross-Neveu model beyond the large-N approximation. Physical Review D, 88(4), 045005,(2013).
  * [25] Kneur, J. L., Pinto, M. B., Ramos, R. O., and Staudt, E. Vector-like contributions from Optimized Perturbation in the Abelian Nambu–Jona-Lasinio model for cold and dense quark matter. International Journal of Modern Physics E, 21(03), 1250017 (2012).
  * [26] Kneur, J. L., Pinto, M. B., and Ramos, R. O. Thermodynamics and phase structure of the two-flavor Nambu–Jona-Lasinio model beyond large N c. Physical Review C,81(6), 065205 (2010).
  * [27] Gomes, Y. M. P., Martins, E., Pinto, M. B., and Ramos, R. O. (2023). First-order phase transitions within Weyl type of materials at low temperatures. Phys. Rev. B 108, 085107 (2023).
  * [28] Anna Okopinska, Nonstandard expansion techniques for the effective potential in $\lambda\phi^{4}$ quantum field theory, Physical Review D 35, 1835 (1987).
  * [29] V. I. Yukalov, Interplay between Approximation Theory and Renormalization Group, Phys. Part. Nucl. 50, 141 (2019).
  * [30] M. Gomes et al., Dynamical Lorentz and CPT symmetry breaking in a 4D four-fermion model, Phys. Rev. D 77, 105002 (2008).
  * [31] J. F. Assuncao, T. Mariz, J. R. Nascimento and A. Yu. Petrov, Dynamical Lorentz symmetry breaking in a 4D massless four-fermion model, Phys. Rev. D 96, 065021 (2017).
  * [32] Assunção, J. F., Mariz, T., Nascimento, J. R., and Petrov, A. Y. Dynamical Lorentz symmetry breaking in a tensor bumblebee model. Physical Review D, 100(8), 085009 (2019).
  * [33] Charneski, B., Gomes, M., Mariz, T., Nascimento, J. R., and da Silva, A. J. Dynamical Lorentz symmetry breaking in 3D and charge fractionalization. Physical Review D, 79(6), 065007 (2009).


[1] [1]Faculty of Information Technology, Monash University, Melbourne, VIC,
3800, Australia [2]AI Division, Westlake University, Hangzhou, Zhejiang,
310024, P.R. China

[1] [1]

[1]

[2]

[1]Corresponding author: Shilin Qu, Weiqing Wang

# Hypergraph Node Representation Learning with One-Stage Message Passing

Shilin Qu<EMAIL_ADDRESS>Weiqing Wang<EMAIL_ADDRESS>Yuan-Fang
Li<EMAIL_ADDRESS>Xin Zhou<EMAIL_ADDRESS>Fajie Yuan
<EMAIL_ADDRESS>
###### Abstract

Hypergraphs as an expressive and general structure have attracted considerable
attention from various research domains. Most existing hypergraph node
representation learning techniques are based on graph neural networks, and
thus adopt the two-stage message passing paradigm (i.e. node $\rightarrow$
hyperedge $\rightarrow$ node). This paradigm only focuses on local information
propagation and does not effectively take into account global information,
resulting in less optimal representations. Our theoretical analysis of
representative two-stage message passing methods shows that, mathematically,
they model different ways of local message passing through hyperedges, and can
be unified into one-stage message passing (i.e. node $\rightarrow$ node).
However, they still only model local information. Motivated by this
theoretical analysis, we propose a novel one-stage message passing paradigm to
model both global and local information propagation for hypergraphs. We
integrate this paradigm into HGraphormer, a Transformer-based framework for
hypergraph node representation learning. HGraphormer injects the hypergraph
structure information (local information) into Transformers (global
information) by combining the attention matrix and hypergraph Laplacian.
Extensive experiments demonstrate that HGraphormer outperforms recent
hypergraph learning methods on five representative benchmark datasets on the
semi-supervised hypernode classification task, setting new state-of-the-art
performance, with accuracy improvements between 2.52% and 6.70%. Our code and
datasets are available111https://anonymous.4open.science/r/HGraphormer-DF3F.

###### keywords:

Hypergraph Graph TransformerNode Representation

##  1 Introduction

Compared with regular graphs, hypergraphs Yadati et al. (2019); Feng et al.
(2019); Yoon et al. (2020); Amburg et al. (2020); Huang and Yang (2021); Liu
et al. (2021a) is a more flexible and general data structure that allows an
edge to join any number of nodes, which can naturally capture high-order
relationships among multiple nodes. Take the paper-authorship hypergraph in
Figure 1(left) as an example, nodes ($v_{1},v_{2},\ldots$) represent papers,
and papers written by the same author form a hyperedge ($e_{1},e_{2},\ldots$).
Hypergraph has attracted much attention in a wide range of fields, including
computer vision Gao et al. (2022); Liu et al. (2020), natural sciences Gu et
al. (2020), social networks Yu et al. (2021a); Sun et al. (2021), and
recommendation systems Xia et al. (2021); Xue et al. (2021); Cheng et al.
(2022).

Figure 1: (left) Paper-Authorship Hypergraph. The nodes represent papers and
multiple papers written by one author form a hyperedge. (right) The incidence
matrix $\mathbf{H}$ and hypergraph Laplacian $\mathbf{L}$ of Paper-Authorship
Hypergraph. $e_{1},...,e_{4}$ and $v_{1},...,v_{7}$ are hyperedges and nodes,
respectively.

Message passing is the foundation of graph neural networks (GNNs) Kipf and
Welling (2017); Veličković et al. (2018); Yun et al. (2019); Ying et al.
(2021) and hypergraph neural networks (HGNNs) Gao et al. (2022); Xue et al.
(2021); Cheng et al. (2022); Liu et al. (2020); Yadati et al. (2019); Feng et
al. (2019); Huang and Yang (2021). The ultimate goal of this mechanism is to
learn good node representations for downstream tasks. (Hyper)edges serve as a
bridge for information transmission, transfer and aggregation. Conceptually,
GNNs utilises one-stage message passing, directly from nodes to nodes (i.e.,
node$\rightarrow$node), as illustrated in Figure 2. In contrast, HGNNs utilise
two-stage message passing Choe et al. (2022); Bera et al. (2022); Lee et al.
(2021), firstly from nodes to hyperedges, then from hyperedges to nodes (i.e.,
node$\rightarrow$hyperedge$\rightarrow$node), as illustrated in Figure 3. Such
two-stage message passing is intuitively motivated by the nature of
hypergraphs, that a node can be connected by multiple nodes through a
hyperedge. Compared with one-stage message passing, two-stage message passing
transfers information between nodes indirectly through hyperedges.

Figure 2: One-stage message passing for GNNs. $\mathbf{v}_{i}$ is the features
of node $v_{i}$. $a_{(v_{i},v_{j})}$ is the weight between node $v_{i}$ and
node $v_{j}$. Figure 3: Two-stage message passing widely used in HGNNs.
$\mathbf{v}_{i}$ and $\mathbf{e}_{j}$ are the features of node $v_{i}$ and
hyperedge $e_{j}$ respectively.

As one of the most popular representation learning frameworks, Transformer
Vaswani et al. (2017) has demonstrated state-of-the-art performance in many
domains, such as natural language processing Hou et al. (2022); Yu et al.
(2021b); Fei et al. (2021); Zhao et al. (2020a, b), audio Haghani et al.
(2018); Chung and Glass (2020), computer vision Arnab et al. (2021); Khan et
al. (2021); Liu et al. (2021b), and graphs Ying et al. (2021); Dwivedi and
Bresson (2020); Hu et al. (2020); Zhang et al. (2020). By carefully designing
position embeddings and the attention matrix, graph structures can be
naturally injected into Transformer Hu et al. (2020); Yang et al. (2020);
Nguyen et al. (2022). For instance, Graphormer Ying et al. (2021) integrates
attention and adjacency matrix to control the message passing between nodes
and achieves state-of-the-art results on multiple real-world datasets (e.g.,
MolHIV, MolPCBA, ZINC).

Following the two-stages message passing paradigm, some recent works on
hypergraph representation learning employ Transformer as the encoder module at
each stage of message passing Georgiev et al. (2022); Chien et al. (2022).
More specifically, they first use Transformer to encode nodes to obtain the
hyperedge representations, then use Transformer to encode hyperedges to
generate node representations.

A fundamental deficiency of the two-stage message passing paradigm is its
inability to effectively incorporate global information. Global information
means the directed interaction (latent correlation) between nodes no matter
whether there is a path between them. However, in two-stage message passing,
interaction between nodes only happen via shared hyperedge and there must be
at least one path between these two nodes. In other words, it only enables
local information passing but lacks global information. For example, in Figure
1(left), node $v_{2}$ must undergo four message passing steps to interact with
node $v_{5}$, through the path $v_{2}\to e_{2}\to v_{4};v_{4}\to e_{3}\to
v_{5}$. Furthermore, there is no path to connect node $v_{2}$ and node
$v_{6}$. No matter how many times the message is propagated, no information
interaction will happen between node $v_{2}$ and node $v_{6}$. But in
practice, $v_{2}$ is an improvement work on $v_{5}$ and $v_{6}$, which means
node $v_{2}$, $v_{5}$ and $v_{6}$ are very closely related. In other words,
such latent semantic correlation (global information) between nodes is also
essential for modeling node representation.

In this paper, we conduct a theoretical analysis on the two-stage messaging
passing mechanism and derive that it is a special case of one-stage message
passing (Sec.4). In addition, we also observe that the Transformer
architecture is amenable to one-stage message passing.

Based on these findings, we propose HGraphormer, a novel framework that deeply
integrates HyperGraph and Transformer that is able to directly and fully
exploit Transformer’ strong representation powers (Sec. 5) instead of simply
using Transformer as encoders like existing Transformer+hypergraphs works
Georgiev et al. (2022); Chien et al. (2022). Distinct from all existing
hypergraph node representation learning techniques, HGraphormer employs the
one-stage message passing paradigm, which is able to incorporate both local
and global interactions between nodes, resulting in better node
representations. This is achieved by injecting a hypergraph’s structure
information into Transformer via the combination of hypergraph Laplacian and
attention matrix of the Transformer. HGraphormer treats hypergraph Laplacian
and attention matrix as local and global connections, respectively. The local
connections represent the local structure of a hypergraph, which is exploited
by existing two-stage message passing techniques. The global connection
enhances information flow between nodes, even though some nodes have no direct
or undirect connection, which further improves the representation power of
HGraphormer. The advantage of incorporating global interactions is
experimentally validated in Sec.6.3.

In summary, our contributions are fourfold.

  * •

We propose a new one-stage message passing paradigm for hypergraph node
representation learning. It can model global and local information regardless
whether there is a direct or indirect connection between nodes. To the best of
our knowledge, we are the first work in the field of hypergraphs to use one-
stage message passing.

  * •

We propose HGraphormer, a Transformer-based hypergraph node representation
learning framework, which injects structure information by combining the
attention matrix and the hypergraph Laplacian.

  * •

We give theoretical proof that, for hypergraphs, two-stage message passing can
be transformed into one-stage message passing. Compared with two-stage, one-
stage message passing is more flexible to be extend, which allows the
structure of hypergraphs to be more naturally injected into Transformers.

  * •

Through extensive experiments on five real-world benchmark datasets about node
classification task, our HGraphormer technique outperforms strong baseline
methods, setting new state-of-the-art performance.

##  2 Related Work

This section briefly reviews existing works of hypergraph learning and
Transformer on (hyper)graph.

###  2.1 Hypergraph Learning

Learning node representation based on GNNs with graph-based message passing is
mature Kipf and Welling (2017); Veličković et al. (2018); Yun et al. (2019);
Ying et al. (2021). Whereas emulating a graph-based message passing framework
for hypergraphs is not straightforward, a hyperedge may consist of more than
two nodes, making the interactions among hyperedge complex. HyperSAGE Arya et
al. (2020) uses a two-stage message passing strategy to propagate information
through hypergraphs. First, it facilitates neighbourhood node sampling to form
hyperedge representations. Then, node representations are aggregated by
hyperedges’ representations with a power mean function. Similarly, AllSet
Chien et al. (2022) also implements hypergraph neural network layers as
compositions of two multiset functions, aggregating node sets and hyperedge
sets respectively. Huang and Yang (2021) further points out that GNNs can be
extended to hypergraphs. Based on the graph-based message passing paradigm of
GCN Kipf and Welling (2017), GAT Veličković et al. (2018), GIN Yun et al.
(2019) and GraphSAGE Hamilton et al. (2017), Huang and Yang (2021) proposes a
series of UniGNN (UniGCN, UniGAN, UniGIN, and UniSAGE) following two-stage
message passing paradigm. To the best of our knowledge, this series of UniGNN
are the state-of-the-art methods for hypergraph node representation learning.

###  2.2 Transformer on (hyper)graph

Transformer has demonstrated a great power in modelling graph-structured data
Min et al. (2022); Yao et al. (2020); Georgiev et al. (2022); Chien et al.
(2022). Two typical ways to inject the graph information into the vanilla
Transformer are designing positional embedding or attention matrix from
graphs. Arya et al. (2020) adopts Laplacian eigenvectors as positional
embeddings in Graph Transformer. Hussain et al. (2021) employs pre-computed
SVD vectors of the adjacency matrix as the positional embeddings. While
positional embedding is convenient to incorporate graph with Transformer,
compressing graph structure into fixed-size vectors (positional embedding)
suffers from information lossGraves et al. (2016). To overcome this issue,
another group of works attempts to design attention matrix based on the graph
structure. Min et al. (2022) uses the adjacent matrix to mask the attention
matrix. Yao et al. (2020) computes the attention matrix based on the extended
Levi graph. However, attention module can only capture pairwise relationships
rather than high-order relationships of hypergraphs. Recent Transformer-based
hypergraph works Georgiev et al. (2022); Chien et al. (2022) employ
Transformer as an encoder module for each stage of message passing. They first
encode nodes to form hyperedge representations, then encode hyperedges to
produce node representations. Nevertheless, above single-stage encoder
strategy does not fully exploit the ability of global modelling of
Transformer.

##  3 Preliminaries

In this section, we introduce the main concepts and formulate the problem. The
important notations used in this paper are summarized in Table 1.

###  3.1 Hypergraph

A hypergraph is defined as $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}$ is the node set and $\mathcal{E}$ is the hyperedge set. A
hyperedge $e\in\mathcal{E}=\\{v_{1},v_{2},...\\}$ connects a set of nodes
instead of only two in regular graphs. Here each $v_{i}$ is a node and
$\left|e\right|\geq 2$. A weighted hypergraph is defined as
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{W})$, where a positive number
$\mathcal{W}_{e}$, the weight of hyperedge $e$, is associated with each $e$. A
hyperedge $e$ is said to be incident with a node $v$ when $v\in e$. A
hypergraph $\mathcal{G}$ can be denoted by an $N\times M$ incidence matrix
$\mathbf{H}$, where $N=\left|\mathcal{V}\right|$ and
$M=\left|\mathcal{E}\right|$, with entries defined as follows:

$\begin{split}H_{v,e}=\left\\{\begin{array}[]{ll}1,&\text{ if }v\in e\\\
0,&\text{ otherwise}\end{array}\right.\end{split}$ (1)

For a node $v\in\mathcal{V}$ and a hyperedge $e\in\mathcal{E}$, their degrees
are defined as $d(v)=\sum_{e\in\mathcal{E}}\mathcal{W}_{e}H_{v,e}$ and
$d(e)=\sum_{v\in\mathcal{V}}H_{v,e}$. Further, $\mathbf{D}_{e}$ and
$\mathbf{D}_{v}$ are diagonal matrices denoting the hyperedge degrees and the
node degrees, respectively. $\mathbf{W}$ is the diagonal matrix containing the
weights of hyperedges. The initial features for all nodes are denoted as
$\mathbf{X}$.

###  3.2 Hypergraph Laplacian

For a given hypergraph $\mathcal{G}$, the incidence matrix $\mathbf{H}$
represents its structure (see Figure 1 top right). Each row in $\mathbf{H}$
shows a node’s connection to each hyperedge, representing high-order
relationships. Based on the incidence matrix $\mathbf{H}$, Zhou et al. (2006)
proposes hypergraph Laplacian $\mathbf{L}$ (see Figure 1 bottom right) where
each element represents the connection between nodes. $\mathbf{L}$ is defined
as follows,

$\mathbf{L}=\mathbf{D}_{v}^{-1/2}\mathbf{HW}\mathbf{D}_{e}^{-1}\mathbf{H}^{\top}\mathbf{D}_{v}^{-1/2}$
(2)

where $\mathbf{L}\in\mathbb{R}^{N\times N}$. The element $L_{i,k}$ of
$\mathbf{L}$ denotes the weight between node $v_{i}$ and node $v_{k}$.

Table 1: Notations and Definitions Notations | Definitions
---|---
$b,\mathbf{b},\mathbf{B},\mathcal{B}$ | Scalar, vector, matrix and set
$\mathcal{G}$ |  |  $\mathcal{G}=(\mathcal{V},\mathcal{E})$ indicates a hypergraph. $\mathcal{V}$ are $\mathcal{E}$
---
are node set and hyperedge set, respectively.
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{W})$ indicates a weighted
hypergraph,
a positive number $\mathcal{W}_{e}$ associated with each
hyperedge $e$.
$\mathcal{E}$ | Hyperedges set of $\mathcal{G}$
$\mathcal{V}$ | Nodes set of $\mathcal{G}$
$\mathcal{W}$ | Hyperedge weights set of $\mathcal{G}$
$v_{i},e_{j}$ | Node $v_{i}$ and hyperedge $e_{j}$
$\mathcal{E}_{v_{i}}$ |  |  $\mathcal{E}_{v_{i}}=\left\\{e\in\mathcal{E}|v_{i}\in e\right\\}$ denotes the incident-edges
---
of node $v_{i}$.
$M$ | The number of hyperedges on $\mathcal{G}$, $M=\left|\mathcal{E}\right|$
$N$ | The number of nodes on $\mathcal{G}$, $N=\left|\mathcal{V}\right|$
$\mathbf{H}$ | The incidence Matrix of $\mathcal{G}$ , $\mathbf{H}\in\mathbb{R}^{N\times M}$
$\mathbf{L}$ |  | The hypergraph Laplacian of $\mathcal{G}$,
---
$\mathbf{L}=\mathbf{D}_{v}^{-1/2}\mathbf{HW}\mathbf{D}_{e}^{-1}\mathbf{H}^{\top}\mathbf{D}_{v}^{-1/2}$,
$\mathbf{L}\in\mathbb{R}^{N\times N}$.
$\mathbf{W},\mathbf{D}_{e},\mathbf{D}_{v}$ |  | The diagonal matrix of hyperedge weights,
---
hyperedge degrees and node degrees on $\mathcal{G}$,
respectively. $\mathbf{W}\in\mathbb{R}^{M\times M}$,
$\mathbf{D}_{e}\in\mathbb{R}^{M\times M}$
and $\mathbf{D}_{v}\in\mathbb{R}^{N\times N}$.
$\mathbf{X}$ |  | Initial features for all nodes on $\mathcal{G}$ , $\mathbf{X}\in\mathbb{R}^{N\times c}$ ,
---
$c$ is feature dimension.
$\mathbf{M}$ |  | The attention matrix of Scaled Dot-Product
---
Laplacian Attention, $\mathbf{M}\in\mathbb{R}^{N\times N}$
$\mathcal{N}(v_{i})$ |  | Node $v_{i}$’s neighborhood nodes across
---
all hyperedges.
$d_{h}$ | Hidden dimension of HGraphomer
$d_{k},d_{q}$ | Dimension of attention Key and attention Value
$\mathbf{Z}^{l}$ | The output of $l$th HGraphomer layer. $\mathbf{Z}^{l}\in\mathbb{R}^{N\times d_{h}}$

###  3.3 Transformer

The Transformer architecture consists of Transformer layers. The core of each
Transformer layer is the self-attention module. Let
$\mathbf{Z}^{l}=[{\mathbf{z}^{l}_{1}}^{T},...,{\mathbf{z}^{l}_{N}}^{T}]^{T}\in\mathbb{R}^{N\times
d_{h}}$ denotes the input of self-attention module of layer $l$, where $d_{h}$
is the hidden dimension of $\mathbf{Z}^{l}$ and
$\mathbf{z}^{l}_{i}\in\mathbb{R}^{1\times d_{h}}$ is the hidden representation
of layer $l$. The input $\mathbf{Z}^{l}$ is projected by three matrices
$\mathbf{W}_{Q}\in\mathbb{R}^{d_{h}\times d_{k}}$,
$\mathbf{W}_{K}\in\mathbb{R}^{d_{h}\times d_{k}}$ and
$\mathbf{W}_{V}\in\mathbb{R}^{d_{h}\times d_{q}}$ to the corresponding
representation matrices $\mathbf{Q,K,V}$. Then the self-attention is
calculated as,

$\mathbf{Q}=\mathbf{Z}^{l}\mathbf{W}_{Q},\quad\mathbf{K}=\mathbf{Z}^{l}\mathbf{W}_{K},\quad\mathbf{V}=\mathbf{Z}^{l}\mathbf{W}_{V},$
(3) $\mathbf{M}=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}})$ (4)
$Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{M}\mathbf{V}$ (5)

We call $\mathbf{M}$ as attention matrix which captures the similarity between
queries and keys. As a result, self-attention in Transformer can only model
pairwise relationships. Furthermore, the extension to the multi-head attention
is standard and straightforward.

###  3.4 Problem Formulation

Given a hypergraph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{W})$, our
task is to learn the representations of nodes $v\in\mathcal{V}$ for downstream
tasks, such as node classification.

##  4 Message Passing in Hypergraph

In this section, we first analyse the existing two-stage message passing
paradigm on hypergraphs, based on which we propose our first intuitive version
of one-stage message passing paradigm for hypergraphs which will be extended
to a “Plus” version in Sec. 5.1.

Most existing HGNNs follow a two-stage message passing paradigm and aggregate
node representations and hyperedge representations in stages iteratively.
Firstly, node representations are aggregated to generate hyperedge
representations through a predefined node aggregation function. Then, a
predefined hyperedge aggregation function aggregates hyperedge representations
to update node representations. The representation learning formulations for
hyperedges and nodes are defined as follows,

$\displaystyle\mathbf{e}_{j}$
$\displaystyle=\phi_{1}\left(\left\\{\mathbf{v_{i}}\right\\}_{v_{i}\in
e_{j}}\right)$ (6) $\displaystyle\hat{\mathbf{v}_{i}}$
$\displaystyle=\phi_{2}\left(\left\\{\mathbf{e}_{j}\right\\}_{e_{j}\in\mathcal{E}_{v_{i}}}\right)$
(7)

where $\mathbf{v}_{k}$ is the node ${v}_{k}$’s initial representation.
$\phi_{1}$ and $\phi_{2}$ are permutation-invariant functions for updating
hyperedge representation $\mathbf{e}_{j}$ and node representation
$\hat{\mathbf{v}_{i}}$ from corresponding nodes and hyperedges respectively.
$\mathcal{E}_{v_{i}}=\left\\{e\in\mathcal{E}|v_{i}\in e\right\\}$ denotes the
set of incident hyperedges of node $v_{i}$.

###  4.1 Two-stage Message Passing

####  4.1.1 HyperSAGE

As a representative HGNNs-based model, HyperSAGE Arya et al. (2020) follows
the two-stage message passing paradigm. With $\mathcal{N}(v_{i})$ denotes node
$v_{i}$’s neighborhood nodes across all hyperedges, HyperSAGE aggregates
information with the following formulations,

$\displaystyle\mathbf{e}_{j}$
$\displaystyle=(\frac{1}{d_{e_{j}}}\sum_{v_{k}\in
e_{j}}\mathbf{v}^{p}_{k})^{\frac{1}{p}}$ (8)
$\displaystyle{\hat{\mathbf{v}_{i}}}$
$\displaystyle=(\frac{1}{d_{v_{i}}}\sum_{e_{j}\in\mathcal{E}_{i}}\frac{d_{e_{j}}}{\left|\mathcal{N}(v_{i})\right|}(\mathbf{e}_{j})^{p})^{\frac{1}{p}}$
(9)

where $d_{e_{j}}$ and $d_{v_{i}}$ are the degree of hyperedge $e_{j}$ and node
$v_{i}$ respectively; and $p$ is a power value. Following prior work Arya et
al. (2020), we set $p=1$ as it achieves the best performance.

This work focuses on the node representation learning task, so we eliminate
${e}_{j}$ by plugging Eq. (8) into Eq. (9). Finally, Eq. (9) can be rewritten
as,

$\begin{split}{\hat{\mathbf{v}_{i}}}=\frac{1}{d_{v_{i}}\left|\mathcal{N}(v_{i})\right|}\sum_{e_{j}\in\mathcal{E}_{v_{i}}}\sum_{v_{k}\in
e_{j}}\mathbf{v}_{k}\end{split}$ (10)

Eq. (10) indicates that node representation $\hat{\mathbf{v}_{i}}$ is a
weighted sum of representations of all nodes connecting $v_{i}$ through
hyperedges. The weight between $v_{k}$ and $v_{i}$ is their co-occurs in
hyperedges normalized by $v_{i}$’s degree and the number of its neighbour
nodes.

####  4.1.2 UniGCN

UniGCN Huang and Yang (2021) is a recent representative work of two-stage
message passsing paradigm. Similar to GCN Kipf and Welling (2017), it
propagates message using weighted sum function,

$\displaystyle\mathbf{e}_{j}$ $\displaystyle=\frac{1}{d_{e_{j}}}\sum_{v_{k}\in
e_{j}}\mathbf{v}_{k}$ (11) $\displaystyle\hat{\mathbf{v}_{i}}$
$\displaystyle=\frac{1}{\sqrt{d_{v_{i}}}}\sum_{e_{j}\in\mathcal{E}_{i}}\frac{1}{\sqrt{d_{e_{j}}}}w_{d_{e_{j}}}\mathbf{e}_{j}$
(12)

where $w_{d_{e_{j}}}$ is a specific weight to the hyperedge degrees. Similar
to Eq. (10), we merge Eq. (11) and Eq. (12) as follows:

$\begin{split}\hat{\mathbf{v}_{i}}=\frac{1}{\sqrt{d_{v_{i}}}}\sum_{e_{j}\in\mathcal{E}_{i}}\sum_{v_{k}\in
e_{j}}\frac{w_{e_{j}}}{{d_{e_{j}}}^{\frac{3}{2}}}\mathbf{v}_{k}\end{split}$
(13)

Similar to Eq. (10), the node representation $\hat{\mathbf{v}_{i}}$ in Eq.
(13) is also a weighted sum of representations of the nodes connected through
hyperedges.

###  4.2 One-stage Message Passing

For the two-stage message passing paradigm, messages flow alternatively
between nodes and hyperedges (i.e. nodes $\rightarrow$ hyperedges
$\rightarrow$ nodes $\rightarrow\ldots$), in which hyperedges play the role of
bridges. Information flows not only start at nodes, but also end at nodes.
Thus, a natural question arises, _why not model message passing between nodes
directly?_ Motivated by our derivation in Eq. (10) and Eq. (13), we propose an
intuitive one-stage message passing paradigm to learn node-to-node
interactions directly.

We derive one-stage message passing equations from two-stage message passing
equations in Section 4.1, which simplify the message passing flow in
hypergraphs. Eq. (10) and Eq. (13) show different functions to learn weights
between two connected nodes on incident hyperedges. Hence, an intuitive one-
stage message passing paradigm can be formulated as follows,

$\begin{split}\hat{\mathbf{v}_{i}}=\sum_{v_{k}\in\mathcal{V}}w(\\{e_{j}|e_{j}\in\mathcal{E}_{v_{k},v_{i}}\\},v_{k},v_{i})\mathbf{v}_{k}\end{split}$
(14)

where $w(\\{e_{j}|e_{j}\in\mathcal{E}_{v_{k},v_{i}}\\},v_{k},v_{i})$ is a
weight function associated with $v_{k}$ and $v_{i}$ on concurrent hyperedges
set of
$\mathcal{E}_{v_{k},v_{i}}=\mathcal{E}_{v_{k}}\bigcap\mathcal{E}_{v_{i}}$,
which controls the amount of message passing between $v_{k}$ and $v_{i}$. Eq.
(14) can be illustrated through Figure 4. However, one disadvantage of both
the traditional two-stage message passing and the above simple one-stage
message passing as shown in Eq. (14) is that there is no message passing
between $v_{k}$ and $v_{i}$ if
$\mathcal{E}_{v_{k}}\bigcap\mathcal{E}_{v_{i}}=\varnothing$. In other words,
Eq. (14) represents a _local_ message passing paradigm, which is not optimal,
as in many real-world situations, implicit relationships between nodes may
exist even if there is no connection between them. Therefore, messages passing
should also be _global_ to flow over implicit relationships.

Figure 4: An example of our proposed one-stage message passing paradigm for
hypergraphs, where $\mathbf{v}_{i}$ represents the features of node $v_{i}$.
$w(\\{e_{j}|e_{j}\in\mathcal{E}_{v_{k},v_{i}}\\},v_{k},v_{i})$ controling
message passing between node $v_{i}$ and node $v_{k}$.

In the next section, we introduce our Transformer-based hypregraph
representation method that supports both local and global message passing in
the one-stage message passing paradigm.

##  5 HGraphormer

In this section, we present our HGraphormer for hypergraph node representation
learning. We first propose a new message passing paradigm on hypergraphs which
is a “Plus” version of the basic one-stage message passing paradigm in Sec.
4.2. One-stage message passing Plus can model both global and local feature at
the same time. Then, we elaborate several vital designs by injecting
hypergraph structure into the attention module of Transformer to learn
hypergraph node representations, and detail the implementations of
HGraphormer.

###  5.1 One-stage Message Passing Plus

Recall Eq. (14) updates $\hat{\mathbf{v}_{i}}$ with only nodes that are
directly connected to node ${v}_{i}$, which limits the model’s ability in
capturing global information (e.g. unconnected nodes) in hypergraphs. However,
it has been demonstrated Ying et al. (2021) that global information is
important in graphs. To exploit this global information, we define one-stage
message passing “Plus” paradigm in Eq. (15), which enables HGraphormer to
capture both global and local information.

$\begin{split}\hat{\mathbf{v}_{i}}=\sum_{v_{k}\in\mathcal{V}}(w^{global}_{ki}+w^{local}_{ki})\mathbf{v}_{k}\end{split}$
(15)

where $w^{global}_{ki}$ captures the implicit relationship between $v_{k}$ and
$v_{i}$. At the same time, $w^{local}_{ki}$ captures the local connections
between $v_{k}$ and $v_{i}$, which is hypergraph structure. The next
subsection will illustrate how to compute $w^{global}$ and $w^{local}$.

###  5.2 Structural Injection in HGraphormer

The self-attention matrix $\mathbf{M}$ defined in Eq. (4) represents the
semantical pairwise correlations between any nodes, which can capture the
global information in hypergraphs. This is also the reason why Transformer
achieve global receptive field. However, $\mathbf{M}$ does not involve the
local relational (structure) information of hypergraph.

Hypergraph Laplacian $\mathbf{L}$, as illustrated in Sec. 3.2, is the spatial
view of hypergraph structure, which highlights the relationships between
connected nodes. Hence, we inject hypergraph Laplacian $\mathbf{L}$ to the
attention matrix $\mathbf{M}$ in TransformerThe structured attention matrix
$\mathbf{A}$ is defined as,

$\mathbf{A}=\gamma\mathbf{M}+(1-\gamma)\mathbf{L}$ (16)

where $\gamma$ is a hyper-parameter to balance semantic correlation between
nodes and structure of hypergraphs, and we share $\gamma$ and $\mathbf{L}$
across all layers. Here, $M$ and $L$ represent $w^{global}$ and $w^{local}$ in
Eq. 15 respectively.

###  5.3 Design Details

HGraphormer is built upon the original implementation of classical Transformer
encoder Vaswani et al. (2017). It consists of Scaled Dot-Product Laplacian
Attention, Multi-Head Laplacian Attention and HGraphormer Layer.

####  5.3.1 Scaled Dot-Product Laplacian Attention

As the core module of HGraphomer, scaled dot-product Laplacian attention
encodes the semantic correlations of nodes and the hypergraph structure
through one-stage message passing, which is defined as follows,

$\begin{split}LaplacianAtt(\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{L},\gamma)&=\mathbf{A}\mathbf{V}\\\
&=\left(\gamma\mathbf{M}+(1-\gamma)\mathbf{L}\right)\mathbf{V}\end{split}$
(17)

where $\mathbf{M},\mathbf{Q},\mathbf{K},\mathbf{V}$ can be caculated through
Eq. (3) and Eq. (4) with the input $\mathbf{Z}^{l}$ which is node
representations from the $l$-th layer. Figure 5 is an illustration of this
module.

Figure 5: Scaled Dot-Product Laplacian Attention. Table 2: Statistic of five real-world hypergraph datasets. $\left|\mathcal{V}\right|$ and $\left|\mathcal{E}\right|$ denotes the number of nodes and edges, respectively. $avg.\ d_{v}=\frac{1}{\left|\mathcal{V}\right|}\sum_{v\in\mathcal{V}}d_{v}$ denotes average node degree. $avg.\ d_{e}=\frac{1}{\left|\mathcal{E}\right|}\sum_{e\in\mathcal{E}}d_{e}$ denotes average hyperedge degree. $avg.\ d_{Neig}=\frac{1}{\left|\mathcal{V}\right|}\sum_{v\in\mathcal{V}}\left|\mathcal{N}(v_{i})\right|$ denotes average neighbourhood node number. $c$ the dimension of node features, $C$ is the number of classes, and $\eta$ denotes the label rate. Method | Co-authorship Data | Co-citation Data
---|---|---
DBLP |  Cora-ca |  Pubmed |  Citeseer |  Cora-cc
$\left|\mathcal{V}\right|$ | 43,413 | 2,708 | 19,717 | 3,312 | 2708
$\left|\mathcal{E}\right|$ | 22,535 | 1,072 | 7,963 | 1,079 | 1,579
$avg.\ d_{v}$ | 4.7 | 4.2 | 4.3 | 3.2 | 3.0
$avg.\ d_{e}$ | 2.41 | 1.92 | 9.01 | 2.69 | 3.33
$avg.\ d_{Neig}$ | 21.94 | 13.51 | 65.49 | 6.91 | 6.94
$C$ | 6 | 7 | 3 | 6 | 7
$c$ | 1,425 | 1,433 | 500 | 3,703 | 1,433
$\eta$ | 4.0% | 5.2% | 0.8% | 4.2% | 5.2%

####  5.3.2 Multi-Head Laplacian Attention

Multi-Head Laplacian Attention allows HGraphormer to jointly process
information from different representation subspaces. It is defined as follows,

$MultiHead(\mathbf{Z}^{l},\mathbf{L},\gamma)=Concat(head_{1},...,head_{h})\mathbf{W}_{Z}$
(18)

where
$head_{i}=LaplacianAtt(\mathbf{Z}^{l}\mathbf{W}_{Q},\mathbf{Z}^{l}\mathbf{W}_{K},\mathbf{Z}^{l}\mathbf{W}_{V},\mathbf{L},\gamma)$.
$\mathbf{Z}^{l}$ is node representations from the $l$-th layer.
$\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V}$ are linear mapping. $h$ is the
number of attention heads, and $\mathbf{W}_{Z}\in\mathbb{R}^{hd_{q}\times
d_{h}}$ is a linear function to reduce dimensions. Figure 6 illustrates its
mechanism.

Figure 6: Multi-Head Laplacian Attention. Multi-Head consists of several
attention layers running in parallel.

####  5.3.3 HGraphormer Layer

Unlike the design of the original Transformer encoder, we only apply residual
layers and layer normalization (LN) after mutil-head Laplacian attention222We
find the feed-forward blocks here lead to over-fitting.. We formally
characterize the HGraphormer layer as follows,

$\begin{split}\mathbf{Z}^{l+1}&=HGraphormer(\mathbf{Z}^{l},\mathbf{L},\gamma))\\\
&=LN(MultiHead(\mathbf{Z}^{l},\mathbf{L},\gamma))+\mathbf{Z}^{l}\end{split}$
(19)

Figure 7: Framwork of HGraphormer

where $\mathbf{Z}^{0}=FFD(\mathbf{X}$), and $\mathbf{X}\in\mathbb{R}^{N\times
c}$ is the initial feature of nodes. $FFD$ represents feed-forward blocks and
$c$ is the dimension of node features. HGraphormer can be built by stacking
multiple HGraphormer layers. Figure 7 shows the overall framework of
HGraphormer. For the node classification task, the stacked layers are followed
by a dense layer and a softmax layer and the loss function is a standard
cross-entropy on nodes with ground-truth class labels.

##  6 Experiments

In this section, we conduct experiments on five real-world datasets to answer
the following research questions:

  * •

(Q1) Does our proposed HGraphormer achieve state-of-art performance in
hypergraph node representation learning tasks?

  * •

(Q2) Does the "Plus" version of one-stage message passing paradigm outperform
that of the basic version which models the local information only and how do
the global features and local features impact the performance of our
HGraphormer framework?

  * •

(Q3) What are the effects of hyperparameters and different modules on the
performance?

###  6.1 Experimental Setup

####  6.1.1 Datasets.

We use five widely-used hypergraph datasets Yadati et al. (2019); Huang and
Yang (2021); Sen et al. (2008); Rossi and Ahmed (2015): DBLP, Cora-ca and
Cora-cc from Cora, Pubmed, and CiteSeer. Nodes in these hypergraphs represent
publications, and hyperedges represent co-authorship/co-citation
relationships. The co-authorship hypergraphs are built from DBLPand Cora,
whose hyperedges represent all documents published by the same author. The co-
citation hypergraphs are constructed from PubMed, Citeseer, and Cora, whose
hyperedges connect all documents cited by the same documents. Since Cora
dataset contains both two relationships, we denote Cora as Cora-ca and Cora-cc
for co-authorship and co-citation separately. These datasets are of varying
sizes, with the number of nodes ranges from 2,708 to 43,413, and the number of
hyperedges ranges from 1,072 to 22,535, allowing us to evaluate model
performance comprehensively. Other properties also show a great variability
amongst the datasets, as can be seen in Table 2.

Table 3: Testing accuracy (%) of of HGraphormer and other hypergraph learning methods on co-authorship and co-citation datasets for Semi-supervised Hypernode Classification. The best and second best results are bolded and underlined respectively for each dataset. The final row shows the improvement of HGraphormer compared with the second best results. Method | Co-authorship Data | Co-citation Data
---|---|---
DBLP |  Cora-ca |  Pubmed |  Citeseer |  Cora-cc
MLP+HLR Zhou et al. (2006) | 63.6 ± 4.7 | 59.8 ± 4.7 | 64.7 ± 3.1 | 56.1 ± 2.6 | 61.0 ± 4.1
HGNN Feng et al. (2019) | 69.2 ± 5.1 | 63.2 ± 3.1 | 66.8 ± 3.7 | 56.7 ± 3.8 | 70.0 ± 2.9
FastHyperGCN Yadati et al. (2019) | 68.1 ± 9.6 | 61.1 ± 8.2 | 65.7 ± 11.1 | 56.2 ± 8.1 | 61.3 ± 10.3
HyperGCN Yadati et al. (2019) | 70.9 ± 8.3 | 63.9 ± 7.3 | 68.3 ± 9.5 | 57.3 ± 7.3 | 62.5 ± 9.7
HyperSAGE Arya et al. (2020) | 77.4 ± 3.8 | 72.4 ± 1.6 | 72.9 ± 1.3 | 61.8 ± 2.3 | 69.3 ± 2.7
UniGAT Huang and Yang (2021) | 88.7 ± 0.2 | 75.0 ± 1.1 |  $\underline{74.7}$ ± 1.2 |  $\underline{63.8}$ ± 1.6 | 69.2 ± 2.9
UniGCN Huang and Yang (2021) |  $\underline{88.8}$ ± 0.2 |  $\underline{75.3}$ ± 1.2 | 74.4 ± 1.0 | 63.6 ± 1.3 | 70.1 ± 1.4
UniSAGE Huang and Yang (2021) | 88.5 ± 0.2 | 75.1 ± 1.2 | 74.3 ± 1.0 |  $\underline{63.8}$ ± 1.3 |  $\underline{70.2}$ ± 1.5
AllSetTransformer Huang and Yang (2021) | 90.25 ± 0.1 | 76.3 ± 1.00 | 85.1 ± 0.22 | 67.3 ± 0.81 | 72.1 ± 0.91
HGraphormer |  92.3 ± 0.19 |  77.2 ± 1.9 |  77.2 ± 2.3 |  67.5 ± 2.6 |  74.9 ± 2.8
Improvement | 4.29%$\uparrow$ | 2.52%$\uparrow$ | 3.35%$\uparrow$ | 5.80%$\uparrow$ | 6.70%$\uparrow$
Figure 8: Test accuracy by varying $\gamma$ in
$\gamma\mathbf{M}+(1-\gamma)\mathbf{L}$ (Eq. 16) on five datasets, where $Acc$
is accuracy

####  6.1.2 Task and Evaluation Protocols.

We use node classification to evaluate the effectiveness of our representation
learning model following the existing hypergraph node representation learning
works Feng et al. (2019); Gao et al. (2022); Jiang et al. (2019). More
specifically, the task is to predict the topic (e.g., CS and Math) for each
node (multi-class classification). Node class label rate $\eta$ on five
datasets ranges from 0.8% to 5.2%, which means that essentially our task here
is semi-supervised node classification task. We adopt the 10-fold cross
validation in previous works Yadati et al. (2019); Huang and Yang (2021) for a
fair and standard evaluation. More specifically, in each split, for each
topic, the labels of 10% of the nodes belonging to that topic are hidden.
Evaluation is performed on this set of 10% of nodes without labels. This
process in repeated 10 times to obtain the mean and standard deviation of
classification accuracy for each model.

####  6.1.3 Baseline Methods.

We compare HGraphormer method with eight strong baseline models, which are
Multi-Layer Perceptron with explicit Hypergraph Laplacian Regularization
(MLP+HLR) Zhou et al. (2006), HyperGraph Neural Network (HGNN) Feng et al.
(2019), Fast HyperGraph Convolutional Network (FastHyperGCN) Yadati et al.
(2019), HyperGraph Convolutional Network (HyperGCN), HyperSAGE Arya et al.
(2020), UniGAT, UniGCN and UniSAGE Huang and Yang (2021) with full batch. On
all datasets, the feature of each node is initialised with the bag-of-words
representation of documents. We adopt the settings of all baseline models from
Srivastava et al. (2014), which are the optimal settings for these models.

####  6.1.4 Parameter Settings.

We use the Adam optimizer with a learning rate of 0.01 and the weight decay of
0.0005. Activation function is PReLU He et al. (2015). To prevent overfitting,
we add a dropout layer after each HGraphormer layer output. $d_{k},d_{q}$ are
tuned in the range of {16, 32, 64, 128}.

###  6.2 Overall Results (Q1)

We implemented HGraphormer for the downstream task of semi-supervised node
classification on hypergraphs to compare with state-of-the-art methods. Table
3 summarizes the mean and standard deviation of classification accuracy of all
the compared methods on the five datasets. We present the major findings
below.

  * •

HGraphormer consistently outperforms all the baselines with a noticeable
performance lift, achieving a new state-of-the-art in the node classification
task on hypergraphs. Compared with all the second best methods over the five
datasets, the average improvement of HGraphormer is 4.53%. This considerable
performance improvement demonstrates the effectiveness of the overall design
of HGraphormer.

  * •

HGraphormer gains more significant improvements on the datasets Cora-cc and
Citeseer compared with baselines, with increases of 6.70% and 5.80% over the
second best models respectively, while the average improvement is 4.53% across
all five datasets. The possible reason is that by modelling the global
information, HGraphormer is able to better mitigate the node connection
sparsity problem as the node connections on these two datasets are sparser
compared with other datasets. The average number of neighbor nodes is usually
used to measure the connectivity between nodes in (hyper)graphs. As shown in
Table 2, $avg.\ d_{Neig}$, which denotes the average number of neighbourhood
nodes, is 6.91 and 6.94 on Citeseer and Cora-cc respectively, which are
significantly lower than 21.94, 13.51 and 65.49 of the other three datasets.

  * •

MLP+HLR Zhou et al. (2006) performs consistently the worst across all
datasets. The possible reason is that all the other methods are based on the
message passing scheme except MLP+HLR. This indicates the strength of message
passing on hypergraph representation learning.

  * •

UniGAT, UniGCN and UniSAGE achieve the second-best performance on different
datasets. This shows that, apart from our proposed HGraphormer, there is no
best baseline which performs consistently best across all datasets. This
further validates the superiority of HGraphormer which is able to produce
consistent best performance on all datasets.

Figure 9: Accuracy influenced by different heads settings

###  6.3 Effect of Global and Local Features (Q2)

A core contrlibution of HGraphomer is the injection of structure information
into Transformer through $\gamma\mathbf{M}+(1-\gamma)\mathbf{L}$, which
incorporates the structure of the hypergraph and semantic correlations between
nodes. To answer Q2, we vary the value of the hyperparameter $\gamma$ as it
controls the balance between global and local message passing. When
$\gamma=0$, only local information is modelled as only the Laplacian is fed to
the model; while when $\gamma=1$, only global information is modelled as the
hypergraph Laplacian, which captures the structure information, is completely
disgarded from the model. For each dataset, we tune hyperparameter $\gamma$
from 0 to 1 with the step size of 0.1, i.e.
$\gamma\in\\{0,0.1,0.2,\ldots,0.9,1.0\\}$. In Figure 8, we observe a
consistent trend among all datasets, and elaborate our findings below.

  * •

HGraphomer achieves optimal performance when both local and global information
are taken into account, i.e., $\gamma$ is neither 0 or 1. This shows that both
local information and global information are essential to a model’s
performance, which validates the motivation and effectiveness of our design of
one-stage message passing plus compared with its basic version that only
models local information.

  * •

On all five datasets, the results when $\gamma=0$ are consistently better than
when $\gamma=1$. This shows that compared with global module $\mathbf{M}$, the
local structural module $\mathbf{L}$ has a higher impact on the performance of
HGraphomer. The direct connection between nodes is the main path of
information transmission.

  * •

The global structural module $\mathbf{M}$ based on semantic correlation plays
a complementary role of local information in the one-stage message passing
plus paradigm. From $\gamma=0$ to $\gamma=\gamma_{optimal}$, the accuracy
rising slowly with the increase of $\gamma$. The accuracy drops off rapidly as
$\gamma$ increases beyond $\gamma_{optimal}$ to 1. This observation
demonstrates the importance of incorporating global information, while making
sure local information is sufficiently represented.

  * •

Global information (semantic correlations) can achieve competitive results in
the scenarios where local information (real structure) cannot be obtained.
When $\gamma=1$, only global information is used. However, HGraphomer still
outperforms HyperSAGE on the DBLP datast (0.77) and UniSAGE on the Cora-cc
dataset (0.72) with the accuracy of 0.77 and 0.74 respectively.

###  6.4 Parameter Sensitivity Study (Q3)

We further investigate the parameters and modules of HGraphomer in order to
provide insights on how to best optimize our framework for node representation
learning.

####  6.4.1 Residual Layer

Table 4 summarizes the accuracy influenced by different residual and layer
settings. We observe a consistent trend in all datasets that HGraphomer with
residual shows considerable improvement than it without residual, especially
when $L>1$. And before $L$ reaches a certain critical value, the accuracy rise
with the increase of $L$, because residual layer makes HGraphformer deeper and
deep networks allow information to travel further along the hypergraph
structure.

Table 4: Accuracy influenced by different residual and layer settings. Best result is marked bold. $L$ denotes the number of HGraphormer layers. $R$ denotes residual. W/O and W/ denote "Without" and "With" residual. $R$ | $L$ | DBLP | Cora-ca | Pubmed | Citeseer | Cora-cc
---|---|---|---|---|---|---
W/O | 1 | 0.9013 | 0.7583 | 0.7606 | 0.6666 | 0.7473
2 | 0.8802 | 0.7204 | 0.7551 | 0.3589 | 0.7407
3 | 0.5732 | 0.3878 | 0.4569 | 0.2636 | 0.7038
4 | 0.2863 | 0.3287 | 0.4515 | 0.2490 | 0.2624
5 | 0.2632 | 0.3156 | 0.4454 | 0.2493 | 0.2581
6 | 0.2513 | 0.3131 | 0.4453 | 0.2448 | 0.2393
7 | 0.2354 | 0.3044 | 0.4409 | 0.2571 | 0.1982
W/ | 1 | 0.9220 | 0.7550 | 0.7495 | 0.6320 | 0.7474
2 | 0.9231 | 0.7652 | 0.7576 | 0.6484 | 0.7493
3 | 0.9099 | 0.7722 | 0.7587 | 0.6620 | 0.7408
4 | 0.8824 | 0.4943 | 0.7645 | 0.6641 | 0.7435
5 | 0.8704 | 0.3605 | 0.7722 | 0.6755 | 0.7402
6 | 0.8579 | 0.3250 | 0.7685 | 0.6593 | 0.7340
7 | 0.8079 | 0.3095 | 0.7442 | 0.6435 | 0.7291

####  6.4.2 Heads

Figure 9 shows the accuracy influenced by different attention head settings.
We observe that when HGraphormer achieves optimal, the number of head is $>1$.
And it happened cross all five datesets. This observation confirms Multi-Head
Laplacian Attention allows HGraphormer to jointly process information from
different representation subspaces.

##  7 CONCLUSION AND FUTURE WORK

In this work, we have designed a one-stage message passing paradigm to model
both global and local information for node representation learning on
hypergraphs. Under this paradigm, we propose HGraphormer, a novel framework
that injects hypergraph structure into Transformer, which consistently
performs better than recent state-of-the-art methods on five real-world
datasets. Our future work will extend HGraphormer for broader downstream tasks
of hypergraphs, such as hyperedge prediction and hypergraph classification.

## References

  * Amburg et al. (2020) Amburg, I., Veldt, N., Benson, A., 2020.  Clustering in graphs and hypergraphs with categorical edge labels, in: Proceedings of The Web Conference 2020\.  URL: https://doi.org/10.1145/3366423.3380152, doi:10.1145/3366423.3380152.
  * Arnab et al. (2021) Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C., 2021\.  Vivit: A video vision transformer.  arXiv preprint arXiv:2103.15691 .
  * Arya et al. (2020) Arya, D., Gupta, D.K., Rudinac, S., Worring, M., 2020\.  Hypersage: Generalizing inductive representation learning on hypergraphs.  arXiv preprint arXiv:2010.04558 .
  * Bera et al. (2022) Bera, S.K., Bhattacharya, S., Choudhari, J., Ghosh, P., 2022\.  A new dynamic algorithm for densest subhypergraphs, in: Proceedings of the ACM Web Conference 2022.  URL: https://doi.org/10.1145/3485447.3512158, doi:10.1145/3485447.3512158.
  * Cheng et al. (2022) Cheng, D., Chen, J., Peng, W., Ye, W., Lv, F., Zhuang, T., Zeng, X., He, X., 2022.  Ihgnn: Interactive hypergraph neural network for personalized product search.  URL: https://doi.org/10.1145/3485447.3511954, doi:10.1145/3485447.3511954.
  * Chien et al. (2022) Chien, E., Pan, C., Peng, J., Milenkovic, O., 2022.  You are allset: A multiset function framework for hypergraph neural networks, in: International Conference on Learning Representations.  URL: https://openreview.net/forum?id=hpBTIv2uy_E.
  * Choe et al. (2022) Choe, M., Yoo, J., Lee, G., Baek, W., Kang, U., Shin, K., 2022.  Midas: Representative sampling from real-world hypergraphs, in: Proceedings of the ACM Web Conference 2022\.  URL: https://doi.org/10.1145/3485447.3512157, doi:10.1145/3485447.3512157.
  * Chung and Glass (2020) Chung, Y.A., Glass, J., 2020\.  Generative pre-training for speech with autoregressive predictive coding, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.
  * Dwivedi and Bresson (2020) Dwivedi, V.P., Bresson, X., 2020\.  A generalization of transformer networks to graphs.  arXiv preprint arXiv:2012.09699 .
  * Fei et al. (2021) Fei, H., Ren, Y., Wu, S., Li, B., Ji, D., 2021.  Latent target-opinion as prior for document-level sentiment classification: A variational approach from fine-grained perspective, in: Proceedings of the Web Conference 2021\.  URL: https://doi.org/10.1145/3442381.3449789, doi:10.1145/3442381.3449789.
  * Feng et al. (2019) Feng, Y., You, H., Zhang, Z., Ji, R., Gao, Y., 2019\.  Hypergraph neural networks.  Proceedings of the AAAI Conference on Artificial Intelligence URL: https://ojs.aaai.org/index.php/AAAI/article/view/4235, doi:10.1609/aaai.v33i01.33013558.
  * Gao et al. (2022) Gao, Y., Zhang, Z., Lin, H., Zhao, X., Du, S., Zou, C., 2022.  Hypergraph learning: Methods and practices.  IEEE Transactions on Pattern Analysis and Machine Intelligence doi:10.1109/TPAMI.2020.3039374.
  * Georgiev et al. (2022) Georgiev, D., Brockschmidt, M., Allamanis, M., 2022.  Heat: Hyperedge attention networks.  arXiv preprint arXiv:2201.12113 .
  * Graves et al. (2016) Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J., et al., 2016\.  Hybrid computing using a neural network with dynamic external memory.  Nature 538, 471–476.
  * Gu et al. (2020) Gu, X., Chen, L., Krenn, M., 2020.  Quantum experiments and hypergraphs: Multiphoton sources for quantum interference, quantum computation, and quantum entanglement.  Phys. Rev. A URL: https://link.aps.org/doi/10.1103/PhysRevA.101.033816, doi:10.1103/PhysRevA.101.033816.
  * Haghani et al. (2018) Haghani, P., Narayanan, A., Bacchiani, M., Chuang, G., Gaur, N., Moreno, P., Prabhavalkar, R., Qu, Z., Waters, A., 2018.  From audio to semantics: Approaches to end-to-end spoken language understanding, in: 2018 IEEE Spoken Language Technology Workshop (SLT), IEEE.
  * Hamilton et al. (2017) Hamilton, W., Ying, Z., Leskovec, J., 2017.  Inductive representation learning on large graphs, in: Advances in Neural Information Processing Systems.  URL: https://proceedings.neurips.cc/paper/2017/file/5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf.
  * He et al. (2015) He, K., Zhang, X., Ren, S., Sun, J., 2015.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision.
  * Hou et al. (2022) Hou, M., Xu, C., Li, Z., Liu, Y., Liu, W., Chen, E., Bian, J., 2022.  Multi-granularity residual learning with confidence estimation for time series prediction, in: Proceedings of the ACM Web Conference 2022.  URL: https://doi.org/10.1145/3485447.3512056, doi:10.1145/3485447.3512056.
  * Hu et al. (2020) Hu, Z., Dong, Y., Wang, K., Sun, Y., 2020.  Heterogeneous graph transformer, in: Proceedings of The Web Conference 2020.  URL: https://doi.org/10.1145/3366423.3380027, doi:10.1145/3366423.3380027.
  * Huang and Yang (2021) Huang, J., Yang, J., 2021.  Unignn: a unified framework for graph and hypergraph neural networks, in: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence.  URL: https://doi.org/10.24963/ijcai.2021/353, doi:10.24963/ijcai.2021/353.
  * Hussain et al. (2021) Hussain, M.S., Zaki, M.J., Subramanian, D., 2021.  Edge-augmented graph transformers: Global self-attention is enough for graphs.  arXiv preprint arXiv:2108.03348 .
  * Jiang et al. (2019) Jiang, J., Wei, Y., Feng, Y., Cao, J., Gao, Y., 2019\.  Dynamic hypergraph neural networks, in: Proceedings of the 28th International Joint Conference on Artificial Intelligence.
  * Khan et al. (2021) Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M., 2021\.  Transformers in vision: A survey.  arXiv preprint arXiv:2101.01169 .
  * Kipf and Welling (2017) Kipf, T.N., Welling, M., 2017\.  Semi-supervised classification with graph convolutional networks.  International Conference on Learning Representations URL: https://openreview.net/forum?id=SJU4ayYgl.
  * Lee et al. (2021) Lee, G., Choe, M., Shin, K., 2021.  How do hyperedges overlap in real-world hypergraphs? \- patterns, measures, and generators, in: Proceedings of the Web Conference 2021.  URL: https://doi.org/10.1145/3442381.3450010, doi:10.1145/3442381.3450010.
  * Liu et al. (2021a) Liu, M., Veldt, N., Song, H., Li, P., Gleich, D., 2021a.  Strongly local hypergraph diffusions for clustering and semi-supervised learning.  doi:10.1145/3442381.3449887.
  * Liu et al. (2020) Liu, S., Lv, P., Zhang, Y., Fu, J., Cheng, J., Li, W., Zhou, B., Xu, M., 2020.  Semi-dynamic hypergraph neural network for 3d pose estimation, in: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence.  URL: https://doi.org/10.24963/ijcai.2020/109, doi:10.24963/ijcai.2020/109.
  * Liu et al. (2021b) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021b.  Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision.
  * Min et al. (2022) Min, E., Rong, Y., Xu, T., Bian, Y., Zhao, P., Huang, J., Luo, D., Lin, K., Ananiadou, S., 2022.  Masked transformer for neighhourhood-aware click-through rate prediction.  arXiv preprint arXiv:2201.13311 .
  * Nguyen et al. (2022) Nguyen, D.Q., Nguyen, T.D., Phung, D., 2022.  Universal graph transformer self-attention networks, in: Companion Proceedings of the Web Conference 2022.  URL: https://doi.org/10.1145/3487553.3524258, doi:10.1145/3487553.3524258.
  * Rossi and Ahmed (2015) Rossi, R., Ahmed, N., 2015.  The network data repository with interactive graph analytics and visualization, in: Proceedings of the AAAI Conference on Artificial Intelligence.
  * Sen et al. (2008) Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T., 2008\.  Collective classification in network data.  AI magazine .
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research .
  * Sun et al. (2021) Sun, X., Yin, H., Liu, B., Chen, H., Meng, Q., Han, W., Cao, J., 2021.  Multi-level hyperedge distillation for social linking prediction on sparsely observed networks, in: Proceedings of the Web Conference 2021.  URL: https://doi.org/10.1145/3442381.3449912, doi:10.1145/3442381.3449912.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017\.  Attention is all you need, in: Advances in Neural Information Processing Systems.
  * Veličković et al. (2018) Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y., 2018\.  Graph Attention Networks.  International Conference on Learning Representations URL: https://openreview.net/forum?id=rJXMpikCZ.
  * Xia et al. (2021) Xia, X., Yin, H., Yu, J., Shao, Y., Cui, L., 2021.  Self-supervised graph co-training for session-based recommendation, in: Proceedings of the 30th ACM International Conference on Information and Knowledge Management.  URL: https://doi.org/10.1145/3459637.3482388, doi:10.1145/3459637.3482388.
  * Xue et al. (2021) Xue, H., Yang, L., Rajan, V., Jiang, W., Wei, Y., Lin, Y., 2021.  Multiplex bipartite network embedding using dual hypergraph convolutional networks, in: Proceedings of the Web Conference 2021.  URL: https://doi.org/10.1145/3442381.3449954, doi:10.1145/3442381.3449954.
  * Yadati et al. (2019) Yadati, N., Nimishakavi, M., Yadav, P., Nitin, V., Louis, A., Talukdar, P., 2019\.  Hypergcn: A new method for training graph convolutional networks on hypergraphs, in: Advances in Neural Information Processing Systems.  URL: https://proceedings.neurips.cc/paper/2019/file/1efa39bcaec6f3900149160693694536-Paper.pdf.
  * Yang et al. (2020) Yang, L., Ng, T.L.J., Smyth, B., Dong, R., 2020\.  Html: Hierarchical transformer-based multi-task learning for volatility prediction, in: Proceedings of The Web Conference 2020.  URL: https://doi.org/10.1145/3366423.3380128, doi:10.1145/3366423.3380128.
  * Yao et al. (2020) Yao, S., Wang, T., Wan, X., 2020.  Heterogeneous graph transformer for graph-to-sequence learning, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7145–7154.
  * Ying et al. (2021) Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., Liu, T.Y., 2021.  Do transformers really perform badly for graph representation?, in: Advances in Neural Information Processing Systems.
  * Yoon et al. (2020) Yoon, S.e., Song, H., Shin, K., Yi, Y., 2020.  How much and when do we need higher-order information in hypergraphs? a case study on hyperedge prediction, in: Proceedings of The Web Conference 2020.  URL: https://doi.org/10.1145/3366423.3380016, doi:10.1145/3366423.3380016.
  * Yu et al. (2021a) Yu, J., Yin, H., Li, J., Wang, Q., Hung, N.Q.V., Zhang, X., 2021a.  Self-supervised multi-channel hypergraph convolutional network for social recommendation, in: Proceedings of the Web Conference 2021.  URL: https://doi.org/10.1145/3442381.3449844, doi:10.1145/3442381.3449844.
  * Yu et al. (2021b) Yu, P., Fei, H., Li, P., 2021b.  Cross-lingual language model pretraining for retrieval, in: Proceedings of the Web Conference 2021.  URL: https://doi.org/10.1145/3442381.3449830, doi:10.1145/3442381.3449830.
  * Yun et al. (2019) Yun, S., Jeong, M., Kim, R., Kang, J., Kim, H.J., 2019\.  Graph transformer networks, in: Proceedings of the 33rd International Conference on Neural Information Processing Systems.
  * Zhang et al. (2020) Zhang, J., Zhang, H., Xia, C., Sun, L., 2020.  Graph-bert: Only attention is needed for learning graph representations.  arXiv preprint arXiv:2001.05140 .
  * Zhao et al. (2020a) Zhao, X., Wang, L., He, R., Yang, T., Chang, J., Wang, R., 2020a.  Multiple knowledge syncretic transformer for natural dialogue generation, in: Proceedings of The Web Conference 2020.  URL: https://doi.org/10.1145/3366423.3380156, doi:10.1145/3366423.3380156.
  * Zhao et al. (2020b) Zhao, X., Xiao, F., Zhong, H., Yao, J., Chen, H., 2020b.  Condition aware and revise transformer for question answering, in: Proceedings of The Web Conference 2020.  URL: https://doi.org/10.1145/3366423.3380301, doi:10.1145/3366423.3380301.
  * Zhou et al. (2006) Zhou, D., Huang, J., Schölkopf, B., 2006.  Learning with hypergraphs: Clustering, classification, and embedding, in: Advances in Neural Information Processing Systems.  URL: https://proceedings.neurips.cc/paper/2006/file/dff8e9c2ac33381546d96deea9922999-Paper.pdf.