# Study of the Coulomb and nuclear breakup of 11Be using a Halo-EFT
description at N2LO

L.-P. Kubushishi<EMAIL_ADDRESS>Institut für Kernphysik, Johannes
Gutenberg-Universität Mainz, D-55099 Mainz, Germany  P. Capel  pcapel@uni-
mainz.de Institut für Kernphysik, Johannes Gutenberg-Universität Mainz,
D-55099 Mainz, Germany

###### Abstract

Background

The halo effective field theory (Halo-EFT) provides a very efficient
description of loosely-bound nuclei in models of reaction. It offers a very
systematical ranking of the significance of nuclear-structure observables in
reaction calculations. This greatly helps to infer reliable structure
information from reaction cross sections. However, for a meaningful analysis,
the Halo-EFT scheme needs to have converged.

Purpose

In a previous study [P. Capel, D. R. Phillips, and H.-W. Hammer, Phys. Rev. C
98, 034610 (2018)], NLO descriptions of 11Be have been developed and lead to
excellent agreement with existing breakup data. However, the convergence of
the scheme at NLO was not fully demonstrated. Moreover, a significant
dependence on the regulator of the effective 10Be-$n$ interaction has been
observed.

Method

We develop Halo-EFT descriptions of 11Be at N2LO and use them in an accurate
breakup-reaction code. We compare our theoretical cross sections with
experiment on Pb and C targets at about 70 MeV/nucleon.

Results

On Pb, the N2LO descriptions of 11Be lead to little change to the NLO results
of the previous study, confirming the convergence of that scheme. On C, the
reaction is significantly affected by the presence of $d$ resonances in the
low-energy spectrum of 11Be. In the Halo-EFT power counting these resonances
appear only at N2LO; our new descriptions include them naturally. Going to
N2LO removes also the cutoff dependence observed in the previous study.

Conclusions

We demonstrate the convergence of the Halo-EFT description of 11Be at NLO for
Coulomb breakup and at N2LO for nuclear-dominated dissociation. The
reliability of the nuclear-structure information inferred in the previous
study is thus confirmed.

###### pacs:

##  I Introduction

Halo nuclei exhibit one of the most peculiar quantal structure Tanihata
(1996). They have a large matter radius compared to their isobars. This
unusual size arises from their very loose binding of one or two valence
neutrons, which can thus tunnel far away from the other nucleons. Accordingly,
halo nuclei can be seen as a compact core, which contains most of the
nucleons, surrounded by a diffuse neutron halo Hansen and Jonson (1987). This
structure challenges modern nuclear models. It is thus at the center of many
experimental and theoretical studies Tanihata (1996).

Because they are found close to the neutron dripline, halo nuclei are mostly
studied through reactions Al-Khalili and Nunes (2003); Baye and Capel (2012).
To infer valuable information from experimental cross sections, a reliable
model of the reaction coupled to a realistic description of the projectile is
needed. The halo effective field theory (Halo-EFT) provides a way to describe
halo nuclei by exploiting the clear separation of scales observed in these
nuclei. The ratio of the small size of the core to the broad extension of the
halo provides a naturally small parameter along which the core-halo
Hamiltonian can be expanded Bertulani  _et al._ (2002); Bedaque  _et al._
(2003) (see Ref. Hammer  _et al._ (2017) for a recent review). Extending this
development at several orders, from the leading order (LO) to the next-to-
leading order (NLO) and beyond, provides a systematic expansion scheme. The
low-energy constants (LECs) of this expansion are adjusted to physical
observables taken either from experiment or from accurate nuclear-structure
calculations. This procedures therefore offers a natural scaling of the
structure observables of halo nuclei. Using a Halo-EFT description of the
projectile in an accurate reaction code enables us to study the effect of
these observables upon reaction cross sections and to confront the predictions
of nuclear-structure models to experiment. Originally considered for the
breakup of 11Be, the archetypical one-neutron halo nucleus Capel  _et al._
(2018), the idea has then been successfully extended to transfer Yang and
Capel (2018) and knockout Hebborn and Capel (2021), as well as to other halo
nuclei: 15C Moschini  _et al._ (2019); Hebborn and Capel (2021) and 19C Capel
_et al._ (2023).

These various analyses have not only led to excellent agreement with
experiment for various kinds of cross sections, but they also have enabled us
to figure out accurately and systematically to which structure observables
these reactions are sensitive. In most cases, it seems that a NLO Halo-EFT
description of the projectile is sufficient, which confirms that the reactions
are mostly sensitive to the binding energy and asymptotic normalization
constant (ANC) of the initial bound state as well as the phase shift in the
continuum Capel and Nunes (2007). However, as pointed out during the program
_“Living Near Unitarity”_ held at the KITP in Santa Barbara in 2022, the good
agreement with experiment is not sufficient to conclude that the scheme has
actually converged hammer (2022). Moreover, the “beyond-NLO” expansion
developed in Ref. Capel  _et al._ (2018) to include the
$\mbox{$\frac{5}{2}$}^{+}$, $\mbox{$\frac{3}{2}$}^{-}$, and
$\mbox{$\frac{3}{2}$}^{+}$ resonances in the description of 11Be exhibits a
significant dependence on the regulator $\sigma$ of the effective 10Be-neutron
interaction. This is particularly true for the $p_{3/2}$ phase shift (see Fig.
10 of Ref. Capel  _et al._ (2018)), which affects significantly the Coulomb-
breakup cross section (see Fig. 12(a) of that reference). This seeming
dependence to the short-range physics of the projectile is neither physical
Capel and Nunes (2007) nor expected within an EFT.

To address these two issues—lack of proof of convergence and $\sigma$
dependency of the description of 11Be—we develop a fully N2LO description of
11Be and repeat the reaction calculations performed in Ref. Capel  _et al._
(2018). To fit the LECs of this new interaction, we consider the experimental
energies of the five lowest states in the 11Be spectrum, and results of the
_ab initio_ calculation of Calci _et al._ Calci  _et al._ (2016). For the
bound states, we use their predictions of the ANC. For the resonant states, we
use the experimental widths. To constrain the additional LEC in the $s$ and
$p$ waves, we use the _ab initio_ prediction for the phase shift in these
partial waves at low energy.

This natural extension of the work initiated in Ref. Capel  _et al._ (2018)
confirms the convergence of the Halo-EFT scheme at NLO for Coulomb breakup.
The N2LO expansion also enables us to eliminate the cutoff dependency in the
$p_{3/2}$ phase shift and hence confirms that breakup cross sections are not
sensitive to the short-range physics of the projectile. Moreover, the
excellent agreement with experiment validates the predictions of Calci _et
al._ Calci  _et al._ (2016).

This paper is structured as follows. After a short presentation of the Halo-
EFT at N2LO (see Sec. II), we detail in Sec. III how the LECs of that
expansion have been fitted. Section IV presents the results of our breakup
calculations first on Pb at 69 MeV/nucleon (Sec. IV.1) and then on C at 67
MeV/nucleon (Sec. IV.2). We conclude this work in Sec. V.

##  II Halo EFT at N2LO

To describe the breakup of 11Be, we consider the usual few-body model of the
reaction. The projectile $P$ is described as a halo neutron $n$ loosely bound
to a 10Be core $c$ assumed in its $0^{+}$ ground state, and of which the
internal structure is ignored. The Hamiltonian for this two-body structure
reads

$\displaystyle H_{0}=-\frac{\hbar^{2}}{2\mu}\Delta+V(r),$ (1)

where $r$ is the $c$-$n$ relative coordinate and $\mu$ is the $c$-$n$ reduced
mass. The potential $V$ simulates the 10Be-$n$ interaction. Following the
original idea of Ref. Capel  _et al._ (2018), we consider a Halo-EFT
interaction Bertulani  _et al._ (2002); Bedaque  _et al._ (2003), see Ref.
Hammer  _et al._ (2017) for a recent review. However, as mentioned before, in
the present work, we go one step further and consider a Halo-EFT description
of 11Be up to N2LO. This interaction is fitted partial wave per partial wave
up to $d$ waves.

Expanding the development of Ref. Capel  _et al._ (2018), we parametrize the
10Be-$n$ interaction in the $s$ and $p$ waves in the following way

$\displaystyle V_{lj}(r)$ $\displaystyle=$ $\displaystyle
V_{lj}^{(0)}e^{-\frac{r^{2}}{2\sigma^{2}}}+V_{lj}^{(2)}\,r^{2}e^{-\frac{r^{2}}{2\sigma^{2}}}+V_{lj}^{(4)}\,r^{4}e^{-\frac{r^{2}}{2\sigma^{2}}},$
(2)

where $l$ and $j$ are the quantum numbers of the orbital and total angular
momenta, respectively. The low-energy constants (LECs) $V_{lj}^{(0)}$,
$V_{lj}^{(2)}$, and $V_{lj}^{(4)}$ are fitted to know experimental
observables, such as the neutron separation energy, or theoretical
predictions. The fitting procedure is detailed in Sec. III.

Following the Halo-EFT power counting, in the $d$ waves, the $c$-$n$
interaction contains only the first two terms of the expression (2); viz.
$V_{dj}^{(4)}=0,\forall j$. For $l>2$, the $c$-$n$ interaction is assumed to
be nil.

In the usual few-body model of breakup, the target $T$ is seen as a
structureless cluster. Its interactions with the projectile constituents, $c$
and $n$, are described by optical potentials $V_{cT}$ and $V_{nT}$ chosen in
the literature. We consider the same interactions as in Ref. Capel  _et al._
(2018), see Sec. VI A of that reference.

This model leads to the resolution of a three-body Schrödinger equation, of
which the Hamiltonian reads

$\displaystyle
H=-\frac{\hbar^{2}}{2\mu_{PT}}\Delta_{R}+H_{0}+V_{cT}(R_{cT})+V_{nT}(R_{nT}),$
(3)

where $R$ is the relative coordinate between the projectile center of mass and
the target, $\mu_{PT}$ is the $P$-$T$ reduced mass, and $R_{cT}$ and $R_{nT}$
are, respectively, the $c$-$T$ and $n$-$T$ distances.

Equation (3) has to be solved with the condition that the 11Be projectile is
initially in its ground state $\phi_{n_{0}l_{0}j_{0}m_{0}}$ of energy
$E_{n_{0}j_{0}l_{0}}$, with $n_{0}$ the number of nodes in the radial wave
function. Accordingly, the three-body wave function $\Psi$ behaves as

$\displaystyle\Psi^{(m_{0})}(\mbox{\boldmath$r$},\mbox{\boldmath$R$})\mathop{\longrightarrow}\limits_{{Z}\rightarrow{-\infty}}e^{iKZ+\ldots}\phi_{n_{0}l_{0}j_{0}m_{0}}(\mbox{\boldmath$r$}),$
(4)

with the $Z$ coordinate chosen along the incoming beam axis, $m_{0}$ is the
projection of the total angular momentum $j_{0}$ of the initial state, and the
incoming wave number is related to the total energy $\cal E$ in the $P$-$T$
center-of-mass rest frame $\hbar^{2}K^{2}/2\mu_{PT}={\cal
E}-E_{n_{0}l_{0}j_{0}}$. The “$\ldots$” in the incoming wave indicates that,
even at large distances, the $P$-$T$ motion is affected by the Coulomb
interaction.

To solve Eq. (3) with the initial condition (4), we follow Ref. Capel  _et
al._ (2018) and use the dynamical eikonal approximation (DEA) Baye  _et al._
(2005); Goldstein  _et al._ (2006). This approximation is very successful in
describing breakup reactions at intermediate energy. In particular it is
perfectly suited to study the Coulomb and nuclear-dominated breakup reactions
measured at RIKEN Fukuda  _et al._ (2004), which we re-analyze in this paper.

##  III Description of 11Be

###  III.1 $s_{1/2}$ partial wave

In the usual single-particle description of 11Be, the
$\mbox{$\frac{1}{2}$}^{+}$ ground state corresponds to a $1s_{1/2}$ neutron
bound to a 10Be core in its $0^{+}$ ground state. It is seen as an intruder
from the $sd$ shell into the $p$ shell.

$\sigma$ | $V^{(0)}_{s1/2}$ | $V^{(2)}_{s1/2}$ | $V^{(4)}_{s1/2}$ | $E_{1s1/2}$ | ${\cal C}_{1s1/2}$
---|---|---|---|---|---
(fm) | (MeV) | (MeV fm-2) | (MeV fm-4) | (MeV) | (fm-1/2)
1.2 | $-382.895$ | $151.3$ | $-13.0$ | $-0.5033$ | $0.7858$
1.5 | $-155.3167$ | $21.35$ | $-1.0$ | $-0.5031$ | $0.7858$
2 | $-100.7272$ | $7.84$ | $-0.15$ | $-0.5031$ | $0.7861$
_Ab initio_ |  |  |  | $-0.5$ | $0.786$
Table 1: Depths of the Halo-EFT 10Be-$n$ interaction (2) in the $s_{1/2}$
partial wave at N2LO, and the corresponding ground-state energy and ANC. The
corresponding _ab initio_ predictions of Calci _et al._ Calci  _et al._ (2016)
used in the adjustment are listed in the last line.

Figure 1: Halo-EFT calculations of 11Be in the $s_{1/2}$ partial wave at N2LO.
(a) Reduced radial wave function of the $\mbox{$\frac{1}{2}$}^{+}$ ground
state described as a $1s_{1/2}$ bound state in the potential given by Eq. (2).
(b) Phase shift as a function of the 10Be-$n$ relative energy $E$. The results
are shown for the cutoffs $\sigma=1.2$, 1.5, and 2 fm. The _ab initio_
predictions of Ref. Calci  _et al._ (2016) against which the LECs of Eq. (2)
have been fitted are shown in blue dash-dotted lines.

At N2LO, the Halo-EFT 10Be-$n$ potential includes three terms in the $s$ wave,
see Eq. (2). To fit the three LECs of that interaction, wee need three
physical observables. As in Ref. Capel  _et al._ (2018), we consider the
experimental one-neutron separation energy $S_{n}(^{11}{\rm Be})=0.503$ MeV as
the first constraint. The second one is the ANC predicted by the _ab initio_
calculation of Calci _et al._ Calci  _et al._ (2016): ${\cal C}_{1s1/2}=0.786$
fm-1/2. This value leads to excellent agreement with experiments on Coulomb
and nuclear breakup of 11Be Capel  _et al._ (2018); Moschini and Capel (2019),
${}^{10}{\rm Be}(d,p)$ transfer Yang and Capel (2018), and one-neutron
knockout off 11Be Hebborn and Capel (2021). It can therefore be considered as
very reliable. As third fitting observable, we consider the
$\mbox{$\frac{1}{2}$}^{+}$ phase shifts predicted by Calci _et al._ Calci  _et
al._ (2016). At NLO, fixing $S_{n}(^{11}{\rm Be})$ and ${\cal C}_{1s1/2}$
naturally produces an $s_{1/2}$ phase shift in agreement with the _ab initio_
calculation up to 1.5 MeV Capel  _et al._ (2018); Sparenberg  _et al._ (2010).
To extend the energy range of that agreement, we use the method of least
squares to constrain the LECs and minimize the square of the residuals between
the $s_{1/2}$ phase shifts of the Halo-EFT Hamiltonian (2) and the _ab initio_
ones up to $E=2$ MeV.

We adjust the LECs of this interaction considering the same three cutoffs as
in Ref. Capel  _et al._ (2018): $\sigma=1.2$, 1.5, and 2 fm. The results of
this fitting are provided in Table 1, which lists, in addition to the depths
$V_{s1/2}^{(0)}$, $V_{s1/2}^{(2)}$, and $V_{s1/2}^{(4)}$, the precise values
obtained by these potentials for $S_{n}$ and ${\cal C}_{1s1/2}$. Figure 1
shows (a) the reduced radial wave function $u_{1s1/2}$ obtained from these
fits, and (b) the corresponding phase shift $\delta_{s1/2}$. The added value
of the N2LO fit can be observed by comparing these plots with, respectively,
Figs. 1 and 2 of Ref. Capel  _et al._ (2018); for clarity, we use the same
line style and color code as in that reference. The calculations with
$\sigma=1.2$ (solid red lines), 1.5 (green dashed lines), and 2 fm (magenta
dotted lines) are compared to the _ab initio_ results (blue dash-dotted
lines).

Thanks to the additional parameter in the N2LO Halo-EFT interaction, we extend
the agreement with the _ab initio_ phase shift up to 2.5 MeV; so beyond the
0–2 MeV range within which it was fitted. Interestingly, this improved
description in the phase shift does not lead to wave functions much closer to
the _ab initio_ overlap wave function. However, the new radial wave functions
leave their asymptotic behavior (thin dashed black line) at the same radius as
the _ab initio_ one, around $r\simeq 5$ fm. At NLO, they held their asymptotic
behavior down to smaller radii. Interestingly, compared to the NLO
interactions of Ref. Capel  _et al._ (2018), the N2LO potentials lead to more
significant differences between the different cutoffs in the interior of the
wave function. In particular the position of the node changes significantly
with $\sigma$. Accordingly, any small sensitivity to the internal part of the
wave function will be noticeable in the breakup cross sections.

$\sigma$ | $V^{(0)}_{p1/2}$ | $V^{(2)}_{p1/2}$ | $V^{(4)}_{p1/2}$ | $E_{0p1/2}$ | ${\cal C}_{0p1/2}$
---|---|---|---|---|---
(fm) | (MeV) | (MeV fm-2) | (MeV fm-4) | (MeV) | (fm${-1/2}$)
1.2 | $361.7838$ | $-198.65$ | $15.4$ | $-0.1843$ | $0.1291$
1.5 | $-46.1204$ | $-9.7$ | $0.9$ | $-0.1841$ | $0.1290$
2 | $-64.0397$ | $5.065$ | $-0.065$ | $-0.1841$ | $0.1290$
_Ab initio_ |  |  |  | $-0.1848$ | $0.1291$
Table 2: Depths of the Halo-EFT 10Be-$n$ interaction (2) in the $p_{1/2}$
partial wave at N2LO, and the corresponding energy and ANC. The corresponding
_ab initio_ predictions of Calci _et al._ Calci  _et al._ (2016) used in the
adjustment are listed in the last line.

Figure 2: Halo-EFT calculations of 11Be in the $p_{1/2}$ partial wave at N2LO.
(a) Reduced radial wave function of the $\mbox{$\frac{1}{2}$}^{-}$ excited
state described as a $0p_{1/2}$ bound state in the potential given by Eq. (2).
(b) Phase shift as a function of the 10Be-$n$ relative energy $E$. The results
are shown for the cutoffs $\sigma=1.2$, 1.5, and 2 fm. The _ab initio_
predictions of Ref. Calci  _et al._ (2016) against which the LECs of Eq. (2)
have been fitted are shown in blue dash-dotted lines.

###  III.2 $p_{1/2}$ partial wave

Within an extreme shell model, the $p_{1/2}$ partial wave should host 11Be
valence neutron in the ground state. As mentioned earlier, it does not.
Instead, it corresponds to the $\mbox{$\frac{1}{2}$}^{-}$ excited bound state
of the nucleus. Here also, the three LECs of the Halo-EFT interaction are
fitted to reproduce the experimental energy of that state, 184 keV under the
one-neutron separation threshold, the _ab initio_ prediction for this state’s
ANC ${\cal C}_{0p1/2}=0.1291$ fm-1/2, and the low-energy phase shift in that
partial wave up to $E=2$ MeV Calci  _et al._ (2016). The fitted depths and
precise values of the parameters they provide are listed in Table 2. The
reduced radial wave functions $u_{0p1/2}$ and phase shifts $\delta_{p1/2}$ are
shown in Figs. 2(a) and (b), respectively.

In this partial wave, the changes compared to the NLO calculation of Ref.
Capel  _et al._ (2018) are striking. Even though all NLO potentials reproduce
reasonably well the _ab initio_ $p_{1/2}$ phase shift (see Fig. 4 of Ref.
Capel  _et al._ (2018)), the dependence of the corresponding $\delta_{p1/2}$
on the regulator $\sigma$ is significantly larger than in the $s_{1/2}$
partial wave. Adding an extra term to the Halo-EFT potential enables us to
reduce that dependence to a level similar to the N2LO $s_{1/2}$ potential [see
Fig. 1(b)]. Here, all $\sigma$ provides nearly identical phase shift up to 3
MeV, and the agreement with the _ab initio_ prediction is excellent up to 2
MeV.

We observe that the additional term in the N2LO potential reduces the
difference between the radial wave functions down to $r\simeq 4$ fm, see Fig.
2(a). However, this does not improve the agreement between the Halo-EFT wave
function and the _ab initio_ overlap wave function in the range $r=4$–7 fm.
Clearly, even at N2LO, the short-range nature of the Halo-EFT interaction (2)
cannot reproduce this feature of the _ab initio_ prediction. This confirms the
mean-field nature of this state Pei  _et al._ (2006).

###  III.3 $p_{3/2}$ partial wave

The major source of uncertainty in the “beyond-NLO” reaction calculations
presented in Sec. VII of Ref. Capel  _et al._ (2018) comes from the $p_{3/2}$
partial wave. In that article, the two LECs of the potential had been adjusted
to reproduce the experimental energy and width of the
$\mbox{$\frac{3}{2}$}^{-}$ resonance of 11Be. However, with only two
parameters, the resulting $p_{3/2}$ phase shift strongly depends on the cutoff
$\sigma$ and differs significantly from the _ab initio_ prediction, see Fig.
10 of Ref. Capel  _et al._ (2018). This dependency affects the Coulomb breakup
cross section by more than 10%, see Fig. 12(a) of Ref. Capel  _et al._ (2018).
To reproduce the _ab initio_ $\delta_{p3/2}$, the cutoff was used as an
additional fitting parameter. A good agreement was found using $\sigma=1$ fm.
This led to a good agreement with breakup data on both Pb and C, see Figs.
12(b) and 14.

$\sigma$ | $V^{(0)}_{p3/2}$ | $V^{(2)}_{p3/2}$ | $V^{(4)}_{p3/2}$ | $E_{p3/2}$ | $\Gamma_{p3/2}$
---|---|---|---|---|---
(fm) | (MeV) | (MeV fm-2) | (MeV fm-4) | (MeV) | (keV)
1.2 | $-686.666$ | $210.06$ | $-7.3$ | $2.150$ | $212$
1.5 | $-498.698$ | $120.10$ | $-4.125$ | $2.150$ | $220$
2 | $-299.57$ | $45.00$ | $-1.0$ | $2.150$ | $217$
_Ab initio_ |  |  |  | $2.15$ | $190$
Experiment |  |  |  | $2.15\pm 0.01$ | $206\pm 8$
Table 3: Depths of the Halo-EFT 10Be-$n$ interaction (2) in the $p_{3/2}$
partial wave at N2LO, and the corresponding energy and width of the
$\mbox{$\frac{3}{2}$}^{-}$ state. The _ab initio_ predictions of Calci _et
al._ Calci  _et al._ (2016) and experimental values Kelley  _et al._ (2012)
used in the adjustment are listed in the last two lines.

Such a dependence of calculations on the cutoff is not expected in Halo-EFT
because the description of the nucleus should not depend on the short-range
physics. One way to counter this is to extend the description to N2LO. We thus
repeat the fit made in Ref. Capel  _et al._ (2018), but using the additional
term in Eq. (2) within a least-square method to reproduce the _ab initio_
$p_{3/2}$ phase shift at low-energy, viz. to force our $\delta_{p3/2}$ to stay
close to zero in the range 0–1.5 MeV. The corresponding LECs, as well as the
position and width of the $p_{3/2}$ resonance, which they produce are listed
in Table 3. The $p_{3/2}$ phase shift is plotted in Fig. 3 as a function of
the $c$-$n$ relative energy $E$.

Figure 3: Halo-EFT phase shift the $p_{3/2}$ partial wave at N2LO as a
function of the 10Be-$n$ relative energy $E$. The results are shown for the
cutoffs $\sigma=1.2$, 1.5, and 2 fm. The _ab initio_ predictions of Ref. Calci
_et al._ (2016) against which the LECs of Eq. (2) have been fitted are shown
in blue dash-dotted lines.

Adding an order to the Halo-EFT expansion clearly solves the $\sigma$
dependency in the $p_{3/2}$ partial wave observed in Ref. Capel  _et al._
(2018). The N2LO interactions not only reproduce the energy and width of the
known $\mbox{$\frac{3}{2}$}^{-}$ resonance, but the $p_{3/2}$ phase shift they
generate is also independent of $\sigma$ up to nearly 2.5 MeV. By construction
they perfectly agree with the _ab initio_ prediction below that energy.

The 10Be-$n$ interactions constructed here within Halo-EFT at N2LO solve both
issues mentioned in the Introduction. First, they enable us to check that the
Halo-EFT expansion converges. Second, they remove the significant dependence
on the regulator $\sigma$ noted in Ref. Capel  _et al._ (2018). In additon,
they enable us to better reproduce the _ab initio_ predictions. In particular,
the phase shifts in the $s$ and $p$ waves are now in perfect agreement with
the calculations of Calci _et al._ up to 2–3 MeV. Accordingly, these new
10Be-$n$ interactions offer us the possibility to check that the Halo-EFT
scheme actually converges and to extend the test of the _ab initio_ prediction
to larger energies in the 10Be-$n$ continuum. In the next section, we test
this on actual reaction data using accurate calculations of the Coulomb and
nuclear breakup of 11Be.

##  IV Breakup calculations of 11Be

###  IV.1 On Pb at 69 MeV/nucleon

Using the N2LO 10Be-$n$ interactions developed in the previous section, we
repeat the reaction calculation for the Coulomb breakup of 11Be on Pb at 69
MeV/nucleon. Besides the 10Be-$n$ interaction, the inputs (optical potentials)
and numerical conditions are identical to those of Ref. Capel  _et al._
(2018). That reaction has been measured at RIKEN by Fukuda _et al._ Fukuda
_et al._ (2004). We can thus compare our calculations to these accurate data
to test the reliability of this extended Halo-EFT description of 11Be.

Figure 4 presents the breakup cross section as a function of the relative
energy $E$ between the 10Be core and the neutron after dissociation. It is
equivalent to Fig. 12(a) of Ref. Capel  _et al._ (2018). The DEA calculations
are shown for the three regulators $\sigma=1.2$, 1.5, and 2 fm. In addition to
the total cross section, the dominant contributions of the $p_{3/2}$,
$p_{1/2}$, and $d_{5/2}$ partial waves are shown as well. To analyze these
results without any filter, the theoretical results have not been folded with
the experimental resolution; this is done in Fig. 5. We nevertheless show the
RIKEN data for comparison.

Figure 4: Theoretical breakup cross section of ${11}$Be on Pb at 69
MeV/nucleon plotted as a function of the 10Be-$n$ relative energy $E$ after
dissociation. The contributions of the dominant partial waves $p_{3/2}$,
$p_{1/2}$, and $d_{5/2}$ are shown as well. The calculations with the three
cutoffs $\sigma=1.2$, 1.5, and 2 fm are nearly identical, illustrating the
independence of the reaction process to the short-range physics of 11Be. The
data of Ref. Fukuda  _et al._ (2004) are shown for comparison.

Compared to the calculations shown in Fig. 12(a) of Ref. Capel  _et al._
(2018), these N2LO results exhibit nearly no sensitivity to the regulator
$\sigma$. This is true in all major partial-wave contributions. The only
slight sensitivity to $\sigma$ is noted at the energy of the $p_{3/2}$
resonance, where the breakup cross section obtained with $\sigma=2$ fm is a
bit larger than the others. This can be traced back to the small difference in
the $p_{3/2}$ phase shift above 2 MeV, see Fig. 3.

This clear independence on $\sigma$ is obtained despite the strong variation
observed in the internal part of the radial wave function of the $1s_{1/2}$
ground state, see Fig. 1(a). This confirms previous results, which have shown
the peripherality of breakup reactions, i.e. that they probe only the
asymptotic part of the projectile wave functions Capel and Nunes (2007); Capel
_et al._ (2018). Accordingly they are sensitive to the ANC of the bound state
and the phase shift in the continuum Capel and Nunes (2006). The good
agreement with experiment confirms the validity of the _ab initio_ predictions
of Calci _et al._ Calci  _et al._ (2016). However, for a meaningful comparison
with data, the theoretical cross section should be folded with the
experimental resolution; this is performed in Fig. 5.

Figure 5: Breakup of 11Be on Pb at 69 MeV/nucleon. The theoretical cross
sections of Fig. 4 are compared to the data of Ref. Fukuda  _et al._ (2004)
after folding with the experimental energy resolution.

After folding, the small bumps due to the presence of resonances in the
$d_{5/2}$ and $p_{3/2}$ partial waves are washed out. The different
calculations are barely distinguishable now, and agree very well with
experiment. We can thus conclude that extending the Halo-EFT to N2LO enables
us to efficiently remove the dependence of the description of 11Be to the
cutoff $\sigma$. Moreover it shows that the expansion scheme has converged.
The breakup cross section obtained at N2LO is not significantly better than
the one obtained at NLO, see Fig. 6 of Ref. Capel  _et al._ (2018). As stated
in that previous work, a NLO description of 11Be is both necessary and
sufficient to describe the Coulomb breakup of 11Be. The presence of the $d$
and $p_{3/2}$ resonances in the nucleus’ spectrum has no visible effect on the
calculation. Only a much precise energy resolution could reveal these features
in the cross section.

This new analysis also confirms the reliability of the _ab initio_ predictions
of Calci _et al._ Calci  _et al._ (2016). Knowing that this reaction is purely
peripheral Capel and Nunes (2006, 2007); Capel  _et al._ (2018), and that the
structure observables, which affect the calculations, correspond to these
predictions, the excellent agreement we obtain with the data validates the
predictions of Ref. Calci  _et al._ (2016).

###  IV.2 On C at 67 MeV/nucleon

We extend our analysis to nuclear-dominated breakup. That reaction is also
peripheral Capel and Nunes (2007), but it is more sensitive to the resonant
continuum Fukuda  _et al._ (2004); Capel  _et al._ (2004). In Ref. Capel  _et
al._ (2018), in an attempt to describe the RIKEN data Fukuda  _et al._ (2004),
the $\mbox{$\frac{5}{2}$}^{+}$ and $\mbox{$\frac{3}{2}$}^{+}$ resonances have
been added as single-particle resonances within the $d_{5/2}$ and $d_{3/2}$
partial waves, respectively. Following the natural power counting of Halo-EFT,
this description of 11Be goes “beyond NLO”. In the present work, we use the
N2LO description of the projectile, which naturally includes these two
$d$-wave resonances. Our goal is to confirm the convergence of the Halo-EFT
scheme and check that, at N2LO, the dissociation on a light target are also
independent of the cutoff.

Figure 6 displays the results of our DEA calculations for the collision of
11Be on C at 67 MeV/nucleon, which corresponds to the conditions of the RIKEN
experiment Fukuda  _et al._ (2004). For the three cutoffs, the breakup cross
section is plotted as a function of the relative energy $E$ between the 10Be
core and the halo neutron $n$ after dissociation. In addition to the total
cross section, the main partial-wave contributions ($p_{3/2}$, $d_{5/2}$, and
$d_{3/2}$) are shown separately.

As already discussed in Refs. Fukuda  _et al._ (2004); Capel  _et al._ (2004,
2018), the presence of the $d_{5/2}$ resonance significantly affects the
breakup process. It favors the dissociation of 11Be at that energy leading to
a clear and significant peak at that resonant state. A similar effect, though
much less ample is also observed around the $d_{3/2}$ resonance. Using the
N2LO description developed in Sec. III, the breakup cross section is
completely independent of $\sigma$, but in the $d$-wave resonances as ralready
noted in Refs. Capel and Nunes (2007); Capel  _et al._ (2018). This result
confirms the convergence of the Halo-EFT scheme; going to the full N2LO does
not lead to significant change compared to the previous calculation, see Fig.
13 of Ref. Capel  _et al._ (2018). It also shows that, although being
dominated by the short-ranged nuclear interaction, this reaction is also
purely peripheral Capel and Nunes (2007) and, as such, does not depend on the
short-range physics of the projectile, but for the resonant breakup Capel  _et
al._ (2018).

Figure 6: Theoretical breakup cross section of 11Be on C at 67 MeV/nucleon
plotted as a function of the 10Be-$n$ relative energy $E$ after dissociation.
The contributions of the dominant partial waves $p_{3/2}$, $d_{5/2}$, and
$d_{3/2}$ are shown as well. The near-equality of the results obtained with
the three different cutoffs $\sigma=1.2$, 1.5, and 2 fm shows the independence
of the reaction calculation to the short-range physics of 11Be.

To compare our calculations with the experiment, we fold our theoretical
results with the energy resolution provided in Ref. Fukuda  _et al._ (2004).
The corresponding cross sections are shown in Fig. 7 alongside the data; see
the bottom curves labelled $V_{\rm 3b}=0$. They reproduce the general features
of the experimental cross sections, but miss the peaks at the
$\mbox{$\frac{5}{2}$}^{+}$ and $\mbox{$\frac{3}{2}$}^{+}$ energies. The
folding has smoothened the narrow peaks of the $d$-wave resonances seen in
Fig. 6. The $d_{3/2}$ one is completely washed out, and the $d_{5/2}$ one is
not high enough to correctly reproduce the data.

Figure 7: Nuclear breakup of 11Be on C at 67 MeV/nucleon; comparison of the
theoretical cross sections with the experimental data of Ref. Fukuda  _et al._
(2004). Our calculations are shown without (V${}_{\text{3b}}=0$, lower curves)
and with (V${}_{\text{3b}}\neq 0$, upper curves) the effective three-body
force simulating core excitation Capel  _et al._ (2022). The theoretical cross
sections have been folded with the experimental energy resolution Fukuda  _et
al._ (2004).

As shown by Moro and Lay, a significant part of the resonant breakup comes
from the excitation of the 10Be core Moro and Lay (2012). The role played by
the $2^{+}_{1}$ excited state of 10Be dominates the breakup towards the
$\mbox{$\frac{3}{2}$}^{+}$ resonance. The role played by the core excitation
in the breakup towards the $\mbox{$\frac{5}{2}$}^{+}$ is smaller, yet non
negligible. This degree of freedom is not included within our Halo-EFT, which
explains why we cannot reproduce the data.

The virtual excitation of the core during the breakup reaction can be
simulated by an effective three-body force Capel  _et al._ (2022). We
therefore repeat the calculations using that three-body force with a depth
$V_{\rm 3b}=-100$ MeV. This significantly increases the breakup strength in
the energy ranges of both resonances; see the upper curves labelled $V_{\rm
3b}\neq 0$ in Fig. 7. In particular, the cross section obtained for the
regulator $\sigma=2$ fm is in fair agreement with the data at the energy of
the $\mbox{$\frac{5}{2}$}^{+}$ resonance; the other two cutoffs lead to too
small resonant-breakup cross sections. However, the results of Ref. Capel  _et
al._ (2022) suggest that this depth can be adjusted to reproduce the data. The
$\sigma$-dependence of these new results confirms that the resonant breakup is
dominated by short-ranged physics, which is not included in the Halo-EFT
considered here. Including explicitly the core excitation in the model of 11Be
is the best way to confirm that Moro and Lay (2012). As shown by the present
and previous Capel  _et al._ (2018); Yang and Capel (2018); Moschini and Capel
(2019); Moschini  _et al._ (2019); Hebborn and Capel (2021) studies, Halo EFT
is a powerful tool to analyze reactions involving halo nuclei. Extending this
framework to add the core degree of freedom would certainly be an asset in the
description of nuclear reactions. A first step has already been taken in this
direction Kubushishi and Capel (2024).

##  V Conclusion

Halo-EFT provides a very efficient tool to analyze nuclear reactions involving
halo nuclei Capel  _et al._ (2018). Their very systematic expansion scheme
offers a natural scaling of the structure observables, which influence the
reaction cross sections. This interesting feature has been observed for
various reactions, from breakup Capel  _et al._ (2018); Moschini and Capel
(2019) to transfer Yang and Capel (2018) and knockout Hebborn and Capel
(2021), and on different nuclei Capel  _et al._ (2018); Moschini and Capel
(2019); Yang and Capel (2018); Moschini  _et al._ (2019); Capel  _et al._
(2023); Hebborn and Capel (2021). Moreover the LECs of the expansion can be
fitted to predictions of accurate structure calculations. Halo-EFT therefore
enables us to test the validity of these predictions by confronting them to
experimental data.

The attractive features of this effective description of halo nuclei however
requires that (i) the expansion schemes has properly converged hammer (2022),
and (ii) the result of the fit of the LECs be insensitive to the regulator
$\sigma$. In Ref. Capel  _et al._ (2018), we have obtained excellent agreement
with reaction data for the Coulomb breakup of 11Be at 69 MeV/nucleon using a
NLO description of the projectile. The same description fails to reproduce the
breakup cross section on 12C at 67 MeV/nucleon in the energy region of the
$\mbox{$\frac{5}{2}$}^{+}$ and $\mbox{$\frac{3}{2}$}^{+}$ resonances. The
extension of that description “beyond NLO” proposed in Ref. Capel  _et al._
(2018) to solve that issue leads to a significant sensitivity to the cutoff
$\sigma$. Such a seeming dependence on the short-range physics of the nucleus
is not expected in Halo-EFT.

To ensure the convergence of the Halo-EFT expansion at NLO for Coulomb breakup
and avoid the $\sigma$ dependency of our reaction calculations, we propose a
rigorous N2LO description of 11Be and run DEA calculations for the breakup of
11Be on both Pb and C.

The addition of a term in the Halo-EFT potential (2) enables us to reduce
significantly the dependence of our structure observables to the regulator
$\sigma$. This is particularly true for the $p_{3/2}$ phase shift, for which
$\sigma$ had to be used as a fitting parameter in Ref. Capel  _et al._ (2018).
Moreover, these new N2LO potentials significantly extend the agreement between
our single-particle description of 11Be and its _ab initio_ calculation by
Calci _et al._ Calci  _et al._ (2016).

The breakup cross sections obtained with this new Halo-EFT description of 11Be
are now nearly independent of the regulator $\sigma$. Extending the
description to N2LO removes the seeming dependence to the short-range physics
observed in Ref. Capel  _et al._ (2018). Moreover, this result is obtained
despite the significant differences observed in the initial bound-state wave
functions for $r\lesssim 5$ fm. This confirms, if that is still needed, that
these reactions are purely peripheral, in the sense that they probe only the
tail of the wave functions: the ANC in the ground state and the phase shift in
the continuum.

On the lead target, our N2LO calculations are nearly identical to the NLO
calculations of Ref. Capel  _et al._ (2018), confirming that for this reaction
a NLO description of 11Be is both needed, as shown in Ref. Capel  _et al._
(2018), and sufficient, as demonstrated by the present work.

The breakup reaction on the carbon target cannot be properly described at NLO.
The significant enhancement of the breakup strength at the resonance energies
cannot be accounted for without including these states in the expansion
scheme. In a single-particle description, they correspond to one-neutron $d$
resonances, and therefore require an expansion up to, at least, N2LO . As
already observed in Ref. Capel  _et al._ (2018), adding these single-particle
resonances is not enough to fully reproduce the data. Moro and Lay have shown
that a significant amount of the breakup cross section can be related to the
core excitation Moro and Lay (2012). This effect can also be effectively
simulated by a three-body force Capel  _et al._ (2022). Our present analysis
shows that the strength of the three-body force depends on the cutoff $\sigma$
of the 10Be-$n$ interaction. This is not surprising since this effect
corresponds to physics within the projectile description. Ideally, this
resonant breakup should be studied using a description that explicitly
accounts for that degree of freedom. Such a description is currently under
development Kubushishi and Capel (2024).

In conclusion, we have demonstrated that to describe the Coulomb breakup of
11Be at intermediate energy, a NLO description of the projectile is both
needed and sufficient. For that reaction a N2LO description does not improve
the reaction calculation. To properly describe the breakup on 12C, however,
including the $d$-wave resonances is necessary. This requires a N2LO
description. This extended description also enables us to avoid the $\sigma$
dependency observed in $p_{3/2}$ partial wave at beyond-NLO in Ref. Capel  _et
al._ (2018). However, even at N2LO the breakup strength at the
$\mbox{$\frac{5}{2}$}^{+}$ and $\mbox{$\frac{3}{2}$}^{+}$ resonances is not
enough to reproduce the RIKEN data. As shown by Moro and Lay Moro and Lay
(2012), this requires to explicitly account for the core excitation Kubushishi
and Capel (2024).

## Acknowledgement

We thank H. W. Grießhammer for his constructive comments on the previous NLO
calculations. This project has received funding from the Deutsche
Forschungsgemeinschaft within the Collaborative Research Center SFB 1245
(Projektnummer 279384907) and the PRISMA+ (Precision Physics, Fundamental
Interactions and Structure of Matter) Cluster of Excellence. This research was
supported in part by grant NSF PHY-1748958 to the Kavli Institute for
Theoretical Physics (KITP).

## References

  * Tanihata (1996) I. Tanihata, J. Phys. G 22, 157 (1996).
  * Hansen and Jonson (1987) P. G. Hansen and B. Jonson, Europhys. Lett. 4, 409 (1987).
  * Al-Khalili and Nunes (2003) J. Al-Khalili and F. Nunes, J. Phys. G 29, R89 (2003).
  * Baye and Capel (2012) D. Baye and P. Capel,  _Clusters in Nuclei, Vol. 2_ , Lecture Notes in Physics 848, 121 (2012), Ed. C. Beck.
  * Bertulani  _et al._ (2002) C. Bertulani, H.-W. Hammer, and U. van Kolck, Nucl. Phys. A712, 37 (2002).
  * Bedaque  _et al._ (2003) P. Bedaque, H.-W. Hammer, and U. van Kolck, Phys. Lett. B569, 159 (2003).
  * Hammer  _et al._ (2017) H.-W. Hammer, C. Ji, and D. R. Phillips, J. Phys. G 44, 103002 (2017).
  * Capel  _et al._ (2018) P. Capel, D. R. Phillips, and H.-W. Hammer, Phys. Rev. C 98, 034610 (2018).
  * Yang and Capel (2018) J. Yang and P. Capel, Phys. Rev. C 98, 054602 (2018).
  * Hebborn and Capel (2021) C. Hebborn and P. Capel, Phys. Rev. C 104, 024616 (2021).
  * Moschini  _et al._ (2019) L. Moschini, J. Yang, and P. Capel, Phys. Rev. C 100, 044615 (2019).
  * Capel  _et al._ (2023) P. Capel, , D. R. Phillips, A. Andis, M. Bagnarol, B. Behzadmoghaddam, F. Bonaiti, R. Bubna, Y. Capitani, P.-Y. Duerinck, V. Durant, N. Döpper, A. E. Boustani, R. Farrell, M. Geiger, M. Gennari, N. Goldberg, J. Herko, T. Kirchner, L.-P. Kubushishi, Z. Li, S. S. L. Muli, A. Long, B. Martin, K. Mohseni, I. Moumene, N. Paracone, E. Parnes, B. Romeo, V. Springer, I. Svensson, O. Thim, and N. Yapa, Eur. Phys. J. A 59, 273 (2023).
  * Capel and Nunes (2007) P. Capel and F. M. Nunes, Phys. Rev. C 75, 054609 (2007).
  * hammer (2022) H. W. G. hammer, (2022), comment during the program _“Living Near Unitarity”_ held at the KITP in Santa Barbara (California, USA).
  * Calci  _et al._ (2016) A. Calci, P. Navrátil, R. Roth, J. Dohet-Eraly, S. Quaglioni, and G. Hupin, Phys. Rev. Lett. 117, 242501 (2016).
  * Baye  _et al._ (2005) D. Baye, P. Capel, and G. Goldstein, Phys. Rev. Lett. 95, 082502 (2005).
  * Goldstein  _et al._ (2006) G. Goldstein, D. Baye, and P. Capel, Phys. Rev. C 73, 024602 (2006).
  * Fukuda  _et al._ (2004) N. Fukuda, T. Nakamura, N. Aoi, N. Imai, M. Ishihara, T. Kobayashi, H. Iwasaki, T. Kubo, A. Mengoni, M. Notani, H. Otsu, H. Sakurai, S. Shimoura, T. Teranishi, Y. X. Watanabe, and K. Yoneda, Phys. Rev. C 70, 054606 (2004).
  * Moschini and Capel (2019) L. Moschini and P. Capel, Phys. Lett. B790, 367 (2019).
  * Sparenberg  _et al._ (2010) J.-M. Sparenberg, P. Capel, and D. Baye, Phys. Rev. C 81, 011601 (2010).
  * Pei  _et al._ (2006) J. Pei, F. Xu, and P. Stevenson, Nucl. Phys. A765, 29 (2006).
  * Kelley  _et al._ (2012) J. Kelley, E. Kwan, J. Purcell, C. Sheu, and H. Weller, Nucl. Phys. A880, 88 (2012).
  * Capel and Nunes (2006) P. Capel and F. M. Nunes, Phys. Rev. C 73, 014615 (2006).
  * Capel  _et al._ (2004) P. Capel, G. Goldstein, and D. Baye, Phys. Rev. C 70, 064605 (2004).
  * Moro and Lay (2012) A. M. Moro and J. A. Lay, Phys. Rev. Lett. 109, 232502 (2012).
  * Capel  _et al._ (2022) P. Capel, D. Phillips, and H.-W. Hammer, Phys. Lett. B825, 136847 (2022).
  * Kubushishi and Capel (2024) L.-P. Kubushishi and P. Capel, “Expanding Halo-EFT to core excitation for one-neutron halo nuclei: an application to 11Be,” (2024), (In preparation).


# Adaptive Transfer Learning on Graph Neural Networks

Xueting Han  Microsoft Research Asia<EMAIL_ADDRESS>,  Zhenhuan Huang
Beihang University Microsoft Research Asia<EMAIL_ADDRESS>,  Bang An
University of Maryland, College Park Microsoft Research Asia<EMAIL_ADDRESS>and  Jing Bai  Microsoft Research Asia<EMAIL_ADDRESS>
(2021)

###### Abstract.

Graph neural networks (GNNs) is widely used to learn a powerful representation
of graph-structured data. Recent work demonstrates that transferring knowledge
from self-supervised tasks to downstream tasks could further improve graph
representation. However, there is an inherent gap between self-supervised
tasks and downstream tasks in terms of optimization objective and training
data. Conventional pre-training methods may be not effective enough on
knowledge transfer since they do not make any adaptation for downstream tasks.
To solve such problems, we propose a new transfer learning paradigm on GNNs
which could effectively leverage self-supervised tasks as auxiliary tasks to
help the target task. Our methods would adaptively select and combine
different auxiliary tasks with the target task in the fine-tuning stage. We
design an adaptive auxiliary loss weighting model to learn the weights of
auxiliary tasks by quantifying the consistency between auxiliary tasks and the
target task. In addition, we learn the weighting model through meta-learning.
Our methods can be applied to various transfer learning approaches, it
performs well not only in multi-task learning but also in pre-training and
fine-tuning. Comprehensive experiments on multiple downstream tasks
demonstrate that the proposed methods can effectively combine auxiliary tasks
with the target task and significantly improve the performance compared to
state-of-the-art methods.

Graph Neural Networks; Transfer Learning; Multi Task Learning; GNN Pre-
Training; Graph Representation Learning

††copyright: rightsretained††journalyear: 2021††conference: Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2021; Virtual Event, Singapore††booktitle: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), August
14–18, 2021, Virtual Event, Singapore††doi: 10.1145/3447548.3467450††isbn:
978-1-4503-8332-5/21/08††ccs: Computing methodologies Learning latent
representations††ccs: Computing methodologies Transfer learning††ccs:
Computing methodologies Neural networks

##  1\. Introduction

Graph neural networks (GNNs) (Bruna et al., 2014; Kipf and Welling, 2017)
attract a lot of attention in representation learning for graph-structured
data. Recent work demonstrates various GNN architectures (Kipf and Welling,
2017; Velickovic et al., 2018; Hamilton et al., 2017; Xu et al., 2019) achieve
state-of-the-art performances in many graph-based tasks, such as node
classification (Kipf and Welling, 2017), link prediction (Zhang and Chen,
2018), and graph classification (Ying et al., 2018b). These GNNs have been
proven effective to learn powerful representations in a variety of scenarios,
including recommendation system (Ying et al., 2018a) and social network (Fan
et al., 2019).

GNNs are often trained in an end-to-end manner with supervision which usually
requires abundant labeled data, but task-specific labels can be scarce for
many graph datasets. Some recent works (Hu et al., 2020b) study GNN pre-
training and employing auxiliary tasks (Hwang et al., 2020) to solve such
problems. Their goal is to transfer the learned knowledge from self-supervised
tasks to downstream tasks, and these methods prove making use of abundant
unlabeled data could further improve graph representation. However, due to the
difference of optimization objectives and data distribution between self-
supervised tasks and downstream tasks, there is an inherent gap between pre-
training and fine-tuning. In addition, conventional pre-training methods are
not aware of which self-supervised tasks benefit more for downstream tasks and
do not make any adaptation for them. This leads to knowledge transfer not
effective enough for downstream tasks. There is little exploration in
improving the effectiveness of transfer learning on GNNs, fully leveraging
information in self-supervision, and enabling sufficient adaptation for
downstream tasks.

In this work, we explore effective knowledge transfer between self-supervised
tasks and downstream tasks, which is an under-investigated problem in GNNs
compared to natural language processing (NLP). Optimizing the effectiveness of
knowledge transfer and alleviating the gap between the pre-trained model and
the target task have been the focus of recent studies in NLP (Howard and
Ruder, 2018; Chronopoulou et al., 2019; Gururangan et al., 2020).
(Chronopoulou et al., 2019) addresses this by using LM as an auxiliary task to
prevent catastrophic forgetting.

Inspired by these works, we propose a new generic transfer learning paradigm
that adaptively selects and combines various auxiliary tasks with the target
task in fine-tuning stage to achieve a better adaptation for downstream tasks.
In other words, we combine the optimization objective from the target task
with multiple objectives from auxiliary tasks. This preserves sufficient
knowledge captured by self-supervised tasks while improving the effectiveness
of transfer learning on GNNs.

Figure 1. Performance of adding two auxiliary tasks with different weights
based on HGT (Hu et al., 2020a) (Reddit).

Challenges. Transferring knowledge from graph-based self-supervised tasks is a
powerful tool for learning a better graph representation. Many self-
supervision methods on graphs, such as DGI (Velickovic et al., 2019), edge
generation (Hu et al., 2020b), attribute masking (Hu et al., 2020c) and meta-
path prediction (Hwang et al., 2020) have been studied and their effectiveness
has been proved. The basic principle, based on which additional auxiliary
tasks can benefit the target task, is that auxiliary tasks share some unknown
information with the target task, and to some extent, help learn features that
are beneficial to the target task. However, the correlation between auxiliary
tasks and the target task can be large or small and may change along with
training. An auxiliary task may be helpful to the target task at the beginning
of training, but may no longer be useful in the later stages of training (Lin
et al., 2019). If an auxiliary task is assigned with an inappropriate weight,
it may even hurt the target task, which means negative transfer (Pan and Yang,
2009) occurs.

Graph preserves diverse information and downstream tasks are also diverse,
single auxiliary task could not fully explore all organic signals. Previous
work involving multiple self-supervised tasks in graph learning usually
manually sets the weight for each task using prior intuition, e.g. assuming
that all auxiliary tasks matter equally or considering the magnitude of the
loss for each task (Hu et al., 2020b). We analyze the effect of different
combinations of auxiliary tasks. Figure 1 shows that the weights of tasks are
critical, different combinations of weights lead to different results on the
target task and bad combinations lead to negative transfer. We also find the
same set of weights behave differently in two transfer schemes, multi-task
learning from scratch and pre-training and fine-tuning (in this scheme, we add
auxiliary tasks in fine-tuning stage). Therefore, the challenge of using
auxiliary tasks lies in the difficulty in selecting and combining different
auxiliary tasks especially when the number of tasks grows. It is still an open
question that has been less explored in graph transfer learning, especially in
adding auxiliary tasks in fine-tune stage after pre-train.

Some related works try to adaptively tune the weights of each task during the
training process (Chen et al., 2018; Liu et al., 2019c; Kendall et al., 2018;
Du et al., 2018; Lin et al., 2019; Hwang et al., 2020). We compared some of
these methods in two schemes, one is multi-task learning and the other is
adding auxiliary tasks in fine-tune stage after pre-train. We find that such
weighting strategies of auxiliary tasks may be effective in multi-task
learning, but may fail in pre-training and fine-tuning scheme. We argue that
it’s because the importance of auxiliary tasks to the target task changes in
different learning stages. In multi-tasking learning, both the target task and
auxiliary tasks are trained from scratch; whereas in pre-training and fine-
tuning, models are already learned by abundant pre-train data.

Contributions. To summarize, this work makes the following major
contributions.

(i) We propose a new generic transfer learning paradigm on GNNs which
adaptively selects and combines various auxiliary tasks on graphs. It can
better adapt to target tasks and improve the effectiveness of knowledge
transfer. We quantify the consistency between an auxiliary task and the target
task by measuring the cosine similarity between the gradients of the auxiliary
task and the target task, and we design an adaptive auxiliary loss weighting
model to learn the weights of auxiliary tasks by considering this signal.
Besides, the weighting model could be optimized through meta-learning.

(ii) We conduct comprehensive experiments on both multi-task learning and pre-
training and fine-tuning across multiple datasets. Our methods outperform
other methods and achieve state-of-the-art results. It is worth mentioning
that our methods perform well in pre-training and fine-tuning while other
methods may be not effective. Our approach is not limited to GNNs scenarios
but is a generic transfer learning paradigm that could be flexibly applied to
various transfer learning scenarios such as in NLP or computer vision (CV).

(iii) We further conduct a thorough analysis to study the advantages of our
methods compared with other baseline methods.

##  2\. Related Work

###  2.1. Pre-Training and Multi-Task Learning on Graph

Pre-Training (Devlin et al., 2019) is a common and effective way to transfer
knowledge learned from related tasks to a target task to improve
generalization. A general way is to pre-train a language model (LM) on a large
corpus of unlabeled text and then fine-tune the model on supervised data.
Recently, researchers have explored pre-training for GNNs to enable more
effective learning on graphs from abundant unlabeled graph data (Hu et al.,
2020c; Hu et al., 2020b). Multi-task learning (Hwang et al., 2020) is another
common method to achieve shared representations by simultaneously training a
joint objective of the supervised task and the self-supervised tasks. (Hwang
et al., 2020) designs a group of auxiliary tasks to learn the heterogeneity of
graphs and try to transfer such knowledge from auxiliary tasks to a target
task. (You et al., 2020) tries to compare three schemes of self-supervision on
GCNs.

However, knowledge transfer for GNN is challenging due to the differences
between self-supervised tasks and downstream tasks on optimization objectives
and data distribution. Optimizing the effectiveness of knowledge transfer and
alleviating the gap between the pre-trained model and the target task has
attracted a lot of attention in recent NLP research (Howard and Ruder, 2018;
Chronopoulou et al., 2019; Gururangan et al., 2020). (Howard and Ruder, 2018)
shows continued pre-train of an LM on the unlabeled data of a given task is
beneficial for the downstream task. Towards the same direction, (Chronopoulou
et al., 2019) addresses this by using LM as an auxiliary task to prevent
catastrophic forgetting. Besides, (Gururangan et al., 2020) show the
importance of further adaptation with large corpora of in-domain data and
task’s unlabeled data. Inspired by these works, we argue that during the
transfer process, knowledge from self-supervised tasks is required for both
pre-training and fine-tuning.

###  2.2. Self-Supervised Tasks on Graph

Self-supervision is a promising direction to learn more transferable,
generalized and robust representations. Research of self-supervised tasks on
graphs attracts a lot of attention recently (Kipf and Welling, 2016; Hamilton
et al., 2017; Sun et al., 2020a; Velickovic et al., 2019; Sun et al., 2020b).
Graph-structured data provides rich information that enables us to design
self-supervised tasks from various perspectives, e.g. node attributes (Hu et
al., 2020c; Hu et al., 2020b), underlying graph structure (Hamilton et al.,
2017; Peng et al., 2020; Sun et al., 2020b) or heterogeneity of graph (Hwang
et al., 2020), etc. For example, edge generation (Hu et al., 2020b) and
context prediction (Hu et al., 2020c) map nodes appearing in local contexts to
nearby embeddings. K-hop context prediction (Peng et al., 2020) tries to
preserve the global topology of graph to better characterize the similarity
and differentiation between nodes. There are also methods (DGI (Velickovic et
al., 2019), InfoGraph (Sun et al., 2020a), etc.) introduce Graph Informax
which maximizes the mutual information between local and global
representations. Besides, attribute generation (Hu et al., 2020c) ensure
certain aspects of node attributes are encoded in the node embeddings.

Some recent work utilizes multiple self-supervised tasks together on graphs.
(Hu et al., 2020b) proposes to use attribute generation and edge generation
tasks to capture the inherent dependency between node attributes and graph
structure during the generative process. (Hwang et al., 2020) transfers the
knowledge of heterogeneity of graph to the target task via adding auxiliary
tasks of predicting meta-paths. However how to effectively select and combine
different self-supervised tasks are less explored in GNNs.

Some other works (Jin et al., 2020; Lu et al., 2021) have studied that the
same auxiliary task behaves differently on different downstream tasks, and
sometimes it may even cause negative-transfer. We argue that multiple kinds of
information on graphs could be captured by different auxiliary tasks and their
interactions and combinations provide big opportunities for us to learn a more
powerful representation.

###  2.3. Auxiliary Task Weighting

Past work has proven that, multi-task learning or auxiliary learning which
simultaneously learns multiple related tasks could achieve better
generalization by sharing a representation (Liu et al., 2019c; Hwang et al.,
2020; Liu et al., 2019b). The goal of such methods includes achieving best
performance across all tasks and only for the target task. In this paper, we
only discuss the later one. The success of these approaches depends on how
well aligned the auxiliary tasks are with the target task and how to combine
these losses with the target task. However, the usefulness of auxiliary tasks
is non-trivial to know and may change through the training.

The weight for each of the auxiliary task is usually manually set using prior
intuition or through hyper-parameter tuning in above mentioned GNN self-
supervised works. It becomes much harder to determine the weights as the
number of auxiliary tasks grows.

Recent works study various weighting strategies to balance tasks. Some of them
treat all tasks equally and adapt the weights based on the gradient norm (Chen
et al., 2018) or task uncertainty (Kendall et al., 2018). Some other works
evaluate the usefulness of auxiliary tasks to the target task and adapt the
weights accordingly so that the more useful ones receive higher weights (Du et
al., 2018; Lin et al., 2019). (Du et al., 2018) proposes to use gradient
similarity between an auxiliary task and the target task on shared parameters
to determine the usefulness of auxiliary tasks. Some recent works (Liu et al.,
2019a; Hwang et al., 2020) use meta-learning in auxiliary task selection. (Liu
et al., 2019a) proposes an elegant solution to generate new auxiliary tasks by
collapsing existing classes. (Hwang et al., 2020) use meta-learning to
automatically select and balance tasks and transfer the knowledge from
auxiliary tasks to the target task. In this paper, we use meta-learning to
learn a weigh function and the consistency of auxiliary tasks to the target
task is considered in this function.

##  3\. Methods

In the following, we first explain the existence of a gap between pre-training
and fine-tuning, and to address the challenges, we propose our Generic
Auxiliary loss Transfer Learning Framework. The goal of this framework is to
effectively leverage self-supervised tasks as auxiliary tasks to help the
target task. Later we introduce our Adaptive Auxiliary Loss Weighting
Algorithm using Task Similarity, in which, we design an adaptive auxiliary
loss weighting model to learn the weights of auxiliary tasks by quantifying
task similarity and we optimize the model through meta-learning. Finally, we
discuss the generality and effectiveness of our methods.

###  3.1. Generic Auxiliary Loss Transfer Learning Framework

We use $\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathcal{X}\\}$ to denote a
graph, where $\mathcal{V}$ represents the node set with $|\mathcal{V}|$ nodes,
$\mathcal{E}$ stands for the edge set with $|\mathcal{E}|$ edges. Denoting
$\boldsymbol{X}\in\mathbb{R}^{|\mathcal{V}|\times d}$ as the node feature
matrix, and $\boldsymbol{A}\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}$
as the adjacency matrix or related variant. A GNN model $f_{\theta}$ encodes
node attributes and contexts into a embedding
$\boldsymbol{Z}\in\mathbb{R}^{|\mathcal{V}|\times d^{\prime}}$ by iteratively
aggregating messages from node’s neighbors.

A classic paradigm of pre-training on graphs is to first pre-train a general
GNN model $f_{\theta}$ by self-supervised tasks on abundant unlabeled data to
achieve a good initialization for downstream tasks. The network is trained
with the objective as follows:

$\displaystyle\theta^{*},\boldsymbol{\Theta_{p}}^{*}$
$\displaystyle=\underset{\theta,\boldsymbol{\Theta_{p}}}{\operatorname{argmin}}\mathcal{L}_{\mathrm{pre}}(\theta,\boldsymbol{\Theta_{p}})$
(1)
$\displaystyle=\underset{\theta,\boldsymbol{\Theta_{p}}}{\operatorname{argmin}}\mathcal{L}_{\mathrm{pre}}(\hat{\boldsymbol{Y}},{h_{\boldsymbol{\Theta_{p}}}}({f_{\boldsymbol{\theta}}}(X,A))),$

where $\mathcal{L}_{\mathrm{pre}}$ denotes the losses of pre-training tasks,
$\hat{\boldsymbol{Y}}$ denotes the pseudo labels of self-supervised data and
$\boldsymbol{\Theta_{p}}$ represents task-specific parameters for pre-train
tasks, e.g. a classifier to predict pseudo labels.

Then in the fine-tuning process, shared GNN model $f_{\theta}$ is initialized
by pre-trained model and fine-tuned on downstream tasks by supervised data.
The objective of fine-tuning is to optimize the following:

$\displaystyle\theta^{*},\boldsymbol{\Theta_{t}}^{*}$
$\displaystyle=\underset{\theta,\boldsymbol{\Theta_{t}}}{\operatorname{argmin}}\mathcal{L}_{\mathrm{sup}}(\theta,\boldsymbol{\Theta_{t}})$
(2)
$\displaystyle=\underset{\theta,\boldsymbol{\Theta_{t}}}{\operatorname{argmin}}\mathcal{L}_{\mathrm{sup}}(\boldsymbol{Y},{h_{\boldsymbol{\Theta_{t}}}}({f_{\boldsymbol{\theta}}}(X,A))),$

where $\mathcal{L}_{\mathrm{sup}}$ is the loss of fine-tuning on the
downstream task, $\boldsymbol{Y}$ denotes labels of supervised data, and
$\boldsymbol{\Theta_{t}}$ represents task-specific parameters for the target
task.

The goal of pre-training is to learn some general and critical features that
can be transferred to downstream tasks; whereas, there is a gap between
training objectives of self-supervised tasks and supervised tasks and there is
not enough adaptation for downstream tasks in this paradigm of pre-training,
which leaves the knowledge transfer not effective enough. As to our knowledge,
there is only one work (Lu et al., 2021) studies this problem on graphs, it
proposes a method to structure the pre-training stage to simulate the fine-
tuning process on downstream tasks, which directly optimizes the pre-trained
model’s quick adaptability to downstream tasks. We propose a new transfer
learning paradigm on GNNs which would adaptively select and combine different
auxiliary tasks with the target task in fine-tuning stage to achieve a better
adaptation for downstream tasks and improve the effectiveness of knowledge
transfer. Our method can automatically select the most useful tasks for
downstream tasks from a task set and balances their weights. It’s a principled
and generic approach that can be applied to all kinds of auxiliary tasks.

Figure 2. Generic auxiliary loss transfer learning framework.

Figure 2 illustrates our proposed Generic Auxiliary Loss Transfer Learning
Framework. In our approach, we first pre-train a GNN model by self-supervised
tasks. We then transfer the parameters of the pre-trained model and fine-tune
them with a joint loss. The joint loss is the weighted sum of the target
task’s loss and multiple auxiliary losses. We also add multiple task-specific
layers for such auxiliary tasks which are randomly initialized. Adding
auxiliary losses in fine-tuning can avoid catastrophic forgetting and better
adapt for the target task, resulting in more powerful representations. The new
joint loss for the fine-tuning process can be formulated as:

(3)
$\mathcal{L}(\theta_{t})=w_{0}\mathcal{L}_{sup}(\theta_{t})+\sum_{i=1}^{K}w_{i}\mathcal{L}_{aux,i}(\theta_{t}),$

where $\mathcal{L}_{sup}$ is the loss of the target task that we want to
complete and $\mathcal{L}_{aux,i}$ represents the loss of auxiliary task
$\boldsymbol{i}$ and $\boldsymbol{i}\in\\{1,2,...,K\\}$. The losses of both
the target task and auxiliary tasks are calculated on shared model parameters
$\theta_{t}$ and their task-specific layers (omitted in the above formula) at
each training step $\boldsymbol{t}$. $w_{i}$ is the weight for task
$\boldsymbol{i}$ which is generated by the weighting model, especially $w_{0}$
means the weight for the target task. We will introduce more details about the
weighting model in Section 3.2. Assuming that we use gradient descent to
update the model, the optimization of shared parameters $\theta_{t}$ on this
joint loss is as follows:

(4)
$\theta_{t+1}=\theta_{t}-\alpha\nabla_{\theta_{t}}\mathcal{L}(\theta_{t}).$

If a large number of auxiliary tasks are used, some auxiliary tasks may play a
more important role than others in the feature representation learning for the
target task; therefore, the weight $w_{i}$ of each auxiliary task needs to be
selected carefully.

###  3.2. Adaptive Auxiliary Loss Weighting using Task Similarity

In previous work, the process of selecting auxiliary tasks on graphs seems to
be too heuristic, and there is not any patterns to follow, which may raise
some issues. First, computation will be much expensive when the number of
auxiliary tasks K increases. Besides, the usefulness of each auxiliary task
may change during the training; using a fixed value as weight may limit or
even hurt the performance.

We propose Adaptive Auxiliary Loss Weighting using Task Similarity (AUX-TS),
which quantifies the similarity of each auxiliary task to the target task and
uses the information obtained after applying the auxiliary task to learn
weights during training.

(1) Weighting model quantifies the consistency of each auxiliary task to the
target task

Our approach designs a weighting model to learn weights for different
auxiliary tasks that benefit the target task most. The weighting model is a
2-layer MLP network, which takes signals that related to current data and task
as input and outputs the weight.

(Du et al., 2018) propose to use the cosine similarity of gradients between
tasks to represent the relatedness between tasks and prove the effectiveness
of the method. However, this method only uses the signal of cosine similarity
in a rule-based way to determine whether adopt the gradients from an auxiliary
task. The weights in this method cannot be optimized. Inspired by this work
(Du et al., 2018), our method follows the principle that a useful auxiliary
task should provide gradient directions which helps to decrease the loss of
the target task.

Specifically, we quantify the consistency of an auxiliary task with the target
task by measuring the cosine similarity between the gradients of the auxiliary
task and the target task. And we design an adaptive auxiliary loss weighting
model to learn the weights of auxiliary tasks by considering this signal. We
use $g([sim,type\\_emb,loss];w)$ to denote the weighting model ($g(sim;w)$ is
used in the following formulas as a simplified expression), where
$[sim,type\\_emb,loss]$ represents all the signals that are taken as the input
to weighting model, it’s the concatenation of the gradient similarity, loss
value and type embedding of the current task. $w$ represents the parameters of
the weighting model. Since the target task is of our interest, the weighting
model is optimized by minimizing the loss of the target task
$\mathcal{L}_{sup}$ at each training step $t$.

(2) Optimize weighting model via Meta-learning

Some recent works (Liu et al., 2019a; Hwang et al., 2020) propose the idea of
using meta-learning to learn meta-parameters, which themselves are used in
another learning procedure. In our method, we treat the weighting model as
meta-parameters and optimize it according to the target task during the
training. Specifically, we train two networks in two stages. The first stage
is to optimize the weighting model which generates weights for all tasks and
the second stage is to optimize the shared GNN networks on the combination of
the target and auxiliary tasks in a multi-task learning scheme.

In the first stage, the weighting model is optimized by the loss of the target
task, encouraging to generate a set of weights, so that if the joint loss
weighted by these weights is used to train GNN, the performance of the target
task could be maximized.

Gradient similarity between tasks is a strong indicator of the consistency of
tasks. We denote the weighting model as $g(sim_{i};w)$ which tasks gradients
similarity between an auxiliary task and the target task as input:

(5)
${sim}_{i}=\cos(\nabla_{\theta}\mathcal{L}_{sup}({\theta}),\nabla_{\theta}\mathcal{L}_{aux,i}({\theta})),$

where $\nabla_{\theta}\mathcal{L}_{sup}({\theta})$ is the gradient vector
evaluated at $\theta$ by the loss of the target task and
$\nabla_{\theta}\mathcal{L}_{aux,i}({\theta})$ is the gradient vector
evaluated by the loss of auxiliary task $i$. Using $g(sim_{i};w)$ to generate
weight for each task, the joint loss can be represented as:

(6)
$\mathcal{L}(\theta;{w})=g(sim_{0};w)\mathcal{L}_{sup}(\theta)+\sum_{i=1}^{K}g(sim_{i};w)\mathcal{L}_{aux,i}(\theta).$

After one update of GNN by above total loss, we leverage the loss of the
target task on the updated GNN to optimize the weighting model. Therefore, the
weighting model is updated as follows:

(7) $\underset{w}{\operatorname{argmin}}\mathcal{L}_{sup}(\hat{\theta}({w})).$

$\hat{\theta}({w})$ denotes the updated parameters of the GNN after one
gradient update using the joint loss $\mathcal{L}(\theta;{w})$ (defined in
Equation 6 )

(8)
$\hat{\theta}_{t}({w}_{t})=\theta_{t}-\alpha\nabla_{\theta}\mathcal{L}(\theta_{t};{w}_{t}),$

where $\alpha$ is the learning rate for GNN. We do not really update the
parameters $\theta$, but plug it in ${L}_{sup}$ to optimize $w$.

In the second stage, GNN is trained by joint losses of both the target and
auxiliary tasks, given the weights of different tasks are determined by the
weighting model, we define the multi-task objective of GNN as follows:

(9)
$\underset{\theta}{\operatorname{argmin}}g(sim_{0};w)\mathcal{L}_{sup}(\theta)+\sum_{i=1}^{K}g(sim_{i};w)\mathcal{L}_{aux,i}(\theta).$

We train both networks in an iterative manner until convergence. Assuming that
we use gradient descent to update both networks, our proposed method is
illustrated in Algorithm 1.

Algorithm 1 AUX-TS

0: training data for the target task/auxiliary tasks
$D^{sup},D^{aux}$;learning rate: $\alpha,\beta$, max iterations $N$;the number
of auxiliary tasks $K$;

1: Initialize ${\theta}_{0}$, ${w}_{0}$, ${sim}_{0}=1$

2: for $t=0$ to $N$ do

3: fetch one batch of training data: $D^{sup}$, $D^{aux}$

4: # step 1: optimize the weighting model via meta-learning

5: $D^{sup(train)},D^{sup(meta)}\leftarrow\text{Split}(D^{sup})$

6: for $i=1$ to $K$ do

7: # compute gradient similarity

8:
${sim}_{i}\leftarrow\cos(\nabla_{\theta_{t}}\mathcal{L}_{sup}({\theta}_{t}),\nabla_{\theta_{t}}\mathcal{L}_{aux,i}({\theta}_{t}))$

9: end for

10: $\mathcal{L}(\theta_{t};{w}_{t})\leftarrow
g(sim_{0};w_{t})\mathcal{L}_{sup}(\theta_{t})+$$\qquad\qquad\qquad\sum_{i=1}^{K}g(sim_{i};w_{t})\mathcal{L}_{aux,i}(\theta_{t})$

11: compute:
$\hat{\theta}_{t}({w}_{t})\leftarrow\theta_{t}-\alpha\nabla_{\theta}\mathcal{L}(\theta_{t};{w}_{t})$
with $D^{sup(train)}\cup D^{aux}$

12: Update
${w}_{t+1}\leftarrow{w}_{t}-\beta\nabla_{{w}}\mathcal{L}_{sup}(\hat{\theta}_{t}({w}_{t}))$
with $D^{sup(meta)}$

13: # step 2: optimize the GNN via joint loss

14: for $i=1$ to $K$ do

15: # compute gradient similarity

16:
${sim}_{i}\leftarrow\cos(\nabla_{\theta_{t}}\mathcal{L}_{sup}({\theta}_{t}),\nabla_{\theta_{t}}\mathcal{L}_{aux,i}({\theta}_{t}))$

17: end for

18: $\mathcal{L}(\theta_{t})\leftarrow
g(sim_{0};w_{t+1})\mathcal{L}_{sup}(\theta_{t})+$$\qquad\qquad\quad\sum_{i=1}^{K}g(sim_{i};w_{t+1})\mathcal{L}_{aux,i}(\theta_{t})$

19: Update
$\theta_{t+1}\leftarrow\theta_{t}-\alpha\nabla_{\theta_{t}}\mathcal{L}(\theta_{t})$
with $D^{sup}\cup D^{aux}$

20: end for

###  3.3. Discussion

First, our proposed Generic Auxiliary Loss Transfer Learning Framework could
alleviate the gap between pre-training and fine-tuning and improve the
effectiveness and adaptability of transferring knowledge to downstream tasks
significantly. In addition, it has good generality and can be applied to all
kinds of self-supervised tasks set and all kinds of GNN models.

Secondly, our proposed AUX-TS algorithm dynamically adjusts the weights for
each auxiliary task during the training via meta-learning. We verify our
algorithm in two transfer learning schemes. One is multi-task learning, in
which GNN model is initialized randomly. The other is pre-training and fine-
tuning, in which GNN model is initialized by relatively well pre-trained
parameters. Considering the difference between these two learning schemes, the
usefulness of auxiliary tasks in different learning stages also changes. In
multi-task learning, some auxiliary tasks may contribute more to the joint
loss. After the knowledge of the pre-trained model is transferred, the target
task may become more important and some of the auxiliary tasks may become less
important along with training. Our method uses gradient similarity between an
auxiliary task and the target task as an informative signal to learn the
usefulness of an auxiliary task. So that it could adaptively generate weights
and perform well in both schemes.

##  4\. Evaluation

We evaluate our proposed methods on four datasets to prove the effectiveness
of our Generic Auxiliary loss Transfer Learning Framework and adaptive
auxiliary task weighting algorithm (AUX-TS). To evaluate the generality of our
methods, we conduct experiments on two auxiliary tasks, multiple GNN models,
and two transfer learning schemes. At last, we conduct a comprehensive
analysis to study the advantages of our methods compared with other baseline
methods.

###  4.1. Datasets and Baseline

Datasets  We utilize four public datasets from different domains to evaluate
the effectiveness of our methods. The detailed statistics are summarized in
Table 1.

  * •

Open Academic Graph (OAG) (Tang et al., 2008; Zhang et al., 2019) is the
largest public academic dataset to date which contains more than 178 million
nodes and 2.236 billion edges. Each paper in OAG is associated with fields
(e.g. Computer Science, Medicine) and publication date ranges from 1900 to
2019. We use Paper-Field (L2) prediction on the Computer Science dataset (CS)
as the downstream task and evaluate the performance by MRR and NDCG, where MRR
is the default one.

  * •

Reddit (Hamilton et al., 2017) is a social network constructed with Reddit
posts in different topical communities. We use the node classification task as
the downstream task, which is to classify Reddit posts to different
communities and use micro F1-score as the evaluation metric.

  * •

Book-Crossing (Wang et al., 2018, 2019) contains book ratings from 0 to 10 in
the Book-Crossing community. The link prediction of user-item is used as the
downstream task and is measured by AUC.

  * •

Last-FM (Wang et al., 2018, 2019) contains musician listening information from
a set of 2 thousand users from Last-FM online music system. The link
prediction task is used as the downstream task and is measured by AUC.

Table 1. Datasets statistics.

| Datasets | # Nodes | # Edges |
---|---|---|---|---
Node classification | OAG_CS | 1,116,163 | 28,427,508 |
Reddit | 232,965 | 114,615,892 |
Link prediction | Last-FM | 15,084 | 73,382 |
Book-Crossing | 110,739 | 442,746 |

Baseline  We compare the following approaches:

  * •

No Auxiliary Losses denotes our base learning algorithm, which updates the
model by the target task only instead of any auxiliary tasks.

  * •

Fixed Weights updates the model by both the target task and auxiliary tasks,
with a group of pre-defined fixed weights. We assign all the tasks the same
weight of 1.

  * •

SELAR learns weight functions that consider the loss value to select auxiliary
tasks and balance them with the target task via meta-learning.

  * •

Cosine Similarity (unweighted) uses gradient similarity to determine whether
adopt the gradients from an auxiliary task (Du et al., 2018). Specifically, it
combines the gradients from an auxiliary task and the target task to update
the shared parameters $\theta$ only if
$\cos(\nabla_{\theta}\mathcal{L}_{sup}({\theta}),\nabla_{\theta}\mathcal{L}_{aux}({\theta}))\geq
0$.

  * •

AUX-TS, as described in Algorithm 1. We have two variants, AUX-TS (one sim)
corresponds to adding current task’s similarity in weighting model, and AUX-TS
(all sims) corresponds to adding all tasks’ similarities in weighting model.

Our proposed auxiliary weighting methods can be applied to two transfer
learning schemes, one is multi-task learning (MTL), the other is pre-training
and fine-tuning (P&F). We evaluate our methods with baseline methods on these
two schemes. In all the experiments, we report the mean of test performance
based on three independent runs.

Table 2. Experimental results of different methods over OAG_CS and Reddit datasets. Dataset | OAG_CS | Reddit
---|---|---
| MTL | P&F | MTL | P&F
Metric | NDCG | MRR | NDCG | MRR | F1 score | F1 score
No Auxiliary Losses | 0.4387$\pm$0.0054 | 0.4941$\pm$0.0091 | 0.4584$\pm$0.0039 | 0.5284$\pm$0.0065 | 0.8474$\pm$0.0072 | 0.8605$\pm$0.0073
Fixed Weights | 0.4470$\pm$0.0040 | 0.5117$\pm$0.0077 | 0.4594$\pm$0.0033 | 0.5319$\pm$0.0079 | 0.8497$\pm$0.0056 | 0.8655$\pm$0.0050
Cosine Similarity | 0.4419$\pm$0.0028 | 0.5031$\pm$0.0095 | 0.4584$\pm$0.0037 | 0.5330$\pm$0.0089 | 0.8469$\pm$0.0082 | 0.8648$\pm$0.0064
SELAR | 0.4512$\pm$0.0041 | 0.5197$\pm$0.0073 | 0.4617$\pm$0.0037 | 0.5382$\pm$0.0063 | 0.8499$\pm$0.0111 | 0.8723$\pm$0.0099
AUX-TS (all sims) | 0.4518$\pm$.0033 | 0.5215$\pm$0.0076 | 0.4634$\pm$0.0033 | 0.5412$\pm$0.0074 | 0.8591$\pm$0.0116 | 0.8744$\pm$0.0092
AUX-TS (one sim) | 0.4524$\pm$0.0033 | 0.5218$\pm$0.0073 | 0.4634$\pm$0.0033 | 0.5406$\pm$0.0073 | 0.8595$\pm$0.0092 | 0.8738$\pm$0.0090
Table 3. Experimental results of different methods over Last-FM and Book-Crossing datasets. Dataset | Task | Base GNNs |  | No Auxiliary
---
Losses
| Fixed
---
Weights
| Cosine
---
Similarity
SELAR |  | AUX-TS
---
(all sims)
| AUX-TS
---
(one sim)
Last-FM | MTL | GCN | 0.7762 | 0.7985 | 0.7770 | 0.7999 | 0.8003 | 0.8015
GAT | 0.7983 | 0.8132 | 0.7990 | 0.8214 | 0.8226 | 0.8289
GIN | 0.7969 | 0.8125 | 0.7855 | 0.8258 | 0.8265 | 0.8252
Avg. Gain | - | 2.23% | -0.42% | 3.19% | 3.29% | 3.55%
P&F | GCN | 0.8448 | 0.8426 | 0.8393 | 0.8461 | 0.8440 | 0.8430
GAT | 0.8395 | 0.8441 | 0.8411 | 0.8493 | 0.8501 | 0.8497
GIN | 0.8283 | 0.8374 | 0.8275 | 0.8470 | 0.8515 | 0.8528
Avg. Gain | - | 0.46% | -0.19% | 1.19% | 1.31% | 1.31%
Book-Crossing | MTL | GCN | 0.6816 | 0.6850 | 0.6683 | 0.6976 | 0.7050 | 0.7169
GAT | 0.6597 | 0.6362 | 0.6531 | 0.6493 | 0.6307 | 0.6774
GIN | 0.6724 | 0.7338 | 0.7030 | 0.7501 | 0.7437 | 0.7453
Avg. Gain | - | 2.05% | 0.53% | 4.14% | 3.26% | 6.25%
P&F | GCN | 0.6818 | 0.6906 | 0.6811 | 0.7165 | 0.7098 | 0.7131
GAT | 0.6716 | 0.6746 | 0.6674 | 0.6825 | 0.6818 | 0.6839
GIN | 0.7044 | 0.7471 | 0.7169 | 0.7429 | 0.7469 | 0.7495
Avg. Gain | - | 2.65% | 0.37% | 4.09% | 3.92% | 4.31%

###  4.2. Experiments Setup and Results

Transfer learning with edge generation and attribute generation.  GCN (Kipf
and Welling, 2017) is chosen as the base model for OAG_CS and Reddit. We use
edge generation and attribute generation as auxiliary tasks and follow the
hyper-parameter settings in GPT-GNN (Hu et al., 2020b), and the full labeled
data is used in our experiments. For multi-task learning, we train the model
from scratch for 500/500 epochs, while for pre-training and fine-tuning, we
first pre-train a GNN model for 20/50 epochs and then fine-tune it on
downstream tasks for 500/500 epochs on OAG_CS/Reddit. We adopt the Time
Transfer settings (Hu et al., 2020b) for the pre-training and fine-tuning on
OAG_CS, specifically, we use data from different time spans for pre-training
(before 2014) and fine-tuning (since 2014). We randomly split Reddit, 70% of
which is used for pre-training, and the other 30% is used for fine-tuning
(Train/Valid/Test Data each account for 10%). Table 2 summarizes the
performance of different methods on OAG_CS and Reddit.

Transfer learning with meta-path prediction . We evaluate our methods on Book-
Crossing and Last-FM with multiple GNN architectures: GCN (Kipf and Welling,
2017), GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019). We use five
auxiliary tasks of meta-path prediction and follow most hyper-parameter
settings in SELAR (Hwang et al., 2020), hyper-parameters such as learning rate
and batch size are tuned using validation sets for all models. For multi-task
learning, we train the model from scratch for 100/50 epochs, while for pre-
training and fine-tuning, we first pre-train a GNN model for 100/50 epochs and
then fine-tune it on downstream tasks for 100/50 epochs on Last-FM/Book-
Crossing. Since the sizes of these two datasets are relatively small, we pre-
train and fine-tune on the same graph. Results on Book-Crossing and Last-FM
are summarized in Table 3. Details of implementation and auxiliary tasks can
be found in Appendix A/B.

Overall, the proposed methods significantly enhance the performance for all
downstream tasks on all datasets. Compared with No Auxiliary Losses baseline,
our new transfer learning framework which adaptively selects and combines
various auxiliary tasks with the target task effectively improves target
tasks. Especially under the pre-training and fine-tuning, it alleviates the
gap between pre-training and fine-tuning and achieves a better adaptation for
target tasks. On average, our methods achieve (relative) performance gains of
5.61%/2.42% and 1.43%/1.61% over No Auxiliary Losses baseline under MTL/P&F
respectively on OAG_CS and Reddit, and 3.55%/1.31% and 6.25%/4.31% on Last-FM
and Book-Crossing. These numbers prove the effectiveness of our framework.
Moreover, AUX-TS consistently outperforms other auxiliary task weighting
methods on all datasets.

Comprehensive experiments evaluate the generality of our methods.

Different transfer schemes:  considering the difference between multi-task
learning and pre-training and fine-tuning, we verify our methods on both
transfer schemes. As observed from Table 2, Fixed Weights and Cosine
Similarity are not as effective in pre-training and fine-tuning as in multi-
task learning on OAG, while AUX-TS consistently perform well under both
schemes.

Diff auxiliary tasks sets:  we evaluate on two sets of auxiliary tasks and
find AUX-TS greatly improves the results, which proves that AUX-TS can be
flexibly applied to all kinds of combinations of tasks.

Different GNN model:  Table 3 shows that AUX-TS consistently improves the
performance of target tasks for all three GNNs compared with other baseline
methods.

Figure 3. Performance curves of different methods on GCN and HGT under MTL
(OAG_CS). Figure 4. Loss and weight curves of AUX-TS for different tasks under
P&F (OAG_CS). Figure 5. Gradient similarity curves of Fixed Weights under MTL
and P&F (OAG_CS).

###  4.3. Further Analysis

The effectiveness and generality of our approach are proved above. We conduct
further analysis to understand why our methods work and to study the
advantages of our methods compared to other baseline methods. The following
analysis is based on our best-performed experiments.

AUX-TS avoids negative transfer. We compare the performance (MRR) curve of our
methods with other baseline methods on GCN (Kipf and Welling, 2017) and HGT
(Hu et al., 2020a) in Figure 3. HGT is a heterogeneous model and is much more
complex than GCN. AUX-TS performs better than Fixed Weights and Cosine
Similarity on both models, and these two methods hurt the performance on HGT.
We can observe that 1) auxiliary tasks are not always helpful to the target
task; 2) negative transfer is more likely to occur on the more complex model;
and 3) AUX-TS can successfully avoid negative transfer in both models.

AUX-TS adaptively assigns appropriate weights for different tasks during the
training. From the loss and weight curves of AUX-TS under pre-training and
fine-tuning in Figure 4, we observe that the weights of different tasks change
during the training. The weight of the target task increases with the
training, while the weights of the two auxiliary tasks increase first and then
decrease and AUX-TS assigns higher weights to the attribute generation task.
Meanwhile, both the losses of the target task and the attribute generation
task decrease continually while the loss of the edge generation task starts to
increase at a certain stage instead. Although the loss of the edge generation
task increases, we have achieved the best results in the target task. This
proves that AUX-TS could learn the usefulness of different auxiliary tasks and
can adaptively select auxiliary tasks with appropriate weights for the target
task.

Gradient similarity is a good indicator to describe the consistency between
tasks, and it changes with learning. The curve of gradient similarity for
Fixed Weights under two transfer schemes (as shown in Figure 5) reflects that,
in multi-task learning, the gradient similarities between auxiliary tasks and
the target task rise rapidly in the early training stage, and then decrease
with the progress of training. In fine-tuning stage, after the knowledge of
the pre-trained model is transferred to the downstream task, the gradient
similarities drop to a much smaller value. Fixed weights or inappropriate
weights of auxiliary tasks may not be helpful or even cause negative transfer,
especially in fine-tuning stage. This further explains why some other
weighting methods are not effective on OAG_CS when the model is pre-trained on
a huge graph. Whereas AUX-TS can perform well in both schemes since it
considers the consistency of an auxiliary task to the target task.

Cosine Similarity method also uses the gradient similarity to adjust the
weight. However, the gradient similarity fluctuates greatly, and there is not
a simple positive correlation between the usefulness of auxiliary tasks and
gradient similarity. Not only data with large gradient similarity can help the
target task, but small ones can also be beneficial. Cosine similarity uses
only those whose gradient similarity is greater than zero. So this rule-based
method is not effective enough and far from optimal to utilize this important
signal to learn weights. Whereas in AUX-TS it’s learnable via meta-learning
and the advantage of AUX-TS is that it can dynamically update the weighting
model to adapt to the latest training situation. We find that in multi-task
learning, the gradient similarity is relatively large (-0.3 $\sim$ 0.9), the
weight change curve and the gradient similarity curve show a certain positive
correlation, but in pre-training and fine-tuning, the similarity is relatively
small (-0.2 $\sim$ 0.4), this correlation becomes weak or even the opposite.

Table 4. Performance (NDCG/MRR) of using different signals in weighting model
on four groups of loss scales (OAG_CS, P&F).

Loss Scales | No Auxiliary Losses | SELAR | AUX-TS (all sims)
---|---|---|---
edge=1 attr=1 | 0.4584/0.5284 | 0.4617/0.5382 | 0.4634/0.5412
edge=1 attr=5 | 0.4586/0.5332 | 0.4637/0.5426
edge=5 attr=1 | 0.4545/0.5258 | 0.4641/0.5415
edge=5 attr=5 | 0.4569/0.5287 | 0.4642/0.5431

Figure 6. Auxiliary task weight curves for AUX-TS on four groups of loss
scales (OAG_CS, P&F).

AUX-TS performs well regardless of the loss scale. To further prove advantages
of AUX-TS over SELAR which only uses loss in the weighting model without
gradient similarity, we adjust loss scales of the two auxiliary tasks. Table 4
shows that SELAR is very sensitive to the loss scale and performs poorly on
readjusted losses, while AUX-TS consistently performs well in various loss
scale settings.

As shown in Figure 6, by comparing the weight curves under different loss
scales, it is proved that AUX-TS could adaptively balance weights of different
tasks under different loss scales. It assigns lower weights to tasks with high
loss scales, which is different from SELAR. We observe a certain consistency
between weight and loss in SELAR. SELAR tends to give high weights to hard
examples with a large loss more than easy ones. However, the loss which
represents the difficulty of a task is insufficient to characterize the
importance of the task. The reason why AUX-TS could outperform SELAR
significantly is that it not only consider the difficulty of a task but also
consider the consistency with the target task.

##  5\. Conclusion

In this work, we propose a new generic transfer learning paradigm on GNNs
which would adaptively add multiple types of auxiliary tasks in the fine-
tuning stage to continually incorporate various priors . It can improve the
effectiveness of knowledge transfer and narrow the gap between pre-training
and fine-tuning. Furthermore, we design an adaptive auxiliary loss weighting
model to learn the weights of auxiliary tasks by quantifying the consistency
between auxiliary tasks and the target task. The model is learnable through
meta-learning. Comprehensive experiments on various downstream tasks and
transfer learning schemes prove the generality and superiority of our methods
compared with other methods. The effectiveness of our approach is demonstrated
on GNNs but not limited to GNNs scenarios. It’s a generic transfer learning
paradigm that could be flexibly extended to various transfer learning
scenarios such as in NLP or computer vision (CV).

## References

  * (1)
  * Bruna et al. (2014) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014.  Spectral Networks and Locally Connected Networks on Graphs. In _ICLR_.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In _ICML_. 794–803.
  * Chronopoulou et al. (2019) Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019.  An embarrassingly simple approach for transfer learning from pretrained language models. In _NAACL_. 2089–2095.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_. 4171–4186.
  * Du et al. (2018) Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. 2018.  Adapting auxiliary losses using gradient similarity.  _arXiv preprint arXiv:1812.02224_ (2018).
  * Fan et al. (2019) Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019\.  Graph neural networks for social recommendation. In _WWW_. 417–426.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In _ACL_. 8342–8360.
  * Hamilton et al. (2017) William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive Representation Learning on Large Graphs. In _NeurIPS_. 1024–1034.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal Language Model Fine-tuning for Text Classification. In _ACL_. 328–339.
  * Hu et al. (2020c) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. 2020c.  Strategies for Pre-training Graph Neural Networks. In _ICLR_.
  * Hu et al. (2020b) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020b.  Gpt-gnn: Generative pre-training of graph neural networks. In _KDD_. 1857–1867.
  * Hu et al. (2020a) Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020a.  Heterogeneous Graph Transformer. In _WWW_. 2704–2710.
  * Hwang et al. (2020) Dasol Hwang, Jinyoung Park, Sunyoung Kwon, Kyung-Min Kim, Jung-Woo Ha, and Hyunwoo J. Kim. 2020\.  Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs. In _NeurIPS_. 10294–10305.
  * Jin et al. (2020) Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. 2020.  Self-supervised learning on graphs: Deep insights and new direction.  _arXiv preprint arXiv:2006.10141_ (2020).
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In _CVPR_. 7482–7491.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Variational Graph Auto-Encoders.  _NeurIPS Workshop on Bayesian Deep Learning_ (2016).
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _ICLR_.
  * Lin et al. (2019) Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. 2019.  Adaptive Auxiliary Task Weighting for Reinforcement Learning. In _NeurIPS_. 4772–4783.
  * Liu et al. (2019a) Shikun Liu, Andrew Davison, and Edward Johns. 2019a.  Self-Supervised Generalisation with Meta Auxiliary Learning. In _NeurIPS_. 1677–1687.
  * Liu et al. (2019c) Shikun Liu, Edward Johns, and Andrew J Davison. 2019c.  End-to-end multi-task learning with attention. In _CVPR_. 1871–1880.
  * Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b.  Multi-Task Deep Neural Networks for Natural Language Understanding. In _ACL_. 4487–4496.
  * Lu et al. (2021) Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. 2021\.  Learning to Pre-train Graph Neural Networks. In _AAAI_.
  * Pan and Yang (2009) Sinno Jialin Pan and Qiang Yang. 2009.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ 22, 10 (2009), 1345–1359.
  * Peng et al. (2020) Zhen Peng, Yixiang Dong, Minnan Luo, Xiao-Ming Wu, and Qinghua Zheng. 2020.  Self-supervised graph representation learning via global context prediction.  _arXiv preprint arXiv:2003.01604_ (2020).
  * Sun et al. (2020a) Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. 2020a.  InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In _ICLR_.
  * Sun et al. (2020b) Ke Sun, Zhouchen Lin, and Zhanxing Zhu. 2020b.  Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes. In _AAAI_. 5892–5899.
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  Arnetminer: extraction and mining of academic social networks. In _KDD_. 990–998.
  * Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _ICLR_.
  * Velickovic et al. (2019) Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019.  Deep Graph Infomax. In _ICLR_.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems. In _CIKM_. 417–426.
  * Wang et al. (2019) Hongwei Wang, Fuzheng Zhang, Mengdi Zhang, Jure Leskovec, Miao Zhao, Wenjie Li, and Zhongyuan Wang. 2019.  Knowledge-aware graph neural networks with label smoothness regularization for recommender systems. In _KDD_. 968–977.
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019\.  How Powerful are Graph Neural Networks?. In _ICLR_.
  * Ying et al. (2018a) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018a.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In _KDD_. 974–983.
  * Ying et al. (2018b) Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. 2018b.  Hierarchical Graph Representation Learning with Differentiable Pooling. In _NeurIPS_. 4805–4815.
  * You et al. (2020) Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. 2020\.  When does self-supervision help graph convolutional networks?. In _ICML_. 10871–10880.
  * Zhang et al. (2019) Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Bin Shao, Rui Li, and Kuansan Wang. 2019.  OAG: Toward Linking Large-scale Heterogeneous Entity Graphs. In _KDD_. 2585–2595.
  * Zhang and Chen (2018) Muhan Zhang and Yixin Chen. 2018.  Link prediction based on graph neural networks. In _NeurIPS_. 5171–5181.

##  Appendix A Implementation Details

The weighting model is a 2-layer MLP network, the first layer uses ReLU as the
activation function, and the last layer uses softplus. We set the hidden
dimensions to 1000.

For OAG_CS and Reddit datasets, we run experiments with three seeds and
evaluate 40 batches for each run to get more stable results.

For hyper-parameters used in OAG_CS and Reddit, we set the batch size to 256,
the learning rate for GNNs to 1e-3/5e-4 respectively for pre-training and
fine-tuning, and set the learning rate for the weighting model to 1e-5. We
tried sigmoid and softplus as the activation function for the last layer of
the weighting model. In most cases, softplus can achieve better performance.
We do not use cross validation for experiments on OAG_CS and Reddit, but use
3-fold cross validation on Last-FM and Book-Crossing datasets.

##  Appendix B Auxiliary Tasks Details

We use five meta-path prediction tasks as auxiliary tasks for Last-FM and
Book-Crossing datasets, and use the meta-paths label data published by SELAR.
According to SELAR, these auxiliary tasks are listed in 5.

Table 5. Auxiliary tasks on Last-FM and Book-Crossing datasets. Meta-paths (Last-FM) | Meta-paths (Book-Crossing)
---|---
user-item-actor-item | user-item-literary.series-item-user
user-item-appearing.in.film-item | item-genre-item
user-item-instruments-item | user-item-user-item
user-item-user-item | user-item-user
user-item-artist.origin-item | item-user-item