# Simultaneous measurements of unstable and stable Alfvén Eigenmodes in JET

R.A. Tinguely1 ***Author to whom correspondence should be addressed:
<EMAIL_ADDRESS>J. Gonzalez-Martin2, P.G. Puglia3, N. Fil4, S. Dowson4, M.
Porkolab1, I. Kumar5, M. Podestà6, M. Baruzzo4, A. Fasoli3, Ye.O. Kazakov7,
M.F.F. Nave8, M. Nocente9 10, J. Ongena7, Ž. Štancar11, and JET Contributors*
1 Plasma Science and Fusion Center, Massachusetts Institute of Technology,
Cambridge, MA, USA
2 Department of Physics and Astronomy, University of California, Irvine,
California 92697, USA
3 Ecole Polytechnique Fédérale de Lausanne (EPFL), Swiss Plasma Center (SPC),
CH-1015 Lausanne, Switzerland
4 Culham Centre for Fusion Energy, Culham Science Centre, Abingdon, UK
5 Durham University, Durham, UK
6 Princeton Plasma Physics Laboratory, Princeton, NJ, USA
7 Laboratory for Plasma Physics, LPP-ERM/KMS, TEC Partner, 1000 Brussels,
Belgium
8 Instituto de Plasmas e Fusão Nuclear, Instituto Superior Técnico, Univ. de
Lisboa, Lisbon, Portugal
9 Dipartimento di Fisica, Universitá di Milano-Bicocca, 20126 Milan, Italy
10 Institute for Plasma Science and Technology, National Research Council,
20125, Milan, Italy
11 Jožef Stefan Institute, Ljubljana, Slovenia
* See author list of J. Mailloux _et al_ 2022 _Nucl. Fusion_ , doi:10.1088/1741-4326/ac47b4

###### Abstract

In this paper, we report the novel experimental observation of both unstable
and stable Toroidicity-induced Alfvén Eigenmodes (TAEs) measured
simultaneously in a JET tokamak plasma. The three-ion-heating scheme
(D-DNBI-${}^{3}\mathrm{He}$) is employed to accelerate deuterons to MeV
energies, thereby destabilizing TAEs with toroidal mode numbers $n=3{-}5$,
each decreasing in mode amplitude. At the same time, the Alfvén Eigenmode
Active Diagnostic resonantly excites a stable $n=6$ TAE with total normalized
damping rate $-\gamma/\omega_{0}\approx 1\%{-}4\%$. Hybrid kinetic-MHD
modeling with codes NOVA-K and MEGA both find eigenmodes with similar
frequencies, mode structures, and radial locations as in experiment. NOVA-K
demonstrates good agreement with the $n=3$, 4, and 6 TAEs, matching the
damping rate of the $n=6$ mode within uncertainties and identifying radiative
damping as the dominant contribution. Improved agreement is found with MEGA
for _all_ modes: the unstable $n=3{-}5$ and stable $n=2,6$ modes, with the
latter two stabilized by higher intrinsic damping and lower fast ion drive,
respectively. While some discrepancies remain to be resolved, this unique
validation effort gives us confidence in TAE stability predictions for future
fusion devices.

Keywords: Alfvén Eigenmodes, stability, three-ion-heating, Ion Cyclotron
Resonance Heating, Neutral Beam Injection

##  1 Introduction

Alfvén Eigenmodes (AEs), destabilized by fast ions (FIs) in tokamak plasmas,
have been the subject of intense investigation due to their potential to
enhance the transport and thus degrade the confinement of said FIs [1]. This
should be avoided in future fusion devices, like ITER and SPARC, which aim to
maximize confinement and fusion power.

Unstable AEs are often easily observable in high-frequency fluctuation data,
e.g. from passive magnetic probes, interferometry, reflectometry, soft x-ray
(SXR), and other signals. However, the measurement of _stable_ AEs requires a
system like JET’s Alfvén Eigenmode Active Diagnostic (AEAD) [2], comprising
two in-vessel, toroidal arrays of four antennas each [3]. The antennas are
independently powered and phased by amplifiers with currents up to
${\sim}10~{}\mathrm{A}$, frequencies ranging $25{-}250~{}\mathrm{kHz}$, and a
toroidal mode number spectrum $|n|<20$ [4]. When the AEAD’s frequency scan
intersects the resonant frequency $\omega_{0}=2\pi f_{0}$
$\mathrm{[rad\,s^{-1}]}$ of a stable†††Note that the AEAD _can_ interact with
unstable AEs, but for sufficiently destabilized mode amplitudes, the growth
rate cannot be straightforwardly determined from the response. AE, the plasma
responds like a weakly damped, driven harmonic oscillator; the damping rate
$\gamma$ $\mathrm{[s^{-1}]}$ and $n$-number are then deduced from the magnetic
response measured by up to fourteen fast magnetic probes [5].

Many past studies have analyzed unstable AEs and compared with modeling
efforts (see [6, 7] and references therein). It has been found [8] that many
codes can identify the same AE and calculate similar contributions of _drive_
to the total growth rate. However, the evaluation of the contribution from
_damping_ is more variable. Therefore, much work has gone into the analysis of
stable AEs and their net damping rates, in both JET [2, 9, 10, 11] (and
references therein) and Alcator C-Mod [12, 13, 14]. Still, there can be a
disconnect of sorts between un/stable AEs if they are not measured under the
same plasma conditions.

In this paper, we report (for the first time, to the authors’ knowledge) the
_simultaneous_ measurements of three destabilized Toroidicity-induced Alfvén
Eigenmodes (TAEs) and one stabilized TAE in a JET deuterium plasma discharge.
Experimental values of the net growth rates, frequencies, and mode numbers and
locations allow direct comparison with AE stability codes. Agreement between
experiment and models gives us greater confidence in the prediction of AE
stability for future devices; in particular, complementary analyses from the
recent JET DT campaign [15] will identify and validate the contribution of
alphas to AE drive.

We note here that such a measurement of concurrent unstable and stable AEs can
be difficult to achieve in JET: First, the AEAD perturbation only resonates
with a stable AE if their frequencies match and the mode location is
accessible. Then, the fast magnetics only measure the resonance if the signal
is sufficiently strong and the damping rate not too large. In JET, the
magnetic field strength is typically too high for FIs from Neutral Beam
Injection (NBI) to drive AEs unstable; thus, AEs are usually only destabilized
during Ion Cyclotron Resonance Heating (ICRH). However, the AEAD is not
efficient during H-mode [11], so the AEAD measures only few stable AEs during
ICRH (and NBI), regardless of unstable AEs being observed at the same time.

The outline of the rest of the paper is as follows: In Section 2, the
experimental measurements of both stable and unstable AEs are presented.
Sections 3 and 4 then show simulation results from hybrid kinetic-MHD codes
NOVA-K [16, 17, 18] and MEGA [19], respectively. Finally, a discussion and
summary is given in Section 5.

##  2 Experimental measurements of AE stability

In this section, we report the novel experimental observation of stable and
unstable AEs _simultaneously_ measured in JPN 94700. This pulse is part of the
three-ion-heating scenario development experiments at JET [20]; in fact,
extensive analyses of the following, similar plasma discharge, JPN 94701, are
reported in [21]. Experimental results are presented here, and further
kinetic-MHD simulations with NOVA-K [16, 17, 18] and MEGA [19] are given in
Sections 3 and 4, respectively.

The time evolution of plasma parameters for JPN 94700 are shown in Fig. 1(a).
During the time range of interest, $t=7.5{-}11~{}\mathrm{s}$ (shaded), steady-
state plasma parameters are $B_{0}=3.7~{}\mathrm{T}$,
$I_{\mathrm{p}}=2.5~{}\mathrm{MA}$, $n_{\mathrm{e}0}=6\times
10^{19}~{}\mathrm{m^{-3}}$, and $T_{\mathrm{e}0}=5.5{-}7~{}\mathrm{keV}$,
although note the large sawtooth crashes at $t\approx 8~{}\mathrm{s}$ and
$8.8~{}\mathrm{s}$. Auxiliary heating powers are Neutral Beam Injection (NBI)
of $P_{\mathrm{NBI}}=6~{}\mathrm{MW}$, with ${\sim}300~{}\mathrm{ms}$ “blips”
up to $8~{}\mathrm{MW}$ at $t\approx 9~{}\mathrm{s}$ and $9.6~{}\mathrm{s}$,
and Ion Cyclotron Resonance Heating (ICRH) of
$P_{\mathrm{ICRH}}=4~{}\mathrm{MW}$ stepping up to $6~{}\mathrm{MW}$ at
$t\approx 8.5~{}\mathrm{s}$. The concentration of ${}^{3}\mathrm{He}$ is
$n_{\mathrm{He3}}/n_{\mathrm{e}}\approx 27\%$ as part of the
D-(DNBI)-${}^{3}\mathrm{He}$ heating scenario [21].

(a)

(b)

Figure 1: (a) Plasma parameters for JPN 94700: toroidal magnetic field
(dashed), plasma current (solid), central electron density from Thomson
Scattering (TS) and temperature from Electron Cyclotron Emission, and heating
powers from NBI (solid) and ICRH (dashed). Both stable and unstable AEs were
measured during the shaded time interval. (b) Profiles at
$t=9.9~{}\mathrm{s}$: safety factor from EFIT [22] constrained by pressure
only (solid) and additionally polarimetry (dashed) with $q=1$ dotted, electron
density and temperature from TS, and rotation frequency from
${}^{3}\mathrm{He}$ charge exchange. Experimental data are shown as circles
with uncertainties as error bars, while solid lines are fits to the data.

Profiles of various plasma parameters are shown in Fig. 1(b) for one time of
interest, $t=9.9~{}\mathrm{s}$. The electron density and temperature profiles,
from Thomson scattering, match the fitted profiles from TRANSP [23, 24, 25]
well. Note here that an effective ion density profile, including the
relatively high concentration of ${}^{3}\mathrm{He}$, is used in the
simulations of the following sections. The rotation profile is obtained from
${}^{3}\mathrm{He}$ charge exchange spectroscopy.

The safety factor profile is solved iteratively with EFIT [22], including
constraints from kinetic pressure and additionally polarimetry; both
$q$-profiles are similar in Fig. 1(b). It is important to note here that this
can be one of the largest sources of uncertainty in the analysis below. The
observation of Reverse Shear AEs (RSAEs), or Alfvén Cascades, starting at
$t\approx 8.9~{}\mathrm{s}$ (see Fig. 2) indicates that a non-monotonic
$q$-profile exists for some time period following the sawtooth crash at
$t\approx 8.8~{}\mathrm{s}$. Further constraints from Motional Stark Effect
data have been used to identify reverse shear in similar plasmas [26];
however, such data were unfortunately not available for this pulse. That said,
the upward-chirping RSAEs attain their maximum frequencies around $t\approx
9.9~{}\mathrm{s}$, meaning that relaxation to a monotonic $q$-profile is
plausible by this time. A central safety factor $q_{0}\approx 0.8$ is
consistent with the previously sawtoothing profile, but with fast ion
stabilization. As will be shown, good agreement between experiment and
modeling is attained with the $q$-profile shown in Fig. 1(b).

TAEs are destabilized by the FI population during $t=7.5{-}11~{}\mathrm{s}$,
as seen in the spectrogram of Fig. 2(a). This is the Fourier decomposition of
magnetic fluctuation data in both time and toroidal angle; thus, we can
clearly identify coherent TAEs in time, frequency, and with a specific
$n$-value. At first ($t\approx 7.5{-}8~{}\mathrm{s}$ and
$P_{\mathrm{ICRH}}=4~{}\mathrm{MW}$), only $n=2,3$ TAEs are driven unstable,
and then the modes quickly disappear with the sawtooth crash at $t\approx
8~{}\mathrm{s}$. This is likely due to the changing $q$-profile and/or
redistribution of FIs.

(a)

(b)

Figure 2: Fourier decomposition of magnetics data with (a) toroidal and (b)
poloidal mode number analysis for JPN 94700. Note that AEAD signals and ICRH
beat frequencies have been (almost) subtracted from the background in (b).

As ICRH increases to $P_{\mathrm{ICRH}}=6~{}\mathrm{MW}$, $n=2{-}4$ TAEs are
destabilized from $t\approx 8.3{-}8.8~{}\mathrm{s}$, again stabilizing with
the sawtooth crash at $t\approx 8.8~{}\mathrm{s}$. From $t\approx
9{-}11~{}\mathrm{s}$, the FIs destabilize $n=3-5$ TAEs and stabilize the
sawteeth; interestingly, note that the $n=2$ mode has now stabilized. The
$n=1$ and $6$ TAEs are also not observed in the spectrogram during this time
interval at all. Furthermore, the frequency-spacing between mode numbers is
$\Delta f\approx 10{-}20~{}\mathrm{kHz}$, which is slightly larger than the
on-axis rotation frequency (see Fig. 1(b)); the additional shift could be
explained by different mode locations within the TAE gap.

From the magnetics data, the amplitudes of the $n=3{-}5$ modes decrease by
approximately an order of magnitude with each increasing $n$, and this is
qualitatively seen in the spectrograms of Fig. 2. However, deducing true mode
amplitudes with edge diagnostics is difficult, especially with fields
decreasing approximately as $\propto r^{-m}$. That said, the $n=2,3$ TAEs are
also observed as low-amplitude fluctuations in the SXR array, localizing them
approximately within $q\sim 2$;‡‡‡Reflectometry data were not available for
the pulse of interest, JPN 94700; however, those data from a similar
discharge, JPN 95683, localized similar TAEs just outside $q\sim 1$ [21].
$n=4,5$ TAEs are not seen in the SXR data, likely due to a too low signal-to-
noise ratio.

Throughout this discharge, the AEAD scans in frequency, sometimes resonantly
exciting stable AEs. This scan can be seen in Fig. 2(a) as the triangular
waveform. Interestingly, the predominant mode number identified by the
magnetic probes for the AEAD perturbation is $n=5$, although all antennas have
the same phase; therefore, power _should_ be injected into a spectrum of mode
numbers peaked around $n=0$, with a higher likelihood of exciting even than
odd $n$-numbers.

A poloidal mode number ($m$) analysis is also performed, with the resulting
spectrogram shown in Fig. 2(b). As with the $n$-calculation, this is a chi-
square minimization of the phase difference between probes, iterating over all
probes as the reference [5]. We calculate the poloidal angle of each probe
with respect to the magnetic axis from EFIT, but ignore the mode location in
the evaluation. We additionally include an inverse variance weighting,
assuming that the standard deviation is the distance from the poloidal probe
position to the magnetic axis. Furthermore, we restrict the poloidal harmonic
range to $m\in[n,3n]$, assuming the mode is located within $q\leq 3$ for
computational efficiency. This assumption is valid for this analysis, as will
be shown in the next sections, but is an overall limitation; thus, there could
be possible effects from aliasing.

While there are certainly larger uncertainties associated with this
$m$-calculation, the results align with expectations: a mix of $m=4,5$ (brown,
red) is seen for the $n=2$ TAE , and $m=3$ (green) dominates for $n=3$; these
are then located within $q<5/2$, agreeing approximately with the SXR data.
Many $m$-numbers appear for both $n=4$ and $5$, but $m=12$ (yellow) and $14$
(orange), respectively, might stand out as the clearest poloidal harmonics;
this suggests localizations closer to $q\sim 3$, although they could not be
identified in the SXR signal. Finally, note that both the AEAD and ICRH beat
frequencies are (almost) subtracted from the background of this spectrogram;
this was done only as part of a larger database analysis of unstable TAEs in
JET.

The AEAD frequency scan is reproduced in Fig. 3; here, the amplitude of the
magnetic response (summed over all probes) is also shown, and two pairs of
broad peaks are visible around $t\approx 9.5~{}\mathrm{s}$ and
$10.1~{}\mathrm{s}$. These are identified as four measurements of the same
stable AE resonance, with resonant frequency $f_{0}\approx
245~{}\mathrm{kHz}$. This is higher than the unstable $n=5$ TAE in Fig. 2(a),
indicating that it could be a Doppler-shifted $n=6$ TAE. Unfortunately, this
mode could only be measured by a few magnetic probes; the limited number of
probes and their toroidal separation did not enable a reliable determination
of the mode’s toroidal mode number. The (normalized) damping rate measurements
range from $-\gamma/\omega_{0}\approx 2{-}3\%$, with relatively large
uncertainties $\Delta(\gamma/\omega_{0})\approx\pm 1\%$.§§§Note that $\gamma$
has units $\mathrm{s^{-1}}$, and $\omega_{0}$ has units $\mathrm{rad\,s^{-1}}$
and not $\mathrm{Hz}$. A stable $n=6$ mode is expected since the $n=5$ mode is
low amplitude and could be only marginally unstable, i.e.
$\gamma/\omega_{0}\sim 0$.

Figure 3: Stable AE resonance measurements: the magnetic response amplitude
(summed over all probes), AEAD (dashed) and resonant frequencies (circles),
and normalized damping rates with uncertainties as error bars.

The deuterium FI distribution function is calculated from TRANSP [23, 24, 25],
using the NUBEAM [27] and TORIC [28] modules, and is shown in Fig. 4 as a
function of radius, energy, and pitch. The NBI population is seen with a birth
energy ${\sim}100~{}\mathrm{keV}$, and the ICRH-accelerated tail extends to
energies ${\sim}2.5~{}\mathrm{MeV}$ with a dominant volume-averaged pitch
$v_{\parallel}/v\approx 0.5$. As will be discussed in the following sections,
analytic fits of the FI distribution function are performed for both NOVA-K
and MEGA simulations.

(a)

(b)

Figure 4: Deuterium FI distribution functions (logarithmic) from TRANSP for
JPN 94700 and integrated over $t=9.7{-}9.9~{}\mathrm{s}$: (a) averaged over
all pitches, and (b) at $\sqrt{\psi_{\mathrm{N}}}\approx 0.5$.

##  3 Kinetic-MHD simulations with NOVA-K

The kinetic-MHD code NOVA-K [16, 17, 18] is used to identify AEs and assess
their stability at one time, $t=9.9~{}\mathrm{s}$. This time is selected
because it occurs during a relatively steady-state period of the pulse (see
Fig. 1(a)), i.e. after sawtooth stabilization, and in between the measurements
of stable AEs by the AEAD (see Fig. 3). The plasma profiles (see Fig. 1(b))
are used in the calculation of the Alfvén continua and eigenmode structures
for $n=2{-}6$, with a subset shown in Fig. 5. Note that the TAE
eigenfrequencies are in the _lab_ frame, as the experimental rotation profile
is included in NOVA-K; they span $f\approx 195{-}225~{}\mathrm{kHz}$ (see
Table 1), which is only slightly lower than those observed experimentally (see
Fig. 2). This ${\sim}10\%$ difference is likely the result of uncertainties in
the density, safety factor, and rotation profiles; the adiabatic index used in
NOVA-K could also play a role [29].

(a)

(b)

(c)

(d)

Figure 5: Continua (thin lines) and poloidal mode structure (solid, dot-
dashed) from NOVA-K at $t=9.9~{}\mathrm{s}$ for $n=$ (a) 3, (b) 4, (c) 5, (d)
6. The frequency (lab-frame) is indicated by the horizontal dotted line.
$\psi_{\mathrm{N}}$ is the normalized poloidal flux. While the ranges of
poloidal harmonics ($m$) vary for each subplot, the colors and line styles are
the same for all.

The poloidal mode structures from NOVA-K for each $n$ (see Fig. 5) are
primarily dominated by harmonics $m=n,n+1$, i.e. localizing the TAEs around
$q\approx 1$ and consistent with the SXR data, although the overall structures
are quite global. From the NOVA-K results, we may have expected $m=4$ to be
identified as the dominant poloidal harmonic for the $n=3$ TAE in Fig. 2(b),
but instead $m=3$ is found experimentally. Similarly, the higher poloidal
harmonics $m=12$ and $14$ for $n=4$ and $5$ TAEs, respectively, are not
dominant in NOVA-K, but perhaps their closer location to the poloidal probes
increases their detection probability in experiment. For example, see the
$m=14$ harmonic for the $n=5$ TAE in Fig. 5(c). Unfortunately, the SXR array
can help localize these modes, identified by frequency, but their low signals
and measurement by only a few SXR lines-of-sight make a mode number evaluation
infeasible. This demonstrates the difficulty of measuring some properties of
core modes from edge diagnostics.

The stability of the $n=2{-}6$ TAEs is computed including the effects of the
FIs. In NOVA-K, an energy threshold¶¶¶Note that the threshold was varied among
$125~{}\mathrm{keV}$, $150~{}\mathrm{keV}$, and $200~{}\mathrm{keV}$ with
little impact on the stability results. is used to discriminate the NBI and
ICRH populations within the FI distribution function from TRANSP and NUBEAM ,
which is important here because this three-ion-heating scenario specifically
heats the DNBI ions. Then, for each radial position, the local NBI population
is fit with an anisotropic slowing down distribution, while the ICRH tail is
fit with an anisotropic equivalent temperature [13]. Furthermore, the ICRH
resonance layer is modeled as a Gaussian profile, in this case with its peak
at the magnetic axis ($R_{0}\approx 3~{}\mathrm{m}$) and width $\Delta
R\approx 0.24~{}\mathrm{m}$, similar to that shown in [21] (see Fig. 12a
therein).

The breakdown of various drive and damping mechanisms is given in Table 1.
Radiative damping is the largest contribution; this has also been identified
in other recent analyses of (asynchronous) unstable [30, 31] and stable [10,
11] AEs in JET. While the implementation of radiative damping, which is
averaged over the radial mode structure in NOVA-K, comes from [32, 33], Ref.
[34] can help us estimate the _relative_ uncertainty:
$\Delta(\gamma/\omega_{0})_{\mathrm{rad}}\propto(1/3)(\gamma/\omega_{0})_{\mathrm{rad}}\left(\mathrm{d}\ln
T+\mathrm{d}\ln q\right)$, which is ${\sim}8\%$ assuming 10% uncertainties in
the plasma profiles. Continuum damping can also be a significant contributor,
which is clear from the eigenmodes’ intersecting the continuum around
$\sqrt{\psi_{\mathrm{N}}}\approx 0.8$ in Fig. 5, although absolute
uncertainties can be of order $\Delta(\gamma/\omega_{0})\sim 0.1\%$ [35].

Table 1:  Growth rates (%, damping $<0$) calculated from NOVA-K. The frequency is in the lab frame; the _absolute_ uncertainty in continuum damping is $\pm 0.1\%$, while the _relative_ uncertainty in radiative damping is $\pm 8\%$. Totals are given with and _without_ the contributions from fast ions, which include finite Larmor radius effects. Toroidal mode number ($n$) | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---
Frequency (kHz, lab frame) | 179.9 | 195.9 | 206.5 | 215.1 | 224.7
Damping/drive mechanism | Growth rate $\gamma/\omega_{0}$ (%)
Continuum | -0.06 | -0.03 | -0.77 | -0.33 | -0.31
Radiative | -1.03 | -2.01 | -2.84 | -4.44 | -4.76
Electron collisional | -0.32 | -0.39 | -0.04 | -0.04 | -0.01
Electron Landau | -0.09 | -0.11 | -0.07 | -0.12 | -0.12
Ion Landau | -0.02 | -0.01 | -0.04 | -0.01 | -0.01
NBI fast ions | -0.07 | -0.12 | -0.11 |  $\sim$0 | -0.05
ICRH fast ions | 4.08 | 4.30 | 4.45 | 2.01 | 2.34
_Total (w/o fast ions)_ | _-1.53_ | _-2.54_ | _-3.76_ | _-4.94_ | _-5.22_
Total (w/ fast ions) | 2.48 | 1.64 | 0.58 | -2.93 | -2.92

Due to the high magnetic field (and thus high Alfvén speed), NBI ions actually
Landau damp these modes in JET [36], but ICRH FIs provide significant drive,
$\gamma/\omega_{0}\approx 2{-}4\%$. Thus, $n=2{-}4$ TAEs are predicted by
NOVA-K to be destabilized, with decreasing total growth rates, while $n=5,6$
TAEs are stabilized with similar total damping rates. The trend of decreasing
drive with $n$ is at least consistent for $n=3{-}5$ modes, although of course
$n=5$ is _unstable_ in experiment (see Fig. 2). Additionally, the $n=2$ TAE is
stable at $t=9.9~{}\mathrm{s}$ in experiment, not strongly driven. These are
currently unresolved discrepancies. That said, the net damping rate
$\gamma/\omega_{0}\approx-2.9\%$ for the $n=6$ TAE agrees within experimental
uncertainties.

##  4 Hybrid kinetic-MHD simulations with MEGA

Despite NOVA-K modeling successfully reproducing many of the characteristics
of the observed TAEs (such as mode frequency and radial location), the
simulations of the previous section predicted an unstable $n=2$ TAE at
$t=9.9~{}\mathrm{s}$, while such a mode is not detected by any fluctuation
diagnostic at that time. On the other hand, large radiative and continuum
damping are predicted for the $n=5$ mode, thus stabilizing it, yet the TAE is
clearly observed in experiment.

To further investigate these features and resolve these discrepancies, a more
advanced hybrid kinetic-MHD code MEGA [19] is employed. MEGA self-consistently
solves the evolution of a nonlinear, resistive, full-MHD background and the
dynamics of the FI population. The FIs are simulated using gyro-kinetic
markers that include finite Larmor radius effects. Both the fluid plasma and
FIs are coupled together by means of including the FI current density term in
the MHD momentum equation applied in the particle-in-cell algorithm. The model
for the thermal plasma is derived by Hazeltine and Meiss in [37], and its
implementation includes diamagnetic drifts and plasma rotation, providing good
accuracy of the simulated TAE frequency.

As with the NOVA-K simulations described in the previous section, the electron
temperature, plasma rotation, and ion density profiles used in these
simulations are taken from TRANSP (see Fig. 1). As mentioned, the density
profile is corrected to account for the ${}^{3}\mathrm{He}$ concentration,
which will affect the simulated mode frequency. For these simulations, the
cylindrical grid used to solve the MHD equations has a resolution of
$N_{R}\times N_{\phi}\times N_{Z}=128\times 192\times 128$; this resolution in
the poloidal plane has good convergence and has been used previously to
investigate harmonics $m>10$ [38]. All modes above $n=6$ are filtered out, so
a minimum of 32 grid points in the toroidal direction per harmonic is ensured.
The simulation time step is 4% of a gyro-period of a deuterium ion at the
magnetic axis. For the dissipative parameters (resistivity $\eta$, viscosity
$\nu$, and diffusivity $\chi$), normalized values
$\eta/\mu_{0}=\nu=\chi=5\times 10^{-7}v_{A}R_{0}$ are used, where $v_{A}$ is
the Alfvén velocity at the magnetic axis and $R_{0}$ is the major radius.

Approximately 6.2 million markers (${\sim}2$ per simulation grid point) are
employed to resolve the FI population using the $\delta f$ method. A quasi-
analytical∥∥∥In MEGA, markers are uniformly distributed in real and velocity
space, with weightings based on energy, normalized pitch angle, and poloidal
flux., anisotropic, single slowing-down distribution is used, where the
maximum energy is set at $E_{\mathrm{max}}=2~{}\mathrm{MeV}$ and the critical
velocity $v_{c}$ is radially dependent on $T_{e}(r)$. In order to match the
ICRH absorption layer with the turning point of the trapped FIs, the
normalized pitch of the simulated FI distribution is centered at $\Lambda=\mu
B_{0}/E=0.8$, with a width of $\Delta\Lambda=0.3$, producing a real-space-
integrated velocity space similar to that depicted in Fig. 4(b).

A simulation is performed where the different harmonics grow linearly together
and then saturate. Figure 6(a) shows the poloidal projection of the radial
velocity perturbation associated with the $n=4$ instability. The toroidal
harmonics $n=3{-}5$ are found to be unstable, in agreement with the
experiment. The poloidal harmonics of each toroidal harmonic are depicted in
Fig. 6(b), and good agreement is seen when comparing to the mode structures
from NOVA-K in Fig. 5. The locations of these harmonics are also consistent
with the observed locations in this and previous experiments.

Figure 6: (a) MEGA-simulated poloidal structure of the radial velocity
perturbation of the unstable $n=4$ TAE. Radial profiles of the (b) radial
velocity and (c) magnetic field perturbation for the unstable $n=3{-}5$ modes,
with the $q$-profile overlaid and surface $q=(n+1/2)/n$ marked.

Next, another simulation is performed with twice the FI pressure from TRANSP,
i.e. $2\beta_{\mathrm{fi}}$, aiming to observe the toroidal harmonics that
remained stable in the previous simulation. The evolution of these fields are
Fourier decomposed and plotted together with Alfvén continua in Fig. 7;
frequencies are provided in Table 2. The modes $n=3-6$, as expected, can be
classified as TAEs since they are located within the continuum gap calculated
by the ALCON code [39]. The frequency of the ALCON 1-D estimation of the
Alfvén continuum for each harmonic is Doppler-shifted by the experimentally
measured toroidal rotation included in the simulation. One can observe that
the $n=2$ instability strongly intersects the Alfvén continuum; thus, this
mode can be classified as an energetic particle mode (EPM) at the time of
interest. However, this does not mean that the unstable $n=2$ mode observed
experimentally earlier in time (see Fig. 2, $t\approx 7.5-9~{}\mathrm{s}$) is
also an EPM.

Figure 7: Fourier-decomposed radial velocity from MEGA together with the
Alfvén continuum calculated by the ALCON code [39]. The modes are located
within the TAE gap, except the $n=2$ EPM.

Comparing Figs. 2, 1 and 2, we see that MEGA’s predicted mode frequencies are
too low compared to experimental values for $n=2{-}4$, but closer than NOVA-K
for $n=5,6$. In addition, it is interesting that MEGA does not find an $n=2$
TAE in the gap indicated by ALCON (see Fig. 7) around
${\sim}180~{}\mathrm{kHz}$, while the EPM is destabilized instead.

Since MEGA reproduces the TAEs satisfactorily and finds $n=3{-}5$ unstable as
in the experiment, a scan in $\beta_{\mathrm{fi}}$ is performed to further
investigate the intrinsic damping associated with each mode and how sensitive
the FI drive is to the applied FI pressure. To disentangle these effects for
each TAE, single-$n$ runs are performed, simulating only a fraction of the
toroidal angle $\phi\in[0,2\pi/n]$. The toroidal resolution is reduced to
$N_{\phi}=32$ for these runs. Seven values of $\beta_{\mathrm{fi}}$ are
simulated, ranging from the value inferred from TRANSP up to
$2.25\beta_{\mathrm{fi}}$. Figure 8(a) shows the temporal evolution of the
energy of the $n=3$ TAE for these different values of FI pressure, while Fig.
8(b) shows the temporal evolution of the different toroidal harmonics
simulated for the case with $2\beta_{\mathrm{fi}}$.

Figure 8: (a) Temporal evolution of the $n=3$ TAE energy for different values
of FI pressure $\beta_{\mathrm{fi}}$ in MEGA. (b) Temporal evolution of
different toroidal harmonics for $2\beta_{\mathrm{fi}}$. (c) Growth rates as a
function of $\beta_{\mathrm{fi}}$: $n=3{-}6$ have similar values of inferred
damping (at $\beta_{\mathrm{fi}}=0$) while $n=2$ has the largest. The slope of
$n=6$ is lower, indicating that the mode is stable due to poor FI drive. The
stability of an $n=3$ TAE with increased resistivity ($10\eta$) is also
included (dashed), producing a line almost parallel to the reference $n=3$
case, indicating that the resistivity primarily affects the mode damping, not
drive.

The growth rate of the instability of each simulation is evaluated during the
linear phase and plotted in Fig. 8(c) as a function of the FI pressure;
specific values are also given in Table 2. Note that $n=2$ and 6 are only
observed for simulations with pressures ${\geq}1.75\beta_{\mathrm{fi}}$
because those modes remain stable for lower values. A linear fit is performed
to the growth rates of each toroidal mode number. The crossing point with the
ordinate ($\beta_{\mathrm{fi}}=0$) indicates the negative growth rate of the
instability in the case without FIs, thereby indirectly estimating the total,
or “intrinsic,” mode damping.

Table 2:  Total growth rates (%, damping $<0$) calculated from MEGA for different values of the fast ion pressure $\beta_{\mathrm{fi}}$ from TRANSP . Toroidal mode number ($n$) | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---
Frequency (kHz, lab frame) | 125 | 170 | 200 | 230 | 240
Fast ion beta | Total growth rates $\gamma/\omega_{0}$ (%)
$0.0\beta_{\mathrm{fi}}$ | -17.9 $\pm$ 0.9 | -6.9 $\pm$ 0.4 | -8.0 $\pm$ 0.6 | -8.9 $\pm$ 0.4 | -4.7 $\pm$ 1.0
$0.7\beta_{\mathrm{fi}}$ | -7.5 $\pm$ 1.0 | 2.3 $\pm$ 0.5 | 1.7 $\pm$ 0.7 | 0.1 $\pm$ 0.5 | -1.2 $\pm$ 1.1
$1.0\beta_{\mathrm{fi}}$ | -3.0 $\pm$ 1.0 | 6.2 $\pm$ 0.5 | 5.9 $\pm$ 0.7 | 3.7 $\pm$ 0.5 | 0.3 $\pm$ 1.1

It is interesting, although perhaps expected, that the $n=3{-}6$ TAEs have
similar values of the intrinsic damping, i.e. $-\gamma/\omega_{0}\approx
5\%{-}10\%$, while $n=2$ EPM is almost twice as large. In view of these
computational results, one could argue that $n=2$ and 6 modes are stable in
the experiment but for different reasons: the $n=2$ EPM has a larger intrinsic
damping due to the strong interaction with the SAW continuum, while the $n=6$
TAE has a lower FI drive, which is reflected by the lower slope in Fig. 8(c).
This is only partially found in the NOVA-K results (see Table 1): The total
intrinsic damping is predicted to increase with $n$, in disagreement with
MEGA. On the other hand, the drive of the $n=2{-}4$ modes is about twice that
of the $n=6$ mode, in agreement with MEGA, although the same applies to $n=5$,
which is a discrepancy. Moreover, there is a factor of 1-10 difference in the
intrinsic damping values predicted by NOVA-K and MEGA, but part of this may be
due to the resistivity used in MEGA, discussed further below.

At the experimental value of $\beta_{\mathrm{fi}}$, the $n=3{-}5$ TAEs are
unstable and decreasing in their growth rates. This qualitatively agrees with
decreasing (saturated) mode amplitudes observed by the magnetic probes in
experiment (see Fig. 2). The $n=2$ mode is heavily damped around
${\sim}\beta_{\mathrm{fi}}$, also in agreement with experiment. At the TRANSP
value of $\beta_{\mathrm{fi}}$, the linear extrapolation of the $n=6$ TAE
growth rate would suggest a marginally _unstable_ mode with
$\gamma/\omega_{0}\approx+0.3\%$, although the uncertainty is significant
${\sim}1\%$ (see Table 2). To best match the measured net damping rate of
$-\gamma/\omega_{0}\approx 2\%\pm 1\%$, we would need to decrease the FI
pressure to ${\sim}0.7\beta_{\mathrm{fi}}$, at which point the $n=5$ TAE would
be only marginally unstable, also in better agreement with experiment.

It is important to note that the validity of the estimated mode damping
depends on the resistivity $\eta$ used in these simulations; however, a value
larger than the Spitzer value was chosen to accommodate MEGA’s high-resolution
grid [40]. To test the sensitivity to $\eta$, the $\beta_{\mathrm{fi}}$ scan
was run again for the $n=3$ harmonic using a resistivity ten times larger. The
resulting growth rates are shown in Fig. 8(c). As expected, the inferred
damping rate has increased by ${\sim}50\%$, but not enough to exceed the
damping estimated for the $n=2$ EPM . The slope of the (dashed) line is very
similar to the reference case, suggesting that $\eta$ has little effect on the
FI drive. In reality, using the true (lower) resistivity would _decrease_ the
net damping. Thus, while variations of $\eta$ might slightly change the
inferred damping, the conclusion that the $n=2$ mode remains stable due to
larger intrinsic damping and $n=6$ due to lower FI drive is still valid.

##  5 Summary

In this work, we presented novel measurements of both unstable and stable TAEs
observed simultaneously in a JET deuterium plasma, with steady-state
parameters $B_{0}=3.7~{}\mathrm{T}$, $I_{\mathrm{p}}=2.5~{}\mathrm{MA}$,
$n_{\mathrm{e}0}=6\times 10^{19}~{}\mathrm{m^{-3}}$, and
$T_{\mathrm{e}0}=5.5{-}7~{}\mathrm{keV}$ (see Fig. 1). The three-ion-heating
scheme (D-DNBI-27% ${}^{3}\mathrm{He}$) was used in this discharge to
accelerate ${\sim}100~{}\mathrm{keV}$ NBI-deuterons to ${\sim}$MeV energies
(see Fig. 4) with a dominant pitch $v_{\parallel}/v\approx 0.5$. This fast ion
(FI) population destabilized $n=2{-}5$ TAEs, and their saturated mode
amplitudes decreased with toroidal mode number $n$ (see Fig. 2), from which
decreasing FI drive was inferred. Synchronously, JET’s in-vessel Alfvén
Eigenmode Active Diagnostic (AEAD) resonantly excited a stable AE with
normalized damping rate $-\gamma/\omega_{0}\approx 2\%\pm 1\%$ (see Fig. 3).
Its Doppler-shifted frequency, $f_{0}\approx 245~{}\mathrm{kHz}$, indicated
that it was likely an $n=6$ TAE.

Simulations with two hybrid kinetic-MHD codes, NOVA-K and MEGA, both
identified AEs with similar eigenfrequencies, poloidal mode structures, and
radial locations as those observed in experiment. NOVA-K’s calculation of AE
stability demonstrated good agreement with the $n=3$, 4, and 6 TAEs (see Fig.
5), matching the experimentally measured damping rate of the $n=6$ mode within
uncertainties and identifying radiative damping as the dominant mechanism (see
Table 1). However, there were two major discrepancies between NOVA-K and
experiment: the stable $n=2$ TAE, i.e. not seen in fluctuation data, was
predicted by NOVA-K to be strongly driven, while the marginally unstable $n=5$
TAE was predicted to be strongly damped.

Improved agreement was then obtained with MEGA for all modes: the unstable
$n=3{-}5$ and stable $n=2,6$ modes (see Figs. 6 and 7). By scanning the FI
pressure $\beta_{\mathrm{fi}}$ in MEGA, an $n=2$ EPM and $n=6$ TAE were
destabilized and identified. The intrinsic damping was extrapolated to
$\beta_{\mathrm{fi}}=0$ (see Figs. 8 and 2), and the $n=2,6$ modes were found
to be stabilized for two different reasons: the former EPM by higher intrinsic
damping due to strong interaction with the Alfvén continuum, and the latter
TAE by lower fast ion drive. Yet the best agreement of the MEGA growth rates
with experiment was not found at the value of $\beta_{\mathrm{fi}}$ inferred
from TRANSP, but rather at ${\sim}0.7\beta_{\mathrm{fi}}$.

These disagreements between experiment and simulation still need to be
resolved. One resolution could be found by manipulating the various inputs
within experimental error bars, and that exploration is left for future work.
The codes also include different physics (see [8] and others for further
details): NOVA-K is a perturbative eigenvalue code with some contributions to
the linear growth/damping rate calculated analytically, whereas MEGA is a
first-principle, resistive, fully nonlinear initial value code. The benefit of
NOVA-K is the identification of various drive and damping mechanisms and their
uncertainties, whereas MEGA gives only the total - though perhaps more
accurate - growth rate.

The codes may also need to improve their stability calculations of intrinsic
damping and drive. These data, as well as large databases of stable AEs (see
[5, 10, 11] and references therein), could help guide this effort.
Nevertheless, the trends of decreasing growth rate for $n=3{-}6$ TAEs in both
MEGA and NOVA-K is promising, and so is the relatively good match with the
experimentally measured damping rate. Preliminary results from the orbit-
following code ORBIT [41] indicate some strong interactions of the TAEs with
trapped fast ions, and this should be looked into further. Additional upcoming
work will include similar analyses of AE stability in DT plasmas from the
recent JET 2021 campaign, during which the AEAD actively excited over 2000
stable AEs.

Looking toward future tokamaks, like ITER and SPARC, a similar modeling
strategy could be used to predict and optimize AE stability: Multiple codes,
e.g. NOVA-K and MEGA, should be used to identify AEs (and other FI-driven
modes), and a cross-check performed among them to confirm similar mode
frequencies, structures, locations, etc. The findings of this work suggest
that scanning the FI pressure ${\pm}30\%$ may be required (at least in MEGA)
to best match reality, although $+30\%$ may be too conservative and
uncertainties must be considered. Both the MEGA $\beta_{\mathrm{fi}}$-scan and
NOVA-K breakdown can then identify the dominant contributions to damping and
drive, and various parameters could be tuned to help stabilize those modes,
e.g. plasma elongation or ICRH resonance location.

## Acknowledgments

The authors are grateful to the reviewers whose comments improved this paper.
The authors also thank P. Bonofiglo, N. Dreval, N. Gorelenkov, W.W. Heidbrink,
and Y. Todo for fruitful discussions. This work was supported by US DOE grants
DE-SC0014264, DE-AC02-09CH11466, DE-SC0020412, and DE-SC0020337, as well as
the Brazilian agency FAPESP Project 2011/50773-0. This work has been carried
out within the framework of the EUROfusion Consortium, funded by the European
Union via the Euratom Research and Training Programme (Grant Agreement No
101052200 – EUROfusion). Views and opinions expressed are however those of the
author(s) only and do not necessarily reflect those of the European Union or
the European Commission. Neither the European Union nor the European
Commission can be held responsible for them. This research used resources of
the National Energy Research Scientific Computing Center (NERSC), a U.S.
Department of Energy Office of Science User Facility located at Lawrence
Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231
using NERSC award FES-ERCAP20598.

## References

## References

  * [1] A. Fasoli, C. Gormenzano, H. L. Berk, B. Breizman, S. Briguglio, D. S. Darrow, N. Gorelenkov, W. W. Heidbrink, A. Jaun, S. V. Konovalov, R. Nazikian, J. M. Noterdaeme, S. Sharapov, K. Shinohara, D. Testa, K. Tobita, Y. Todo, G. Vlad, and F. Zonca.  Progress in the ITER Physics Basis Chapter 5: Physics of energetic ions.  2007 Nuclear Fusion 47 S264.
  * [2] A. Fasoli, D. Borba, G. Bosia, D. J. Campbell, J. A. Dobbing, C. Gormezano, J. Jacquinot, P. Lavanchy, J. B. Lister, P. Marmillod, J. M. Moret, A. Santagiustina, and S. Sharapov.  Direct measurement of the damping of toroidicity-induced Alfvén eigenmodes.  Physical Review Letters, 75(4):645–648, 1995.
  * [3] T. Panis, D. Testa, A. Fasoli, A. Klein, H. Carfantan, P. Blanchard, and JET-EFDA Contributors.  Optimization of the active MHD spectroscopy system on JET for the excitation of individual intermediate and high-n Alfvén eigenmodes.  Nuclear Fusion, 50(084019), 2010.
  * [4] P. Puglia, W. Pires de Sa, P. Blanchard, S. Dorling, S. Dowson, A. Fasoli, J. Figueiredo, R. Galvão, M. Graham, G. Jones, C. Perez von Thun, M. Porkolab, L. Ruchko, D. Testa, P. Woskov, M.A. Albarracin-Manrique, and JET Contributors.  The upgraded JET toroidal Alfvén eigenmode diagnostic system.  Nuclear Fusion, 56(11):112020, nov 2016.
  * [5] R A Tinguely, P G Puglia, N Fil, S Dowson, M Porkolab, A Fasoli, D Testa, and Jet Contributors.  Results from the Alfvén Eigenmode Active Diagnostic during the 2019- 2020 JET deuterium campaign.  Plasma Phys. Control. Fusion, 62:11, 2020.
  * [6] King-Lap Wong.  A review of Alfvén eigenmode observations in toroidal plasmas.  Plasma Physics and Controlled Fusion, 41(1):R1–R56, jan 1999.
  * [7] W. W. Heidbrink.  Basic physics of Alfvén instabilities driven by energetic particles in toroidally confined plasmas.  Physics of Plasmas, 15(5):055501, may 2008.
  * [8] S. Taimourzadeh, E.M. Bass, Y. Chen, C. Collins, N.N. Gorelenkov, A. Könies, Z.X. Lu, D.A. Spong, Y. Todo, M.E. Austin, J. Bao, A. Biancalani, M. Borchardt, A. Bottino, W.W. Heidbrink, R. Kleiber, Z. Lin, A. Mishchenko, L. Shi, J. Varela, R.E. Waltz, G. Yu, W.L. Zhang, and Y. Zhu.  Verification and validation of integrated simulation of energetic particles in fusion plasmas.  Nuclear Fusion, 59(6):066006, apr 2019.
  * [9] D. Borba, A. Fasoli, N.N. Gorelenkov, S. Gunter, Ph. Lauber, N. Mellet, R. Nazikian, T. Panis, S.D. Pinches, D. Spong, D. Testa, and JET-EFDA Contributors.  The Influence of Plasma Shaping on the Damping of Toroidal Alfvén Eigenmodes.  (Proc. 23rd Int. Conf. Daejeon, 2010) (Vienna: IAEA) CD-ROM file THW/P7-08 and http://www-naweb.iaea.org/napc/physics/FEC/FEC2010/index.htm
  * [10] R A Tinguely, P G Puglia, N Fil, S Dowson, M Porkolab, A Dvornova, A Fasoli, M Fitzgerald, V Guillemot, G T A Huysmans, M Maslov, S Sharapov, and D Testa.  Experimental studies of plasma-antenna coupling with the JET Alfvén Eigenmode Active Diagnostic.  Nuclear Fusion, 61:026003, 2021.
  * [11] R.A. Tinguely, N. Fil, P.G. Puglia, S. Dowson, M. Porkolab, V. Guillemot, M. Podestà, M. Baruzzo, R. Dumont, A. Fasoli, M. Fitzgerald, Ye.O. Kazakov, M.F.F. Nave, M. Nocente, J. Ongena, S.E. Sharapov, Ž. Štancar, and JET Contributors.  A novel measurement of marginal alfvén eigenmode stability during high power auxiliary heating in JET.  Nuclear Fusion, 62(7):076001, apr 2022.
  * [12] J. A. Snipes, D. Schmittdiel, A. Fasoli, R. S. Granetz, and R. R. Parker.  Initial active MHD spectroscopy experiments exciting stable Alfvén eigenmodes in Alcator C-Mod.  Plasma Physics and Controlled Fusion, 46(4):611–620, apr 2004.
  * [13] J. A. Snipes, N. Basse, C. Boswell, E. Edlund, A. Fasoli, N. N. Gorelenkov, R. S. Granetz, L. Lin, Y. Lin, R. Parker, M. Porkolab, J. Sears, S. Sharapov, V. Tang, and S. Wukitch.  Active and fast particle driven alfvén eigenmodes in alcator c-mod.  Physics of Plasmas, 12(5):056102, 2005.
  * [14] J. A. Snipes, N. N. Gorelenkov, and J. A. Sears.  A comparison of measured and calculated toroidal Alfvén eigenmode damping rates in Alcator C-Mod.  Nuclear Fusion, 46(12):1036–1046, dec 2006.
  * [15] Joelle Mailloux.  Overview of JET results for optimising ITER operation.  Nuclear Fusion 62 042026, 2022.
  * [16] CZ Cheng.  Kinetic extensions of magnetohydrodynamics for axisymmetric toroidal plasmas.  Physics Reports, 211(1):1–51, 1992.
  * [17] GY Fu and CZ Cheng.  Excitation of high-n toroidicity-induced shear Alfvén eigenmodes by energetic particles and fusion alpha particles in tokamaks.  Physics of Fluids B: Plasma Physics, 4(11):3722–3734, 1992.
  * [18] NN Gorelenkov, Chio-Zong Cheng, and GY Fu.  Fast particle finite orbit width and Larmor radius effects on low-n toroidicity induced Alfvén eigenmode excitation.  Physics of Plasmas, 6(7):2802–2807, 1999.
  * [19] Y. Todo and T. Sato.  Linear and nonlinear particle-magnetohydrodynamic simulations of the toroidal Alfvén eigenmode.  Physics of Plasmas, 5:1321–1327, 1998.
  * [20] Ye O Kazakov, Massimo Nocente, MJ Mantsinen, Jozef Ongena, Yuriy Baranov, Teddy Craciunescu, Mykola Dreval, Remi Dumont, Jacob Eriksson, Jeronimo Garcia, et al.  Plasma heating and generation of energetic D ions with the 3-ion ICRF+ NBI scenario in mixed HD plasmas at JET-ILW.  Nuclear Fusion, 60(11):112013, 2020.
  * [21] Ye. O. Kazakov, J. Ongena, J. C. Wright, S. J. Wukitch, V. Bobkov, J. Garcia, V. G. Kiptily, M. J. Mantsinen, M. Nocente, M. Schneider, H. Weisen, Y. Baranov, M. Baruzzo, R. Bilato, A. Chomiczewska, R. Coelho, T. Craciunescu, K. Crombé, M. Dreval, R. Dumont, P. Dumortier, F. Durodié, J. Eriksson, M. Fitzgerald, J. Galdon-Quiroga, D. Gallart, M. Garcia-Muñoz, L. Giacomelli, C. Giroud, J. Gonzalez-Martin, A. Hakola, P. Jacquet, T. Johnson, A. Kappatou, D. Keeling, D. King, K. K. Kirov, P. Lamalle, M. Lennholm, E. Lerche, M. Maslov, S. Mazzi, S. Menmuir, I. Monakhov, F. Nabais, M. F. F. Nave, R. Ochoukov, A. R. Polevoi, S. D. Pinches, U. Plank, D. Rigamonti, M. Salewski, P. A. Schneider, S. E. Sharapov, Ž. Štancar, A. Thorman, D. Valcarcel, D. Van Eester, M. Van Schoor, J. Varje, M. Weiland, and N. Wendler.  Physics and applications of three-ion icrf scenarios for fusion research.  Physics of Plasmas, 28(2):020501, 2021.
  * [22] L.L. Lao, H. St. John, R.D. Stambaugh, A.G. Kellman, and W. Pfeiffer.  Reconstruction of current profile parameters and plasma shapes in tokamaks.  Nuclear Fusion, 25(11):1611–1622, 1985.
  * [23] Joshua Breslau, Marina Gorelenkova, Francesca Poli, Jai Sachdev, and Xingqiu Yuan.  TRANSP.  [Computer Software] https://doi.org/10.11578/dc.20180627.4, jun 2018\.
  * [24] RJ Hawryluk.  An empirical approach to tokamak transport.  In Coppi et al, editor, Physics Close to Thermonuclear Conditions, Ed. B, volume 19. Brussels: Commission of the European Communities, Brussels, 1980.
  * [25] JPHE Ongena, I Voitsekhovitch, M Evrard, and D McCune.  Numerical transport codes.  Fusion Science and Technology, 61(2T):180–189, 2012.
  * [26] Mykola Dreval, Sergei E Sharapov, Yevgen Kazakov, Jozef Ongena, Massimo Nocente, Rui Calado, Rui Coelho, Jorge Ferreira, Antonio C A Figueiredo, Michael Fitzgerald, Jeronimo Garcia, Carine Giroud, Nicholas Hawkes, Vasily Kiptily, Fernando Nabais, Maria Filomena Ferreira Nave, Henri Weisen, Teddy Craciunescu, Mirko Salewski, and Ziga Stancar.  Alfvén cascade eigenmodes above the TAE-frequency and localization of Alfvén modes in D-3He plasmas on JET.  Nuclear Fusion 62 056001, 2022.
  * [27] Alexei Pankin, Douglas McCune, Robert Andre, Glenn Bateman, and Arnold Kritz.  The tokamak monte carlo fast ion module nubeam in the national transport code collaboration library.  Computer Physics Communications, 159(3):157–184, 2004.
  * [28] M Brambilla.  Numerical simulation of ion cyclotron waves in tokamak plasmas.  Plasma Physics and Controlled Fusion, 41(1):1–34, jan 1999.
  * [29] M.A. Van Zeeland, W.W. Heidbrink, S.E. Sharapov, D. Spong, A. Cappa, Xi Chen, C. Collins, M. García-Muñoz, N.N. Gorelenkov, G.J. Kramer, P. Lauber, Z. Lin, and C. Petty.  Electron cyclotron heating can drastically alter reversed shear Alfvén eigenmode activity in DIII-D through finite pressure effects.  Nuclear Fusion, 56(11):112007, jul 2016.
  * [30] F. Nabais, V. Aslanyan, D. Borba, R. Coelho, R. Dumont, J. Ferreira, A. Figueiredo, M. Fitzgerald, E. Lerche, J. Mailloux, M. Mantsinen, P. Rodrigues, M. Porkolab, P. Puglia, S.E. Sharapov, and JET Contributors.  TAE stability calculations compared to TAE antenna results in JET.  Nuclear Fusion, 58(8):082007, aug 2018.
  * [31] V. Aslanyan, S. Taimourzadeh, L. Shi, Z. Lin, G. Dong, P. Puglia, M. Porkolab, R. Dumont, S. E. Sharapov, J. Mailloux, M. Tsalas, M. Maslov, A. Whitehead, R. Scannell, S. Gerasimov, S. Dorling, S. Dowson, H. K. Sheikh, T. Blackman, G. Jones, A. Goodyear, K. K. Kirov, P. Blanchard, A. Fasoli, and D. Testa.  Gyrokinetic simulations of toroidal Alfvén eigenmodes excited by energetic ions and external antennas on the Joint European Torus.  Nuclear Fusion, 59(2) 026008, 2019.
  * [32] H. L. Berk, R. R. Mett, and D. M. Lindberg.  Arbitrary mode number boundary-layer theory for nonideal toroidal Alfvén modes.  Physics of Fluids B, 5(11):3969–3996, nov 1993.
  * [33] G. Y. Fu, C. Z. Cheng, R. Budny, Z. Chang, D. S. Darrow, E. Fredrickson, E. Mazzucato, R. Nazikian, K. L. Wong, and S. Zweben.  Analysis of alpha particle-driven toroidal Alfvén eigenmodes in Tokamak Fusion Test Reactor deuterium-tritium experiments.  Physics of Plasmas, 3(11):4036–4045, nov 1996.
  * [34] R. R. Mett and S. M. Mahajan.  Kinetic theory of toroidicity-induced Alfvén eigenmodes.  Physics of Fluids B, 4(9):2885–2893, sep 1992.
  * [35] G. W. Bowden, A. Könies, M. J. Hole, N. N. Gorelenkov, and G. R. Dennis.  Comparison of methods for numerical calculation of continuum damping.  Physics of Plasmas, 21(5):052508, 2014.
  * [36] D Borba, B Alper, R.V Budny, A Fasoli, R.F Heeter, W Kerner, S.E Sharapov, and P Smeulders.  Beam driven alfvén eigenmodes and fishbones in JET.  Nuclear Fusion, 40(4):775–783, apr 2000.
  * [37] R. D. Hazeltine and J. D. Meiss.  Shear-Alfvén dynamics of toroidally confined plasmas.  Physics Reports, 121:1–164, 5 1985.
  * [38] Y Todo, M Sato, Hao Wang, M Idouakass, and R Seki.  Magnetohydrodynamic hybrid simulation model with kinetic thermal ions and energetic particles.  Plasma Physics and Controlled Fusion, 63(7):075018, jun 2021.
  * [39] W. Deng, Z. Lin, I. Holod, Z. Wang, Y. Xiao, and H. Zhang.  Linear properties of reversed shear Alfvén eigenmodes in the DIII-D tokamak.  Nuclear Fusion, 52(4) 043006, 2012.
  * [40] A. Bierwage, Y. Todo, N. Aiba, and K. Shinohara.  Sensitivity study for N-NB-driven modes in JT-60U: boundary, diffusion, gyroaverage, compressibility.  Nuclear Fusion, 56:106009, 8 2016.
  * [41] R. B. White and M. S. Chance.  Hamiltonian guiding center drift orbit calculation for plasmas of arbitrary cross section.  The Physics of Fluids, 27(10):2455–2467, 1984.


# CIKQA: Learning Commonsense Inference with a Unified
Knowledge-in-the-loop QA Paradigm

Hongming Zhang1,2, Yintong Huo3, Yanai Elazar4,5, Yangqiu Song1, Yoav
Goldberg4,5, Dan Roth2
1HKUST, 2UPenn, 3CUHK, 4Bar Ilan University, 5AI2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, the community has achieved substantial progress on many commonsense
reasoning benchmarks. However, it is still unclear what was learned from the
training process: the knowledge, how to do inference, or both? We argue that
due to the large scale of commonsense knowledge, it is infeasible to annotate
a large enough training set for each task to cover all commonsense for
learning. Thus we should separate the commonsense knowledge acquisition and
inference over commonsense knowledge as two separate tasks. In this work, we
focus on investigating models’ commonsense inference capabilities from two
perspectives: (1) Whether models can know if the knowledge they have is enough
to solve the task; (2) Whether models can learn commonsense inference
capabilities, that generalize across commonsense tasks. We first align
commonsense tasks with relevant knowledge from commonsense knowledge bases and
ask humans to annotate whether the knowledge is enough or not. Then, we
convert different commonsense tasks into a unified question answering format
to evaluate models’ generalization capabilities. We name the benchmark as
Commonsense Inference with knowledge-in-the-loop Question Answering (CIKQA).

CIKQA: Learning Commonsense Inference with a Unified
Knowledge-in-the-loop QA Paradigm

Anonymous ACL submission

##  1 Introduction

Figure 1: CIKQA demonstration. Models need to learn that all pronouns “I”
refer to the same person and then solve the question based on the knowledge
that “one may fall into sleep if he/she rests on a bench.” Yanai: this is not
very clear. I think we should provide a more high level description, of what
we’re trying to do, rather than the specific solution to this problem (or in
addition).

Understanding human language requires both the language knowledge (e.g.,
grammar and semantics) and world knowledge, which can be further divided into
factual and commonsense knowledge Katz1963-KATTSO-3. Recently, the community
has made great progress on helping machines acquire and apply language and
factual knowledge. However, how to help machines acquire and inference over
commonsense is still unclear. To answer this question, many commonsense
reasoning datasets DBLP:conf/aaaiss/RoemmeleBG11; sakaguchi2019winogrande;
DBLP:conf/naacl/TalmorHLB19; DBLP:conf/cvpr/ZellersBFC19;
DBLP:conf/emnlp/LinLKR20 have been proposed. Even though they target different
knowledge types, modalities, and come in different formats, they often follow
a standard supervised learning setting, and aim at helping machines to solve a
specific task with the training data. However, two limitations of this
learning paradigm have limited the development of commonsense reasoning
systems.

First, there is no clear separation between knowledge and inference. As
discussed in DBLP:journals/corr/abs-2104-08161, a common phenomenon is that
larger training data will lead to better performance, mainly because richer
knowledge is covered. However, due to the large scale of commonsense
knowledge, it is infeasible to annotate a large enough training data for each
task, and the responsibility of the training data should be teaching models
how to do inference rather than acquiring the commonsense knowledge. Several
recent works have explored using structured knowledge for commonsense
reasoning tasks DBLP:conf/emnlp/LinCCR19; DBLP:conf/aaai/LvGXTDGSJCH20;
DBLP:conf/emnlp/PaulF20. However, as these works did not clearly analyze the
coverage of the structured knowledge (i.e., knowledge graphs), it is still
unclear what the performance means, better knowledge coverage or better
inference capability. To dig into what is behind this learning process, we
propose to equip each question with supporting knowledge. By doing so, we
could evaluate whether models can know if there provided knowledge is
sufficient or not and how well they can conduct inference over the provided
knowledge to solve the task.

Second, the supervised learning may force the model to learn the distribution
of the training data rather than a universal inference model. As a result, the
model may perform well on the test set that follows the same distribution but
fail on other tasks DBLP:journals/corr/abs-2011-09159. Previously, as
different tasks have different formats, it is hard to evaluate the
generalization ability of commonsense reasoning models. Motivated by existing
trends of using a unified format (i.e., question answering) for different
tasks DBLP:conf/emnlp/KhashabiMKSTCH20, we propose to convert various
commonsense reasoning tasks into a unified QA format such that we can easily
and fairly evaluate the generalization ability of learned commonsense
reasoning models.

Combining these two lines of effort, we propose a new commonsense inference
evaluation benchmark Knowledge-based Commonsense Inference with QA (CIKQA). An
example is shown in Figure 1. We equip several popular commonsense reasoning
tasks with the supporting knowledge from existing commonsense knowledge graphs
and convert them into a unified QA format. We leverage human annotation to
label whether the provided knowledge is enough to answer the question. With
CIKQA, we are interested in answering three questions: (1) Whether current
models can distinguish the knowledge is gold or not; (2) Whether current
models can learn to conduct inference over provided knowledge; (3) Can current
commonsense inference models generalize across different commonsense reasoning
tasks.

Experiments with several recent knowledge-based commonsense reasoning models
and a proposed baseline JointI, which jointly encodes the knowledge and
question with a single model, show that even though inference over commonsense
knowledge is challenging, models can learn to conduct simple inference after
training with a few examples and better answer the questions than not using
the knowledge. As a comparison, learning to distinguish gold knowledge Yanai:
what is gold knowledge? is still a more challenging task. Last but not least,
even though current models demonstrate the encouraging generalization ability
across the three tasks we consider, they still cannot learn complex inference
(i.e., compare multiple paths) very well. Yanai: what’s missing then? why is
our method good? We hope that our benchmark could motivate more advanced
commonsense inference methods in the future.

##  2 Related Work

To help machines understand commonsense, the community has devoted great
efforts in constructing commonsense knowledge bases with either crowdsourcing
(e.g., ConceptNet liu2004conceptnet and ATOMIC sap2019atomic) or information
extraction techniques (e.g., ASER zhang2019aser). Typically, crowd-sourced
knowledge bases are of higher quality, and the auto-constructed ones have
larger coverage. Besides acquiring commonsense knowledge, the community also
developed many commonsense reasoning datasets to train and test models’
commonsense reasoning abilities. Even though these datasets may have different
formats (e.g., slot fitting in Winogrande sakaguchi2019winogrande and question
answering in CommonsenseQA DBLP:conf/naacl/TalmorHLB19), knowledge types
(e.g., causal commonsense in COPA DBLP:conf/aaaiss/RoemmeleBG11 and numerical
commonsense in NumerSense DBLP:conf/emnlp/LinLKR20), or modalities (e.g,
visual commonsense in VCR DBLP:conf/cvpr/ZellersBFC19 and textual commonsense
in many others), they follow a standard supervised learning setting, and aim
at helping machines to solve a specific commonsense task in an end-to-end
manner. Given this setting, it is often difficult to tell what has been
learned during the training process. Was it used to acquire commonsense
knowledge, learn to conduct commonsense inference, or both? Such ambiguity
limits our progress in solving these commonsense reasoning tasks. In this
work, we connect the efforts on commonsense acquisition and inference by
creating a commonsense inference benchmark CIKQA , where the models can focus
on learning to perform inference over the supporting commonsense knowledge
graph (KG). Yanai: the commonsense knowledge graph was not introduced until
this point. this is confusing

Answering questions in natural language based on a knowledge base (KB) is a
mature research topic in the NLP community, which is also known as the KBQA
problem clark1999knowledge; DBLP:conf/acl/YihCHG15; DBLP:conf/acl/YihRMCS16;
DBLP:conf/esws/UsbeckNHKRN17; DBLP:journals/pvldb/CuiXWSHW17. Previous works
mainly focus on factual knowledge, which is stored in the format of triplets,
and the main challenge is to parse the question and then precisely and
effectively identify the correct path over a large-scale KB to do the
inference. Compared with inference over factual knowledge, inference over
commonsense knowledge brings the following unique challenges: (1) Commonsense
is a kind of preference rather than fixed knowledge, which typically involves
the comparison of several candidates Yanai: this is not clear. As a result,
the ideal commonsense reasoning process could involve the comparison of
multiple paths Yanai: not clear. what are paths? it shouldn’t be related to
the graph yet, and when you relate it to the graph, make it explicit.; (2)
Commonsense is about daily events or objects rather than named entities Yanai:
although it can also be, with famous entities, and thus it is difficult to
find an exact node from the commonsense KB that matches the question and we
may need to conduct inference based on the partial match (i.e., the extracted
nodes are relevant but not identical). Yanai: again, what is the relation to
nodes? I think in general you have in the mind a graph, but you’re not
explicitly connecting it to that. A KB can also be a table. This should be
discussed.

##  3 Task Formulation

In CIKQA, to encourage a generalizable commonsense inference model, we equip
each question with a supporting knowledge graph $G$, and we follow previous
works DBLP:conf/emnlp/KhashabiMKSTCH20; DBLP:journals/corr/abs-2010-04829;
DBLP:conf/acl/WuWYWL20; DBLP:conf/emnlp/DuC20 to unify all selected tasks as a
binary question answering problem ($Q$, $A_{1}$, $A_{2}$). As the auto-
extracted knowledge could contain noise or may not cover all the essential
knowledge for answering the question and humans are capable of saying “I do
not know” when they do not know how to answer a question, we leverage human
annotators to annotate whether the knowledge is gold (i.e., accurate and
enough) for answering the question and test whether current models have the
same commonsense reasoning capability of distinguishing the gold knowledge as
humans. Details about task selection, format unification, support knowledge
extraction, and annotation are as follows.

###  3.1 Task Selection and format Unification

Yanai: the choice of the task is not part of the task formulation. I think we
should either move it from this section, or change the section’s title In
CIKQA, we select the following four popular commonsense reasoning tasks:

  1. 1.

HardPCR: The hard pronoun coreference resolution (HardPCR) task is one of the
most famous commonsense reasoning tasks. For each question, a target pronoun
and two candidate mentions are provided, and the task is to select the correct
mention that the pronoun refers to. Careful expert annotations are conducted
to get rid of the influence of all simple linguistic rules and ask the models
to solve the problem with commonsense reasoning. In CIKQA, we include
instances from WSC levesque2012winograd, DPR DBLP:conf/emnlp/RahmanN12, and
WinoGrande DBLP:conf/aaai/SakaguchiBBC20. To create a question regarding the
target pronoun, we first find the sentence that contains the target pronoun
and then determine whether the pronoun refers to a person or an object. If it
is a person, we will ask who participates. Otherwise, we will ask what
participates. Yanai: why is it called ”hard”?

  2. 2.

CommonsenseQA DBLP:conf/naacl/TalmorHLB19: CommonsenseQA is a commonsense
question answering dataset. For each question-answer pair, four relevant but
wrong concepts are used as the other candidates, and the models are required
to select the correct one out of five candidates. In CIKQA, we randomly sample
a negative answer to make it a binary choice task, which is consistent with
other datasets.

  3. 3.

COPA DBLP:conf/aaaiss/RoemmeleBG11: COPA focuses on evaluating the
understanding of events causality. For each head event, Yanai: what is a head
event? two candidate tail events are provided, and models are asked to predict
the one caused by or the reason for the head event.

  4. 4.

ATOMIC sap2019atomic: The last one is the commonsense knowledge base
completion. Given a head concept (e.g., “eat food”) and a relation (e.g.,
“cause”), we want to predict the tail concept. In CIKQA, we focus on
predicting edges of ATOMIC.

In COPA and ATOMIC, where the task is to predict the relations between two
events or states (e.g., “PersonX eats”-Causes-“PersonX is full”), for each
edge, we randomly sample another event or state (e.g., “PersonX is hungry”) as
the negative tail and ask the model to select the correct one Yanai: what’s an
edge here?. To make the task challenging and avoid sampling irrelevant events
or states, we require the sampled negative event or state to be connected with
the head event or state with a different edge. Yanai: it’s not very clear.
what does it mean to be connected? an example should help clarifying For each
type of relation, we write a simple pattern to generate the question. For
example, for the “Causes” relation, we will ask “What can be caused by
‘PersonX is hungry’?”. Yanai: why hungry? Examples of instances in the
original datasets and their transformed questions and candidate answers are
presented in Appendix Section C.

###  3.2 Supporting Knowledge Extraction

As mentioned in Section 1, a limitation of existing commonsense reasoning
benchmarks is that there is no clear boundary between knowledge and inference.
As such, it is unclear what was learned from the training data, the knowledge,
or how to perform inference. To address this issue and encourage models to
learn inference rather than knowledge from the training data, we propose to
equip each question with supporting knowledge. The question is selected as
part of the dataset only if we find a supporting knowledge for to answer the
question. Since the commonsense knowledge space is huge, it is not likely to
learn it all from current datasets. As such, we wish to present the model with
the relevant knowledge, and focus on the inference. Note that this procedure
serves as a better evaluation setup, and not as a solution to commonsense
reasoning. This section introduces the selected commonsense knowledge graphs
and then introduces how we extract the corresponding commonsense knowledge for
each question.

####  3.2.1 Commonsense KG Selection

Many commonsense knowledge graphs were developed to enhance machines’
commonsense reasoning abilities, including ConceptNet liu2004conceptnet,
ATOMIC sap2019atomic, GLUCOSE mostafazadeh-etal-2020-glucose, and ASER
zhang2019aser. Among these four, ConceptNet, ATOMIC, and GLUCOSE were
constructed via crowd-sourcing while ASER was constructed automatically with
information extraction techniques. Besides ATOMIC, which is used as one of the
tasks, we use the other KBs as supporting knowledge resources.

####  3.2.2 Supporting Graph Extraction

Here we introduce how to extract the supporting knowledge from external
commonsense knowledge bases. For each question, we need to obtain a sub-graph
from supporting knowledge graphs such that it contains the relevant
commonsense knowledge about the question. The sub-graph extraction process
includes the following three steps: (1) Pre-processing: Convert each question
into several key sentences; (2) Matching: Match the sentences into nodes in
the KG; (3) Extraction: Retrieve the supporting sub-graph from the entire
graph.

Data Pre-processing: For each question and the associated candidate answers,
we first replace the question words (e.g., “What”) with the two candidate
answers such that it becomes two declarative sentences. For instance, if the
question is “The fish ate the worm. It was hungry. Who is hungry?” and the
candidates are “Fish” and “Worm,” we will convert the question into the
declarative sentence: “The fish is hungry” and “The worm is hungry.” As a
result, we will get three sentences for this question: “The fish ate the
worm,” “The fish is hungry,” and “The worm is hungry.”

Task Name | # Instance by Knowledge Resource | # Total Instance | # Instance with Gold Knowledge
---|---|---|---
ASER | ConceptNet | GLUCOSE
HardPCR | 2,030 | 202 | 2,143 | 4,375 | 670
CommonsenseQA | 530 | 31 | 37 | 598 | 59
COPA | 103 | 41 | 149 | 293 | 78
ATOMIC | 5,655 | 212 | 3,466 | 9,333 | 2,200
Total | 8,318 | 486 | 5,795 | 14,599 | 3,007
Table 1: CIKQA statistics. We report the number of instances supported by
different knowledge resources and annotated high quality (i.e., Accurate and
Enough) knowledge.

KG Matching: After getting the declarative sentences that contain the question
and key answers, to extract the relevant knowledge, we map them to nodes in
knowledge graphs. Considering that each sentence may have multiple words and
it is often hard to find an exact match, we adopt an embedding-based matching
technique. For each sentence and node in the KG, we treat them as a sentence
and get the corresponding representations with SimCSE DBLP:conf/emnlp/GaoYC21.
For each input sentence, SimCSE encodes the sentence in a vector. A close
distance between two vectors indicates that the two sentences are similar to
each other. We use cosine similarity on the obtained representations to
measure the similarity between two sentences.111We also tried other techniques
such as string match, ROUGE lin2004rouge, and BLEURT DBLP:conf/acl/SellamDP20,
but found them to be either inaccurate or too slow for our scale. Since there
are 287 thousand nodes in GLUCOSE and 194 million nodes in ASER, it is
computationally infeasible to compute the cosine similarity between sentences
pair by pair. Thus we use an approximation. For each extracted sentence, we
first apply Faiss DBLP:journals/corr/JohnsonDJ17, a large-scale similarity-
based matching algorithm that first clusters all KG nodes in the vector space
to increase the matching efficiency when finding the top $N$ nodes in the KG.
We encode all the nodes of the graph and index them using Faiss
DBLP:journals/corr/JohnsonDJ17. Then, we can perform fast and quick retrieval
of the most-similar nodes with each query sentence. Yanai: that’s my
altenative to the previous sentence. make sure it is still correct Yanai: how
are the nodes encoded? it’s an important detail that should be included After
that, we sort the $N$ nodes based on the cosine similarity to find the top $K$
similar nodes. We set $N$ and $K$ to be 60 and 1, respectively. On average, it
takes 25 seconds to retrieve the relevant nodes for each question.

Graph Extraction: In the next step, we construct the sub-graph. We denote the
extracted $m$ nodes as $n_{1},n_{2},...,n_{m}$, and for each of them, we find
$K$ similar nodes from KG. The resulting matched node sets are denoted as
${\mathcal{N}}_{1},{\mathcal{N}}_{2},...,{\mathcal{N}}_{m}$. For any pair of
eventualities $n\in{\mathcal{N}}_{i}$ and $n^{\prime}\in{\mathcal{N}}_{j}$
($i\neq j$), if there exist a path in the KG between $n$ and $n^{\prime}$, we
will keep that path. After merging all paths together, we will get the final
sub-graph. On average, it takes less than two seconds to construct a graph for
each question. Yanai: what does it mean to merge a graph? Yanai: overall, it’s
somewhat hard to read this. It seems like there are many details, and they are
not described in the clearest way currently.

Knowledge Quality Annotation: Since our extraction method is done
automatically, some of the subgraphs may be irrelevant for answer the
questions. As such, we use crowdsourcing to filter out bad extractions. Each
question is inspected by 5 annotators to indicate if @@@. Overall, the average
Inter-annotator agreement (Cohen’s kappa statistic) is 0.83, which indicates
the high-quality of our annotation. We only keep examples where at least four
annotators marked the subgraph is relevant. More annotation details could be
found in Appendix Section A. Yanai: above is my alternative for this paragraph
- also check the @@@, which needs to be completed We annotate whether the
extracted knowledge is accurate and enough. For each question, we invite five
annotators to provide the annotation. The average Inter-annotator agreement
(Cohen’s kappa statistic) is 0.83, which indicates the high-quality of our
annotation. In the end, we apply a strict standard (at least four of five
annotators need to vote for gold) to select the gold knowledge. More
annotation details could be found in Appendix Section A.

###  3.3 CIKQA Statistics

We report the dataset statistics in Table 1. In total, we collect 14,599
instances, and among which Hard PCR and ATOMIC provide the most questions
because their original datasets are much larger than others. According to the
annotation, 16.69% of the supporting knowledge graphs are gold knowledge.
Yanai: what is gold knowledge? Based on our analysis, annotators hold a very
strict standard for selecting the gold knowledge. For each task, we randomly
split the dataset into training, development, and testing set with a standard
8:1:1 splitting. As a result, we get 11,678 training, 1,459 development, and
1,462 testing instances. More detailed statistics, and examples of CIKQA are
presented in Appendix Section B and C, respectively.

##  4 Experiments

In this section, we present the performance of current commonsense inference
models on CIKQA. Besides JointI, we also show the performance of the following
baseline methods:

(1) Vanilla LM: We use the language model (LM) based multiple-choice (MC)
model as the basic baseline. For each candidate answer, we concatenate it with
the question and feed it to the model. After getting the sentence
representation, a linear layer is used to obtain a score and trained with a
cross-entropy loss.

(2) KagNet: As one of the pioneering works that utilized structured knowledge
for solving commonsense reasoning tasks, KagNet DBLP:conf/emnlp/LinCCR19 first
uses a graph convolution network to encode the knowledge graph and then apply
an LSTM based hierarchical attention mechanism to encode the knowledge path
that starts with the concepts corresponding to the question and end with
concepts corresponding to the answer. At the same time, KagNet encodes the
question and answers with pre-trained LMs. In the end, it concatenates all
representations for the final prediction. Yanai: what are these concepts?

(3) Graph Based Reasoning (GBR): Instead of only encoding paths starting with
the question concepts and ending with answer concepts, in GBR
DBLP:conf/aaai/LvGXTDGSJCH20, they proposes to run a depth-first algorithm
over the knowledge graph to generate a sequence of paths as the supporting
knowledge paths.

(4) Multi-Head Knowledge Attention (MHKA): To further utilize the knowledge,
MHKA DBLP:conf/emnlp/PaulF20 uses a transformer network to model the paths
from the question concepts and answer concepts, then concatenates the
knowledge and context representation for the final prediction.

##### Implementation Details

We implement all experiments with Huggingface
DBLP:journals/corr/abs-1910-03771. We select BERT-base
DBLP:conf/naacl/DevlinCLT19 as the base language model for all models. The
batch size is set to be 16. All models are trained for 10,000 steps222All
models converge at 10,000 steps., and the best-performing checkpoints on the
dev set are evaluated. For our model, we set both the number of random walk
paths and walk length to be five. Considering that the auto-extracted
knowledge could contain noise or miss certain knowledge, we add a “gold
knowledge” setting, where only examples with the gold knowledge are used for
training and testing, for all models as the upper bound of their model. All
other hyper-parameters are the same as the base language model. All models are
trained with GTX 2080 and the average running time is 12 hours. Yanai: this
can be moved to the appendix if there’s not enough space

###  4.1 Results

Figure 2: Learning curves of all evaluated models. Models with the “gold”
suffix are evaluated on the gold subset of CIKQA, where only instances with
gold knowledge are used for training and testing. We cannot directly compare
them with other models, but they could serve as a good signal for the upper-
bound of these models when we have a perfect commonsense knowledge base.

For each model, we train it with different numbers of training instances and
report the average performance and standard deviation333Due to the space
limitation, we put the detailed experimental results in Appendix Section D. of
five trails in Figure 2, from which we observe that when explicitly including
the knowledge, all inference models outperform the baseline model without
knowledge, especially JointI. Yanai: what are trails? Yanai: how can we see
that the knowledge helps? is it just in fig 3? it’s not clear. are these the
’gold’ lines? if so, I think we should change the name When the auto-extracted
knowledge and gold knowledge are provided, JointI outperforms the baseline
Vanilla LM model by 4.17 and 15.34, respectively. It supports our assumption
that it is hard to learn all knowledge from the limited training data and
external structured knowledge could help. Moreover, we also notice that when
the knowledge is provided, JointI could learn to answer the questions with
only a small number of examples. This suggests that if we only want to learn
to perform inference over commonsense, we may only need a few training
examples. Besides that, the comparison between auto-extracted knowledge and
gold knowledge also shows that current commonsense knowledge base construction
and retrieval methods are still not optimal and we may need to devote more
effort to these two directions in the future. Last but not least, we can see
that JointI outperforms other inference models among most settings, which
shows that jointly encoding question and knowledge is not just more efficient
but also a more effective strategy than acquiring them separately, and could
serve as a stronger baseline for future works. Due to the simplicity and
efficiency of JointI, we will conduct the rest analysis experiments with
JointI.

###  4.2 Distinguishing the Gold Knowledge

Figure 3: The learning curve of JointI on the gold knowledge identification
task.

Humans have the capability of saying “I do not know” when they find out that
they cannot answer a question with their knowledge. To investigate whether
current deep models have a similar capability, we use JointI as an example to
test whether these deep models can distinguish the gold knowledge. For each
(question, answer, and knowledge) triplet, we train and test JointI with
annotated knowledge quality label. To address the imbalanced distribution
problem, we randomly select the same number of “Not Gold” examples as the
“Gold” ones to make the dataset balanced. From the results in Figure 3, we can
see that the performance of JointI could be improved slightly with the
increase of training data. However, after seeing thousands of examples, it
still can only achieve 0.65 accuracy on a binary classification problem. It
shows that knowing when to say “I do not know” is still a challenging task for
current deep models.

Yanai: I don’t understand this. What is the ”knowledge quality label”? what is
the purpose of this experiment? what is the exact task? Why is it relevant
that models cannot say IDK in this context?

##  5 Generalization Ability

Yanai: Why do we need to experiment with the entire data and not just the
gold? Yanai: it’s hard to compare the tables (and that’s the important part).
Maybe instead of having the gold/full data in the same table, we can have with
and w/o knowledge in 1 table, one for each setup (gold/full data). it’ll be
much easier to compare the results this way

An important assumption and motivation behind CIKQA is that even though the
commonsense could be enormous, the inference rules over commonsense knowledge
should be limited. As a result, even though we could not learn all the
commonsense from limited training data, we can learn how to conduct inference
with several tasks and then generalize to others. In this section, we conduct
experiments with both the “Without Knowledge” and “With Knowledge” models to
show that with our unified formulation, we can gain such generalization
ability across different tasks. To clearly show the effect of the supporting
commonsense KB, we conduct experiments on two settings: (1) Gold Subset: We
only train and test the model on questions, where the supporting graph is
annotated as gold; (2) Full Set: We train and test the model with the whole
dataset. We train the model with questions from a specific task and test it on
all tasks. The results are presented in Table
LABEL:tab:Generalization_ability.

From the results, we can see that the knowledge can help models to generalize
well among CommonsenseQA, COPA, and ATOMIC. The only exception is HardPCR.
This is mainly because the inference needed for solving HardPCR is more
complex than the other tasks, where we do not only need to find the relevant
knowledge but also need to replace the target pronoun with the entity in the
provided knowledge. How to train a model that can learn to conduct such
complex reasoning is a problem worth exploring in the future.

In general, the observed generalization ability is encouraging because if we
can learn a good model on CIKQA, based on the assumption that there exists
limited types of inference, potentially we can solve any commonsense reasoning
tasks as long as the needed inference type is covered by CIKQA. At the same
time, we also notice that current models still cannot learn complex inference
(i.e., compare multiple paths) with few examples, and we leave how to solve
that problem as the future work.

##  6 Conclusion

In this paper, we present CIKQA, a unified commonsense inference benchmark.
Specifically, we first convert several popular commonsense tasks into a
unified QA format and equip each question with a supporting commonsense
knowledge graph. During training on CIKQA, models can thus focus on learning
to perform inference, rather than learn commmonsense knowledge. Experiments
show that models can better learn how to do commonsense inference with a few
examples and significantly outperform the baseline method that does not use
structured knowledge in the data-scarce setting. More interestingly, with our
unified formulation, models demonstrate the encouraging generalization ability
across tasks. As both the format unification and supporting graph extraction
are automatic, we can easily extend to other commonsense reasoning tasks in
the future. All used code and data are submitted as part of the appendix.

## Acknowledgements

The authors of this paper were supported by the Office of the Director of
National Intelligence (ODNI), Intelligence Advanced Research Projects Activity
(IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program, and
by contract FA8750-19-2-1004 with the US Defense Advanced Research Projects
Agency (DARPA). The views expressed are those of the authors and do not
reflect the official policy or position of the Department of Defense or the
U.S. Government. This paper was also supported by the NSFC Fund (U20B2053)
from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520)
from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong with
special thanks to HKMAAC and CUSBLT, and the Jiangsu Province Science and
Technology Collaboration Fund (BZ2021065). Yanai Elazar is grateful to be
supported by the PBC fellowship for outstanding PhD candidates in Data Science
and the Google PhD fellowship.

##  Appendix A Annotation Details

Figure 4: An example of the used survey.

The annotation goal is to determine whether the supporting graph can help
answer the question or not. Thus, for each QA pair, we present the question,
candidate answers, and the supporting sub-graph to annotators444All
annotations follow the ethical guidelines., and then ask them two questions:
(1) What is the correct answer for this question; (2) Whether the provided
commonsense knowledge contains all the essential commonsense for answering
this question. The purpose of the first question is to assess the annotation
quality. A survey example is shown in Figure 4. In beginning of each survey,
we also provide detailed instructions and examples to help annotators
understand our task. We employ annotators from Amazon Mechanical Turk to
provide annotations. To improve the annotation quality, we require the
annotators to be English native speaker and to have an overall acceptance rate
above 90%. For each survey, we invite five annotators to provide the
annotations and pay them $0.1. The average Inter-annotator agreement (Cohen’s
kappa statistic) for Q1 and Q2 are 0.87 and 0.83, respectively. The annotation
results show that humans could provide consistent annotation about whether the
knowledge could be used to answer the questions.

Task Name |  Original Assertion |  Transformed Question |  Answer
---|---|---|---
HardPCR |  The fish ate the worm. It was hungry. |  The fish ate the worm. It was hungry. What was hungry? |  (A) Fish; (B) Worm
CommonsenesQA |  What is a place that someone can go buy a teddy bear? |  What is a place that someone can go buy a teddy bear? |  (A) Toy store; (B) Shelf
COPA |  I drank from the water fountain. |  I drank from the water fountain. What was the cause of this? |  (A) I was thirsty.; (B) I felt nauseous.
ATOMIC |  PersonX buys the bike. |  Before PersonX buys the bike, what did PersonX want? |  (A) To be social.; (B) To have transportation.
Table 3: Demonstration of the original assertion, transformed questions, and
answers. Correct and wrong answers are indicated with blue and red,
respectively.

##  Appendix B Statistics

We report the number of questions that a supported graph can be find, the
average size of supporting graph, and the number of helpful instances of CIKQA
in Table 4. In total, we collect 14,599 instances with the average supported
graph size of 2.75.

Task Name | # Instances | Avg Sub-graph Size (# Edges) | # Helpful Instances
---|---|---|---
Hard PCR | 4,375 | 2.85 | 670
CommonsenseQA | 598 | 3.19 | 59
COPA | 293 | 3.03 | 78
ATOMIC | 9,333 | 2.67 | 2200
Total | 14,599 | 2.75 | 3,007
Table 4: Detailed CIKQA dataset statistics.

##  Appendix C Case Study

Figure 5: CIKQA Case Study. Mapped sentences for the question and answers are
indicated with blue and pink. Other eventualities are white. Edge weights are
in brackets. We only show the relevant part of the graph for the clear
representation. All extracted eventualities are lemmatized, we recover them
for the ease of understanding.

Demonstration of how we convert the original dataset into the unified format
is presented in Table 3. For each task, we use a template to automatically
convert it into the unified QA format. Besides that, we also present several
questions along with knowledge in Figure 5. From the example we can see that,
the reasoning over HardPCR is more challenging than other tasks. In the
HardPCR example, two paths can be found relevant to question: (1) “I am
drunk”$\rightarrow$Co_Occurrence$\rightarrow$“I hit someone”; (2) “I am
drunk”$\rightarrow$Co_Occurrence$\rightarrow$“That is not
fair”$\rightarrow$Co_Occurrence$\rightarrow$“You kick me”. For the correct
inference, we need to know when there is a conflict, we should trust the one-
hop inference more because the additional node in the two-hop path may
introduce extra noise. As a comparison, for other tasks, the main inference we
need is to find the relevant paths, which is relatively easy.

##  Appendix D Detailed Experimental Results

Detailed experimental results are presented in Table 5.

Model | Number of Training Instances
---|---
5 | 10 | 100 | 500 | 1,000 | 5,000 | 11,678
Chance Performance | 50.00 (0.00) | 50.00 (0.00) | 50.00 (0.00) | 50.00 (0.00) | 50.00 (0.00) | 50.00 (0.00) | 50.00 (0.00)
Vanilla LM | 51.16 (1.92) | 55.88 (2.41) | 56.52 (2.37) | 63.67 (2.19) | 66.76 (1.37) | 70.04 (0.58) | 70.11 (0.28)
KagNet DBLP:conf/emnlp/LinCCR19 | 53.29 (2.16) | 55.47 (2.74) | 59.92 (3.05) | 61.97 (1.19) | 65.90 (1.54) | 68.90 (1.21) | 71.50 (1.29)
GBR DBLP:conf/aaai/LvGXTDGSJCH20 | 51.77 (1.75) | 56.57 (3.13) | 59.92 (2.34) | 63.36 (1.62) | 68.06 (0.35) | 67.10 (0.17) | 71.34 (0.31)
MHKA DBLP:conf/emnlp/PaulF20 | 54.89 (2.34) | 60.47 (1.13) | 61.70 (0.41) | 63.82 (0.78) | 67.85 (0.32) | 69.29 (1.58) | 71.30 (1.14)
JointI(Our Model) |  57.25 (0.21) |  62.41 (0.97) |  64.02 (0.99) |  68.54 (0.47) |  71.55 (0.75) |  72.36 (0.56) |  74.28 (0.21)
KagNet-gold | 55.21 (3.21) | 64.36 (0.83) | 68.65 (1.64) | 74.28 (1.31) | 79.05 (0.57) | 80.21 (0.84) | 80.20 (0.21)
GBR-gold | 50.53 (1.62) | 66.34 (1.82) | 69.31 (1.33) | 72.94 (0.35) | 76.24 (0.21) | 80.86 (0.21) | 78.85 (0.13)
MHKA-gold | 58.35 (2.67) | 78.54 (1.32) | 78.55 (0.72) | 79.23 (0.64) | 80.53 (0.50) | 80.52 (0.52) | 81.85 (0.15)
JointI-gold |  61.39 (2.56) |  80.85 (1.35) |  82.18 (0.33) |  82.51 (0.50) |  84.32 (0.42) |  85.81 (0.45) |  85.48 (0.17)
Table 5: Demonstration of different models with different training instances.
We report the average performance of five different random seeds and standard
deviation (in brackets). “-gold” indicates that the models are trained and
tested with instances with gold knowledge. We cannot directly compare them
with the normal setting, but it could serve as the upper-bound for our
learning paradigm. Best performing models under both settings are indicated
with the bold font.