# Improved Upper Limits on Gravitational Wave Emission
from NS 1987A in SNR 1987A

Benjamin J. Owen  Department of Physics and Astronomy, Texas Tech University,
Lubbock, Texas 79409-1051, USA  Lee Lindblom  Center for Astrophysics and
Space Sciences, University of California at San Diego, La Jolla, California
92093-0424  Luciano Soares Pinheiro  Department of Physics and Astronomy,
Texas Tech University, Lubbock, Texas 79409-1051, USA  Binod Rajbhandari
Department of Mathematics, Texas Tech University, Lubbock, Texas 79409-1042,
USA  Department of Physics and Astronomy, Texas Tech University, Lubbock,
Texas 79409-1051, USA  School of Mathematical Sciences and Center for
Computational Relativity and Gravitation, Rochester Institute of Technology,
Rochester, NY 14623, USA

###### Abstract

We report on a new search for continuous gravitational waves from NS 1987A,
the neutron star born in SN 1987A, using open data from Advanced LIGO and
Virgo’s third observing run (O3). The search covered frequencies from 35–1050
Hz, more than five times the band of the only previous gravitational wave
search to constrain NS 1987A [B. J. Owen et al., ApJL 935, L7 (2022)]. It used
an improved code and coherently integrated from 5.10 days to 14.85 days
depending on frequency. No astrophysical signals were detected. By expanding
the frequency range and using O3 data, this search improved on strain upper
limits from the previous search and was sensitive at the highest frequencies
to ellipticities of $1.6\times 10^{-5}$ and $r$-mode amplitudes of $4.4\times
10^{-4}$, both an order of magnitude improvement over the previous search and
both well within the range of theoretical predictions.

##  1 Introduction

Piran & Nakamura (1988) first suggested shortly after SN 1987A that the
neutron star (NS 1987A) probably born in the supernova could be emitting
detectable continuous gravitational waves. Yet the first search to constrain
the behavior of the neutron star through GW upper limits was not performed
until recently (Owen et al., 2022). That search used open data from Advanced
LIGO and Virgo’s second observing run (O2), covered the frequency band 75–275
Hz, and was sensitive in that band to GW signals just below an analog of the
pulsar spin-down limit based on the age of the neutron star (Wette et al.,
2008). Here we describe a search of open data (Abbott et al., 2023) from
Advanced LIGO and Virgo’s third observing run (O3) (Tse et al., 2019; Acernese
et al., 2019) using a new and improved code covering a wider frequency band
(35–1050 Hz).

The detection of neutrinos from SN 1987A (Bionta et al., 1987; Hirata et al.,
1987) suggests that a neutron star, rather than a black hole, was the most
likely product of this event. Located 51.4 kpc away in the Large Magellanic
Cloud (Panagia, 1999), NS 1987A is the youngest neutron star in our galactic
neighborhood. Electromagnetic searches for a pulsar or non-pulsing neutron
star in the remnant SNR 1987A are made difficult by its dust-filled
surroundings. Far infrared observations of SNR 1987A by Cigan et al. (2019),
however, have detected a relatively warm, compact region of dust that could be
powered by a very young cooling neutron star (Page et al., 2020; Dohi et al.,
2023). Greco et al. (2021) and Greco et al. (2022) argue that hard X-ray
emission suggests the presence of a pulsar wind nebula.

The first searches for continuous GW emission from NS 1987A used stochastic
background methods to analyze data from Advanced LIGO and Virgo’s observing
runs—see Abbott et al. (2021a) and references therein. These searches did not
cover a reasonable parameter space. They assumed spin-down rates for NS 1987A
of order $10^{-9}$ Hz/s, which is large by the standards of known pulsars but
is at least an order of magnitude smaller than the spin-down that would be
caused by GW emission at detectable levels (Owen et al., 2022). Stochastic
background searches also did not achieve the needed sensitivity to detect GWs
at the level of the indirect upper limit on GW strain $h_{0}^{\mathrm{age}},$
analogous to the spindown limit for pulsars but based on the age of the object
when pulses are not observed.

This age-based indirect limit was defined by Wette et al. (2008), who
described the basic continuous-wave method for searches for persistent GWs
from supernova remnants where there is evidence for a neutron star but where
pulses are not observed (such as SNR 1987A). Such searches require that wide
bands of frequencies and spin-down parameters (time derivatives of the
frequency) be explored. These continuous GW searches use longer signal
coherence times than stochastic background searches, and therefore generally
require searching over spin-down parameters as well as GW frequencies. The
limit $h_{0}^{\mathrm{age}}$ is a useful figure of merit for search
sensitivity. The method of Wette et al. (2008) has been used to search for
many objects, starting with the central compact object in supernova remnant
Cas A (Abadie et al., 2010) and more recently targeting NS 1987A (Owen et al.,
2022).

Until Owen et al. (2022), searches for NS 1987A did not cover the full
parameter space or reach the sensitivity of $h_{0}^{\mathrm{age}}.$ Narrow
parameter space searches for continuous GWs from NS 1987A were performed by
Sun et al. (2016) using the method described by Chung et al. (2011). Since the
Wette et al. (2008) parameter space required a fourth spin-down parameter for
NS 1987A (then 19 years old), Chung et al. (2011) narrowed the search by
introducing a detailed spin-down model. This is less robust than models making
fewer assumptions. Even with a narrow parameter space, Sun et al. (2016) did
not achieve upper limits comparable to $h_{0}^{\mathrm{age}}.$ Recent all-sky
surveys for continuous GWs such as Dergachev & Papa (2023) do beat that limit
in the direction of NS 1987A, but do not cover spin-down ranges physically
consistent with NS 1987A.

Wette et al. (2008) derived $h_{0}^{\mathrm{age}}$ for mass-quadrupole GW
emission and Owen (2010) extended it to current-quadrupole GW emission from
$r$-mode oscillations [see Glampedakis & Gualtieri (2018) for a summary of
emission mechanisms]. The derivations of these limits assume that GWs dominate
the spindown of the star from birth and that the initial spin frequency was
much higher than present. The Wette et al. (2008) mass-quadrupole age limit on
the GW amplitude $h_{0}$ [a measure called the intrinsic strain (Jaranowski et
al., 1998)] can be written as a frequency-independent expression that depends
on the age of the neutron star $a$, its distance $D$, and moment of inertia
$I$:

$\displaystyle h_{0}^{\mathrm{age}}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{5}{8}}\sqrt{\frac{GI}{c^{3}aD^{2}}}.$ (1)

The analogous indirect limit for GW emission from $r$-modes is given by (Owen,
2010)

$\displaystyle h_{0}^{\mathrm{age}}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{10}{A^{2}(n-1)}}\sqrt{\frac{GI}{c^{3}aD^{2}}},$ (2)

where $n=f\ddot{f}/\dot{f}^{2}$ is the braking index. Here we have modified
the expression of Owen (2010) to leave free the parameter $A,$ which is the
ratio of $r$-mode frequency (in an inertial frame) to stellar spin frequency.

We convert the above expressions to numerical ranges as follows. Owen (2010)
used $A=4/3,$ appropriate for slow rotation and Newtonian gravity, while
general relativistic slow rotation estimates are about 1.39–1.64 (Idrisy et
al., 2015; Ghosh et al., 2023). The moment of inertia depends on the neutron
star equation of state and mass. We take the mass range for neutron stars to
be 1.2–2.1 $M_{\odot}$ (Martinez et al., 2015; Cromartie et al., 2020). For
this mass range and the equations of state used by Abbott et al. (2021b), the
moment of inertia range is about $9.1\times 10^{44}$–$4.7\times 10^{45}$ g
cm${}^{2}.$ Finally we use 6–7 for the physically plausible range of $r$-mode
braking indices (Lindblom et al., 1998; Ho & Lai, 2000). The resulting range
of $h_{0}^{\mathrm{age}}$ for NS 1987A, using an age of 33 years (applicable
for late O3), is about

$2.3\times 10^{-25}\leq h_{0}^{\mathrm{age}}\leq 5.3\times 10^{-25}$ (3)

for mass-quadrupole GW emission using Eq. (1) and

$2.3\times 10^{-25}\leq h_{0}^{\mathrm{age}}\leq 6.8\times 10^{-25}$ (4)

for $r$-mode GW emission using Eq. (2).

Inserting these parameters into the results of Wette et al. (2008) and Wette
(2012) indicates that a coherent search of O3 data using only two spin-down
parameters can surpass the sensitivity of $h_{0}^{\mathrm{age}}$ for a
computing budget of order a million core-hours. This paper describes such a
search, which detected no astrophysical signals but placed direct upper limits
on the GW strain from NS 1987A. These limits beat the indirect limit
$h_{0}^{\mathrm{age}}$ over a physically consistent parameter space that is
considerably larger than the range of frequencies explored in the O2 data
search by Owen et al. (2022).

##  2 Search methods

The search methods used in this paper are similar to those used by Owen et al.
(2022). Highlights and changes are summarized here. Readers are directed to
Owen et al. (2022) and references therein for details.

Derived parameters
---
Name | Value (35–125 Hz) | Value (125–450 Hz) | Value (450–1050 Hz)
Span | 14.85 d | 8.13 d | 5.10 d
Start | 2020-02-28 12:39:33 | 2020-02-24 02:56:27 | 2020-02-27 12:34:01
H1 SFTs | 555 | 330 | 216
L1 SFTs | 603 | 333 | 220
V1 SFTs | 539 | 263 | 169
Table 1:  Data parameters used in this search. Times are UTC.

The Drill pipeline (Owen et al., 2023) version 1.0.0 was used for this search.
It will be described more fully elsewhere. Here we summarize the differences
between Drill and previous codes. Drill is a completely new code with
functionality similar to that used in Owen et al. (2022). It consists of
Python scripts running C codes from LALSuite (LIGO Scientific Collaboration,
2020) (v6.25.1 of lalapps and concurrent versions of other packages) that
implement the multi-detector $\mathcal{F}$-statistic (Jaranowski et al., 1998;
Cutler & Schutz, 2005). Drill is more efficient than the code used in Owen et
al. (2022), vetoes signal candidates based on nonparametric statistics, and
handles upper limits more consistently.

This search used open data (Vallisneri et al., 2015; Abbott et al., 2023) from
O3 in the form of 1800 s Short Fourier Transforms generated from Frame-format
time-domain data sampled at 4 kHz. All SFT logic and data selection were
handled by the lalapps_MakeSFTs program. O3 and our search included data from
the Hanford, WA (H1) and Livingston, LA (L1) 4 km LIGO interferometers and the
Cascina, Italy (V1) 3 km Virgo interferometer. Although V1 was generally less
sensitive, we verified that including its data improved search sensitivity at
fixed computational cost. The observation time spans for our search bands were
also set to achieve a fixed computational cost. Following Jaranowski et al.
(1998) the start time of each span was chosen to maximize the data time
divided by the joint power spectral density (psd) of strain noise, which is
approximately equivalent to maximizing the search sensitivity.

For NS 1987A we used the (J2000) right ascension and declination

$\alpha=05^{\mathrm{h}}\,35^{\mathrm{m}}\,27^{\mathrm{s}}.998,\qquad\delta=-69^{\circ}\,16^{\prime}\,11^{\prime\prime}.107$
(5)

from Cigan et al. (2019). We used an age of 33 yr (suitable for the late O3
data used) and the distance 51.4 kpc (Panagia, 1999) to determine the
parameter space and infer source properties. Observation spans and other
search parameters derived from the age are listed in Table 1.

The signal parameter space was chosen similarly to Owen et al. (2022). The GW
frequency in the solar system barycenter frame was modeled by

$f(t)=f+\dot{f}(t-t_{0})+\frac{1}{2}\ddot{f}(t-t_{0})^{2},$ (6)

where $t_{0}$ is the time at the beginning of the span and the parameters
$(f,\dot{f},\ddot{f})$ are evaluated at epoch $t_{0}$. The ranges of
$(\dot{f},\ddot{f})$ for a given value of $f$ were

$\displaystyle\frac{f}{(n_{\max}-1)a}\leq$ $\displaystyle-\dot{f}$
$\displaystyle\leq\frac{f}{(n_{\min}-1)a},$ (7) $\displaystyle
n_{\min}\frac{\dot{f}^{2}}{f}\leq$ $\displaystyle\ddot{f}$ $\displaystyle\leq
n_{\max}\frac{\dot{f}^{2}}{f},$ (8)

with the braking index $n$ ranging from $n_{\min}=3$ to $n_{\max}=7$ and $a$
being the neutron star’s age. Unlike in previous searches, a minimum braking
index of $n_{\min}=3$ was used to keep the highest values of $-\dot{f}$ below
about $5\times 10^{-7}$ Hz/s, where Owen et al. (2022) found that the SFT
length of 1800 s can become problematic. We verified that the other
consistency checks described in Owen et al. (2022) were satisfied. This range
of braking indices is consistent with the minimum spin-down for a given
$h_{0}$ (Owen, 2010),

$\displaystyle-\dot{f}=1.6\times
10^{-8}\mbox{Hz/s}\left(\frac{A}{2}\right)^{2}\left(\frac{D}{\mbox{51.4
kpc}}\right)^{2}$ $\displaystyle\times\left(\frac{h_{0}}{2\times
10^{-25}}\right)^{2}\left(\frac{f}{\mbox{100
Hz}}\right)\left(\frac{10^{45}\,\mbox{g cm}^{2}}{I}\right)$ (9)

and is greater than the maximum value covered by the all-sky surveys such as
Dergachev & Papa (2023). It does not include some of the more extreme range
recently proposed by Morales & Horowitz (2023), which is more appropriate for
older stars.

The frequency band was split into a low frequency band from 35–125 Hz, a
medium frequency band from 125–450 Hz, and a high frequency band from 450–1050
Hz. The boundary at 125 Hz was chosen so that even the fastest spinning young
pulsar (Marshall et al., 1998) emits GWs in the low band. The boundary at 450
Hz was chosen so that the middle band would avoid noise artifacts due to
violin modes of the LIGO test mass suspensions (Covas et al., 2018; Davis et
al., 2021). The overall lower bound of 35 Hz was chosen so that an a priori
estimate of search sensitivity (Wette, 2012) indicated that Eq. (1) would be
achieved. The overall upper bound of 1050 Hz was chosen for the same reason
and because going much higher in frequency causes difficulties with the
analysis such as the spin-down range mentioned above.

This search ran a total of roughly $1.2\times 10^{6}$ core-hours on the Texas
Tech “Nocona” computing cluster, split into batch jobs of about 8 core-hours
each. Integration times and other parameters are shown in Table 1. Each search
job covered the full range of spin-down parameters appropriate for its
frequency band. Widths of these bands ranged from about 1–37 mHz depending on
frequency. Parameter spacings in $\dot{f}$ and $\ddot{f}$ were of order
$10^{-11}$ Hz/s and $10^{-18}$ Hz/s2 respectively. Search jobs contained about
4–$9\times 10^{8}$ templates each, for totals of about 2, 4, and $6\times
10^{13}$ templates for the low, medium, and high frequency bands respectively.

The approach of Drill to vetoing signal candidates does not rely on time-
frequency behavior or known instrumental lines for a priori vetoes, as in Owen
et al. (2022). Instead it outputs a $2\mathcal{F}$ histogram for each search
job without recording any specific candidates on the first pass. This
ameliorates major storage and input/output issues in noisy bands. From each
histogram Drill takes the loudest $2\mathcal{F}$ (to within the binning
resolution of 0.1) and computes (approximately due to the binning) the
discrete Cramér-von Mises statistic (Choulakian et al., 1994). The continuous
version of the Cramér-von Mises statistic can be written

$\omega^{2}=\int
dC^{*}(2\mathcal{F})[C(2\mathcal{F})-C^{*}(2\mathcal{F})]^{2},$ (10)

where $C$ and $C^{*}$ are the observed and expected cumulative distributions
respectively. For the $\mathcal{F}$-statistic, the latter is a $\chi^{2}$ with
four degrees of freedom. Thus $\omega$ emphasizes the middle of the
distribution (where non-Gaussian noise is concentrated) more than the tail
(where a detectable signal will be) and is a good way of checking for “bad”
noise bands without discarding loud signals.

We determined the threshold of $\omega$ empirically. Even for Gaussian
stationary noise, the output of the LALSuite $\mathcal{F}$-statistic code is
not precisely $\chi^{2}$(4) distributed due to approximations used to increase
computational speed. We checked the behavior of $\omega$ in real noise and
simulated Gaussian noise, with and without injected signals. In Gaussian or
nearly Gaussian noise, we found that $\omega$ has a mean of 0.0035 and a
standard deviation of 0.0012. Therefore $\omega=0.02$ corresponds to about
thirteen standard deviations and should not veto clean noise bands even with
our large number of trials. Loud enough signals should cause a high $\omega$
due to many templates triggering at high $2\mathcal{F}$ with slightly wrong
parameters. Through injection studies we found that signals are not spuriously
vetoed until $2\mathcal{F}\sim 10^{6},$ which is orders of magnitude above the
physical limit $h_{0}^{\mathrm{age}}.$

##  3 Search results

We checked the search results for candidate signals as follows. First we
vetoed the entirety of each search job which produced $\omega\geq 0.02,$
totaling about 20 Hz or 2% of the total search band. We calculated the 95%
confidence threshold in idealized Gaussian noise for each wide band
($2\mathcal{F}$ about 75.6, 76.5, and 76.4 respectively) and recorded which
jobs exceeded it (4, 6, and 36 jobs respectively). Then we visually inspected
histograms of the surviving jobs by the same criteria as in Aasi et al.
(2015), looking for fat tails rather than the thin tails indicative of
injected signals. All histograms but one were rejected at a glance, and the
remaining one (with $2\mathcal{F}=79.8$ and $\omega=0.001$) hinted at
abnormality on closer inspection. Also, that job searched around 998.4 Hz, a
frequency known to be contaminated (in LIGO data) by violin modes of the test
mass suspensions (Covas et al., 2018; Davis et al., 2021). Nevertheless, we
followed it up. That search job was rerun, keeping detailed information on all
templates with $2\mathcal{F}\geq 40.$ Plotting $2\mathcal{F}$ versus frequency
showed multiple peaks indicative of noise lines. We then searched the entire
frequency band of that job with double the integration time. A real signal
would produce double the value of $2\mathcal{F},$ or about 160. The double-
length followup search produced several $2\mathcal{F}$ peaks only a little
over 100, and every search job dramatically failed the Cramer-von Mises test
($\omega\sim 0.05$). Therefore we concluded that we found no astrophysical
signals.

We then set upper limits on $h_{0}$ in 1 Hz bands, similar to Owen et al.
(2022). We considered a population of signals with fixed $h_{0}$ and random
values of intrinsic parameters $(f,\dot{f},\ddot{f})$ as well as extrinsic
parameters described in Jaranowski et al. (1998). We estimated what $h_{0}$
would be detected at a rate of 90% with $2\mathcal{F}$ larger than the largest
non-vetoed search result in that upper limit band. A semi-analytic estimate of
$h_{0}$ was checked by software-injecting 1000 signals per 1 Hz upper limit
band. The Drill pipeline does this more efficiently by cutting down on disk
input/output bottlenecks, corrects a minor inconsistency in the way vetoed
bands were incorporated into the upper limits in previous analyses, and
computes upper limits for all bands that were not too heavily vetoed. A
heavily vetoed band is one where eliminating the search jobs exceeding the
$\omega$ threshold vetoes more than 10% of the upper limit band, and therefore
a 90% upper limit is not meaningful. These amounted to 60 upper limit bands
out of the 1015 covered by the search, or about 6%.

Figure 1:  The left panel displays observational 90% confidence upper limits
on $h_{0}$ from NS 1987A in 1 Hz bands as a function of frequency. The (red)
horizontal solid lines show the range of $h_{0}^{\mathrm{age}}$ from mass-
quadrupole GW emission, Eq. (3), and the (green) horizontal dashed lines show
the range for $r$-mode GW emission, Eq. (4). (Note that the
$h_{0}^{\mathrm{age}}$ lower limits coincide.) The right panel shows
corresponding upper limits on the dimensionless neutron-star ellipticity
$\epsilon$ and $r$-mode amplitude $\alpha.$

The left panel of Fig. 1 displays our 90% confidence upper limits on $h_{0}$
as a function of frequency, except in the 60 heavily vetoed bands. The
discontinuities at 125 and 450 Hz are caused by differences in integration
times in the three search bands. The (red) horizontal solid lines in the left
panel of Fig. 1 show the range of $h_{0}^{\mathrm{age}}$ from mass-quadrupole
GW emission, Eq. (3), and the (green) horizontal dashed lines show the range
for $r$-mode GW emission, Eq. (4). Note that the lower red and green lines
coincide. The observed upper limits on $h_{0}$ are less (better) than the
average values of the indirect limits $h_{0}^{\mathrm{age}}$ over the full
frequency band. Our search places limits on GW emission from NS 1987A that are
better than the strictest $h_{0}^{\mathrm{age}}$ estimates over the
astrophysically most interesting part of the frequency band, 50–600 Hz.

The efficiency of our search can be expressed in terms of the sensitivity
depth (Behnke et al., 2015)

$\mathcal{D}=\sqrt{S_{h}/h_{0}},$ (11)

where $S_{h}$ is the harmonic mean strain noise power spectral density. For
our search $\mathcal{D}$ is about 29 Hz${}^{-1/2},$ 22 Hz${}^{-1/2},$ and 18
Hz-1/2 in the low, middle, and high frequency bands respectively. This is
comparable to or somewhat worse than Owen et al. (2022), as one would expect
from the short integration times (Wette, 2023).

Upper limits on $h_{0}$ imply upper limits on the fiducial neutron star
ellipticity $\epsilon$ (Jaranowski et al., 1998)

$\epsilon\simeq 9.5\times 10^{-5}\left(\frac{h_{0}}{1.2\times
10^{-24}}\right)\left(\frac{D}{\mbox{1 kpc}}\right)\left(\frac{\mbox{100
Hz}}{f}\right)^{2},$ (12)

and upper limits on the $r$-mode amplitude $\alpha$ defined by Lindblom et al.
(1998) using Owen (2010),

$\alpha\simeq 0.028\left(\frac{h_{0}}{10^{-24}}\right)\left(\frac{\mbox{100
Hz}}{f}\right)^{3}\left(\frac{D}{\mbox{1 kpc}}\right).$ (13)

The fiducial values in these expressions are uncertain by roughly a factor of
three due to uncertainties in the neutron star mass and equation of state.
Moreover, general relativity complicates these expressions in ways that have
not yet been calculated. These upper limit estimates on $\epsilon$ and
$\alpha$ for NS 1987A from this search are shown in the right panel of Fig 1.

##  4 Conclusions

We have performed a search for continuous GWs from NS 1987A over a much wider
range of frequencies than the only previous physically consistent search (Owen
et al., 2022) using an improved code on improved data. While we did not detect
any astrophysical signal, we set upper limits which constrained the behavior
of NS 1987A over a much broader parameter space than ever before.

When translated into fiducial neutron star ellipticity or $r$-mode amplitude,
our upper limits (at the highest frequencies) are an order of magnitude better
than the highest frequency limits set by Owen et al. (2022). Our best
ellipticities are just above $10^{-5}.$ Such elastic deformations are possible
for quark stars and quark-baryon hybrid stars (Owen, 2005; Johnson-McDaniel &
Owen, 2013), and are skirting the maximum predicted for normal neutron stars
(Morales & Horowitz, 2022). For a magnetic deformation, our strain upper
limits imply a limit on the internal magnetic field of a few times $10^{15}$
or $10^{14}$ G if the protons in the core are or are not superconducting
respectively (Lander, 2014; Ciolfi & Rezzolla, 2013). Our best upper limits on
$r$-mode amplitude are starting to enter the range of theoretical predictions
(Bondarescu et al., 2009).

These comparisons show that our search had a sensitivity to GW emission from
NS 1987A compatible with a variety of theoretical predictions, not just the
most extreme ones as in Owen et al. (2022). We used simple coherent
integrations of relatively short spans of O3 data. Searches of better data
from the current Advanced LIGO and Virgo’s fourth observing run (O4) to Cosmic
Explorer (Evans et al., 2021; Gupta et al., 2023) will further improve on this
sensitivity, especially with more sophisticated data analysis techniques
(Wette, 2023).

This research has made use of data or software obtained from the Gravitational
Wave Open Science Center (gwosc.org), a service of the LIGO Scientific
Collaboration, the Virgo Collaboration, and KAGRA. This material is based upon
work supported by NSF’s LIGO Laboratory which is a major facility fully funded
by the National Science Foundation, as well as the Science and Technology
Facilities Council (STFC) of the United Kingdom, the Max-Planck-Society (MPS),
and the State of Niedersachsen/Germany for support of the construction of
Advanced LIGO and construction and operation of the GEO600 detector.
Additional support for Advanced LIGO was provided by the Australian Research
Council. Virgo is funded, through the European Gravitational Observatory
(EGO), by the French Centre National de Recherche Scientifique (CNRS), the
Italian Istituto Nazionale di Fisica Nucleare (INFN) and the Dutch Nikhef,
with contributions by institutions from Belgium, Germany, Greece, Hungary,
Ireland, Japan, Monaco, Poland, Portugal, Spain. KAGRA is supported by
Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan
Society for the Promotion of Science (JSPS) in Japan; National Research
Foundation (NRF) and Ministry of Science and ICT (MSIT) in Korea; Academia
Sinica (AS) and National Science and Technology Council (NSTC) in Taiwan. This
research was supported in part by NSF grant 2012857 to the University of
California at San Diego, NSF grants 1912625 and 2309305 to Texas Tech
University, and NSF grant 2110460 to the Rochester Institute of Technology.
The authors acknowledge computational resources provided by the High
Performance Computing Center (HPCC) of Texas Tech University at Lubbock
(http://www.depts.ttu.edu/hpcc/). This paper has LIGO Document number
P2300319.

## References

  * Aasi et al. (2015) Aasi, J., et al. 2015, Astrophys. J., 813, 39, doi: 10.3847/1538-4357/ac1f2d
  * Abadie et al. (2010) Abadie, J., et al. 2010, Astrophys. J., 722, 1504, doi: 10.1088/0004-637X/722/2/1504
  * Abbott et al. (2021a) Abbott, R., et al. 2021a, Phys. Rev. D, 104, 022005, doi: 10.1103/PhysRevD.104.022005
  * Abbott et al. (2021b) —. 2021b, Astrophys. J., 922, 71, doi: 10.3847/1538-4357/ac0d52
  * Abbott et al. (2023) —. 2023, Astrophys. J. Suppl., 267, 29, doi: 10.3847/1538-4365/acdc9f
  * Acernese et al. (2019) Acernese, F., et al. 2019, Phys. Rev. Lett., 123, 231108, doi: 10.1103/PhysRevLett.123.231108
  * Behnke et al. (2015) Behnke, B., Papa, M. A., & Prix, R. 2015, Phys. Rev. D, 91, 064007, doi: 10.1103/PhysRevD.91.064007
  * Bionta et al. (1987) Bionta, R. M., et al. 1987, Phys. Rev. Lett., 58, 1494, doi: 10.1103/PhysRevLett.58.1494
  * Bondarescu et al. (2009) Bondarescu, R., Teukolsky, S. A., & Wasserman, I. 2009, Phys. Rev. D, 79, 104003, doi: 10.1103/PhysRevD.79.104003
  * Choulakian et al. (1994) Choulakian, V., Lockhart, R., & Stephens, M. 1994, Canadian Journal of Statistics, 22, 125, doi: https://doi.org/10.2307/3315828
  * Chung et al. (2011) Chung, C., Melatos, A., Krishnan, B., & Whelan, J. T. 2011, Mon. Not. Roy. Astron. Soc., 414, 2650, doi: 10.1111/j.1365-2966.2011.18585.x
  * Cigan et al. (2019) Cigan, P., et al. 2019, Astrophys. J., 886, 51, doi: 10.3847/1538-4357/ab4b46
  * Ciolfi & Rezzolla (2013) Ciolfi, R., & Rezzolla, L. 2013, Mon. Not. Roy. Astron. Soc., 435, L43, doi: 10.1093/mnrasl/slt092
  * Covas et al. (2018) Covas, P. B., et al. 2018, Phys. Rev. D, 97, 082002, doi: 10.1103/PhysRevD.97.082002
  * Cromartie et al. (2020) Cromartie, H. T., Fonseca, E., Ransom, S. M., et al. 2020, Nature Astronomy, 4, 72, doi: 10.1038/s41550-019-0880-2
  * Cutler & Schutz (2005) Cutler, C., & Schutz, B. F. 2005, Phys. Rev. D, 72, 063006, doi: 10.1103/PhysRevD.72.063006
  * Davis et al. (2021) Davis, D., et al. 2021, Class. Quant. Grav., 38, 135014, doi: 10.1088/1361-6382/abfd85
  * Dergachev & Papa (2023) Dergachev, V., & Papa, M. A. 2023, Phys. Rev. X, 13, 021020, doi: 10.1103/PhysRevX.13.021020
  * Dohi et al. (2023) Dohi, A., Greco, E., Nagataki, S., et al. 2023, Astrophys. J., 949, 97, doi: 10.3847/1538-4357/acce3f
  * Evans et al. (2021) Evans, M., et al. 2021.  https://arxiv.org/abs/2109.09882
  * Ghosh et al. (2023) Ghosh, S., Pathak, D., & Chatterjee, D. 2023, ApJ, 944, 53, doi: 10.3847/1538-4357/acb0d3
  * Glampedakis & Gualtieri (2018) Glampedakis, K., & Gualtieri, L. 2018, Astrophys. Space Sci. Libr., 457, 673, doi: 10.1007/978-3-319-97616-7_12
  * Greco et al. (2021) Greco, E., Miceli, M., Orlando, S., et al. 2021, Astrophys. J. Lett., 908, L45, doi: 10.3847/2041-8213/abdf5a
  * Greco et al. (2022) Greco, E., Miceli, M., Orlando, S., et al. 2022, ApJ, 931, 132, doi: 10.3847/1538-4357/ac679d
  * Gupta et al. (2023) Gupta, I., et al. 2023.  https://arxiv.org/abs/2307.10421
  * Hirata et al. (1987) Hirata, K., et al. 1987, Phys. Rev. Lett., 58, 1490, doi: 10.1103/PhysRevLett.58.1490
  * Ho & Lai (2000) Ho, W. C. G., & Lai, D. 2000, Astrophys. J., 543, 386, doi: 10.1086/317085
  * Idrisy et al. (2015) Idrisy, A., Owen, B. J., & Jones, D. I. 2015, Phys. Rev. D, 91, 024001, doi: 10.1103/PhysRevD.91.024001
  * Jaranowski et al. (1998) Jaranowski, P., Krolak, A., & Schutz, B. F. 1998, Phys. Rev. D, 58, 063001, doi: 10.1103/PhysRevD.58.063001
  * Johnson-McDaniel & Owen (2013) Johnson-McDaniel, N. K., & Owen, B. J. 2013, Phys. Rev. D, 88, 044004, doi: 10.1103/PhysRevD.88.044004
  * Lander (2014) Lander, S. K. 2014, Mon. Not. Roy. Astron. Soc., 437, 424, doi: 10.1093/mnras/stt1894
  * LIGO Scientific Collaboration (2020) LIGO Scientific Collaboration. 2020, LIGO Algorithm Library - LALSuite, free software (GPL), doi: 10.7935/GT1W-FZ16
  * Lindblom et al. (1998) Lindblom, L., Owen, B. J., & Morsink, S. M. 1998, Phys. Rev. Lett., 80, 4843, doi: 10.1103/PhysRevLett.80.4843
  * Marshall et al. (1998) Marshall, F. E., Gotthelf, E. V., Zhang, W., Middleditch, J., & Wang, Q. D. 1998, Astrophys. J. Lett., 499, L179, doi: 10.1086/311381
  * Martinez et al. (2015) Martinez, J. G., Stovall, K., Freire, P. C. C., et al. 2015, ApJ, 812, 143, doi: 10.1088/0004-637X/812/2/143
  * Morales & Horowitz (2022) Morales, J. A., & Horowitz, C. J. 2022, Mon. Not. Roy. Astron. Soc., 517, 5610, doi: 10.1093/mnras/stac3058
  * Morales & Horowitz (2023) —. 2023.  https://arxiv.org/abs/2309.04855
  * Owen (2005) Owen, B. J. 2005, Phys. Rev. Lett., 95, 211101, doi: 10.1103/PhysRevLett.95.211101
  * Owen (2010) —. 2010, Phys. Rev. D, 82, 104002, doi: 10.1103/PhysRevD.82.104002
  * Owen et al. (2022) Owen, B. J., Lindblom, L., & Soares Pinheiro, L. 2022, Astrophys. J. Lett., 935, L7, doi: 10.3847/2041-8213/ac84dc
  * Owen et al. (2023) Owen, B. J., Lindblom, L., Soares Pinheiro, L., & Rajbhandari, B. 2023, Drill for continuous gravitational waves, free software (GPL), doi: https://github.com/bjowen/drill_public
  * Page et al. (2020) Page, D., Beznogov, M. V., Garibay, I., et al. 2020, Astrophys. J., 898, 125, doi: 10.3847/1538-4357/ab93c2
  * Panagia (1999) Panagia, N. 1999, in New Views of the Magellanic Clouds, ed. Y. H. Chu, N. Suntzeff, J. Hesser, & D. Bohlender, Vol. 190, 549
  * Piran & Nakamura (1988) Piran, T., & Nakamura, T. 1988, Prog. Theor. Phys., 80, 18, doi: 10.1143/PTP.80.18
  * Sun et al. (2016) Sun, L., Melatos, A., Lasky, P. D., Chung, C. T. Y., & Darman, N. S. 2016, Phys. Rev. D, 94, 082004, doi: 10.1103/PhysRevD.94.082004
  * Tse et al. (2019) Tse, M., et al. 2019, Phys. Rev. Lett., 123, 231107, doi: 10.1103/PhysRevLett.123.231107
  * Vallisneri et al. (2015) Vallisneri, M., Kanner, J., Williams, R., Weinstein, A., & Stephens, B. 2015, J. Phys. Conf. Ser., 610, 012021, doi: 10.1088/1742-6596/610/1/012021
  * Wette (2012) Wette, K. 2012, Phys. Rev. D, 85, 042003, doi: 10.1103/PhysRevD.85.042003
  * Wette (2023) —. 2023, Astropart. Phys., 153, 102880, doi: 10.1016/j.astropartphys.2023.102880
  * Wette et al. (2008) Wette, K., et al. 2008, Class. Quant. Grav., 25, 235011, doi: 10.1088/0264-9381/25/23/235011

  *[GW]: gravitational wave
  *[O2]: Advanced LIGO and Virgo’s second observing run
  *[O3]: Advanced LIGO and Virgo’s third observing run
  *[GWs]: gravitational wave
  *[SFT]: Short Fourier Transform
  *[H1]: Hanford, WA
  *[L1]: Livingston, LA
  *[V1]: Cascina, Italy
  *[psd]: power spectral density
  *[O4]: Advanced LIGO and Virgo’s fourth observing run


# Contextual embedding and model weighting by fusing domain knowledge on
Biomedical Question Answering

Yuxuan Lu  0000-0002-8520-0540 Beijing University of Technology100
PingleyuanChaoyang QuBeijing ShiChina<EMAIL_ADDRESS>,  Jingya Yan
0000-0003-0373-1696 Beijing University of Technology100 PingleyuanChaoyang
QuBeijing ShiChina<EMAIL_ADDRESS>,  Zhixuan Qi  0000-0001-5333-0335
Beijing University of Technology100 PingleyuanChaoyang QuBeijing ShiChina
<EMAIL_ADDRESS>,  Zhongzheng Ge  0000-0003-2002-6251 Beijing University
of Technology100 PingleyuanChaoyang QuBeijing ShiChina<EMAIL_ADDRESS>and  Yongping Du  0000-0001-6867-2063 Beijing University of Technology100
PingleyuanChaoyang QuBeijing ShiChina<EMAIL_ADDRESS>
###### Abstract.

Biomedical Question Answering aims to obtain an answer to the given question
from the biomedical domain. Due to its high requirement of biomedical domain
knowledge, it is difficult for the model to learn domain knowledge from
limited training data. We propose a contextual embedding method that combines
open-domain QA model AoA Reader and BioBERT model pre-trained on biomedical
domain data. We adopt unsupervised pre-training on large biomedical corpus and
supervised fine-tuning on biomedical question answering dataset. Additionally,
we adopt an MLP-based model weighting layer to automatically exploit the
advantages of two models to provide the correct answer. The public dataset
biomrc constructed from PubMed corpus is used to evaluate our method.
Experimental results show that our model outperforms state-of-the-art system
by a large margin.

biomedical question answering, contextual embedding, model weighting, domain
knowledge.

##  1\. Introduction

Question answering is a classic task in Natural Language Processing, requiring
a model to understand natural lang-uages. Cloze-style question answering
problem has been a popular task because it is relatively easier to build
cloze-style datasets. The cloze style question aims to select the best
candidate answer regarding the specified context and fill in the blank of the
question. Multiple cloze-style datasets have been published, such as CNN/Daily
Mail (Hermann et al., 2015), Children’s Book Test (Hill et al., 2016), etc.
Models based on neural networks are proposed, such as AS READER (Kadlec et
al., 2016), CAS READER (Cui et al., 2016), AoA Reader (Cui et al., 2017)and
BERT (Devlin et al., 2019).

These models have achieved good performance on several datasets. However, they
do not perform well when facing domain-oriented problems. The main reason is
that domain-oriented questions require more background knowledge to give an
answer, and a large dataset is needed to allow the models to learn the
required domain knowledge.

We make improvements to the existing model, AoA Reader, and validate our
results on the biomrc dataset(Pappas et al., 2020), the public biomedical
dataset constructed from corpus from PubMed. We put forward the Contextual
Word Embedding method and the MLP-based model weighting strategy for the
biomedical question answering task. By combining the open-domain QA model and
domain-oriented contextual word embedding, the proposed method outperforms
state-of-the-art system on biomedical domain question answering significantly,
setting up a new state-of-the-art system.

The main contributions of this paper are listed as follows:

  * •

Combining BioBERT and AoA Reader, which can take full advantage of contextual
word embedding model pre-trained on large domain corpus and mining semantic
and contextual information to choose the best answer. In particular, multiple
aggregation methods are adopted and evaluated.

  * •

An MLP-based model weighting strategy is proposed, which can automatically
learn the preferences and biases of different models and exploit the
advantages of both models to provide the correct answer.

  * •

Our method is evaluated on the biomrc dataset, and the results show that it
outperforms state-of-the-art system significantly. Our code is available at
https://github.com/leoleoasd/MLP-based-weighting.

##  2\. Related Work

The research of Question Answering has made rapid progress, which benefits
from the publication of large-scale and high-quality datasets. Richadson et
al. release MCTest (Richardson et al., 2013), a multiple choice machine
reading comprehension dataset that opened up research on statistical-based
machine learning models. Hermann et al. release the CNN Daily Mail dataset
(Hermann et al., 2015), which includes over 1 million cloze-style data. More
high-quality datasets have been released since then, such as SQuAD (Rajpurkar
et al., 2018, 2016), Facebook Children’s Book Test (Hill et al., 2016), etc.

Models based on deep learning technologies significantly outperform
traditional models in extracting context information. Hermann et al. (Hermann
et al., 2015) propose an attention-based neural network and proves that the
incorporation of attention mechanism is more effective than traditional,
statistical-based baselines. Seo et al. propose the BiDAF (Seo et al., 2017)
model, which uses different levels of encoding for linguistic representation
and uses a bidirectional attention flow mechanism to obtain the query-aware
context representation. Kadlec et al. (Kadlec et al., 2016) propose a simple
model, the _Attention Sum Reader_ (AS Reader), which uses attention to
directly pick answer. Fu et al. propose Ea-Reader (Fu and Zhang, 2019), whose
memory updating rule is able to maintain the understanding of document through
read, update and write operations. Chen et al. propose McR2 (Chen et al.,
2020), which enables relational reasoning on candidates based on fusion
representations of document, query and candidates. Fu et al. propose ATNet (Fu
et al., 2019), which utilities both intra-attention and inter-attention to
answer close-style questions over documents. Cui et al. propose _Attention-
over-Attention Reader_ ( AoA READER) (Cui et al., 2017), which puts another
level of document-to-query attention on top of query-to-document attention,
achieving state-of-the-art performance on multiple datasets.

In recent years, researchers have focused on combining the unsupervised pre-
training on large corpus and supervised fine-tuning on the specific task.
Vaswani et al. propose the Transformer (Vaswani et al., 2017) model, which
uses attention mechanism to replace the CNN and RNN parts of the traditional
model to improve the model while speeding up the training process. Devlin et
al. build a large-scale unsupervised pre-training model BERT (Devlin et al.,
2019) on top of Transformer to pre-train the language representation model
using the masked language model task and the next sentence prediction task,
and innovatively propose a training strategy that separates pre-training and
fine-tuning. While sharing the same pre-trained weights, BERT achieves state-
of-the-art performance on many different downstream tasks and datasets. Other
BERT-based model is proposed, for instance, Liu et al. propose RoBERTa(Liu et
al., 2019), providing servel techniques to robustly pre-train language models.
Lan et al. propose ALBERT(Lan et al., 2019), providing two parameter-reduction
techniques to lower memory consumption and increase the training speed of
BERT.

In the biomedical domain, Tsatsaronis et al. launch the BioASQ (Tsatsaronis et
al., 2015) challenges. It contains multiple subtasks, including article /
snippet retrieval, document classification and question answering. Pappas et
al. construct two cloze-style datasets, bioread(Pappas et al., 2018) and
biomrc (Pappas et al., 2020), and compare the accuracy of experts, non-expert
human and different baseline models and neural network models. The results
show that the baseline methods fail to correctly answer questions of the
biomrc dataset while neural MRC models perform well, indicating that the
biomrc dataset is less noisy and has enough features for the model to learn.
Tang et al. initiate the CORD-19 (Tang et al., 2020) dataset at the beginning
of the global COVID-19 pandemic to help researchers in the biomedical field
retrieve articles quickly.

Figure 1. Model structure based on Pre-training and Weighting Strategy

Traditional neural network models require large-scale, high-quality supervised
training data to obtain better results. However, it is challenging to build a
large-scale and high-quality dataset for domain-oriented tasks because it
requires domain experts’ annotations. Recent researchers prove that combining
the pre-training on large corpus and fine-tuning on supervised training data
can achieve better performance on domain-oriented tasks. Gururangan et al.
(Gururangan et al., 2020) proves that a second phase of pre-training in domain
(domain-adaptive pre-training) leads to performance gains. Gu et al. propose
BLURB (Biomedical Language Understanding & Reasoning Benckmark) (Gu et al.,
2021) to test the biomedical language understanding ability of language
models. Lee et al. propose BioBERT (Lee et al., 2019) and Cohan et al. propose
SciBERT (Beltagy et al., 2019), both of which learn the corpus representation
of papers on a large-scale corpus of biomedical papers or scientific papers
and have obtained better results on natural language processing tasks in
biomedical domains. We aim to further improve the performance by using BioBERT
to obtain biomedical contextual and semantic information and by using model
weighting layers to combine different models.

##  3\. Method

We propose a pre-training strategy based on the scientific pre-training model
(SciBERT) and open-domain QA model (AoA Reader) to obtain the final answer to
the question. In particular, different embedding and weighting strategies are
used in the training process. fig. 1 shows the full structure of our model.

###  3.1. Formal Task Description

This model is aiming at tasks that comprise cloze-style questions, to which
answers are closely related to the comprehension of the context documents
included in the problems. A set of candidate answers are also provided
alongside, and the model is supposed to choose an answer from the candidates.
This task can be formalized as a triplet
$\left\langle\mathcal{C},\mathcal{Q},\mathcal{A}\right\rangle$ that is
inclusive of the given context $\mathcal{C}=\\{w_{1},w_{2},\dots,w_{n}\\}$
made of words $w_{i}$, a query
$\mathcal{Q}=\\{q_{1},q_{2},\dots,[MASK],\dots,q_{m}\\}$ where the special
token [MASK] marks the position where the answers are supposed to be placed,
and answer candidates $\mathcal{A}=\\{a_{1},a_{2},\dots,a_{o}\\}$. A function
$F$ is expected to be learned by the model to predict the answer $\mathcal{A}$
of question $\mathcal{Q}$ based on its comprehension of the proffered context
$\mathcal{C}$:

(1) $\displaystyle\forall
a\in\mathcal{A},P(a|\mathcal{C},\mathcal{Q})=\begin{cases}1&a\text{ is the
correct answer}\\\ 0&a\text{ is not the correct answer}\end{cases}$ (2)
$\displaystyle
F(\mathcal{C},\mathcal{Q},\mathcal{A})=\max_{a\in\mathcal{A}}P(a|\mathcal{C},\mathcal{Q})$

###  3.2. Training of SciBERT

####  3.2.1. Training data

We use the SciBERT (Beltagy et al., 2019), which has been pre-trained on the
Semantic Scholar corpus. The corpus consists of 18% papers from the computer
science domain and 82% from the broad biomedical domain. The unsupervised pre-
training process using the large-scale corpus allows the model to obtain the
semantic information of biomedical texts. Further, in order to let the model
adapt to the cloze-style question answering task, biomrc dataset is used to
fine-tune the model.

####  3.2.2. Answer extraction

We adopt the answer extraction strategy Pappas et al. used in their biomrc
dataset (Pappas et al., 2020). For each context-question pair, we first divide
the context into sentences using NLTK(Bird and Loper, 2004). Each sentence is
concatenated to the question by [SEP] token, and they are fed to SciBERT
respectively. In this way, we obtain the top-level embedding of the candidate
entities and the placeholder in the question. The embeddings of each entity in
the sentence are connected to the placeholder’s embedding and are sent to a
multilayer perceptron to obtain the score for the particular entity. If an
entity appears multiple times in the paragraph, we choose the maximum value of
its score.

###  3.3. Training of AoA Reader

In order to make AoA Reader (Cui et al., 2017) achieve better performance on
domain-oriented tasks, we adopt different contextualized word embedding and
attention aggregation strategies.

####  3.3.1. Contextualized word embedding

The AoA Reader uses direct word embedding. This approach converts each token
in the context $\mathcal{C}$ and the question $\mathcal{Q}$ into a one-hot
vector. It then converts them into a continuous representation using the
shared embedding matrix $W_{e}$. Using this method in biomedical question
answering may result in severe Out-Of-Vocabulary issues, given that most terms
in the biomedical domain are made through several word-formation methods.
Meanwhile, the model cannot learn necessary knowledge and the meanings of
terms due to the lack of domain-oriented training data.

Instead, we adopt BioBERT to generate contextualized word embedding, which has
been pre-trained on a large biomedical corpus from PubMed containing
biomedical literature proposed by Devlin (Devlin et al., 2019). Thus, we can
obtain domain-oriented knowledge and terms.

For tokenization, WordPiece (Wu et al., 2016) through which new words can be
represented by known tokens is used. Further, it can solve the Out-Of-
Vocabulary issue and allow the model to better understand domain terms made by
word-formation methods.

After connecting the context and the question with [SEP] token, it is fed into
BioBERT. If it is longer than length limit (512 tokens), we trim the back of
the context and keep the original question unchanged:

(3)
$E(\mathcal{C,Q})=BERT([\mathrm{[CLS]};\mathcal{C};\mathrm{[SEP]};\mathcal{Q};\mathrm{[SEP]}])$

$\mathrm{E(\mathcal{C,Q})}$ in eq. 3 is the contextual embedding of the
context $\mathcal{C}$ and the query $\mathcal{Q}$.

To obtain the embeddings of context $\mathcal{C}$ and the query $\mathcal{Q}$,
we’ve applied a masking operation on $E(\mathcal{C,Q})$ for segmentation. This
would conceal the representation of the other segment by zero vectors, leaving
the desired half acquired:

(4) $\displaystyle E(\mathcal{C})_{i}$
$\displaystyle=\begin{cases}E(\mathcal{C,Q})_{i}&i\text{ is a context
token}\\\ 0&i\text{ is a question token}\\\ \end{cases}$ (5) $\displaystyle
E(\mathcal{Q})_{i}$ $\displaystyle=\begin{cases}E(\mathcal{C,Q})_{i}&i\text{
is a question token}\\\ 0&i\text{ is a context token.}\\\ \end{cases}$

We adopt bi-directional RNN to further obtain the contextual representations
$h_{context}(\mathcal{C})\in\mathbb{R}^{(|\mathcal{C}|+|\mathcal{Q}|+3)*2d}$
of the context $\mathcal{C}$:

(6)
$\displaystyle\overrightarrow{h_{context}(\mathcal{C})}=\overrightarrow{GRU}(E(\mathcal{C}))$
(7)
$\displaystyle\overleftarrow{h_{context}(\mathcal{C})}=\overleftarrow{GRU}(E(\mathcal{C}))$
(8) $\displaystyle
h_{context}(\mathcal{C})=[\overrightarrow{h_{context}(\mathcal{C})};\overleftarrow{h_{context}(\mathcal{C})}]$

and similarly we obtain
$h_{question}(\mathcal{Q})\in\mathbb{R}^{(|\mathcal{C}|+|\mathcal{Q}|+3)*2d}$
for the question $\mathcal{Q}$.

####  3.3.2. Pair-wise Matching Score

After obtaining the contextual embedding of the context $h_{context}$ and the
question $h_{question}$, we calculate a pairwise matching matrix, which
indicates the relevance between a token in the context and question, by
calculating their dot product:

(9) $M(i,j)=h_{context}(i)^{T}\cdot
h_{question}(j)\quad\;i\in\mathcal{C},\;j\in\mathcal{Q}$

####  3.3.3. Attentions over Attention Mechanism

Context | Because of reports of anaplastic transformation following irradiation, this study examines the incidence of anaplastic transformation and local control of these lesions. This review of seven @entity1 who had @entity189 of the @entity135 that was treated with irradiation shows local control in 71% of cases. There were no cases of anaplastic transformation. This report adds to the literature two cases of ”de-differentiation” to less differentiated @entity957 ; one such case occurred after surgery alone. The literature is reviewed. Overall, anaplastic transformation is reported in 7% of @entity1 who had irradiation. De-differentiation occurs after surgery as well. The rate of local control with irradiation is less than 50%; with surgery it is 85%. It is concluded that surgery should be used if the procedure has acceptable morbidity. Otherwise, irradiation can be used. Failures can be salvaged surgically. ”Anaplastic transformation” should not affect treatment approach.
---|---
Candidate |
Entities |  @entity1: [‘patients’] @entity135: [‘head and neck’] @entity957: [‘squamous carcinomas’] @entity189: [‘verrucous carcinoma’]
Question | Radiotherapy in the treatment of XXXX of the @entity135 .
Answer |  @entity189: [‘verrucous carcinoma’]
Figure 2. A example of the biomrc dataset.

After getting the pair-wise matching matrix M, column-wise softmax is used to
get the context-level attention regarding each token in the question:

(10)
$\displaystyle\alpha(t)=\operatorname{softmax}(M(1,t),\dots,M(|\mathcal{C}|,t))$
(11) $\displaystyle\alpha=[\alpha(1),\alpha(2),\dots,\alpha(|\mathcal{Q}|)]$

We calculate a reversed attention using row-wise softmax to obtain the
“importance” of each token in the question regarding each token in the
context:

(12)
$\displaystyle\beta(t)=\operatorname{softmax}(M(t,1),M(t,2),\dots,M(t,|\mathcal{Q}|)).$

We average all $\beta(t)$ to get a question-level attention $\beta$:

(13) $\displaystyle\beta=\frac{1}{n}\sum_{t=1}^{|\mathcal{C}|}\beta(t)$

Then we adopt the attention-over-attention mechanism, by merging these two
attentions to get the “attended context-level attention”:

(14) $\displaystyle s=\alpha^{T}\cdot\beta$

where $s$ denotes the importance of each token in the context.

####  3.3.4. Final Predictions

AoA Reader regards an entity as a word no matter how many words it has. It
uses _sum attention_ mechanism proposed by Kadlec et al. (Kadlec et al., 2016)
to get the confidence score of each candidate entity. In our model which uses
WordPiece to obtain contextualized word embeddings, an entity may be either
segmented into multiple tokens or composed of multiple words, and each token
of the entity may occur multiple times in the context. So the confidence score
of each candidate answer $a$ is calculated by aggregating all the occurrences
of all its tokens in the context:

(15)
$P(a|\mathcal{C},\mathcal{Q})=\operatorname*{F_{1}}_{t\in\mathcal{T}(a)}(\operatorname*{F_{2}}_{i\in
I(t,\mathcal{C})}(s_{i}))$

where $\mathcal{T}(a)$ is the result of segmenting the candidate answer $a$
using WordPiece; $F_{1}$ and $F_{2}$ are aggregating functions, which can be
either maximum or sum, and $I(t,\mathcal{C})$ indicates the position that the
token $t$ appears in the context $\mathcal{C}$.

###  3.4. Model Weighting strategy

After completing the training of AoA Reader and SciBERT, a model weighting
strategy is used to obtain the final answer by combining the advantages of
both models.

Our previous study (Du et al., 2021) demonstrates that better performance can
be achieved using a dual-model weighting strategy. The weighting process is
performed by calculating a weighted average of the answer’s confidence score
and the similarity of the answer derived from two models. Further, considering
that different models perform differently against data with different
features, we use a simple MLP with one hidden layer to allow the model
weighting layer to _automatically_ learn this difference and to take advantage
of both models.

(16) $score=MLP([score_{a},score_{b}])$

Where $score_{a},score_{b}\in\mathbb{R}^{|\mathcal{A}|}$ is the confidence
score of each model. In this way, the weighting layer would be able to learn
the predilection and biases of each candidate model and achieve better
performance.

##  4\. Experiment

###  4.1. Datasets

We conduct the experiments on the biomrc lite dataset (Pappas et al., 2020) to
verify the effect of our method. biomrc is a biomedical cloze-style dataset
for machine reading comprehension. The contexts in each sample are extracted
from PUBTATOR, a repository containing 25 million abstracts and their
corresponding titles on PubMed. Biomedical entities in the abstract are
extracted to form the candidate entities. The contexts are the abstracts
themselves, and the questions are construct by randomly replacing a biomedical
entity in the title with a placeholder. fig. 2 gives a sample in the biomrc
dataset.

###  4.2. Experiment Settings

Table 1.  The result of different aggregation functions, compared to the
state-of-the-art model and human experts

Method | Occurrence Aggregation |  Token Aggregation | Train Time11footnotemark: 1 | biomrc lite |  |  biomrc tiny 22footnotemark: 2
---|---|---|---|---|---|---
|  |  |  | Dev Acc | Test Acc | Test Acc
AS-READER | - |  - | 16.56hr | 62.29 | 62.38 | 66.67
AoA-READER | - |  - | 60.90hr | 70.00 | 69.87 | 70.00
SCIBERT-MAX-READER | - |  - | 83.22hr | 80.06 | 79.97 | 90.00
Human Experts | - |  - | - | - | - | 85.00
AoA-READER with BioBERT Embedding | max |  max | 1.50hr | 78.54 | 78.11 | 90.00
| max |  sum | 0.88hr | 83.40 | 83.36 | 93.33
| sum |  max | 3.60hr | 80.98 | 81.20 | 90.00
| sum |  sum | 1.76hr | 87.22 | 86.74 | 93.33

1: We conduct some code optimizations on the AoA-Reader model, so the training
time of our implementation with BioBERT can not be compared to their original
implementation.

2: The test set of biomrc tiny dataset only contains 30 samples, and so the
results on it may be unstable. On the other hand, the demonstrated accuracy of
human experts comes from averaging the results of multiple experts, so it is a
bit more stable than other results.

Our experiments are carried out on the machine with Intel i9-10920X (24) @
4.700GHz, GPU of GeForce GTX 3090 24G, using pytorch 1.9.1 as the deep
learning framework. To avoid overfitting, all models are trained for a maximum
of 40 epochs, using early stopping on the dev, with a patience of 3 epochs.

During the process of fine-tuning SciBERT, the batch size is set to 1 and the
top layer of SciBERT is frozen; other layers are trained with the learning
rate of 0.001.

During the process of fine-tuning BioBERT and training AoA Reader, the batch
size is set to 30, the learning rate is set to 0.001, and the learning rate
for BioBERT is set to $10^{-5}$. To reduce GPU memory usage, we use the mixed-
precision training technique (Micikevicius et al., 2018), setting precision to
16 bits.

We train our model on the biomrc lite dataset and evaluate it both on the
biomrc lite  and tiny  dataset, which have 100,000 and 30 samples,
respectively. We use Setting A for biomrc, in which all pseudo-identifier like
_@entity1_ have a global scope, i.e., all biomedical entities have a unique
pseudo-identifier in the whole dataset.

The model weighting layer is implemented after completing the training process
of the two models, SciBERT and AoA Reader. The best weights evaluated by the
_Dev Acc_ are chosen to obtain the individual scores of each sample, which
will later be used to train the model weighting layer.

###  4.3. Results

####  4.3.1. Performance of the Contextualized Word Embedding Strategy

Contextualized word embedding strategy based on BioBERT is used to obtain the
final prediction answer. The selection of aggregating functions is crucial to
the model performance. Therefore, multiple combinations of different
aggregating functions are evaluated, and the results are shown in table 1.

Table 2. The results of two single models and our MLP-based model weighting model, compared to the union accuracy of two single models.  Method | biomrc lite |  | biomrc tiny
---|---|---|---
| Dev Acc | Test Acc | Test Acc
AoA-READER with BioBERT Embedding | 87.22 | 86.74 | 93.33
SCIBERT-MAX-READER | 79.74 | 80.21 | 86.67
MLP-based Weighting Model (ours) | 88.76 | 88.00 | 96.66
The union of two single models (ideal result) | 93.07 | 92.26 | 96.66
Table 3.  The results of our MLP-based Weighting Model, excluding data that
both models failed to answer

Method | biomrc lite |  |  biomrc tiny 11footnotemark: 1
---|---|---|---
| Dev Acc | Test Acc | Test Acc
AoA-READER with BioBERT Embedding | 93.71 | 93.21 | 96.55
SCIBERT-MAX-READER | 86.67 | 86.94 | 89.66
MLP-based Weighting Model (ours) | 95.36 | 95.38 | 100.00

1: The test set of biomrc tiny dataset only contains 30 samples, and so the
results on it may be unstable.

It can be seen that choosing sum as both aggregation functions obtains better
performance, and our model outperforms the state-of-the-art model
significantly, which is about 6.77% absolute improvements on the biomrc lite
test sets.

Our model also shows an improvement on the biomrc tiny dataset, though the
dataset contains only 30 samples, and this result may be unstable. Our
performance on the larger biomrc lite test set still exceeds the average human
expert performance on the biomrc tiny test set.

####  4.3.2. Performance of the Weighting Model

The MLP-based weighting model is used to further achieve better performance.
We implement the SCIBERT-MAX-READER proposed by Pappas et al. (Pappas et al.,
2020), and use the model weighting strategy on top of AoA Reader with BioBERT
embedding and SCIBERT-MAX-READER. The results of our experiments are shown in
table 2. The result of SCIBERT-MAX-READER comes from our implementation of
this model, which is used to train our MLP-based weighting layer. The results
slightly differs to those in table 1.

It can be seen that our MLP-based weighting model improves the accuracy
effectively. Especially, the accuracy on the test set is improved by 1.26%
over the AoA Reader on the biomrc lite test dataset.

Figure 3.  The number of question answered correctly by different models on
the biomrc lite dataset.

There were 6250 questions, and the AoA Reader correctly answered 5421
questions. SciBERT correctly answered 5013 questions, and there were 4668
questions that they both answered correctly. There were 484 questions that
they both did not answer correctly, and the MLP weighted model could answer
all of the questions that they both answered, most of the questions that one
of them answered, and 26 questions that neither model answered.

In order to evaluate the effectiveness of the model weighting layer, we
compare the result to the union accuracy of two single models, i.e., the
percentage of the union of the questions answered correctly by the two models
in the total number of questions.

The results when excluding data that both models failed to answer are shown in
table 3.The number of questions answered correctly by different models on the
biomrc lite dataset is shown in fig. 3. Here, _Both_ in the figure refers to
the the questions correctly answered by AoA-READER and SciBERT, and _Neither_
in the figure refers to questions that cannot be answered correctly by any
model.

As expected, both of the two models correctly answer some questions that the
other model failed to answer. The proposed MLP-based weighting model not only
gives the correct answer to the question that at least one model answers
correctly, but also a small number of questions that both models fail to
answer.

To further corroborate the model’s improvements in performance, we’ve applied
the McNemar test to the results. Letting null hypothesis $h_{0}$ be our model
has the same performance as SciBERT-MAX-READER, the alternate hypothesis
$h_{1}$ would be there is a notable difference between the performance of our
model and SciBERT-MAX-READER. The $h_{1}$ is accepted by the test where n=2,
significance $\alpha$ = 0.025. Since our model has a lower error rate in all
tests, it sufficiently supports the hypothesis that our model has
significantly outperformed the SciBERT-MAX-READER.

In general, our MLP-based weighting model improves the performance by 2.17%
significantly compared to the original single model. These results demonstrate
that the proposed method can automatically learn the biases and preferences
and exploit the strengths of both models to achieve better performance.

Figure 4. The validation accuracy of AoA Reader after each epoch.

The valudation accuracy of AoA Reader with different token aggregation and
occurence aggregation is shown. Here, ‘max-max’ represent token aggregation
and occurence aggregation respectively.

####  4.3.3. Model Training Analysis

In our structure, SCIBERT-MAX-READER and AoA Reader are trained separately,
the outputs of which are then gathered for the final weighting layer to learn.

The BioBERT embedded in the AoA Reader model is pre-trained on PubMed, and
fine-tuned with the AoA Reader model on the BIOMRC dataset. It takes about 26
minutes for the AoA Reader to finish one epoch of learning based on the biomrc
lite dataset.

All models generated in epochs are saved, among which the one that has the
best performance on the dev set will be used to train the weighting layer.

The training process for the AoA reader is illustrated in fig. 4. It can be
seen that model using sum as both token and occurence aggregation converges
faster compated to most models while giving best results.

##  5\. Conclusions

We propose a contextual embedding and model weighting method, which can
combine model pre-trained on a large corpus and open-domain QA model to mine
semantic and contextual information in biomedical question answering.
Especially, we adopt an MLP-based model weighting strategy which can
automatically learn and utilize the preferences and biases of two models to
combine their advantages. The results show that our method outperforms state-
of-the-art system and has higher accuracy than experts. In future work, how to
use the semantic similarity between entity tokens and context tokens in
getting final predictions should be studied, i.e., a context token should
contribute to the score of an entity if its semantic information is similar to
that of entity token.

## References

  * (1)
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3615–3620.  https://doi.org/10.18653/v1/D19-1371
  * Bird and Loper (2004) Steven Bird and Edward Loper. 2004.  NLTK: The Natural Language Toolkit. In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_. Association for Computational Linguistics, Barcelona, Spain, 214–217.  https://doi.org/10.3115/1219044.1219075
  * Chen et al. (2020) Wuya Chen, Xiaojun Quan, Chunyu Kit, Zhengcheng Min, and Jiahai Wang. 2020.  Multi-Choice Relational Reasoning for Machine Reading Comprehension. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 6448–6458.  https://doi.org/10.18653/v1/2020.coling-main.567
  * Cui et al. (2017) Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017\.  Attention-over-Attention Neural Networks for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 593–602.  https://doi.org/10.18653/v1/P17-1055
  * Cui et al. (2016) Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016.  Consensus Attention-based Neural Networks for Chinese Reading Comprehension. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, Osaka, Japan, 1777–1786.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Du et al. (2021) Yongping Du, Jingya Yan, Yiliang Zhao, Yuxuan Lu, and Xingnan Jin. 2021.  Dual Model Weighting Strategy and Data Augmentation in Biomedical Question Answering. In _Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_.
  * Fu et al. (2019) Chengzhen Fu, Yuntao Li, and Yan Zhang. 2019.  ATNet: Answering Cloze-Style Questions via Intra-attention and Inter-attention. In _Advances in Knowledge Discovery and Data Mining_ _(Lecture Notes in Computer Science)_ , Qiang Yang, Zhi-Hua Zhou, Zhiguo Gong, Min-Ling Zhang, and Sheng-Jun Huang (Eds.). Springer International Publishing, Cham, 242–252.  https://doi.org/10.1007/978-3-030-16145-3_19
  * Fu and Zhang (2019) Chengzhen Fu and Yan Zhang. 2019.  EA Reader: Enhance Attentive Reader for Cloze-Style Question Answering via Multi-Space Context Fusion. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 6375–6382.  https://doi.org/10.1609/aaai.v33i01.33016375
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, et al. 2021\.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.  _ACM Transactions on Computing for Healthcare_ 3, 1 (Oct. 2021), 2:1–2:23.  https://doi.org/10.1145/3458754
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, et al. 2020\.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 8342–8360.  https://doi.org/10.18653/v1/2020.acl-main.740
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, et al. 2015\.  Teaching Machines to Read and Comprehend. In _Advances in Neural Information Processing Systems_ , Vol. 28. Curran Associates, Inc.
  * Hill et al. (2016) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016.  The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).
  * Kadlec et al. (2016) Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016.  Text Understanding with the Attention Sum Reader Network. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 908–918.  https://doi.org/10.18653/v1/P16-1086
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In _International Conference on Learning Representations_.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, et al. 2019\.  BioBERT: A Pre-Trained Biomedical Language Representation Model for Biomedical Text Mining.  _Bioinformatics_ (Sept. 2019), btz682.  https://doi.org/10/ggh5qq
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, et al. 2019\.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv:1907.11692 [cs]_ (July 2019).  arXiv:1907.11692 [cs]
  * Micikevicius et al. (2018) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, et al. 2018\.  Mixed Precision Training. In _International Conference on Learning Representations_.
  * Pappas et al. (2018) Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. 2018\.  BioRead: A New Dataset for Biomedical Reading Comprehension. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Miyazaki, Japan.
  * Pappas et al. (2020) Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. 2020.  BioMRC: A Dataset for Biomedical Machine Reading Comprehension. In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_. Association for Computational Linguistics, Online, 140–149.  https://doi.org/10.18653/v1/2020.bionlp-1.15
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Melbourne, Australia, 784–789.  https://doi.org/10.18653/v1/P18-2124
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, 2383–2392.  https://doi.org/10.18653/v1/D16-1264
  * Richardson et al. (2013) Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013.  MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, 193–203.
  * Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Bidirectional Attention Flow for Machine Comprehension. In _The International Conference on Learning Representations_.  arXiv:1611.01603
  * Tang et al. (2020) Raphael Tang, Rodrigo Nogueira, Edwin Zhang, Nikhil Gupta, Phuong Cam, Kyunghyun Cho, et al. 2020\.  Rapidly Bootstrapping a Question Answering Dataset for COVID-19.  _arXiv:2004.11339 [cs]_ (April 2020).  arXiv:2004.11339 [cs]
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, et al. 2015\.  An Overview of the BIOASQ Large-Scale Biomedical Semantic Indexing and Question Answering Competition.  _BMC Bioinformatics_ 16, 1 (April 2015), 138\.  https://doi.org/10.1186/s12859-015-0564-6
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, et al. 2017\.  Attention Is All You Need. In _Advances in Neural Information Processing Systems_ , Vol. 30. Curran Associates, Inc.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, et al. 2016\.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  _CoRR_ abs/1609.08144 (2016).