Sagar ] Indian Institute of Astrophysics, Bangalore 560034 and Aryabhatta
Research Institute of Observational sciencES (ARIES), Manora Peak, Nainital,
263001, India<EMAIL_ADDRESS>
# Indo-Belgian co-operation in Astrophysics: From inception to future
prospects

Ram  [

(15th May 2023)

###### Abstract

In this manuscript, an overview of the accomplishments of the Indo-Belgian co-
operation is presented in the current era of multi-wavelength global
astronomy. About two decades ago, in the field of astronomy and astrophysics,
academicians from India and Belgium embarked on formal interaction and
collaboration. The Belgo-Indian Network for Astronomy & astrophysics (BINA),
initiated in 2014, has been very productive and its activities have set a
landmark for Indo-Belgian co-operation. Under this program, three
international workshops were conducted. Several exchange work visits were also
made among the astronomers of the two countries. Since the necessary
foundation work has already been done, continuation of the BINA activities in
future is strongly recommended.

###### keywords:

Network: BINA; telescopes: ILMT, DOT; astronomy: Galactic and Extra-galactic

##  1 Introduction

The Indo-Belgian Research and Technology co-operation germinated during
2004-2005 and became operative on November 3, 2006, when the Department of
Science and Technology (DST; Govt. of India) and the Belgian Federal Science
Policy Office (BELSPO; Govt. of Belgium) signed a Memorandum of Understanding
(MoU) in New Delhi. This MoU covers a wide range of scientific fields
including physics and astrophysics, one of the key areas for long-term co-
operation. A joint committee (JC) consisting of officials and experts from
both countries was constituted for periodic monitoring and smooth
implementation of the collaborative projects. The first Indo-Belgium JC
meeting was held in Brussels during June 23-28, 2007. The Indian Science
delegation was led by Prof. T. Ramasamy, who was the secretary of DST at that
time. To strengthen the Indo-Belgian collaborations in the area of astronomy
and astrophysics, author in the role of director, ARIES visited a few Belgian
astronomical research institutions from May 11 to 15, 2012. On May 14, 2012,
he also participated in and delivered an invited talk during the FNRS Contact
Group Astronomie & Astrophysique meeting held on astronomy day of the Royal
Observatory of Belgium (ROB) at the planetarium in Brussels. During such
meetings, an open discussion regarding the future of Belgian astronomy takes
place.

Three optical telescopes, namely the 1.3-m Devasthal Fast Optical Telescope
(DFOT), 3.6-m Devasthal Optical Telescope (DOT) and 4-m International Liquid
Mirror Telescope (ILMT), have been successfully installed at the Devasthal
observatory. Its growth path has been chronicled recently by Sagar and Gopal-
Krishna (2023). The observatory is managed and operated by the Aryabhatta
Research Institute of observational sciencES (ARIES) in Nainital (India),
which is an autonomous research institution under DST (Sagar, 2022). The
Belgians are partners in both 3.6-m DOT and 4-m ILMT. An aerial view of the
Devasthal observatory is shown in Fig. 1. The building with the sliding roof
towards the bottom right is the 1.3-m DFOT (Sagar et al., 2011), the
rectangular building towards the bottom left is the 4-m ILMT and the topmost
is the 3.6-m DOT dome and extension buildings.

Figure 1: An aerial view of the Devasthal observatory located in the central
Himalayan region of Nainital, Uttarakhand
(Longitude=$79\mbox{${}^{\circ}$}41\mbox{${}^{\prime}$}04\mbox{${}^{\prime\prime}$}$
E,
Latitude=$29\mbox{${}^{\circ}$}21\mbox{${}^{\prime}$}40\mbox{${}^{\prime\prime}$}$
N and Altitude = $2424\pm 4$ m). The housing of all the optical telescopes,
marked with their sizes, are shown.

In 2014, the Belgo-Indian Network for Astronomy and astrophysics (BINA) was
created under the above-mentioned MoU with an aim to foster collaborations in
space research between different institutions of both countries. The BINA
links researchers of the 13 Indian and 6 Belgian institutions (Joshi and De
Cat, 2019). Indian institutions, led by ARIES, Nainital (PI : Dr. Santosh
Joshi), are located in different parts of the country while both federal and
regional organizations of Belgium are participating in the activities of BINA
under the leadership of the ROB, Brussels (PI : Dr. Peter De Cat). Thus, a
good number of researchers from both countries are collaborating under BINA
and organizing both joint workshops and mutual work visits. Such activities
have strengthened research in the areas of the solar system, galactic and
extra-galactic astronomy and have also contributed to the development of back-
end instruments for Indo-Belgian telescopes.

The next section describes the participation of the Belgians in the
development of the Devasthal observatory. The main accomplishments of the
Indo-Belgian cooperation, including BINA, are summarised in Sec. 3. Finally,
in the last section, we have listed the scientific potential of the BINA along
with the possibilities of future collaborations.

##  2 Indo-Belgium partnership in both 4-m class optical telescopes

Belgium is a partner in both 3.6-m DOT and 4-m ILMT, India’s largest 4-m class
optical observational facilities. On March 27, 2007, ARIES awarded the
contract for the supply and installation of a 3.6-m aperture-size modern
optical telescope at Devasthal to Advanced Mechanical and Optical Systems
(AMOS), a globally well-known Belgian company for the delivery of precision
astronomical instruments. In the year 2008, BELSPO contributed 2 million Euros
in cash to this project. In return, Belgian astronomers are assured of 7%
observing time on this modern observing facility. This Indo-Belgian telescope
was technically activated on March 30, 2016 (Fig. 2), jointly by the premiers
of India, Shri Narendra Modi and of Belgium, Mr. Charles Michel (Omar et al.,
2017; Kumar et al., 2018; Sagar et al., 2019).

Figure 2: A photograph of technical activation of the 3.6-m DOT by the
premiers of India (Shri Narendra Modi) and Belgium (Mr. Charles Michel) on
March 30, 2016.

The University of Liège initiated the 4-m ILMT project and AMOS manufactured
its bowl and mechanical structure. The 4-m ILMT was installed at Devasthal
site in the year 2018 with the infrastructural support provided by ARIES. Fig.
3 shows a top view of the mercury mirror where the mylar cover, preventing the
formation of wavelets on the mercury surface, is clearly seen. The telescope
received its first light on April 29, 2022. During the first commissioning
phase of the telescope in April-May, 2022, photometric observations in $g,r$
and $i$ Sloan filters were acquired. A colour composite image of a small
portion of the sky is shown in Fig. 4. In this image, the NGC 4274 galaxy can
be seen in the top right corner. See Surdej et al. (2022) and Kumar et al.
(2022a) for further details. Based on the first-light ILMT observations, 14
posters were presented during the $3^{\rm rd}$ BINA workshop. The preliminary
results clearly demonstrate the significance of this low-cost telescope for a
wide variety of celestial objects including transients and space debris. The
4-m ILMT in co-ordination with the 3.6-m DOT will strengthen the Indo-Belgian
co-operation significantly. The efforts of Prof. Jean Surdej from the
University of Liège , Belgium toward the success of this project are highly
valued.

Figure 3:  A top view of the ILMT showing the liquid mercury mirror covered by
a mylar film.
Figure 4:  The NGC 4274 galaxy can be seen in the top right corner of this
colour composite image obtained from first light observations of the ILMT.

On March 21, 2023, the ILMT was inaugurated jointly online by Lt. Gen. (Retd.)
Gurmit Singh (honourable Governor of Uttarakhand) and Dr. Jitendra Singh
(honourable Minister of State for Science & Technology and Earth Sciences,
Govt. of India) in adverse weather conditions. To commemorate this momentous
occasion, several dignitaries both from India and Belgium were present either
online or in-person at the Devasthal Observatory. Dr. S. Chandrasekhar
(secretary of DST, Govt. of India) and Prof. A. S. Kiran Kumar (former
secretary of the Department of Space, Govt. of India) participated online in
the function. The event was graced by the physical presence of His Excellency
Mr. Didier Vanderhasselt (ambassador of Belgium in India), Prof. Anne-Sophie
Nyssen (rector of the University of Liège , Belgium), Dr. Ronald van der
Linden (director of the ROB, Belgium), Dr. S.K. Varshney (head of the
international division, DST, Govt. of India), Prof. Jean Surdej (project
investigator of the ILMT from University of Liège , Belgium), Mrs. Brigitte
Decadt (BELSPO, Govt. of Belgium), and Prof. Dipankar Banerjee (director of
ARIES, Nainital). Fig. 5 shows a photo of the ILMT inauguration function.
Congratulatory messages by Shri Pushkar Singh Dhami (honorable chief minister
of Uttarakhand), Shri Ajay Bhatt (minister of state, Govt. of India; member of
parliament, Nainital), and Prof. P. C. Agrawal (chairman of the governing
council, ARIES) were portrayed during the inaugural event. The dignitaries
addressed this watershed event and emphasized the astrophysical significance
of the 4-m ILMT. The global astronomical community, thus, witnessed this
historical event. On this occasion, a public talk titled The 4-m International
Liquid Mirror Telescope: A Short History was delivered by Prof. Jean Surdej.
The ILMT inaugural function ended with a welcome reception hosted by ARIES at
the Country Inn Nature Resort, Bhimtal.

Figure 5: A photo of the ILMT inauguration on 21 March 2023 at Devasthal is
shown.

##  3 Accomplishments of BINA

To achieve the objectives of BINA, three workshops were conducted along with a
number of academic exchange work visits. They are briefly discussed in the
next sub-sections. The scientific outputs of BINA are summarized in the
remaining part of this section. All these indicate that the network has been
successful but needs further strengthening by both countries.

###  3.1  Workshops, work visits and training of next generation under BINA

Three BINA workshops have been organised so far. ARIES hosted the $1^{\rm st}$
BINA Workshop on Instrumentation and Science with the 3.6-m DOT and 4.0-m ILMT
telescopes at Nainital during 15-18 November 2016 (De Cat et al., 2018). Over
100 academicians from 8 countries (India, Belgium, Russia, Japan, China, South
Africa, Thailand and Taiwan) participated in the workshop. The ROB organised
the $2^{\rm nd}$ BINA Workshop in Brussels during 9-12 October, 2018. The
theme of the workshop was BINA as an Expanding International Collaboration (De
Cat et al., 2019). A total of 65 scientists from 7 countries (India, Belgium,
France, Germany, Russia, Spain and Thailand) participated. The ROB also
organized a one-day workshop on 8 October, 2018 during the work visits of
three Indian scientists while an online ILMT workshop was organised by ARIES
from June 29 to July 1, 2020 during the pandemic. The $3^{\rm rd}$ BINA
workshop was hosted by ARIES from March 22 to 24, 2023 at the Bhimtal campus
of the Graphic Era Hill University. Further details of this workshop are
provided by Joshi et al. (2023) while its summary is given by Semenko and
Cuntz (2023). Peer-reviewed manuscripts of all three BINA workshops are
published online in the Bulletin de la Société Royale des Sciences de Liège.

Apart from above mentioned three joint international workshops, twelve work
visits by researchers from both sides were conducted. Mutual visits were
beneficial to strengthen the scientific activities and train the next
generation of young PhD students. As a part of the human resources programme,
about a dozen PhD students were/are jointly supervised by Indian and Belgian
scientists. Two PhD students, namely Brajesh Kumar from ARIES, Nainital and
Bikram Pradhan from Indian Institute of Space science and Technology (IIST),
Thiruvananthapuram were trained for the ILMT project. The University of Liège
, Belgium awarded PhD degrees to them in November 2014 and January 2020,
respectively, under the joint supervision led by Prof. Jean Surdej. About a
dozen of Master and PhD students from both sides were trained towards the
analysis of observational data and interpretation of the results. To date, a
total of 8 PhD students from various Indian institutes received 3-month
Erasmus+ mobility grants for working at the University of Liège, under a
collaborative program coordinated by Michaël De Becker. One of them, Bharti
Arora, is now working as a post-doctoral researcher funded by Wallonia-
Brussels International at the University of Liège , Belgium.

Supervision of PhD work of African students is one of the mandates of the DST
International co-operation division. Under this program, Dr. Otto Trust from
Mbarara University of Science & Technology (MUST), Uganda, completed his PhD
under the joint supervision of the Indian and Belgian PI of the BINA project
while Ms. Dorothy Museo Mwanzia from University of Nairobi and Mr. Kampindi
Felix from MUST, Uganda are registered for their PhD work under Indian and
Belgian researchers.

###  3.2 Key science results obtained under BINA projects

The 3.6-m DOT is the first major project which benefited from the Indo-Belgian
collaboration. Both optical and near infra-red (NIR) observations taken with
this telescope reveal that images of sub-arc-sec resolution can be obtained
for a significant fraction of photometric nights. They also indicate that the
performance of the telescope is comparable with its peers located elsewhere in
the world (Omar et al., 2017; Kumar et al., 2018; Sagar et al., 2019, 2020,
2022). This observing facility has led to a good number of national and
international collaborations including BINA (Joshi and De Cat, 2019). In one
such collaboration, Arora et al. (2021) combined 3.6-m DOT NIR data and
$X-$ray observations taken with the AstroSat Soft $X-$ray telescope to
investigate formation mechanisms of the colliding-wind binary WR125
(WC$7+O9$III). Analyses of these new measurements yield a period of long-term
light variations of 28–29 years and support the recurrence time of episodic
dust production. So far, the 3.6-m DOT has contributed to over 100
publications and 6 PhD theses in a number of front-line galactic and extra-
galactic astrophysical research areas, including optical follow-up of Giant
Meter Radio Telescope (GMRT) and AstroSat sources (Sagar, 2022).

The second telescope built under this collaboration is the 4-m ILMT. It was
successfully installed in 2018 (Surdej et al., 2022; Kumar et al., 2022a) and
has started routine observations after its inauguration on March 21, 2023. It
is a zenith-pointing optical observing facility at Devasthal Observatory. The
ILMT performs multi-band optical imaging of a narrow strip ($\sim$22 arcmin)
of the sky. Software pipelines for the astrometric calibration, using the
astrometric catalogues published in Dukiya et al. (2022) and Mandal et al.
(2020), and the photometric calibration are in the development and testing
phase (Kumar et al., 2022b). Other than transient astronomy, the ILMT survey
will detect and characterize the space debris which is present at altitudes
ranging from low Earth orbits (LEO - an altitude of 2000 km or less) to
geosynchronous orbits. It consists primarily of expired spacecraft, rocket
stages, separation devices, and products of the collision or breakup of
satellites. The study of their size distribution is an important input to risk
analysis for current and future space missions. Pradhan et al. (2019)
investigated these aspects with the 1.3-m DFOT telescope and found that the
4-m ILMT photometric survey may provide detections of objects having diameters
as small as 3 cm in a LEO. The ILMT team has published over a dozen papers and
contributed to two PhD theses, so far.

Optical characterization and Radial velocity monitoring with Belgian and
Indian Telescopes (ORBIT) is an ongoing project under BINA to study the
exoplanets and low-mass eclipsing binary stars. For this, the photometric
observations are taken from ARIES (Nainital; India) (Joshi et al., 2022b)
while the spectroscopic observations are taken with the Hanle Echelle
Spectrograph at the 2-m Himalayan Chandra Telescope (Hanle; India) (Sriram et
al., 2018) and the High Efficiency and high-Resolution Mercator Echelle
Spectrograph (HERMES) mounted at the 1.2-m Mercator Telescope, La Palma;
Canary Islands in Spain (Raskin et al., 2011). The combined observations are
used to estimate basic parameters of the binary systems (e.g. orbital periods,
masses of the components, radii) (Panchal et al., 2023).

At the beginning of the year 2000, a bilateral research project entitled “The
Nainital-Cape survey” was jointly initiated by the scientists from the ARIES
in India and from the South African Astronomical Observatory in South Africa.
It was supported by DST (Govt. of India) and the National Research Foundation
(Govt. of South Africa). This was one of the largest ground-based photometric
surveys ever taken with the aim to search and study the photometric
variability in a large sample of chemically peculiar (CP) stars (Ashoka et al.
2000, Martinez et al. 2001; Joshi et al. 2003, 2006, 2009, 2010, 2012, 2016,
2017). By combining time-series of ground- and space-based photometry with
high-resolution spectroscopy of a set of samples observed using multiple
telescopes around the globe, Joshi et al. (2022a) discovered a new heartbeat
system (HD 73619), for which no pulsation signatures are seen. Such works not
only aim to study the stellar structure and atmospheres of CP stars in the
presence of magnetic fields, inhomogeneities (such as spots), and tidal
interactions but also demonstrate the importance of international (13
countries) co-operation led by Indians and Belgians.

Another active area of the collaboration has been the photometric and
spectroscopic study of eclipsing binaries with the aim to understand the
formation mechanism of these stars at different stages of their evolution. For
example, 56 UMa is a wide binary system containing a CP red giant with a faint
companion. To unravel the nature of its faint companion, Escorza et al. (2023)
revisited the orbital parameters of the system and carried out a detailed
spectral analysis including high-resolution HERMES spectra. This study
indicates that unseen component in 56 UMa has a mass of 1.31$\pm
0.12M_{\odot}$, which is compatible with the mass of both a white dwarf (WD)
and a neutron star. However, some observations are in favor of the latter
interpretation.

Magnetic cataclysmic variables (MCVs) are interacting semi-detached binaries
consisting of a magnetic WD as the primary and a Roche lobe filling star as
the secondary. The magnetic field strength of the WDs divides the MCVs in two
sub-classes namely intermediate polars and polars. Based on detailed optical
and $X-$ray timings and a spectral study of two candidate MCVs, namely 1RXS
J174320.1-042953 and YY Sex, Rawat et al. (2023) conclude that these MCV
candidates belong to the polar sub-class of MCVs.

Gravitational microlensing by compact objects in lensing galaxies is a known
tool for probing the structure of distant quasars on sub-parsec scales. For
this purpose, Hutsemékers et al. (2020) used the Very Large Telescope of the
European Southern Observatory to obtain spectropolarimetric observations of
the two images of the broad absorption line (BAL) quasar SDSS
$J081830.46+060138.0$ at redshift $z=$2.35. These observations indicate that
the underlying source is actually gravitationally lensed and not a binary
quasar. Detection of an absorption system at z=1.0065$\pm$0.0002 might reveal
the lensing galaxy. The authors also found that this is the second BAL quasar
in which an extended source of the rest-frame ultraviolet continuum is found.
This study provides constraints on the BAL flow and finds that the outflow is
seen with a nonzero onset velocity stratified according to ionization.

Benaglia et al. (2020) used the GMRT observations at frequencies 325 and 610
MHz, for a detailed study of 11 early-type stars including both WR and O-type
systems, located in $\sim$ 15 degree2 area of the sky centred on the Cygnus
region and identified two additional particle-accelerating colliding-wind
binaries, namely Cyg OB2 12 and ALS 15108 AB. This project was part of a
collaborative network led by Michël De Becker from Belgium (PANTERA-Stars)
with Indian scientists from National Center of Radio Astronomy, Pune and IIST,
Thiruvananthapuram who are making use of the GMRT observing facility for the
multi-wavelength studies of systems consisting of massive stars active at
accelerating particles up to relativistic velocities.

##  4 Conclusion and summary

Especially during the last decade, the Indo-Belgian co-operation and BINA
activities have been very productive and scientifically rewarding. This
fruitful co-operation helped and lead to the installation of two 4-m class
telescopes at Devasthal Observatory, being the largest optical ones in India.
Not only a good number of next-generation young minds have been trained but
also over 50 joint papers have been published in peer-reviewed high-impact
international journals (including 25 in main journals and 26 in the
proceedings of the $1^{\rm st}$ and $2^{\rm nd}$ BINA workshops). As a result,
BINA has grown into a household name in the astronomical community within
Belgium and India that aims to stimulate all forms of Indo-Belgian research in
astronomy and space sciences. All the network activities, supported by the
funding agencies DST (at the Indian side) and BELSPO (at the Belgian side),
have created a strong foundation for its future. Latest addition to these
activities is the creation of the network Belgo-Indian Projects on precision
Astronomical spectroscopy for Stellar and Solar system bodies (BIPASS) in
2022. It is led by Prof. Shashikiran Ganesh from Physical Research Laboratory,
Ahmadabad (India) and Dr. Laurent Mahy from ROB, Brussels (Belgium). There are
5 Indian and 4 Belgian partners in the BIPASS project. Research in atmospheric
sciences is one of the areas of common interest between Indian and Belgian
scientists that has not been explored yet, but BINA intends to include it in
their domain of activities too. All these portray a bright future for BINA co-
operation.

###### Acknowledgements.

Constructive suggestions made by Prof. Edward Guinan, reviewer of the
manuscript, are thankfully acknowledged. This work is supported by the Belgo-
Indian Network for Astronomy and astrophysics (BINA), approved by the
International Division, Department of Science and Technology (DST, Govt. of
India; DST/INT/BELG/P-09/2017) and the Belgian Federal Science Policy Office
(BELSPO, Govt. of Belgium; BL/33/IN12). The author acknowledges financial
support provided by the organizing committee of the $3^{\rm rd}$ BINA
workshop. He is also grateful to the National Academy of Sciences, Prayagraj,
India for the award of Honorary Scientist position, the Alexander von Humboldt
Foundation (Germany) for the award of long-term group research linkage
program, and director of the Indian Institute of Astrophysics; Bangalore,
India for providing infrastructural support during this work. Valuable inputs
provided by Drs. Peter De Cat, Michël De Becker, Santosh Joshi and Kuntal
Misra are thankfully acknowledged.

0000-0003-4973-4745Ram Sagar

The authors declare no conflict of interest.

## References

  * Arora et al. (2021) Arora, B., Pandey, J. C., De Becker, M., Pandey, S. B., Chakradhari, N. K., Sharma, S. and Kumar, B. (2021) Quest for the Upcoming Periastron Passage of an Episodic Dust Maker and Particle-accelerating Colliding-wind Binary: WR 125.  AJ, 162(6), 257.  https://doi.org/10.3847/1538-3881/ac2506.
  * Ashoka et al. (2000) Ashoka, B. N., Seetha, S., Raj, E., Chaubey, U. S., Gupta, S. K., Joshi, S., Martinez, P., Kurtz, D. W., Sagar, R. and Kasturirangan, K. (2000) Survey for roAp stars in the northern sky from Naini Tal.  BASI, 28, 251–254.
  * Benaglia et al. (2020) Benaglia, P., Ishwara-Chandra, C. H., Intema, H., Colazo, M. E. and Gaikwad, M. (2020) Cygnus survey with the Giant Metrewave Radio Telescope at 325 and 610 MHz: the catalog.  A&A, 642, A136.  https://doi.org/10.1051/0004-6361/202037916.
  * De Cat et al. (2019) De Cat, P., Lampens, P., Joshi, S., De Becker, M. and Jorissen, A. (2019) Proceedings of the 2nd BINA workshop: BINA as an expanding international collaboration.  BSRSL, 88, 1–18.
  * De Cat et al. (2018) De Cat, P., Surdej, J., Omar, A., De Becker, M. and Joshi, S. (2018) Proceedings of the 1st BINA workshop: Instrumentation and Science with the 3.6-m DOT and 4-m ILMT telescopes.  BSRSL, 87, 1–14.
  * Dukiya et al. (2022) Dukiya, N., Misra, K., Pradhan, B., Negi, V., Ailawadhi, B., Kumar, B., Hickson, P. and Surdej, J. (2022) Astrometric and Photometric Standard Candidates for the Upcoming 4-m International Liquid Mirror Telescope Survey.  JAI, 11(4), 2240001.  https://doi.org/10.1142/S2251171722400013.
  * Escorza et al. (2023) Escorza, A., Karinkuzhi, D., Jorissen, A., Van Eck, S., Schmelz, J. T., Verschuur, G. L., Boffin, H. M. J., De Rosa, R. J. and Van Winckel, H. (2023) A neutron star candidate in the long-period binary 56 UMa.  A&A, 670, L14.  https://doi.org/10.1051/0004-6361/202245796.
  * Hutsemékers et al. (2020) Hutsemékers, D., Sluse, D. and Kumar, P. (2020) Spatially separated continuum sources revealed by microlensing in the gravitationally lensed broad absorption line quasar SDSS J081830.46+060138.0.  A&A, 633, A101.  https://doi.org/10.1051/0004-6361/201936973.
  * Joshi and De Cat (2019) Joshi, S. and De Cat, P. (2019) Overview of the BINA Activities.  BSRSL, 88, 19–30.  https://doi.org/10.48550/arXiv.1911.00346.
  * Joshi et al. (2023) Joshi, S., De Cat, P., De Beker, M., Kolenberg, K., Mondal, S., Ganesh, S., Mahy, L., Karinkuzhi, D., Yadav, A. P., Baug, T., Pandey, J. C., Misra, K., Stalin, C. S., Veerabathina, N., Joshi, B., Bergman, D. and Doorsselaere, T. V. (2023) Proceedings of the 3rd BINA Workshop: Scientific Potential of Indo-Belgian Co-operation.  BSRSL, 92, ??–??
  * Joshi et al. (2003) Joshi, S., Girish, V., Sagar, R., Kurtz, D. W., Martinez, P., Kumar, B., Seetha, S., Ashoka, B. N. and Zhou, A. (2003) Discovery of unusual pulsations in the cool, evolved Am stars HD 98851 and HD 102480.  MNRAS, 344(2), 431–438.  https://doi.org/10.1046/j.1365-8711.2003.06823.x.
  * Joshi et al. (2016) Joshi, S., Martinez, P., Chowdhury, S., Chakradhari, N. K., Joshi, Y. C., van Heerden, P., Medupe, T., Kumar, Y. B. and Kuhn, R. B. (2016) The Nainital-Cape Survey. IV. A search for pulsational variability in 108 chemically peculiar stars.  A&A, 590, A116.  https://doi.org/10.1051/0004-6361/201527242.
  * Joshi et al. (2009) Joshi, S., Mary, D. L., Chakradhari, N. K., Tiwari, S. K. and Billaud, C. (2009) The Nainital-Cape Survey. III. A search for pulsational variability in chemically peculiar stars.  A&A, 507(3), 1763–1784.  https://doi.org/10.1051/0004-6361/200912382.
  * Joshi et al. (2006) Joshi, S., Mary, D. L., Martinez, P., Kurtz, D. W., Girish, V., Seetha, S., Sagar, R. and Ashoka, B. N. (2006) The Nainital-Cape Survey. II. Report for pulsation in five chemically peculiar A-type stars and presentation of 140 null results.  A&A, 455(1), 303–313.  https://doi.org/10.1051/0004-6361:20064970.
  * Joshi et al. (2010) Joshi, S., Ryabchikova, T., Kochukhov, O., Sachkov, M., Tiwari, S. K., Chakradhari, N. K. and Piskunov, N. (2010) Time-resolved photometric and spectroscopic analysis of the luminous Ap star HD 103498.  MNRAS, 401(2), 1299–1307.  https://doi.org/10.1111/j.1365-2966.2009.15725.x.
  * Joshi et al. (2012) Joshi, S., Semenko, E., Martinez, P., Sachkov, M., Joshi, Y. C., Seetha, S., Chakradhari, N. K., Mary, D. L., Girish, V. and Ashoka, B. N. (2012) A spectroscopic analysis of the chemically peculiar star HD 207561\.  MNRAS, 424(3), 2002–2008.  https://doi.org/10.1111/j.1365-2966.2012.21340.x.
  * Joshi et al. (2017) Joshi, S., Semenko, E., Moiseeva, A., Sharma, K., Joshi, Y. C., Sachkov, M., Singh, H. P. and Yerra, B. K. (2017) High-resolution spectroscopy and spectropolarimetry of selected $\delta$ Sct pulsating variables.  MNRAS, 467(1), 633–645.  https://doi.org/10.1093/mnras/stx087.
  * Joshi et al. (2022a) Joshi, S., Trust, O., Semenko, E., Williams, P. E., Lampens, P., De Cat, P., Vermeylen, L., Holdsworth, D. L., Garcia, R. A., Mathur, S., Santos, A. R. G., Mkrtichian, D., Goswami, A., Cuntz, M., Yadav, A. P., Sarkar, M., Bhatt, B. C., Kahraman Alicavus, F., Nhlapo, M. D., Lund, M. N., Goswami, P. P., Savanov, I., Jorissen, A., Jurua, E., Avvakumova, E., Dmitrienko, E. S., Chakradhari, N. K., Das, M. K., Chowdhury, S., Abedigamba, O. P., Yakunin, I., Letarte, B. and Karinkuzhi, D. (2022a) Study of chemically peculiar stars-I. High-resolution spectroscopy and K2 photometry of Am stars in the region of M44.  MNRAS, 510(4), 5854–5871.  https://doi.org/10.1093/mnras/stab3158.
  * Joshi et al. (2022b) Joshi, Y., Bangia, T., Jaiswar, M., Pant, J., Reddy, K. and Yadav, S. (2022b) ARIES 130-cm Devasthal Fast Optical Telescope: Operation and Outcome.  JAI, 11(4), 2240004.  https://doi.org/10.1142/S2251171722400049.
  * Kumar et al. (2022a) Kumar, B., Kumar, H., Dangwal, K. S., Rawat, H., Misra, K., Negi, V., Jaiswar, M. K., Dukiya, N., Ailawadhi, B., Hickson, P. and Surdej, J. (2022a) First Light Preparations of the 4m ILMT.  JAI, 11(4), 2240003.  https://doi.org/10.1142/S2251171722400037.
  * Kumar et al. (2022b) Kumar, B., Negi, V., Ailawadhi, B., Mishra, S., Pradhan, B., Misra, K., Hickson, P. and Surdej, J. (2022b) Upcoming 4m ILMT facility and data reduction pipeline testing.  JApA, 43(1), 10.  https://doi.org/10.1007/s12036-021-09795-3.
  * Kumar et al. (2018) Kumar, B., Omar, A., Maheswar, G., Pandey, A. K., Sagar, R., Uddin, W., Sanwal, B. B., Bangia, T., Kumar, T. S., Yadav, S., Sahu, S., Pant, J., Reddy, B. K., Gupta, A. C., Chand, H., Pandey, J. C., Joshi, M. K., Jaiswar, M., Nanjappa, N., Purushottam, Yadav, R. K. S., Sharma, S., Pandey, S. B., Joshi, S., Joshi, Y. C., Lata, S., Mehdi, B. J., Misra, K. and Singh, M. (2018) 3.6-m Devasthal Optical Telescope Project: Completion and first results.  BSRSL, 87, 29–41.
  * Mandal et al. (2020) Mandal, A. K., Pradhan, B., Surdej, J., Stalin, C. S., Sagar, R. and Mathew, B. (2020) Quasar catalogue for the astrometric calibration of the forthcoming ILMT survey.  JApA, 41(1), 22.  https://doi.org/10.1007/s12036-020-09642-x.
  * Martinez et al. (2001) Martinez, P., Kurtz, D. W., Ashoka, B. N., Chaubey, U. S., Girish, V., Gupta, S. K., Joshi, S., Kasturirangan, K., Sagar, R. and Seetha, S. (2001) The Naini Tal - Cape survey for pulsations in chemically peculiar A-type stars. I. Methods and preliminary results.  A&A, 371, 1048–1055.  https://doi.org/10.1051/0004-6361:20010432.
  * Omar et al. (2017) Omar, A., Kumar, B., Gopinathan, M. and Sagar, R. (2017) Scientific capabilities and advantages of the 3.6 meter optical telescope at Devasthal, Uttarakhand.  CSci, 113(4), 682–685.  https://doi.org/10.18520/cs/v113/i04/682-685.
  * Panchal et al. (2023) Panchal, A., Joshi, Y. C., De Cat, P., Lampens, P., Goswami, A. and Tiwari, S. N. (2023) Optical characterization and radial velocity monitoring with Belgian and Indian telescopes (ORBIT): the eclipsing binaries EPIC 211982753 and EPIC 211915147.  MNRAS, 521(1), 677–689.  https://doi.org/10.1093/mnras/stad533.
  * Pradhan et al. (2019) Pradhan, B., Hickson, P. and Surdej, J. (2019) Serendipitous detection and size estimation of space debris using a survey zenith-pointing telescope.  AcA, 164(77).  https://doi.org/htps://doi.org/10.1016/j.actaastro.2019.07.008.
  * Raskin et al. (2011) Raskin, G., van Winckel, H., Hensberge, H., Jorissen, A., Lehmann, H., Waelkens, C., Avila, G., de Cuyper, J. P., Degroote, P., Dubosson, R., Dumortier, L., Frémat, Y., Laux, U., Michaud, B., Morren, J., Perez Padilla, J., Pessemier, W., Prins, S., Smolders, K. and van Eck, S. (2011) HERMES: a high-resolution fibre-fed spectrograph for the Mercator telescope.  A&A, 526, A69.  https://doi.org/10.1051/0004-6361/201015435.
  * Rawat et al. (2023) Rawat, N., Pandey, J. C., Joshi, A., Potter, S. B., Hojaev, A. S., De Becker, M., Rao, S. M. and Yadava, U. (2023) Confirmation of two magnetic cataclysmic variables as polars: 1RXS J174320.1-042953 and YY Sex.  MNRAS, 521(2), 2729–2744.  https://doi.org/10.1093/mnras/stad570.
  * Sagar (2022) Sagar, R. (2022) History of ARIES: A premier research institute in the area of observational sciences.  InJHS, 57, 227–247.  https://doi.org/10.1007/s43539-022-00054-0.
  * Sagar and Gopal-Krishna (2023) Sagar, R. and Gopal-Krishna, K. (2023) Pathway to Devasthal Astronomical Observatory, ARIES.  InJHS, 58, accepted.
  * Sagar et al. (2019) Sagar, R., Kumar, B. and Omar, A. (2019) The 3.6 meter Devasthal Optical Telescope: From inception to realisation.  CSci, 117, 365–381.  https://doi.org/10.18520/cs/v117/i3/365-381.
  * Sagar et al. (2020) Sagar, R., Kumar, B. and Sharma, S. (2020) Observations with the 3.6-meter Devasthal optical telescope.  JApA, 41(1), 33.  https://doi.org/10.1007/s12036-020-09652-9.
  * Sagar et al. (2011) Sagar, R., Omar, A., Kumar, B., Gopinathan, M., Pandey, S. B., Bangia, T., Pant, J., Shukla, V. and Yadava, S. (2011) The new 130-cm Optical telescope at Devasthal, Nainital.  CSci, 101(8), 1020–1023.
  * Sagar et al. (2022) Sagar, R., Yadav, R. K. S., Pandey, S. B., Sharma, S., Lata, S. and Joshi, S. (2022) Optical observations of star clusters NGC 1513 and NGC 4147; white dwarf WD 1145+017 and K band imaging of star-forming region Sh 2-61 with the 3.6-m Devasthal optical telescope.  JApA, 43(1), 31.  https://doi.org/10.1007/s12036-022-09815-w.
  * Semenko and Cuntz (2023) Semenko, E. and Cuntz, M. (2023) Summary of the 3rd BINA Workshop.  BSRSL, 92, ??–??
  * Sriram et al. (2018) Sriram, S., Kumar, A., Surya, A., Sivarani, T., Giridhar, S., Kathiravan, S., Anand, M. N., Jones, D., Grobler, D., Jakobsson, R., Chanumolu, A., Unni, A., Dorje, A., Dorje, T. and Gyalson, T. (2018) Hanle echelle spectrograph: design and performance.  Proceedings of the SPIE 10702, Ground-based and Airborne Instrumentation for Astronomy VII, 10702(10702), 107026K.  https://doi.org/10.1117/12.2313165.
  * Surdej et al. (2022) Surdej, J., Hickson, P., Kumar, B. and Misra, K. (2022) First Light with the 4-m International Liquid Mirror Telescope.  Physics News, 52(3), 25–28.


# A Dataset of Information-Seeking Questions
and Answers Anchored in Research Papers

Pradeep Dasigi♣ Kyle Lo♣ Iz Beltagy♣ Arman Cohan♣
Noah A. Smith♢♣ Matt Gardner♣
♣Allen Institute for AI ♢Paul G. Allen School of CSE, University of Washington
<EMAIL_ADDRESS>
###### Abstract

Readers of academic research papers often read with the goal of answering
specific questions. Question Answering systems that can answer those questions
can make consumption of the content much more efficient. However, building
such tools requires data that reflect the difficulty of the task arising from
complex reasoning about claims made in multiple parts of a paper. In contrast,
existing information-seeking question answering datasets usually contain
questions about generic factoid-type information. We therefore present Qasper,
a dataset of 5,049 questions over 1,585 Natural Language Processing papers.
Each question is written by an NLP practitioner who read only the title and
abstract of the corresponding paper, and the question seeks information
present in the full text. The questions are then answered by a separate set of
NLP practitioners who also provide supporting evidence to answers. We find
that existing models that do well on other QA tasks do not perform well on
answering these questions, underperforming humans by at least 27 $F_{1}$
points when answering them from entire papers, motivating further research in
document-grounded, information-seeking QA, which our dataset is designed to
facilitate.

##  1 Introduction

Machines built to assist humans who engage with texts to seek information
ought to be designed with an awareness of the information need. Abstractly,
the human’s need should define the lens through which the system views the
text in order to find desired information. Existing information-seeking
machine reading datasets (e.g., Kwiatkowski et al., 2019; Clark et al., 2020)
have led to significant progress in reading at scale (e.g., Asai et al., 2020;
Guu et al., 2020; Liu et al., 2020). However, most of those benchmarks focus
on an “open domain” setting where the questions are not anchored in any
particular user context. The result is an emphasis on generic factoid
questions, rather than the full range of information needs people have.

Figure 1: An example instance taken from Qasper. A question about the paper is
written after reading only the title and the abstract. To arrive at the
answer, one finds relevant evidence, which can be spread across multiple
paragraphs. In this example, to answer the question about “baselines”, the
reader must realize from evidence from Sections 3 and 4 that “context
documents” come pre-ranked in the dataset and the paper’s “baselines” select
from these “context documents.”

We present Qasper,111Loosely derived from Question Answering over Scientific
Research Papers. The dataset, baseline code, and other information about the
project can be found at https://allenai.org/project/qasper. an information-
seeking question answering (QA) dataset over academic research papers. Each
question is written as a follow-up to the title and abstract of a particular
paper, and the answer, if present, is identified in the rest of the paper,
along with evidence required to arrive at it. This setup results in questions
requiring more complex document-level reasoning than prior datasets, because
_(i)_ abstracts provide rich prompts for questions that can be asked as
follow-up and _(ii)_ academic research papers naturally trigger questions by
their target readers that require supporting or refuting claims. This evidence
may be spread across the paper, including tables and figures, often resulting
in complex entailment problems. The example in Figure 1 illustrates one such
case where we need to retrieve information from paragraphs in three different
sections to answer the question.

Qasper contains 5,049 questions over 1,585 natural language processing (NLP)
papers, asked by regular readers of NLP papers, and answered by a separate set
of NLP practitioners. Each paper has an average of 3.2 questions, up to a
maximum of 12 questions for a single paper. In addition to providing answers
when the questions are answerable, the annotators were asked to select text,
tables, or figures as evidence required for answering the questions. 55.5% of
the questions require evidence from multiple paragraphs in the paper and 13%
require tables or figures. To the best of our knowledge, Qasper is the first
QA dataset in the academic research domain focusing on entire papers, and not
just abstracts.

To quantify the difficulty of the tasks in Qasper, we apply state-of-the-art
document-level Transformer Vaswani et al. (2017) models to the tasks of
selecting evidence and generating answers, and show that the best model
performance lags behind humans by 27 $F_{1}$ points at answering questions
from entire papers, and 32 $F_{1}$ points at selecting the paragraphs that
provide evidence to answer the questions, indicating that these are both
unsolved problems. Additionally, we experiment with oracles that answer
questions from gold evidence and find that better pretraining and domain-
adaptation might be helpful.

##  2 Building the Qasper Dataset

We now describe our process for constructing the dataset. We began with a set
of open-access NLP papers, recruited NLP practitioners who are regular readers
of research papers, and designed two different data collection interfaces: one
for collecting follow-up questions given titles and abstracts, and another for
obtaining evidence and answers to those questions.

###  2.1 Papers

We filtered S2ORC Lo et al. (2020),222We accessed both release versions
20190928 and 20200705v1. a collection of machine-readable full text for open-
access papers, to _(i)_ those from arXiv with an associated LaTeX source
file,333LaTeX allows us to avoid quality issues with PDF parsing. and _(ii)_
are in the computational linguistics domain.444We chose those either tagged
with the cs.CL arXiv category or published with an ACL Anthology identifier.
We limited our domain to computational linguistics to ensure high quality as
we have access to realistic users through our research network; broader domain
collection is left to future work and should be enabled by the proof-of-
concept of our protocols given in this paper. We used the S2ORC parser (which
normalizes multi-file LaTeX sources and resolves comments and macros) to
convert LaTeX markup to full text while preserving section and paragraph
breaks and math equations. We supplemented the paper text with extracted
images of figures and tables associated with their captions; these were
crawled from Semantic Scholar.555http://semanticscholar.org The result of this
process was a collection of 18K full text papers for annotation.

###  2.2 Decoupled Data Collection

To ensure that our questions are realistic, we decoupled the question-writing
and question-answering phases. For both tasks we recruited graduate students
studying NLP and freelancers practicing NLP through professional networks and
Upwork666https://www.upwork.com/. All the workers were regular readers of NLP
papers, and were paid US$25 per hour on average ($20-$40 based on experience).
We paid them on a per-hour basis and not a per-question basis to prioritize
data quality over quantity. A total of 25 workers wrote questions while 51
answered them.

#### Questions

To ensure that annotators were actually interested in the paper they are
reading, we provided them with a lightweight search interface to search papers
from the aforementioned collection to focus on their papers of interest. The
interface supports entering manual queries and examples of the queries
annotators used include general (e.g., “computer vision”) or specific (e.g.,
“question answering”, “information extraction”) areas of study, specific tasks
(e.g., “language identification”), entities (e.g., “bert”, “transformers”) or
concepts (e.g., “commonsense”, “interpretability”), or domain specifications
(e.g., “medical”, “wikipedia”). Annotators also had the option to not enter
any search queries; in this case, they were shown random papers. Annotators
were displayed only the title and abstracts of relevant papers and asked to
write any number of questions they had about the paper. Annotators were
instructed to only write questions that are _not_ answerable from the title
and abstract but expected to be answered somewhere in the paper. Annotators
also provided basic information about their expertise in NLP and how familiar
they already were with the paper for which they asked questions. Most workers
(about 70%) had some experience in NLP, with 20% having more than five years
of experience. A vast majority (94%) of the abstracts were seen by the
question-writers for the first time.

#### Answers

Annotators were randomly assigned papers with all the corresponding questions
written for that paper. They were shown the paper title, abstract, question,
full text, and all associated figures and tables to answer the questions.
After reading these, annotators were were asked to:

  * •

Make a binary decision as to whether the question is answerable given the
paper.

  * •

If the question is answerable, select the minimal set of _evidence_ snippets
that contains the answer to the question. This could be (possibly
discontiguous) paragraphs from the text and/or figures or tables. Annotators
were asked to prioritize text over figures and tables, unless the information
required was present only in figures or tables. When multiple paragraphs could
serve as evidence, annotators were asked to first prioritize evidence that
adequately answered the question, and then paragraphs that occurred earlier in
the text.

  * •

If the question is answerable, also provide a concise _answer_ to the
question. Annotators were also asked to also indicate whether their concise
answer was _(i)_ extracted from the evidence, _(ii)_ “yes” or “no”, or _(iii)_
abstractively written.

Annotators were allowed to skip any questions they did not feel comfortable
answering. Since the answering task is significantly more complex than the
question-writing task, we designed interactive tutorials and qualification
exams for the workers for this task using CrowdAQ (Ning et al., 2020). Workers
who scored well were invited to work on the task. If the test performance
indicated that the workers did not have sufficient NLP knowledge, or were not
used to reading papers we did not let them work on the task. In cases where
the workers misunderstood the task, but had sufficient background knowledge,
we provided additional training before letting them work on the task.

##  3 Qasper Analysis

Table 3 provides representative examples from Qasper categorized by question,
answer, and evidence types, which we describe here in greater detail.

Question |  Type |  % |  Paper(s)
---|---|---|---
What datasets do they use? |  General |  33.3% |  1; 2; 3
[0.4pt/2pt] What other political events are included in the database? |  Specific |  66.7% |  1706.01875 |
Question |  Answer |  Type |  % |  Paper
What five dialogue attributes were analyzed? |  Model; Confidence; Continuity; Query-relatedness; Repetitiveness; Specificity |  Extractive |  51.8% |  1705.00571
[0.4pt/2pt] Which neural architecture do they use as a base for their attention conflict mechanisms? |  GRU-based encoder, interaction block, and classifier consisting of stacked fully-connected layers. |  Abstractive |  24.2% |  1906.08593
[0.4pt/2pt] Do they ensure the that the architecture is differentiable everywhere after adding the Hungarian layer? |  Yes |  Yes/No |  13.9% |  1712.02555
[0.4pt/2pt] What language are the captions in? |  N/A |  Unanswer. |  10.2% |  1909.09070
Question |  Evidence |  Type |  % |  Paper
What new tasks do they use to show the transferring ability of the shared meta-knowledge? |  To test the transferability of our learned Meta-LSTM, we also design an experiment, in which we take turns choosing 15 tasks to train our model with multi-task learning, then the learned Meta-LSTM are transferred to the remaining one task. The parameters of transferred Meta-LSTM, $\theta^{(s)}_{m}$ in Eq.( 33 ), are fixed and cannot be updated on the new task. |  Text |  81.6% |  1802.08969
[0.4pt/2pt] How much does it minimally cost to fine-tune some model according to benchmarking framework? |  Table 1 |  Table/Figure |  11.6% |  2002.05829
[0.4pt/2pt] Do they recommend translating the premise and hypothesis together? |  N/A |  None |  12.8% |  2004.04721

Table 1: Examples of questions (top), answers (middle), and evidence (bottom)
sampled from Qasper. % are relative frequencies of the corresponding type over
all examples in Qasper. The percentages for evidence types sum over 100% due
to double-counting of 446 answers with both Table/Figure and Text evidence.

#### Question types

We first analyze whether our annotation setup results in questions that are
anchored in the context of the papers. To answer this question, we
manually777Two domain-experts independently judged these, and achieved a
Cohen’s $\kappa$ of 0.94. categorized a set of 200 questions as being
applicable to most papers in the domain (general) vs. being applicable only to
the paper that the question is written about (specific). Table 3 shows that
most of the questions (67%) are specific to the papers they are written about.
This result indicates the advantage of viewing the Qasper task as a question
answering problem, instead of an information extraction problem since a fixed
schema would not be able to handle the long tail of paper-specific information
needs.

#### Answer types

As shown in Table 3, most of the answers in the dataset are extractive. The
average length of the extractive answers is 14.4 words (including all spans),
and that of abstractive spans is 15.6 words.

#### Evidence types

Evidence can include one or more paragraphs from the paper, a figure, or a
table, or a combination of these. Table 3 shows the distribution of these
types. Among the answerable questions with text-only evidence, 55.5% of the
answers have multi-paragraph evidence (Figure 1 is one example). Unanswerable
questions do not have any evidence. Among the answerable ones, (3.0%) have no
evidence when the answer is No, and the evidence is the _lack_ of a mention of
something specific. The last question in Table 4 is one example of such a
case.

#### Distribution of evidence paragraphs

We perform an analysis to identify the main sections of a paper that contain
textual evidence. We assign each evidence paragraph to its containing top-
level888S2ORC provides section hierarchy derived from LaTeX source section,
and perform some section name normalization. We find that among the frequently
used section names such as “Experiments” and “Introduction,” there was not a
single section name that contained a majority of evidence spans, indicating
that the distribution of evidence over section in the paper was more or less
uniform.

#### Inter-annotator agreement

44% of the questions in Qasper have multiple annotated answers. On average,
each question is answered by 1.6 annotators (up to a maximum of 6 annotators
for the same question). Using these multiple annotations, we compute some
measures of agreement between annotators. First, we found that there is a high
level of agreement (90%) regarding answerability of questions. Second, we find
that annotators agreed on the type of the evidence (text vs. figure) in 84.0%
of the cases. Papers often provide the same information both in tables and
text, and agreement over the evidence types could be a consequence of our
clear annotation guidelines regarding selecting evidence.

#### Correctness

To estimate the correctness of the answer annotations in Qasper, we manually
analyzed 100 randomly sampled questions with multiple answer annotations
(averaging 2.73 answers per question). We found that 207 (75.8%) of the
answers were correct. 98% of the questions had at least one correct answer,
and 77% had most of the answers correct.

##  4 Modeling Qasper

This section explains the task, evaluation metrics, and a model addressing
Qasper tasks.

###  4.1 Task Setup

We formally define the Qasper tasks as follows: Given a paper, and a question
about it, the primary task is to determine if the question is answerable, and
output a predicted answer, that is one or more spans in the full-text of the
paper, yes, no or other free-form text. A system built for this will be
evaluated based on the correctness of the predicted answer measured against
the reference answers. Since Qasper also provides labeled evidence for all
questions, the system may also use auxiliary supervision provided by the
evidence.

One such auxiliary task is to predict the evidence required for the question.
The inputs are the same as that of the primary task, but the outputs are
expected to be one or more paragraphs in the full-text, figures, or tables,
and they will be evaluated against labeled evidence spans.

#### Evaluation metrics

As an automatic proxy for the measure of correctness of all types of answers,
we use the span-level $F_{1}$ measure proposed by Rajpurkar et al. (2016). We
convert answers that are multiple selected spans into single comma-separated
strings. For questions with multiple reference answers, we compute the max
span-$F_{1}$ of the predictions over all the references. We evaluate the
performance of a system over the auxiliary task by computing a $F_{1}$ score
over the set of paragraphs, figures, and tables chosen by the system against
the reference evidence, considering a max when there are multiple references.
We refer to these metrics as Answer-$F_{1}$ and Evidence-$F_{1}$,
respectively.

#### Data splits

We split the dataset into train, validation, and test sets, so that each paper
appears in only one of them. Our analysis of correctness of annotations
presented in Section 3 indicates a high likelihood (98%) of evaluating against
a correct reference when evaluation is aggregated over multiple references.
Hence we ensure that most of the questions in validation and test sets have
multiple references (98% in test, and 74% in validation). This resulted in
2,593, 1,005, and 1,451 questions in the three sets, respectively.

#### Estimating human performance

To estimate an upper bound on model performance given our data splits and
metrics, we assess the performance of the workers when evaluated against each
other using the same metrics on a sample of the test set. Since model
performance is evaluated by aggregating over multiple references, we consider
a subset of the test set containing questions with at least three references
(40% of the test set), evaluate each reference against the remaining, and
compute an average over all such combinations. This procedure estimates the
human performance to be 60.9 Answer-$F_{1}$, and 71.6 Evidence-$F_{1}$. Note
that given the disagreements among the workers estimated in Section 3, this is
a lower bound on human performance for two reasons: first, because only two
annotations are used to compute the metric, while systems are evaluated
against all three; and second, because the annotators are NLP practitioners,
not expert researchers, and it is likely that an expert would score higher.
Hence we report these numbers, along with a breakdown over answer types in
Table 2 and Table 3 as human performance lower bounds.

###  4.2 Qasper Model

We base our model on pretrained Transformer Vaswani et al. (2017) models which
currently produce state-of-the-art results on a majority of QA
tasks.999https://paperswithcode.com/task/question-answering Recall that Qasper
introduces two main modeling challenges – different answer types and long
input documents.

First, Qasper includes a variety of answer types, including extractive,
abstractive, yes/no, and unanswerable questions, which means a typical span-
selection BERT-based QA model Devlin et al. (2019) is not sufficient to
support all these answer types. We address this by converting all answer types
into a single task: generating answer text Raffel et al. (2020); Khashabi et
al. (2020).101010We tried a model that predicts answer type, then based on the
type uses a different head to predict the corresponding answer. This model
performed much worse than the proposed seq2seq formulation. This is a
sequence-to-sequence formulation that requires an encoder-decoder Transformer
model where the encoder reads the question and the document and the decoder
generates the answer text.

Second, research papers are much longer than the typical 512 or 1024 token
limit of most BERT-like models, so we need a Transformer model that can
process long inputs. We use the Longformer-Encoder-Decoder (LED; Beltagy et
al., 2020), an encoder-decoder Transformer model that can efficiently process
input sequences thousands of tokens long. With LED’s support for input
sequence length of 16K tokens, we can encode 99% of the paper full texts in
the Qasper dataset without truncation.

#### Longformer-Encoder-Decoder (LED)

LED Beltagy et al. (2020) is a variant of the original Transformer encoder-
decoder model that replaces the Transformer’s full self-attention in the
encoder with the efficient local+global attention pattern of Longformer. This
allows each token to attend to only its local window and a pre-specified set
of global locations of interest, thereby scaling self-attention computation
linearly with the input size (as opposed to quadratically with full context
self-attention). LED has a similar architecture to BART Lewis et al. (2020) in
terms of number of layers and hidden state sizes, with the distinction that it
has a larger position embeddings matrix, allowing it to process inputs of up
to 16K tokens long (up from 1K tokens in the original BART model). In
practice, LED’s parameters are initialized from a pretrained BART model, and
LED copies BART’s position embeddings 16 times to fill the entire 16K position
embeddings matrix. For all experiments we use the LED-base sized model, which
uses BART-base weights.

#### Input and Output Encoding

For the input, we follow the Longformer QA models Beltagy et al. (2020) and
encode the question and context in one concatenated string with “global
attention” over all the question tokens. For the output, all answer types are
encoded as single strings. The string is the text of the abstractive answer, a
comma separated concatenation of the extractive spans, “Yes”, “No”, or
“Unanswerable”.

#### Evidence extraction

To support extracting evidence paragraphs, we prepend each paragraph with a
</s> token and add a classification head over these tokens on LED’s encoder
side. We also add Longformer’s global attention over these tokens to
facilitate direct information flow across the paragraphs. We then train LED
using both loss functions (teacher-forced text generation and paragraph
classification) in a multi-task training setup. For the answer generation, we
use a cross-entropy loss function over the vocabulary. For the evidence
paragraph extraction, we use a cross-entropy loss function with binary 0 or 1
gold labels for evidence/non-evidence paragraph. To account for class
imbalance, we use loss scaling with weights proportional to the ratio of
positive to negative gold paragraphs in the batch, which we found to be
crucial for the model to train. One benefit of multi-task training of evidence
extraction along with answer selection is that tasks can benefit each other
(see Section 5.2).

##  5 Experiments

We evaluate model performance on question answering and evidence selection
tasks, and compare them to estimated lower bounds on human performance. These
human performance estimates are calculated by comparing the answers of
questions for which we have multiple human annotations. For each question, we
choose one annotation as if it were a prediction, and evaluate it against the
rest of the annotations, and consider as human performance the average over
all annotations chosen as predictions. We restrict our experiments to the
subset of questions in Qasper that can be answered from text in the paper,
ignoring those that require figures or tables as evidence (13% of the dataset;
see Section 3) to avoid having to deal with multimodal inputs. We leave
multimodal question answering to future work.

Input | Extractive | Abstractive | Yes/No | Unanswerable | Overall
---|---|---|---|---|---
Dev. | Test | Dev. | Test | Dev. | Test | Dev. | Test | Dev. | Test
Q only | 4.60 | 5.91 | 6.06 | 7.38 | 69.05 | 66.36 | 58.43 | 66.67 | 17.81 | 22.48
Q+Abstract | 6.69 | 7.97 | 7.50 | 8.25 | 69.05 | 63.43 | 51.14 | 62.50 | 18.60 | 22.30
Q+Introduction | 4.40 | 6.60 | 2.52 | 3.16 | 65.87 | 67.28 | 71.00 | 78.07 | 18.30 | 24.08
[.4pt/1pt] Q+Full Text | 26.07 | 30.96 | 16.59 | 15.76 | 67.48 | 70.33 | 28.57 | 26.21 | 29.05 | 32.80
Q+Full Text w/ scaff. | 24.62 | 29.97 | 13.86 | 15.02 | 63.64 | 68.90 | 38.89 | 44.97 | 28.01 | 33.63
[.4pt/1pt] Human (lower bound) | - | 58.92 | - | 39.71 | - | 78.98 | - | 69.44 | - | 60.92
Table 2: LED-base and lower-bound human performance on answering questions in
Qasper, measured in Answer-$F_{!}$. The top three rows are heuristic baselines
that try to predict answers without encoding entire papers. w/ scaff. refers
to the inclusion of the evidence selection scaffold during training.

###  5.1 Training Details

We train all models using the Adam optimizer (Kingma and Ba, 2014) and a
triangular learning rate scheduler Howard and Ruder (2018) with 10% warmup. To
determine number of epochs, peak learning rate, and batch size, we performed
manual hyperparameter search on a subset of the training data. We searched
over {1, 3, 5} epochs with learning rates {$1e^{-5}$, $3e^{-5}$, $5e^{-5}$,
$9e^{-5}$}, and found that smaller batch sizes generally work better than
larger ones. Our final configuration was 10 epochs, peak learning rate of
$5e^{-5}$, and batch size of 2, which we used for all reported experimental
settings. When handling full text, we use gradient checkpointing Chen et al.
(2016) to reduce memory consumption. We run our experiments on a single RTX
8000 GPU, and each experiment takes 30–60 minutes per epoch.

###  5.2 Results

#### Question answering

Table 2 shows the overall performance of the LED-base model111111We trained an
LED-large model as well, but it performed much worse than the base model on
the QA task. on question answering, as well as the performance breakdown on
the different answer types. The table also compares LED-base variants when the
input is heuristically limited to smaller parts of the paper (i.e., no
context, abstract, introduction). We generally observe that, by using more
context, the performance improves. Specifically, as we observe in row 5
encoding the entire context results in significant overall performance
improvement ($\Delta=+9.5$) over the best heuristic (“introduction”). This
signifies the importance of encoding the entire paper. Comparing rows 4 and 5,
we observe that using the evidence prediction as a multi-task scaffolding
objective helps, improving the results by $\Delta=+0.8$ points.

#### Evidence selection

Table 3 illustrates the evidence selection performance of the LED-large and
LED-base models compared with simpler baselines. We observe that LED variants
outperform the simple TF-IDF baseline but there still remains a large gap to
human performance.

#### Varying amounts of training

Figure 2 shows the learning curve that measures the validation Answer-$F_{1}$
and Evidence-$F_{1}$ of the LED-base variants based on training data size. The
learning curve suggests that performance has not reached a plateau, and future
data collection could be useful.

#### Answer prediction from gold evidence

To better isolate the question answering (as opposed to evidence selection)
task performance, we perform oracle experiments where models are given the
gold evidence. For these experiments, we are able to use larger (T5-large;
Raffel et al., 2020) or better task-adapted pretrained models (UnifiedQA-
large; Khashabi et al., 2020), which perform significantly better in the
oracle setting. We did not use them in the non-oracle setting, however, as
Longformer versions of these models are not available, and LED’s ability to
handle the full document without the need for a pipelined retrieval system was
more important. These experiments show that (1) the human lower bound is in
fact a lower bound, as large models exceed it for span answers in this
setting; (2) the majority of the large headroom in the non-oracle setting can
be closed with better evidence selection; and (3) research into making large
pretrained models able to better scale to long documents would be beneficial.

Model | Evidence $F_{1}$
---|---
Dev. | Test
LED-base | 23.94 | 29.85
LED-large | 31.25 | 39.37
[.4pt/1pt] TF-IDF | 10.68 | 9.20
Random paragraph | 2.09 | 1.30
First paragraph | 0.71 | 0.34
[.4pt/1pt] Human (lower bound) | - | 71.62
Table 3: Model and lower-bound human performance on selecting evidence for questions in Qasper Figure 2: Learning curves showing Answer-$F_{1}$ and Evidence-$F_{1}$ on the dev. set while varying training data size. Model | Answer $F_{1}$
---|---
Span | Abstractive | Overall
LED-base | 54.20 | 24.95 | 44.96
T5-large | 65.59 | 29.11 | 60.03
UnifiedQA-large | 67.23 | 28.92 | 61.39
Table 4: Model performance on the Qasper test set on answering questions given
gold evidence. We do not show performance on _Yes/No_ and _Unanswerable_ types
because they can be trivially predicted to a large extent from the absence of
gold evidence.

#### Error analysis

To gain insight into the model’s errors, we sample 67 test questions with
predicted Answer-$F_{1}$ scores below 0.10 from the LED model trained with
evidence prediction scaffolding. We remove four cases in which the predicted
answers are actually correct. Examining gold answers of the remaining 63, we
find 31 are extractive, 24 are abstractive, 3 are “yes”, 3 are “no,” and 2 are
unanswerable. We observe that LED often predicts shorter spans than the gold
answers (9.5 words shorter than gold counterparts, on average). Focusing only
on the 55 questions with either extractive or abstractive gold answers, we
manually categorize error types in Table 5.2.

Error | % | Example question | Gold | Predicted |
---|---|---|---|---|---
| Incorrectly
---
predicts
unanswer-
able
34.5% |  | How is the text segmented?
---
| “dividing documents into chunks
---
before processing”
Unanswerable |
[0.4pt/2pt]  Lacks domain knowledge |  | 23.6%
---
| What other scenarios can the bias
---
mitigation methods be applied to?
| “sentiment analysis, other identity
---
problems like racial”
GRU |
|  | What baselines is the neural
---
relation extractor compared to?
| Berant et al. (2013), Yao and
---
Van Durme (2014), Xu et al.
(2014), Berant and Liang (2014), …
| Freebase, MCCNN
---
|  |  †What hand-crafted features
---
are used?
| “position of sentence, sentence
---
length, tense, qualifying adjectives,
meta-discourse features”
| the Minimum word
---
count is 40, The
number of threads
to run in parallel is 4
and the context
window is 10
[0.4pt/2pt]  Uninfor- mative |  | 20.0%
---
| What do they mean by intrinsic
---
geometry of spaces of learned
representations?
| “the inferred embedding space
---
creates a globally consistent
structured prediction of the
ontology, rather than local relation
predictions”
| intrinsic geometry
---
|  | How does the proposed training
---
framework mitigate the bias
pattern?
| by balancing or, smoothing the
---
artifacts across different classes by
assigning specific weights for every
sample
| By minimizing the
---
impact of the bias
pattern on the dataset
[0.4pt/2pt]  Not com- prehensive |  | 7.3%
---
| Which metrics were considered?
---
| “ter, bleu, rouge, nist, lepor, cider,
---
meteor, Semantic Similarity (sim),
readability and grammaticality”
| Grammar-based
---
metrics (GBMs)
|  | Was permission sought from the
---
bipolar patients to use this data?
| For Twitter and Reddit users,
---
implicit consent is assumed to use
their public tweets. Blog users are
contacted to obtain consent for
using their texts.
| No
---
[0.4pt/2pt]  Lacks specificity |  | 7.3%
---
What are the performance metrics? |  | “Rouge-1, Rouge-2 and
---
Rouge-4 recall”
| Rouge scores
---
|  | What supervised machine
---
learning models do they use?
| “ZeroR, Naïve Bayes, J48, and
---
random forest”
| Weka classifiers
---
[0.4pt/2pt]  | Lacks
---
numeracy
7.3% |  | How many tags are included
---
in the ENE tag set?
“200 fine-grained categories” | 1 |

Table 5: Error analysis of our best model (LED from row 5 from Table 2) on 55
test examples with low $F_{1}$ score (excluding those with “yes,” “no,” or
“unanswerable” gold answers). “Quotations” denote extractive gold answers. We
note _Lacks domain knowledge_ errors are not always solved by better entity
type resolution (see $\dagger$).

##  6 Related Work

#### Information-Verifying QA

A large body of work on question answering follows the information-verifying
paradigm where the writer of the question already knows its answer, and the
questions are written solely for evaluating the knowledge or understanding
capabilities of machines. Some examples include SQuAD Rajpurkar et al. (2016),
TriviaQA Joshi et al. (2017), NarrativeQA Kočiský et al. (2018), WikiHop Welbl
et al. (2018), HotpotQA Yang et al. (2018), CoQA Reddy et al. (2019), DROP Dua
et al. (2019), Quoref Dasigi et al. (2019). Most datasets for QA on academic
research papers also fall within the information-verifying paradigm as they
automatically construct QA examples using extracted entities and relations and
structured knowledge resources, like DrugBank. Some examples include emrQA
Pampari et al. (2018), BioRead Pappas et al. (2018), BioMRC Pappas et al.
(2020), MedHop Welbl et al. (2018). While these datasets enabled significant
progress in machine comprehension, they include biases in questions that may
not reflect real-world settings Kwiatkowski et al. (2019).

#### Information-Seeking QA in General Domain

Recognizing this challenge, others have followed an information-seeking
paradigm where the writer of questions is genuinely interested in finding the
answer to the question, or at least does not have access to the answer.
Examples of such datasets include WikiQA Yang et al. (2015), NewsQA Trischler
et al. (2017), MsMarco Campos et al. (2016), QuAC Choi et al. (2018), Natural
Questions Kwiatkowski et al. (2019), TyDiQA Clark et al. (2020), and IIRC
Ferguson et al. (2020). Unlike Qasper, Natural Questions and
TyDiQA121212TyDiQA uses short snippets to prime annotators to write questions
of interest, but the annotation process does not require workers to write
questions grounded in those snippets. questions are not grounded in any
contexts, and the associated documents are linked to the questions after they
are written. In contrast, Qasper’s questions are real follow-up questions
about a paper that a reader of appropriate domain expertise would have after
reading the title and the abstract. The priming lets the readers ask detailed
questions that are specific to the papers in context, those that require a
deeper understanding of the contexts, like those shown in Figure 1 and Table
3. QuAC used similar data collection method but with focus on entities, which
Qasper does not impose.

#### Domain-Specific Information-seeking QA

Some work has been done on information-seeking QA on academic research papers.
PubmedQA Jin et al. (2019) derives Yes/No/Maybe questions from PubMed paper
titles answered from the conclusion sections of the corresponding abstracts.
BioAsq benchmarks Balikas et al. (2013); Nentidis et al. (2018); Krallinger et
al. (2020) focus on open-domain QA over PubMed abstracts. Like Qasper, BioAsq
answers can take different forms (e.g., yes/no, extracted span(s)). Qasper
differs from BioAsq in that questions are grounded in a single paper of
interest. Furthermore, Qasper uses the paper full text, not just the abstract.
To the best of our knowledge, Qasper is the first information-seeking QA
dataset in a computer science domain, while most prior work using academic
research papers has been in biomedicine. Furthermore, with over 5K annotated
questions, Qasper is also larger than other comparable human-annotated QA
datasets – PubmedQA and BioAsq contain 1K and 3.2K questions, respectively.
Finally, Qasper poses a challenging full document-level task while other
related datasets are abstract-level. Beyond the domain of academic research,
realistic QA datasets have also been built in the privacy policy domain
(Ravichander et al., 2019; Ahmad et al., 2020). These tasks are similar to our
evidence selection task.

##  7 Conclusion

We presented Qasper, an information-seeking QA dataset over NLP research
papers. With natural questions asked as follow-up to titles and abstracts, the
task presented by Qasper requires evidence from multiple paragraphs and/or
figures and tables within the full text of the papers. Our empirical results
show plenty of room for improvement when compared to the estimated human
performance, and suggest that Qasper could serve as a test-bed for evaluating
document-grounded QA research.

## Ethical Considerations

We present a new dataset that uses papers authored by other researchers. To
adhere to copyright, we have restricted ourselves to arXiv papers released
under a CC-BY-* license, as identified via Unpaywall, which was used in the
S2ORC Lo et al. (2020) dataset construction. Due to our choice to use arXiv as
the source of papers, Qasper is almost entirely an English-language dataset,
and QA systems built on Qasper would not be expected to work well on non-
English language research papers.

We have determined the amount we paid the annotators to be well-above the
minimum wage in our local area. While we do collect information about
annotator background in NLP and familiarity with the papers they are
annotating, we have not collected personal identifiable information without
their permission except for payment purposes, and do not include any such
information in the released dataset.

## References

  * Ahmad et al. (2020) Wasi Ahmad, Jianfeng Chi, Yuan Tian, and Kai-Wei Chang. 2020.  PolicyQA: A reading comprehension dataset for privacy policies.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 743–749, Online. Association for Computational Linguistics.
  * Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _International Conference on Learning Representations_.
  * Balikas et al. (2013) Georgios Balikas, Ioannis Partalas, Aris Kosmopoulos, Sergios Petridis, Prodromos Malakasiotis, Ioannis Pavlopoulos, Ion Androutsopoulos, Nicolas Baskiotis, Eric Gaussier, Thierry Artieres, et al. 2013.  Evaluation framework specifications.  _Project deliverable D_ , 4.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _ArXiv_ , abs/2004.05150.
  * Campos et al. (2016) Daniel Fernando Campos, T. Nguyen, M. Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, L. Deng, and Bhaskar Mitra. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _ArXiv_ , abs/1611.09268.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _ArXiv_ , abs/1604.06174.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5925–5932, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ferguson et al. (2020) James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. 2020.  IIRC: A dataset of incomplete information reading comprehension questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1137–1147, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _ArXiv_ , abs/2002.08909.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  PubMedQA: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _ArXiv_ , abs/1412.6980.
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The NarrativeQA reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Krallinger et al. (2020) Martin Krallinger, Anastasia Krithara, A. Nentidis, G. Paliouras, and Marta Villegas. 2020.  Bioasq at clef2020: Large-scale biomedical semantic indexing and question answering.  _Advances in Information Retrieval_ , 12036:550 – 556.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020.  RikiNet: Reading Wikipedia pages for natural question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6762–6771, Online. Association for Computational Linguistics.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The semantic scholar open research corpus.  In _Proceedings of the Association for Computational Linguistics (ACL)_.
  * Nentidis et al. (2018) Anastasios Nentidis, Anastasia Krithara, Konstantinos Bougiatiotis, Georgios Paliouras, and Ioannis Kakadiaris. 2018.  Results of the sixth edition of the BioASQ challenge.  In _Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering_ , pages 1–10, Brussels, Belgium. Association for Computational Linguistics.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasović, and Zhen Nie. 2020.  Easy, reproducible and quality-controlled data collection with CROWDAQ.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 127–134, Online. Association for Computational Linguistics.
  * Pampari et al. (2018) Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018.  emrQA: A large corpus for question answering on electronic medical records.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2357–2368, Brussels, Belgium. Association for Computational Linguistics.
  * Pappas et al. (2018) Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. 2018.  BioRead: A new dataset for biomedical reading comprehension.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Pappas et al. (2020) Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. 2020\.  BioMRC: A dataset for biomedical machine reading comprehension.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_ , pages 140–149, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ravichander et al. (2019) Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019.  Question answering for privacy policies: Combining computational and legal perspectives.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4947–4958, Hong Kong, China. Association for Computational Linguistics.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018.  Constructing datasets for multi-hop reading comprehension across documents.  _Transactions of the Association for Computational Linguistics_ , 6:287–302.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  WikiQA: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.