The Scattering and Neutrino Detector at the LHC

A. Iuliano (On behalf of the SND@LHC Collaboration)

Università di Napoli Federico II and INFN

> SND@LHC is a compact and stand-alone experiment to perform measurements with
> neutrinos produced at the LHC in a hitherto unexplored pseudo-rapidity
> region of $7.2<\eta<8.4$, complementary to all the other experiments at the
> LHC. The experiment is located $480\text{\,}\mathrm{m}$ downstream of ATLAS
> IP1 in the unused TI18 tunnel. The detector is composed of a hybrid system
> based on an 800 kg target mass of tungsten plates, interleaved with emulsion
> and electronic trackers, followed downstream by a calorimeter and a muon
> system. The configuration allows efficiently distinguishing between all
> three neutrino flavours, opening a unique opportunity to probe physics of
> heavy flavour production at the LHC in the region that is not accessible to
> ATLAS, CMS and LHCb. This region is of particular interest also for future
> circular colliders and for predictions of very high-energy atmospheric
> neutrinos. The detector concept is also well suited to searching for Feebly
> Interacting Particles via signatures of scattering in the detector target.
> The first phase aims at operating the detector throughout LHC Run 3 to
> collect a total of $250\mbox{fb}^{-1}$. This presentation reports the
> current status of the analysis, after the data collected in the first year
> of data taking. A new era of collider neutrino physics is just starting.

> PRESENTED AT
>
>
>
>
> DIS2023: XXX International Workshop on Deep-Inelastic Scattering and Related
> Subjects,
> Michigan State University, USA, 27-31 March 2023
>

##  1 Introduction and Motivation for SND@LHC

The LHC (Large Hadron Collider) is currently the highest energy collider in
the world. It provides high energy neutrinos, allowing to scope the currently
unexplored domain in the $O(10^{2}\,\mbox{GeV})$ \- $O(1\,\mbox{TeV})$ range.
Measuring neutrinos of all the flavours can fill the gap between accelerator
measurements and data from cosmic rays. The SND@LHC (Scattering and Neutrino
Detector at the Large Hadron Collider) aims to provide these measurements,
identifying all three neutrino flavours.

The physics program of the experiment covers the following subjects:

  * •

The measurement of the $\sigma_{pp\rightarrow\nu X}$ cross section: $\nu_{e}$
and $\nu_{\tau}$ come mainly from charmed hadron decays, while $\nu_{\mu}$
spectra contain a soft component from $\pi$ and $K$ decay;

  * •

Charm production in proton collisions: The off-axis location of the detector,
in the pseudorapidity range $7.2<\eta<8.4$, focuses on the measurement of the
charmed sources;

  * •

Gluon PDF Measurement: the data collected by the experiment will provide
invaluable input for future colliders and the study of astrophysical sources;

  * •

Lepton flavour universality: obtained from cross section ratio measurements in
neutrino interactions. The ratio between electron and tau neutrinos depends
only on charm hadronization and decay branching fractions;

  * •

Measurement of $\sigma_{NC}/\sigma_{CC}$: the identification of the charged
leptons separates Charged Current(CC) neutrino interactions from Neutral
Current (NC) neutrino interactions. The ratio of the observed events can be
written as a function of the Weinberg angle, providing a control measurement
for the physical accuracy of the experiment.

The experiment will collect, during Run 3 of the LHC, a total integrated
luminosity of $250\text{\,}\mathrm{f}\mathrm{b}^{-1}$. In total, $\sim$ 1690
Charged Current and $\sim$ 555 Neutral Current neutrino interactions are
expected. The spectra of the neutrinos, obtained with Monte Carlo simulations,
are shown in Figure 1.

Figure 1: Neutrino spectras, after the integration of $250\,\mbox{fb}^{-1}$.

##  2 Detector layout

The SND@LHC experiment is located at $480\text{\,}\mathrm{m}$ from the
collision point of the ATLAS experiment (IP1), in the TI18 tunnel of the LHC.
The detector is composed of three main sections: a veto system, to tag
penetrating muons, a vertex detector employing ECC (Emulsion Cloud Chambers)
and SciFi(Scintillating Fibers), also acting as electromagnetic calorimeter,
and a muon identification system and hadronic calorimeter downstream. The
layout is shown in Figure 2.

Figure 2: Layout of the SND@LHC detector, showing the three main components
[1].

After the approval of the experiment in March 2021, the detector has been
installed in one year, in order to be ready for the start of Run 3 in April
2022. During the 2022 data taking, a total integrated luminosity of
$38.7\text{\,}\mathrm{f}\mathrm{b}^{-1}$ has been delivered, with the exposure
of 4 emulsion targets. The luminosity increase with time is reported in Figure
3.

Figure 3: Total delivered and recorded integrated luminosity in the 2022 year.
The amount of the single emulsion targets is also shown.

##  3 Status of the analysis

The first physics measurement performed has been the muon flux, aimed to
validate Monte Carlo simulations and to evaluate the background for the next
neutrino searches. The electronic detectors [2] and the nuclear emulsions
resulted in consistent measurements of the muon flux, amounting to $(1.5\pm
0.1)\times 10^{4}\mbox{\,fb}/\mbox{\,cm}^{2}$ in the brick closest to the Line
Of Sight in the transverse plane. After updating the geometry of the LHC
magnets, consistent yields have been estimated with Monte Carlo simulations.

A search for high energy $\nu_{\mu}$ CCDIS interactions has been performed
using all the data taken by the electronic detectors in 2022. This analysis
has been performed adopting a selection with a strong rejection power,
designed to yield a clean set of events. First, a fiducial region in the
target has been considered, including only events located in the 3rd or 4th
wall in order to enhance rejection power for muon-induced background, also
ensuring that neutrino-induced showers are sampled by at least two SciFi
planes. Then, the event is identified as signal after a cut-based procedure,
requiring large hadronic activity in the calorimeter system, a clean outgoing
muon track reconstructed in the muon system, and hit time distribution
consistent with IP1 direction as source.

The main source of background are muons reaching the detector location. The
muons can either enter the fiducial volume without being vetoed, or interact
in the surrounding material and produce neutral particles that can then mimic
neutrino interactions in the target. The background has been estimated by
measuring the muon flux, the veto inefficiency, amounting to $4.5\times
10^{-4}$ during the 2022 run, and estimating the rate of neutral hadron events
in muon DIS simulations. A total of 8 candidate events, consistent with
$\nu_{\mu}$ CC interactions, has been observed. The total amount of
background, including both muon and neutral hadrons sources, has been
estimated to $(8.6\pm 3.8)\times 10^{-2}$ events, which implies an excess of
$\nu_{\mu}$ CC signal over the background-only hypothesis of 6.8 standard
deviations [3]. One of the selected candidates is shown in Figure 4.

Figure 4: Display of a $\nu_{\mu}$ CC candidate event. Hits in the SciFi are
shown as blue markers, while hits in the hadron calorimeter and muon system
are shown as black bars. The line represents the reconstructed muon track[3].

##  4 Advanced SND@LHC detector

Along with data taking and analysis, the SND@LHC collaboration is planning the
upgrade for the upcoming High Luminosity Large Hadron Collider (HL-LHC). The
current ”Far” detector will be completely renovated (Figure 5), replacing
nuclear emulsions with an electronic vertex detector, more suited to handle
the increased beam luminosity. A magnetic spectrometer will be added, allowing
to measure the charge of produced muons, thus separating muon neutrino CC
interactions from anti-neutrinos, along with tau neutrinos in the
$\tau\rightarrow\mu$ decay channel.

Figure 5: Schematic drawing of the upgraded SND@LHC detector for HL-LHC.

Finally, another detector will be installed. Labeled as ”Near” detector, it
will be placed in pseudo-rapidity range overlapping with LHCb detector, thus
reducing the systematic uncertainty of SND@LHC measurements of charm
production. The expected high statistics and detector accuracy will thus allow
to perform precise measurements of neutrino physics at colliders.

## References

  * [1] The SND@LHC Collaboration, SND@LHC: The Scattering and Neutrino Detector at the LHC, https://arxiv.org/abs/2210.02784 (accepted for publication by JINST) (2022).
  * [2] S. I. Ilieva, Measurement of the muon flux at SND@LHC, SNDLHC-NOTE-2023-001 (2023).
  * [3] R. Albanese et al. (The SND@LHC Collaboration), Observation of collider muon neutrinos with the SND@LHC experiment, Phys. Rev. Lett. 131, 031802 (2023).


# COVID-19: Comparative Analysis of Methods for Identifying Articles Related
to Therapeutics and Vaccines without Using Labeled Data

Mihir Parmar, Ashwin Karthik Ambalavanan, Hong Guan, Rishab Banerjee,
Jitesh Pabla, Murthy Devarakonda
Arizona State University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Here we proposed an approach to analyze text classification methods based on
the presence or absence of task-specific terms (and their synonyms) in the
text. We applied this approach to study six different transfer-learning and
unsupervised methods for screening articles relevant to COVID-19 vaccines and
therapeutics. The analysis revealed that while a BERT model trained on search-
engine results generally performed well, it miss-classified relevant abstracts
that did not contain task-specific terms. We used this insight to create a
more effective unsupervised ensemble.

##  1 Introduction

COVID-19 Open Research Dataset (CORD-19) is a machine-readable collection of
scientific articles relevant to COVID-19. Finding articles relevant to
COVID-19 vaccines and therapeutics in the dataset has many practical uses. Our
goal was to study how different transfer-learning and unsupervised methods
perform on the task since these methods do not require labeled data.

We formulated the task as a classification problem and considered six
different transfer learning or unsupervised techniques for it: (1) BERT’s next
sentence prediction; (2) Model trained on a different dataset to identify
treatments; (2) Clinical semantic text similarity; (4) Lexicon-based semantic
similarity; (5) Model trained on Google scholar search results; (6) TF-IDF
based scorers. The input to the systems was the title, abstract, and journal
name, and the output was one of three labels: vaccine-related, therapeutics-
related, or the other. Where relevant, we used BERT Devlin et al. (2018) and
its variants.

An important challenge was to understand the performance characteristics of
these models at a deeper level than the standard performance metrics so that
the insight can be used to improve a promising method or develop an effective
ensemble. Most error analyses tend to be ad-hoc.

We characterized the performance of the methods for four categories that were
obtained by taking the cross-product of two factors: (1) whether an article
was rated as a positive or negative class; and (2) whether the terms vaccines,
therapeutics, and their synonyms appeared in the article (abstract) or not. We
manually double-annotated a small (203 articles) test set. Results from our
analysis showed that while a Google search-results trained method generally
performed well, it miss-classified large portion of articles in one category,
and this insight helped us to formulate an effective ensemble, which achieved
an F measure of 0.65.

##  2 Methods

###  2.1 Dataset

We used the COVID-19 Open Research Dataset (CORD-19) Wang et al. (2020), which
contains around 59K articles previously screened for COVID-19 and associated
illnesses and were collected from peer-reviewed publications and archival
services. For our experiments, we used the text consisting of [title +
abstract + journal name] for each article. We removed the articles that have
missing information for one of them, which resulted in about 47K articles. We
also manually labeled 203 articles from CORD-19 dataset in three classes: 1)
Vaccine, 2) Therapeutics and 3) Other. Each article was judged by two authors
of this paper separately. The inter-annotator agreement, measured with kappa
score was 0.83. There were 39 vaccine-related articles, 28 therapeutics-
related articles, and 136 other articles in the test dataset.

###  2.2 Systems

####  2.2.1 NSP-based approach

In this approach, we formulated the task as the Next Sentence Prediction (NSP)
in SciBERT Beltagy et al. (2019). We further pre-trained this model using MLM
(Masked Learning Model) on the abstract and title text of the CORD-19 dataset.
We handcrafted a passage each for vaccines and therapeutics, to use as the
first “sentence”. These passages were created from sentences in articles that
were positive for vaccine and therapeutics, respectively, and strongly
indicated relevancy (based on manual analysis). The passages are shown in
Appendix A.

To test if an article is about therapeutics, the therapeutics passage and the
article text (title + abstract + journal name) are used as the “sentence” pair
in the next sentence prediction mode of SciBERT. If SciBERT predicts that an
article is the next sentence with a probability threshold of 0.999 or higher,
then this model outputs a “therapeutics” label. Similarly, the article is
scored for vaccines, using the vaccine passage as the first sentence. The
final predicted label is determined by the higher of the two scores, if the
probabilities were greater than 0.999 for both.

####  2.2.2 Clinical Hedges (CH) trained model

In this scorer, we used a SciBERT model that was previously fine-tuned on the
Clinical Hedges dataset to identify articles describing treatments. Clinical
Hedges Wilczynski NL (2005) is a dataset of articles from MEDLINE which were
manually annotated for the study-purpose, including therapeutics, etiology,
diagnostics and others. Articles were also annotated for other study
characteristics that are not of interest here. We defined the task as binary
classification, i.e. therapeutics or not therapeutics.

We combined this model with the NSP model described in Section 2.2.1 as
follows: we used the text documents categorized as therapeutics or vaccine by
the NSP model as the input to the Clinical Hedges model at the prediction
time. If the Clinical Hedges model categorized the text as therapeutics then
the final label is therapeutics otherwise the label is vaccine. The Clinical
Hedges model consisted of SciBERT (fine-tuned on Clinical Hedges Data) with a
Feed-Forward Network (FFN) head to generate the probability distribution for
the binary classification.

####  2.2.3 STS-based approach

In this clinical Semantic Textual Similarity (STS) approach, we pre-trained
BERT on the Clinical STS dataset from n2c2 Challenge 2019 Cer et al. (2017) to
score semantic similarity between a manually created query and a given text
segment. We input the pair ([CLS] + query + [SEP] + text-segment) to the BERT,
and used the output CLS representation in a linear regression model to predict
similarity in the range of 0 to 5 Cer et al. (2017). Separate queries were
manually created for vaccine and therapeutics and are shown in Appendix B.

The text-segments were obtained by breaking up the concatenation of title,
abstract, and the journal name on the sentence boundaries using NLTK. The
average of the top $n$ ($n=3$) text-segment scores was produced as the
article-level score Yang et al. (2019); Kotzias et al. (2015). As suggested in
Wang et al. (2018), we used $2.0$ (out of $5.0$) as threshold for classifying
an article as positive for a query-type (i.e. therapeutics or vaccine), and
the final label was based on the higher of the two scores if both were above
2.0.

####  2.2.4 Lexicon-based Similarity Scoring (LSS)

The intuition here is that vaccines-related articles will have many tokens
whose embeddings are similar to those of words like “vaccines”. The contextual
embeddings for all tokens are first obtained by processing the [title +
abstracts + journal name] of articles through the BioBERT model. If a token
appear in multiple documents, we take the average of the embeddings of the
token from the documents.

Next, starting with seed words – “vaccine”, “vaccines”, “vaccination” and
“vaccinations” – we find additional tokens (extended-seeds whose embeddings
are within a small cosine distance of the embeddings of the seed words. The
cosine distance threshold is set as the max of the closest 1000 pairs.
Subsequently, we find the top-$k$ ($k=50$) words called representative-words
from the abstract of each article – the top-$k$ words whose embeddings are
closest to any of the extended-seeds.

Lastly, the article-score is the average of the cosine similarity values of
the representative-words that appear in its abstract. The process was repeated
for the seed words for therapeutics. The final label for an article is the
argmax of the vaccine and therapeutics article-scores.

####  2.2.5 Search-results trained approach (GS)

In this approach, we trained a model using results from the Google search
engine, inspired by the knowledge hunting methods Prakash et al. (2019); Emami
et al. (2018). Google Scholar™ was queried with “coronavirus vaccine” and
“coronavirus therapeutics”, and the respective top 1000 results were scraped.
The intersection of the search results for each query and the CORD-19 dataset
was designated as the set of positive samples for the corresponding label.

To obtain negative samples, additional Google Scholar queries that were
semantically unrelated to coronavirus vaccines or therapeutics were carefully
constructed (see Appendix C). The top 1000 Google Scholar™ results from these
queries were intersected with CORD-19 dataset to get the negative samples
required. Any articles common between the two positive samples sets and the
negative samples set were removed from the negative samples. Any common
articles between the vaccine and the therapeutics positive samples were kept
in the set for which the article’s rank is better.

A SciBERT model was fine-tuned on 80% of this weakly labeled dataset, the
remaining 20% was used for validation. Concatenation of article’s title,
abstract and journal name were given as the input. The trained (fine-tuned)
model was used to predict the label for the remaining articles in CORD-19.

####  2.2.6 Micro-scorers (MS)

In this approach, $n$ representative vaccine-related queries
($V_{Q_{1}},V_{Q_{2}},...,V_{Q_{n}}$) and $n$ representative therapeutics-
related queries ($T_{Q_{1}},T_{Q_{2}},...,T_{Q_{n}}$) were manually created,
and their Cartesian product forms a list of query pairs. The title and
abstract were concatenated as the input text for an article. Given an input of
the $k^{th}$ article, we first compute its tf-idf vector $a_{k}$. For each
query pair ($V_{Q_{i}},T_{Q_{j}}$), where $i\in[1,n]$ and $j\in[1,n]$, we
obtained the tf-idf vectors ($v_{i}$, $t_{j}$). The vaccine-score $vs_{k,ij}$,
therapeutics-score $ts_{k,ij}$, and other-score $os_{k,ij}$ are computed as
follow:

$\displaystyle vs_{k,ij}$ $\displaystyle=\frac{a_{k}\cdot
v_{i}}{|a_{k}|\cdot|v_{i}|}$ $\displaystyle ts_{k,ij}$
$\displaystyle=\frac{a_{k}\cdot t_{j}}{|a_{k}|\cdot|t_{j}|}$ $\displaystyle
os_{k,ij}$ $\displaystyle=0.5\cdot(\sqrt{(1-vs_{k,ij})(1-ts_{k,ij})}-$
$\displaystyle\sqrt{(1+vs_{k,ij})(1+ts_{k,ij})})$

The intuition behind the other-score $os_{k,ij}$ is that if the angle between
$a_{k}$ and $vs_{k,ij}$ is $\alpha$, the angle between $a_{k}$ and $ts_{k,ij}$
is $\beta$, then vaccine-score is $\cos\alpha$, therapeutics-score is
$\cos\beta$, other-score is $\cos(\frac{\alpha+\beta}{2}+\pi)$. The
accumulative scores are computed as follow:

$\displaystyle
vs_{k}=\displaystyle\sum_{i,j}vs_{k,ij},ts_{k}=\displaystyle\sum_{i,j}ts_{k,ij},os_{k}=\displaystyle\sum_{i,j}os_{k,ij}$

The label corresponding class of the highest score is the final vote for the
$k^{th}$ article. In our experiment, we selected four different queries, which
are listed in the Appendix B.

####  2.2.7 Majority Voting (MV)

We aggregated the output of all the scorers into one single prediction using
majority voting. We did two majority voting scheme: (1) majority of all
scorers (MV_6); (2) majority of three methods (MV_3) selected based on the
performance analysis described in the next section. Each scorer contributes
one vote as a label - vaccine, therapeutics, or the other - for each article.
The label with the most votes is the overall prediction for the article. Since
there are even number of scorers for majority of all, when there is a tie
between two labels, a positive class is always preferred and between positive
classes one is chosen randomly.

###  2.3 Experiments

First we used the individual systems described earlier and the majority of all
(MV_6) to predict labels (classes) for the manually annotated test dataset. As
is customary, we calculated precision (P), recall (R), and F-measures (F) only
for the positive classes (i.e., Vaccine and Therapeutics).

To characterize performance of the systems compared to the human judgment,
random 119 of the 204 articles were categorized as shown in Table 1. The
categories are a cross-product of two factors: (1) whether or not specific
lexicons are present in the article text (abstract); (2) whether or not the
article was manually judged as positive class (vaccines or therapeutics).

While 58% of the articles are in category 4, there were significant number of
articles in the other categories as well lending this categorization to
interesting analysis of the methods. From the performance of the different
systems for these categories, we could draw preliminary conclusions about the
scorer design vis-a-vis how the subjects of article screening (i.e. vaccines
and therapeutics) were discussed in the abstracts. Based on the analysis, a
majority of three scorers (MV_3) was formed and its performance was analyzed
on the test dataset. Here is the implementation of all the scorers
111https://github.com/md-labs/covid19-kaggle.

Table 1: Different categories for analysis. Count and % out of articles
analyzed (119) are shown in parenthesis.

|  |  | Does the article contain
---
words like vaccine,
therapeutics, or their synonyms?
|  | YES | NO
Does the article belong to a positive class (vacc. or thera.) according to the manual judgement? | YES |  | Category 1
---
(27, 22.69%)
| Category 3
---
(13, 10.92%)
NO |  | Category 2
---
(21, 17.65%)
| Category 4
---
(58, 48.74%)

##  3 Results

Among the individual systems, GS and MS outperformed all the others and
achieved the same best F-score (0.50) (See Table 2). GS achieved the best
precision (0.50) while MS achieved the best recall (0.76). NSP and LSS
performed rather poorly across the board, and CH and STS achieved moderate
performance. As expected, MV_6 (majority of all) performance was in the middle
of the range.

Table 2: Precision, Recall, and F-measures for different systems. MV_6 is the
majority vote of all, and MV_3 is majority vote of GS, MS, and CH.

Systems | NSP | CH | STS | LSS | GS | MS | MV_6 | MV_3
---|---|---|---|---|---|---|---|---
Precision | 0.12 | 0.42 | 0.32 | 0.21 | 0.50 | 0.37 | 0.39 | 0.59
Recall | 0.13 | 0.29 | 0.44 | 0.32 | 0.49 | 0.76 | 0.71 | 0.69
F-measure | 0.12 | 0.34 | 0.37 | 0.15 | 0.50 | 0.50 | 0.49 | 0.65

Table 3 shows performance of the systems for the four categories introduced in
Table 1. We highlighted in bold the best two systems for each category. We
note that not all systems performed uniformly across the four categories.
While GS performed well on categories 1, 2, and 4, it did poorly on category
3. On the other hand, MS and CH performed better on category 3. This suggested
that combining predictions of GS, CH, and MS might result in a system
performing better than any of them. We created majority of three voting system
(MV_3) based on this observation, and found that it achieved the best results
on the test dataset (the highest F measure 0.65 and precision 0.59, and the
second best recall 0.69).

Table 3: Num. of articles correctly classified by each system out of 119
articles analyzed. (Cat. = Category)

Cat. | Total | NSP | CH | STS | LSS | GS | MS | MV_6 | MV_3
---|---|---|---|---|---|---|---|---|---
1 | 27 | 03 | 03 | 13 | 08 | 14 | 18 | 16 | 20
2 | 21 | 12 | 00 | 02 | 01 | 16 | 04 | 02 | 14
3 | 13 | 02 | 09 | 03 | 06 | 01 | 09 | 10 | 08
4 | 55 | 36 | 00 | 18 | 01 | 49 | 15 | 13 | 41

##  4 Discussion

Using the labels obtained from the Google Scholar search results to fine-tune
the SciBERT model, helped GS perform well across most categories. It, however,
performed poorly on the category 3 articles where terms specific to vaccines
and therapeutics were not present. This observation from our analysis suggests
the possibility that the term-based relevance matching commonly used in search
engines may not be adequate for accurate screening (classification) of
articles for a study-topic.

The robust tf-idf based scoring helped MS to make the better decisions on
category 3 articles, in addition to category 1 articles. However, lack of
query term discrimination might have undermined its performance for category 2
and 4. Surprisingly better performance of NSP on categories 2 and 4 was due to
its tendency to classify most articles as “Other”. CH benefited from transfer
learning for category 3. These observations enabled by our analysis method,
helped us to combine GS, MS, and CH as a select majority-voting group and
achieve the best overall performance.

##  5 Conclusions

We proposed a new approach to analyzing performance of text classification
methods. We applied this approach to study six transfer learning and
unsupervised methods for screening articles related to COVID-19 vaccines and
therapeutics. These methods are of particular value since timely results can
be obtained without the time and effort of generating a large labeled dataset.
We used a novel 2x2 categorization of articles to understand performance of
the systems, and from the analysis formulated an effective voting ensemble of
systems. Our methodology showed that while a weakly supervised model based on
search-engine results performed generally well, it miss-classified articles
that did not contain task-specific lexicon. Combining it with a tf-idf and a
transfer learning system yielded better results. The key contribution of this
paper is the novel approach to analyze text classification methods to gain
insights into their performance characteristics.

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3606–3611.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the $11^{th}$ International Workshop on Semantic Evaluation (SemEval-2017)_, pages 1–14, Vancouver, Canada. Association for Computational Linguistics (ACL).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.  {Last Accessed: May 24, 2019}.
  * Emami et al. (2018) Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018.  A knowledge hunting framework for common sense reasoning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1949–1958, Brussels, Belgium. Association for Computational Linguistics.
  * Kotzias et al. (2015) Dimitrios Kotzias, Misha Denil, Nando De Freitas, and Padhraic Smyth. 2015.  From group to individual labels using deep features.  In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 597–606.
  * Prakash et al. (2019) Ashok Prakash, Arpit Sharma, Arindam Mitra, and Chitta Baral. 2019.  Combining knowledge hunting and neural language models to solve the Winograd schema challenge.  In _Proceedings of the $57^{th}$ Annual Meeting of the Association for Computational Linguistics_, pages 6110–6119, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. 2020\.  Cord-19: The covid-19 open research dataset.  _arXiv preprint arXiv:2004.10706_.  {Last Accessed: Apr 25, 2020}.
  * Wang et al. (2018) Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2018.  Medsts: a resource for clinical semantic textual similarity.  _Language Resources and Evaluation_ , pages 1–16.
  * Wilczynski NL (2005) Haynes RB Hedges Team Wilczynski NL, Morgan D. 2005.  An overview of the design and methods for retrieving high-quality studies for clinical care.  _BMC Med Inform Decis Mak._ , (Jun):5–20.
  * Yang et al. (2019) Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Simple applications of bert for ad hoc document retrieval.  _arXiv preprint arXiv:1903.10972_.

##  Appendix A Queries used in NSP-based Approach

Vaccine-related query:

Vaccine is a substance used to stimulate the production of antibodies and
provide immunity against diseases. They are treated to act as an antigen
without inducing the disease. When the virulent version of an agent comes
along, the immune system is prepared to respond due to the generation of B
cells (memory and plasma cells), which will generate antibodies that will bind
to pathogens and destroy them. Vaccine researchers are working on the
development of a vaccine candidate expressing the viral spike protein of SARS-
CoV-2 using a messenger RNA vaccine. Scientists are also focusing on the
development of a chimpanzee adenovirus-vectored vaccine candidate against
COVID-19. In addition, scientists are also working to see if vaccines
developed for SARS coronavirus are effective against COVID-19.

Therapeutics-related query:

Therapeutics is the branch of medicine concerned with the therapeutics of
disease and the action of remedial agents. There is no specific antiviral
therapy and therapeutics given by doctors is largely supportive, consisting of
supplemental oxygen and conservative fluid administration. Drugs like
Chloroquine, Hydroxychloroquine, Lopinavir, Ritonavir, Azithromycin and
Tocilizumab are being prescribed by doctors in ICU testing. The drug
Remdesivir has shown promise against other coronaviruses in animal models.
Patients with respiratory failure require intubation. Patients in shock
require urgent fluid resuscitation and administration of empiric antimicrobial
therapy. Corticosteroid therapy is not recommended for viral pneumonia;
however, use may be considered for patients with refractory shock or acute
respiratory distress syndrome.

##  Appendix B Queries used in Micro-scorers (MS) and STS-based Approach

Vaccine-related query:

  * •

vaccine vaccination dose antitoxin serum immunization inoculation for COVID-19
or coronavirus related research work.

Therapeutics-related query:

  * •

Therapeutics therapeutics therapy drug antidotes cures remedies medication
prophylactic restorative panacea for COVID-19 or coronavirus related research
work.

##  Appendix C Queries used in Search-results trained approach (GS)

  * •

Coronavirus transmission, incubation and environment stability

  * •

Coronavirus ethical and social science considerations

  * •

Coronavirus information sharing and inter-sectoral collaboration