# Powering COVID-19 community Q&A with Curated Side Information

Manisha Verma, Kapil Thadani, and Shaunak Mishra  Yahoo! Research,
VerizonMedia, New York<EMAIL_ADDRESS>
(2019)

###### Abstract.

Community question answering and discussion platforms such as Reddit, Yahoo!
answers or Quora provide users the flexibility of asking open ended questions
to a large audience, and replies to such questions maybe useful both to the
user and the community on certain topics such as health, sports or finance.
Given the recent events around COVID-19, some of these platforms have
attracted 2000+ questions from users about several aspects associated with the
disease. Given the impact of this disease on general public, in this work we
investigate ways to improve the ranking of user generated answers on COVID-19.
We specifically explore the utility of external technical sources of side
information (such as CDC guidelines or WHO FAQs) in improving answer ranking
on such platforms. We found that ranking user answers based on question-answer
similarity is not sufficient, and existing models cannot effectively exploit
external (side) information. In this work, we demonstrate the effectiveness of
different attention based neural models that can directly exploit side
information available in technical documents or verified forums (e.g.,
research publications on COVID-19 or WHO website). Augmented with a
temperature mechanism, the attention based neural models can selectively
determine the relevance of side information for a given user question, while
ranking answers.

††journalyear: 2019

##  1\. Introduction

Question answering systems are key to finding relevant and timely information
about several issues. Community question answering (cQ&A) platforms such as
Reddit, Yahoo! answers or Quora have been used to ask questions about wide
ranging topics. Most of these platforms let users ask, answer, vote or comment
on questions present on the platform. However, question answering platforms
are useful not only for getting public opinions or votes about areas such as
entertainment or sports but can also serve as information hot-spots for more
sensitive topics such as health, injuries or legal topics. Thus, it is
imperative that when the user visits _sensitive_ topics content, answer
ranking also takes into account curated side information from reliable
(external) sources. Most prior work on cQ&A has focused on incorporating
question-answer similarity (LTR_2008, ; semantic_sim2015, ), user reputation
(yang2016beyond, ; hong2009classification, ; user_interaction_2013, ),
integration of multi-modal content (medical_qa2019, ), community interaction
features (comm_interaction2018, ) associated with answers or just the question
answering network (comm_net2019, ) on the platform. However, there is very
limited work on incorporating _curated content_ from external sources.
Existing work only exploits knowledge bases (medical_qa2019, ) that consist of
different entities and relationships between these entities to score answers.
However, there are some limitations of knowledge bases that would make it
difficult to use them for community Q&A for rapidly evolving topics such as
disease outbreaks (e.g. ebola, COVID-19), wild-fires or earthquakes.

Figure 1. Illustrative example of COVID-19 community answer ranking powered by
side information in the form of research papers, and information from verified
sources (such as CDC, WHO, and NHS).

Firstly, knowledge bases contain information about established entities, and
do not rapidly evolve to incorporate new information which makes them
unreliable for novel disease outbreaks such as COVID-19 where information
rapidly changes and its verification is time sensitive. Secondly, it may be
hard to determine what even _constitutes an entity_ as new information arrives
about the topic. To overcome these limitation in this work, we posit that
external _curated_ free-text or semi-structured informational sources can also
be used effectively for cQ&A tasks.

In this work, we demonstrate that free text or semi structured external
information sources such as CDC111https://www.cdc.gov/,
WHO222https://www.who.int/ or NHS333https://www.nhs.uk/ can be very useful for
ranking answers on community Q&A platforms since they contain frequently
updated information about several topics such as ongoing disease outbreaks,
vaccines or resources about other topics such as surgeries, birth control or
historical numerical data about diseases across the world.

We argue that for sensitive topics such as COVID-19, it is useful to use
publicly available _vetted_ information for improving our ranking systems. In
this work, we explore the utility of publicly available information for
ranking answers for questions associated with COVID-19. We specifically focus
on ranking answers for questions in two publicly available _primary_ Q&A
datasets: a) Yahoo! Answers444https://answers.yahoo.com/ and b) recently
released annotated Q&A dataset (emnlp_data2020, ) in presence of two
_external_ semi-structured curated sources: a) TREC-COVID
(voorhees2020treccovid, ) and b) WHO questions and answers
555https://www.who.int/emergencies/diseases/novel-coronavirus-2019/question-
and-answers-hub on COVID-19. We explore the utility of deep learning models
with attention in this work to improve upon existing state-of-the-art systems.

More specifically, we propose a temperature regulated attention mechanism to
rank answers in presence of external (side) information. Our experiments on
10K+ questions from both _source_ datasets on COVID-19 show that our models
can improve ranking quality by a significant margin over question-answer
matching baselines in presence of external information. Figure 1 demonstrates
the overall design of our system. We specifically use attention based neural
architecture with temperature to automatically determine which components in
the external information are useful for ranking user answers with respect to a
question. Ranking performance, when evaluated with three metrics shows that
precision and recall for correct answer retrieval improves by $\sim$17% and
$\sim$9% for both source datasets respectively over several other cQ&A models.

##  2\. Related work

Community Question and Answering (cQ&A) systems is a well researched sub-field
both in information retrieval and NLP communities. Several systems have been
proposed to rank user submitted answers to questions on community platforms
such as Yahoo! answers, Reddit and Quora.

Ranking user submitted answers on community question-answering platforms has
been addressed with several approaches. Primary method is to determine the
relevance of the answer given an input question. Text based matching is one of
the most common approaches to rank answers. Researchers have used several
methods to compute _similarity_ between a question and user generated answers
to determine relevance. For instance, feature based question-answer matching
is used in (LTR_2008, ) with 17 features extracted from unigrams, bigrams and
web correlation features using unstructured user search logs to rank answers.
It is worth noting that user features and community features when incorporated
may still yield further improvements in the performance of these models but
this is not the focus of our work. The authors in (LTR_2008, ) used questions
extracted from Yahoo! answers for their experiments. Researchers have used
different approaches such representation learning, for instance, in
(LSTMans2017IWCS, ; cohen_2016, ) authors use LSTM to represent questions and
answers respectively. Convolutional networks have also been used in
(yang2016beyond, ; zhou2015answer, ) to rank answers. Other approaches such as
doc2vec (nie2017data, ), tree-kernels (severyn2012structural, ), adversarial
learning (yang2019adversarial, ), attention (attention_question2018, ;
medical_qa2019, ; LSTMans2017IWCS, ; attentive2017AAAI, ) or deep belief
networks (wang2010modeling, ) have been used to score question and answer
pairs. There have also been studies exploring community, user interaction or
question based features (yang2016beyond, ; hong2009classification, ;
user_interaction_2013, ; comm_interaction2018, ) to rank answers. While these
approaches are relevant, it is not always evident how one can incorporate
external information when it is either in free-text or semi-structured format
into these systems. We explore some question-answer based matching approaches
as baselines in this work and show that for rapidly evolving topics such as
COVID-19, inclusion of external _curated_ information can boost model
performance.

The line of work most closely related to ours is incorporation of knowledge
bases in Q&A systems. Existing work (rankQA_2019, ; kb_rank2018Sig, ;
medical_qa2019, ), however, approaches different tasks. For instance, authors
in (kb_rank2018Sig, ; rankQA_2019, ) focus on finding factual answers to
questions using a knowledge base. This does not extend easily to cQ&A where
neither the questions nor the answers may request or refer to any facts. Most
recent work is (medical_qa2019, ) on incorporating medical KB for ranking
answers on medical Q&A platforms. They propose to learn path based
representation of _entities (from KB)_ present in question and answers posted
by users. This approach relies on reliable detection of entities first, which
may be absent for emerging topics such as COVID-19 pandemic. Another
limitation of this work is that external knowledge may not always be present
in a _structured_ format. For example, CDC guidelines are usually simple
question-answer pairs posted on the website. This makes it difficult to apply
their approach to our problem. The proposed approach in this work incorporates
semi-structured information directly with help of temperature regulated
attention.

Finally, with the rise of COVID-19, researchers across disciplines are
actively publishing information and datasets to share understanding of the
virus and its impact on people. Researchers routinely organize dedicated
challenges such as SemEval (nakov2019semeval, ) with tasks such as ranking
answers on QA forums. One such initiative is TREC-COVID track
(voorhees2020treccovid, ) which released queries, documents and manual
relevance judgements to power search for COVID related information. Authors in
(cairecovid, ) also released COVID-19 related QA dataset with 100+ questions
and answers pairs extracted from TREC COVID
666https://ir.nist.gov/covidSubmit/data.html initiative. These
questions/answer pairs are not user generated content, hence, do not reflect
real user questions. We also rely on recently released Q&A dataset from
(emnlp_data2020, ) for our task. We also compile a dataset of 2000+ COVID-19
questions with 10K+ answers all submitted by users on Yahoo! answers for this
work.

##  3\. Method

###  3.1. Problem formulation

In this work, we focus on ranking answers for $n$ questions
$q_{1},\ldots,q_{n}$ related to an emerging topics such as COVID. Each $q_{i}$
is associated with a set of two or more answers $A_{i}=\\{a_{ij}:j\geq 2\\}$
and corresponding labels $Y_{i}=\\{y_{ij}:j\geq 2\\}$ representing answer
relevance. We use binary indicator for relevance where relevant judgments
(e.g., favorite, upvoted) are provided by the user, i.e., $y_{ij}\in\\{0,1\\}$
respectively.

We attempt to model the relevance of each answer $a_{ij}$ to its corresponding
question using an external source which may contain free text or semi-
structured information. For example, the _external_ source could consist of
information-seeking queries or questions $eq_{1},\ldots,eq_{m}$ related to a
topic, with each $eq_{k}$ linked to a set of relevant scientific articles or
answers $ED_{k}$, where each answer/document $ed_{1},\ldots,ed_{p}$ may be
judged for relevance by human judges (voorhees2020treccovid, ) or some
experts.

We hypothesize that this semi-structured or free-text information may be
valuable in identifying user answer quality for certain kinds of questions,
although not all. We investigate this with our model to recover the true
labels $y_{ij}$ for each user answer $a_{ij}\in A_{i}$ given its question
$q_{i}$, category information, and information from the _external_ source
$\langle eq_{k},ED_{k}\rangle_{k=1}^{m}$.

###  3.2. Proposed Model

Figure 2. External source augmentation model

In this work, we explore token-level matching mechanism to determine the
relevance of information in the external source that may inform the label
prediction task. Our model (_$\tau$ -att_) aims to match a given user question
with all the submitted answers in the presence of external information about
the same domain. First, the question $q_{i}$, an answer $a_{ij}$ and
additional metadata can be encoded into a $d$-dimensional vector $x_{i}$ using
a text encoder $f_{\text{input}}$. We use LSTM based encoder for both question
and answer in the _primary_ source which can handle input sequences of
variable length.

##### Question Encoding:

Each word $w_{i}^{q}$ in a question is represented as a $K$ dimensional vector
with pre-trained word embeddings. LSTM takes each token embedding as input and
updates hidden state $h_{i}^{q}$ based on previous state $h_{i-1}^{q}$.
Finally, the hidden state is input to a feed forward layer with smaller
dimension $F<K$ to compress question encoding as follows:

(1)
$h_{i}^{q}=LSTM(h_{i-1}^{q},w_{i}^{q}),\>f_{i}^{q}=RELU(h_{i}^{q}W_{q}+b_{q})$

##### Answer Encoding:

Each word $w_{j}^{a}$ in the answer is also represented as a $K$ dimensional
vector with pre-trained word embeddings. LSTM takes each token embedding as
input and updates hidden state $h_{j}^{a}$. We also reduce the dimension of
answer encoding with a feed forward layer with dimension $F<K$ as follows:

(2)
$h_{j}^{a}=LSTM(h_{j-1}^{a},w_{j}^{a}),\>f_{j}^{a}=RELU(h_{j}^{a}W_{a}+b_{a})$

We concatenate the question and answer representations for further processing.

(3) $f_{ij}=[f_{i}^{q},f_{j}^{a}]$

##### External source encoding:

External sources of information can vary from task-to-task. We encode each
segment of data individually. For instance, if there are two segments in the
source (e.g. question/answer or query/document), our system encodes both
segments individually. We use the same encoding architecture used for primary
source question/answer encoding above. Encoding example for two segment
_external_ source is given below.

(4)
$\begin{split}h_{t}^{eq}=LSTM(h_{t-1}^{eq},w_{t}^{eq}),\>f_{t}^{eq}=RELU(h_{t}^{eq}W_{eq}+b_{eq})\\\
h_{t}^{ed}=LSTM(h_{t-1}^{ed},w_{t}^{ed}),\>f_{t}^{ed}=RELU(h_{t}^{ed}W_{ed}+b_{ed})\\\
\end{split}$

Source | Question | Rel answer | Non-rel answer
---|---|---|---
Yahoo! Ans |  3cmI am really scared to go places for St. Patrick’s day because of the coronavirus. what do I do? |  7cmUnfortunately, there’s not enough people that care and will still go out and party despite the coronavirus epidemic. I’m proud of you in that you’re taking extra precautions … Good for you! |  5cmStop being scared of viruses. What’s the problem?
Infobot |  3cmCan corona live on cardboard? |  7cmA recent study shows that the virus can live in the air … On cardboard, it can live up to 24 hours (1 day) |  5cmThe risk is quite low for one to become infected with COVID19 through mail/packages - especially because…(over a period of a few days/weeks).
Table 1. Sample rel/non-rel answers from both sources

We incorporate external source encoding with a temperature $(\tau)$ based
variant of scaled dot-product attention, which provides a straightforward
conditioning approach over a set of query-document pairs. Question encoding
vector $f_{ij}$ serves as a query over keys $f_{t}^{eq}$. If two segments are
present in the external source such as query/document, the model uses the
attention weights over first segment (e.g. query) to determine the importance
of the second segment (e.g. document) respectively. It is easy to extend this
framework to _external_ sources with multiple segments. The two segment
attention is described below.

(5) $z_{it}=\frac{f_{ij}^{\top}f_{t}^{eq}}{\sqrt{d}}\\\
\alpha_{it}=\frac{e^{z_{it}/\tau}}{\sum_{l}e^{z_{l}/\tau}}\\\
s_{itd}^{\prime}=\sum_{d}\alpha_{it}f_{t}^{ed}$

To summarize, temperature $(\tau)$ based attention helps determine the
relevance of each $f_{t}^{ed}$ corresponding $f_{t}^{eq}$ with respect to the
question encoding. Temperature $(\tau)$ parameter helps us control the
uniformity of attention weights $\alpha_{it}$. Finally, labels are predicted
using a multi-layer perceptron over the input vector $f_{ij}$ and the learned
weighted average of side information $s_{itd}^{\prime}$. We use binary cross
entropy loss to train the proposed model.

(6) $\hat{y}_{ij}=F_{\text{output}}([f_{ij};s_{itd}^{\prime}])$

where $F_{\text{output}}$ uses sigmoid activation function. Since community
questions may often be entirely unrelated to external sources, a key aspect of
this approach is determining whether the _external source_ is useful, not
merely attending to its entries that are most relevant. Temperature based
attention mechanism is useful in controlling which external source entries are
useful for user questions. It is worth noting that one will have to experiment
and tune the value of temperature $\tau$ such that ranking performance
improves.

##  4\. Experimental Setup

Given the model architecture, in this section, we provide a detailed overview
of different datasets, metrics and baselines used in our experiments.

(a) Yahoo! ques length

(b) Yahoo! ans length

(c) Infobot ques length

(d) Infobot ans length

Figure 3. Token distribution in different sources

###  4.1. Data

We compiled two question answering datasets. The first was collected from
Yahoo! answers and the second was recently released in (emnlp_data2020, )
where both datasets have questions raised by real users. In this work we focus
specifically on questions associated with COVID-19. Different statistics about
the train and test split of both q&a datasets are given in Table 2
respectively. A pair of relevant and non-relevant answers for a question in
both datasets is also shown in Table 1 for reference. More details about them
is given below.

Stat | Yahoo! Ans | Infobot
---|---|---
Train Q-A | 9341 | 6354
Train ans/q | 6.25$\pm$2.9 | 4.40$\pm$0.77
Train #qwords | 12.71$\pm$5.8 | 6.55$\pm$3.93
Train #awords | 36.31$\pm$93.59 | 92.17$\pm$59.27
Test Q-A | 2232 | 1592
Test ans/q | 5.96$\pm$2.87 | 4.41$\pm$0.76
Test #qwords | 13.07$\pm$5.89 | 6.21$\pm$2.94
Test #awords | 35.64$\pm$80.31 | 92.39$\pm$59.47
Table 2. Train and test data from primary sources

##### Yahoo! Dataset

: We crawled COVID-19 related questions from Yahoo! answers
777https://answers.search.yahoo.com/search?p=coronavirus using several
keywords such as ‘coronavirus’, ‘covid-19’, ‘covid’, ‘sars-cov2’ and ‘corona
virus’ between the period of Jan 2020 to July 2020 to ensure we gather all
possible questions for our experiments. We keep only those questions have two
or more answers. In total, we obtained 1880 questions with 11500 answers. We
used favorite answers as positive labels (similar to previous work (LTR_2008,
)), assuming that users, over time rate answers (with upvotes/downvotes) that
are most relevant to the submitted question. We normalized the question and
answer text by removing a small list of stop words, numbers, links or any
symbols. Figure 3(a) and 3(b) show the distribution of question and answer
lengths respectively. Questions contain $12.7\pm 5.8$ _(qwords)_ words and
answers consist of $36.3\pm 93.5$ (mean$\pm$std) words _(awords)_ respectively
which indicates that user submitted answers can vary widely on Yahoo! answers.
On average, a question has about 6 answers _(ans/q)_ in Yahoo! ans dataset. We
spilt the data into three sets: train (64%, 1196 questions, 7435 answers),
validation (16%, 298 questions, 1858 answers) and test (20%, 374 questions,
2310 answers) set where questions for each set were uniformly sampled.

##### Infobot Dataset (emnlp_data2020, )

: Researchers at JHU (emnlp_data2020, ) have recently compiled a list of user
submitted questions on different platforms and manually labeled 22K+ question-
answer pairs. We cleaned this set by removing questions with less than two
answers or no relevant answers. In total, our dataset contains 8000+ question
answer pairs where each question may have _multiple_ relevant answers which is
not the same as Yahoo! answers dataset. Figure 3(c) and 3(d) show the
distribution of question and answer lengths respectively.

####  4.1.1. External sources

We use two external datasets to rank answers. Details of each dataset are
given below:

##### TREC COVID (voorhees2020treccovid, ):

We use recently released TREC COVID-19 track data with 50 queries which also
contain manually drafted query descriptions and narratives. Expert judges have
labeled over 5000 scientific documents for these 50 queries from the CORD-19
dataset 888https://www.semanticscholar.org/cord19. These documents contain
coronavirus related research. Given the documents are scientific literature,
we initialize document embeddings using SPECTER (cohan2020specter, ).

##### WHO:

We use data released on question and answer hub of WHO
website999https://www.who.int/emergencies/diseases/novel-
coronavirus-2019/question-and-answers-hub to create a list of question-answer
pairs. There are 147 question and answer pairs in this dataset where questions
contain 13.28$\pm$5.36 words and answers contain 133.2$\pm$100.9 words
respectively.

###  4.2. Baselines

We evaluated our model against embedding similarity baseline. We computed four
baselines as follows:

##### Random:

An answer is chosen at random as relevant for a user question. This is
expected to provide a lower bound on retrieval performance.

##### Linear Attention (_att_)

: When $\tau=1.0$, our model defaults to simple linear attention over all the
information present in the external sources. This gives an indication of how
well the model performs when its forced to look at all the information in the
external source.

##### Linear combination (_$\lambda$ -sim_)

: We linearly combine similarities between Yahoo! question-answer and Trec
query-answer as shown below:

(7)
$\emph{$\lambda$-sim}=\lambda\;cos(ya,yq)+(1-\lambda)\;\max_{tq}(cos(ya,tq))$

where $ya$, $yq$ and $tq$ are Yahoo! answer, question and concatenated trec
query, narrative and description embeddings respectively. This is a more crude
version of temperature attention where $\lambda$ controls the contribution of
each component directly. We vary $\lambda$ to determine the optimal
combination. Question-Answer similarity (_qasim_) is similarity between
question and answer embedding i.e. $\lambda=1$. Both question and answer
embeddings are obtained by averaging over their individual token embeddings.

##### BERT Q&A (_bert_)

: Large scale pre-trained transformers (devlin2019bert, ) are widely popular
for NLP tasks. BERT like models have shown effectiveness on Q&A datasets such
as SQUAD 101010https://rajpurkar.github.io/SQuAD-explorer/. We fine-tune BERT
base model with two different answer lengths a) 128 _(bert-sl128)_ and b) 256
tokens _(bert-sl256)_ respectively. The intuition is that large scale pre-
trained models are adept at language understanding and can be fine-tuned for
new tasks with small number of samples. We finetune BERT for both datasets
Yahoo! ans and Infobot respectively. It is non-trivial to include external
information in BERT and we leave this for future work.

| Yahoo! Ans | Infobot
---|---|---
Model | P$@$1 | R$@$3 | MRR | P$@$1 | R$@$3 | MRR
$\tau$-att | 0.393 | 0.644 | 0.598 | 0.673 | 0.868 | 0.802
$\lambda$-sim | 0.3743 | 0.633 | 0.578 | 0.551 | 0.817 | 0.7207
bert-sl256 | 0.406 | 0.657 | 0.615 | 0.581 | 0.803 | 0.744
bert-sl128 | 0.363 | 0.604 | 0.589 | 0.557 | 0.799 | 0.731
att | 0.377 | 0.645 | 0.589 | 0.567 | 0.821 | 0.739
qasim | 0.318 | 0.608 | 0.546 | 0.551 | 0.817 | 0.720
random | 0.21 | - | - | 0.239 | - | -
Table 3. Evaluation with WHO external data

###  4.3. Evaluation Metrics

We evaluate the performance of our model using three popular ranking metrics,
mainly Precision (P$@$1), Mean Reciprocal Rank (MRR), and Recall (R$@$3). Each
metric is described below:

  * •

Precision (P$@$k): Precision at position $k$ evaluates the fraction of
relevant answers retrieved until position k. For, both datasets Yahoo! ans and
Infobot (emnlp_data2020, ), we evaluate whether the top answer i.e. $(k=1)$ in
the ranked list is indeed correct. It is defined as follows:

(8)
$Prec@k=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{\sum_{j=1}^{k}\mathbb{I}\\{rel_{ij}=1\\}}{k}$

where $\mathbb{I}\\{rel_{ij}=1\\}$ indicates whether the answer at position
$j$ is relevant to the $i^{th}$ question.

  * •

Recall (R$@$k): Recall at position $k$ evaluates the fraction of relevant
answers retrieved from all the answers marked relevant for a question. We
report recall averaged for all the queries in test set. For recall, we take a
cutoff as $(k=3)$, which evaluates whether the model is able to retrieve the
correct answers in top 3 positions. It is defined as follows:

(9)
$Recall@k=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{\sum_{j=1}^{k}\mathbb{I}\\{rel_{ij}=1\\}}{|rel_{i}|}$

where $|rel_{i}|$ is the number of relevant answers for the $i$th question.

  * •

MRR (MRR): evaluates the average of the reciprocal ranks corresponding to the
most relevant answer for the questions in test set, which is given by:

(10) $MRR=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{rank_{i}}$

where $|Q|$ indicates the number of queries in the test set and $rank_{i}$ is
the rank of the _first_ relevant answer for the $i^{th}$ query.

###  4.4. Parameter Settings

Both primary datasets, Yahoo! ans and Infobot, were divied into three parts:
train ($\sim$60%), validation and test (20%) respectively. The baseline models
$\lambda$-sim and $att$ are initialized with glove embeddings
111111https://nlp.stanford.edu/projects/glove/ of 100 dimensions. We performed
a parameter sweep over $\lambda$ and $\tau$ for $\lambda$-sim and $\tau$-att
models with step size of 0.1 between $\\{0,1.0\\}$ respectively. We used base
uncased model for $bert$ implementation. We fine-tuned the model between 1-10
epochs and found that 3 epochs gave the best result on validation set. We used
LSTM with 64 hidden units to represent question, answer and all the
information in external datasets. We experimented with higher embedding size
and hidden units, but the performance degraded significantly as the model
tends to overfit on training data. Lastly we used batch size of 64 and trained
the model for 30 epochs with early stopping.

| Yahoo! Ans | Infobot
---|---|---
Model | P$@$1 | R$@$3 | MRR | P$@$1 | R$@$3 | MRR
$\tau$-att | 0.532 | 0.778 | 0.715 | 0.606 | 0.842 | 0.766
$\lambda$-sim | 0.326 | 0.616 | 0.555 | 0.556 | 0.813 | 0.722
bert-sl256 | 0.406 | 0.657 | 0.615 | 0.581 | 0.803 | 0.744
bert-sl128 | 0.363 | 0.604 | 0.589 | 0.557 | 0.799 | 0.731
att | 0.291 | 0.495 | 0.494 | 0.601 | 0.833 | 0.762
qasim | 0.318 | 0.608 | 0.546 | 0.551 | 0.817 | 0.720
random | 0.21 | - | - | 0.239 | - | -
Table 4. Evaluation with TREC-COVID external data

##  5\. Results

In this work, our focus is to evaluate the utility of external information in
improving answer ranking for cQ&A task. Thus, we performed experiments to
answer three main research questions listed below.
RQ1: Does external information improve answer ranking?
RQ2: How does temperature ($\tau$) compare with $\lambda$ parameter?
RQ3: What kind of queries/questions does the model attend to when ranking
relevant/non-relevant answers?

#####  RQ1: Does external information improve answer ranking?

We evaluated different models for ranking answers in Yahoo! ans and Infobot
dataset in presence of TREC and WHO datasets respectively. We found that
temperature regulated attention models that incorporate external sources
indeed outperform the baselines as shown in Table 4 and Table 3 respectively.

Category |  $\tau$-att |  $\lambda$-sim | qasim
---|---|---|---
Entertainment (47) | 0.829 | 0.702 | 0.59
Health (62) | 0.693 | 0.69 | 0.645
Politics (143) | 0.727 | 0.629 | 0.587
Society (38) | 0.578 | 0.473 | 0.42
Family (20) | 0.85 | 0.750 | 0.65
Table 5. Recall$@$3 of models across categories

(_$\tau$ -att_) model beats _bert_ models by $\sim$30% in precision, $\sim$18%
in recall and $\sim$16% in MRR respectively on TREC data. However, (_$\tau$
-att_) does only marginally better than _att_ model in precision and MRR on
Infobot data. We suspect that is due to the large set of query-document pairs
in TREC-COVID data compared to fewer number of question-answer pairs in
Infobot dataset. Our results also clearly suggest that embedding based
matching of question-answer pair (_qasim_) would not yield a good ranker,
though it is better than choosing an answer at random (_random_). When WHO is
used as an external dataset, we find that (_$\tau$ -att_) model is slightly
worse than _bert_. This suggests that not all sources would equally benefit
cQ&A task. Since attention is dependent on the input query and key embedding
lengths, it would be interesting to scale the computation in our model to
incorporate several open external datasets to overcome this limitation in the
future.

Yahoo! ans questions are also assigned categories by users. Category based
breakdown of performance on test set is given in Table 6 and Table 5
respectively, where categories with largest number of questions in test set
are listed. In all the categories, our model outperforms best $\lambda$-sim
and _qasim_ model respectively. The largest improvement happens for questions
in Family category where our model achieves an improvement of 71% over the
$\lambda$-sim model. It seems that ranking answers for questions from society
and politics are harder than other categories. All the models, however, are
able to rank the top answer in first three positions effectively as Recall$@$3
is high for all the categories.

Category |  $\tau$-att |  $\lambda$-sim | qasim
---|---|---|---
Entertainment (47) | 0.446 | 0.382 | 0.297
Health (62) | 0.483 | 0.419 | 0.354
Politics (143) | 0.45 | 0.300 | 0.272
Society (38) | 0.28 | 0.157 | 0.236
Family (20) | 0.6 | 0.350 | 0.40
Table 6. Precision$@$1 of models across categories | Temperature ($\tau$) ¿ 1.0
---|---
| 10 | 100 | 1000 | 10 | 100 | 1000
Src+ Ext | Prec$@$1 | Recall$@$3
Yahoo! + TREC | 0.46 | 0.38 | 0.38 | 0.73 | 0.644 | 0.64
Yahoo! + WHO | 0.37 | 0.38 | 0.36 | 0.64 | 0.65 | 0.64
Infobot + TREC | 0.44 | 0.59 | 0.39 | 0.72 | 0.81 | 0.75
Infobot + WHO | 0.65 | 0.41 | 0.44 | 0.85 | 0.76 | 0.79
Table 7. Variation in P$@$1 and R$@$3 across different temperature values.

(a) Yahoo!+TREC

(b) Yahoo!+WHO

(c) Infobot + TREC

(d) Infobot + WHO

Figure 4. Temperature and $\lambda$ variation impact on Prec$@$1

#####  RQ2: How does temperature ($\tau$) compare with $\lambda$ parameter?

We argued that linearly combining similarities between question-answer in
primary dataset and between question-external source may not be sufficient to
boost performance. We observe that in our results too i.e. $\lambda$-sim
models do not perform better than (_$\tau$ -att_) models. This clearly
indicates that more sophisticated models can learn to combine this information
directly from training data. However, our experiments indicate that optimal
value of (_$\tau$_) varies across primary datasets and external sources of
information. For instance, (_$\tau$ -att_) model performed best when
_$\tau=0.4$_ and _$\tau=0.9$_ for Yahoo! ans and Infobot dataset respectively
when TREC was used as external source. It performed best when _$\tau=0.1$_ and
_$\tau=0.5$_ for Yahoo! ans and Infobot dataset respectively when WHO was used
as external source. We also tried to vary _$\tau$_ beyond 1.0 to determine
whether it yielded a trend as shown in Table 7. Higher values of temperature
seem to degrade model performance. We found that optimal temperature range is
between $[0.1-1]$. Existing research in model distillation
(hinton2015distilling, ) has also empirically found that lower values of
temperature yield better performance.

We also compared model performance in terms of precision when $\lambda$ and
$\tau$ are varied for $\lambda$-sim models and temperature based models
respectively as shown in Figure 4. Temperature based models peak at one value
but do not have a clear trend indicating that one needs to explore different
$\tau$ values at the time of training for better performance. On the other
hand, we observe that adding external information also helps the $\lambda$-sim
models until a certain threshold. Overall, both sets of models show that free-
text external information can be incorporated to improve answer ranking
performance.

#####  RQ3: What kind of queries/questions does the model attend to when
ranking relevant/non-relevant answers?

Attention based models have a very unique feature: they can aid explaining the
internal workings of neural network models. We inspect what kind of
queries/questions in external datasets does our model pay attention to while
ranking relevant or non-relevant answers. Figure 5 shows one such example of
Yahoo! question and incorporation of TREC data. At the time of scoring
relevant answer, the model gives higher weight to some queries compared to
others. In the example, for instance, it assigns more weight to queries
associated with masks or COVID virus response to weather changes. We observe
higher attention weights for questions when relevant answers are ranked than
when non-relevant answers are scored. An example question, a relevant and non-
relevant answer along with model attention weights on TREC queries are shown
from the Infobot data in Figure 6 respectively. It shows a similar trend where
attention weights are high for external queries that are closely associated
with the question answer text.

Overall, our experiments show that curated external information is useful for
improving community question answering task. Our experiments also indicate
that this external knowledge need not always be structured text. However, it
is worth noting that curated and reliable external sources may not always be
available for all domains. We addressed a very niche task in this work, and
further research is required to extend it to incorporate multiple external
sources. We posit that with scalable attention mechanisms, this work can be
easily made tractable for large external sources containing thousands or
millions of entries in the future.

Figure 5. Y! ques, its rel and non-rel ans and questions with $\tau$-att
model’s attention values for TREC queries. Figure 6. Infobot ques, its rel and
non-rel ans and questions with $\tau$-att model’s attention values for TREC
queries.

##  6\. Conclusion

Question answering platforms provide users with effective and easy access to
information. These platforms also provide content on rapidly evolving
_sensitive_ topics such as disease outbreaks (such as COVID-19) where it is
also useful to use external _vetted_ information for ranking answers. Existing
work only exploits knowledge bases which have some limitations that makes it
difficult to use them for community Q&A for rapidly evolving topics such as
wild-fires or earthquakes. In this work, we tried to evaluate the
effectiveness of external (free text or semi-structured) information in
improving answer ranking models. We argue that simple question-answer text
matching may be insufficient and in presence of external knowledge, but
temperature regulated attention models can distill information better which in
turn yields higher performance. Our proposed model with temperature regulated
attention, when evaluated on two public datasets showed significant
improvements by augmenting information from two _external_ curated sources of
information. In future, we aim to expand these experiments to other categories
such as disaster relief and scale the attention mechanism to include multiple
external sources in one model.

## References

  * [1] A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. S. Weld.  Specter: Document-level representation learning using citation-informed transformers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282, 2020.
  * [2] D. Cohen and W. Croft.  End to end long short term memory networks for non-factoid question answering.  pages 143–146, 09 2016.
  * [3] D. H. Dalip, M. A. Gonçalves, M. Cristo, and P. Calado.  Exploiting user feedback to learn to rank answers in q&a forums: A case study with stack overflow.  In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, page 543–552, New York, NY, USA, 2013. Association for Computing Machinery.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [5] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network, 2015.
  * [6] L. Hong and B. D. Davison.  A classification-based approach to question answering in discussion boards.  In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 171–178, 2009.
  * [7] J. Hu, S. Qian, Q. Fang, and C. Xu.  Attentive interactive convolutional matching for community question answering in social multimedia.  In Proceedings of the 26th ACM International Conference on Multimedia, MM ’18, page 456–464, New York, NY, USA, 2018. Association for Computing Machinery.
  * [8] J. Hu, S. Qian, Q. Fang, and C. Xu.  Hierarchical graph semantic pooling network for multi-modal community question answer matching.  In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, page 1157–1165, New York, NY, USA, 2019. Association for Computing Machinery.
  * [9] H. Huang, X. Wei, L. Nie, X. Mao, and X.-S. Xu.  From question to text: Question-oriented feature attention for answer selection.  ACM Transactions on Information Systems, 37:1–33, 10 2018.
  * [10] B. Kratzwald, A. Eigenmann, and S. Feuerriegel.  Rankqa: Neural question answering with answer re-ranking.  CoRR, abs/1906.03008, 2019.
  * [11] P. Nakov, D. Hoogeveen, L. Màrquez, A. Moschitti, H. Mubarak, T. Baldwin, and K. Verspoor.  Semeval-2017 task 3: Community question answering.  arXiv preprint arXiv:1912.00730, 2019.
  * [12] L. Nie, X. Wei, D. Zhang, X. Wang, Z. Gao, and Y. Yang.  Data-driven answer selection in community qa systems.  IEEE transactions on knowledge and data engineering, 29(6):1186–1198, 2017.
  * [13] A. Poliak, M. Fleming, C. Costello, K. W. Murray, M. Yarmohammadi, S. Pandya, D. Irani, M. Agarwal, U. Sharma, S. Sun, et al.  Collecting verified covid-19 question answer pairs.  2020\.
  * [14] A. Rücklé and I. Gurevych.  Representation learning for answer selection with LSTM-based importance weighting.  In IWCS 2017 — 12th International Conference on Computational Semantics — Short papers, 2017.
  * [15] A. Severyn and A. Moschitti.  Structural relationships for large-scale learning of answer re-ranking.  In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 741–750, 2012.
  * [16] Y. Shen, Y. Deng, M. Yang, Y. Li, N. Du, W. Fan, and K. Lei.  Knowledge-aware attentive neural network for ranking question answer pairs.  In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 901–904, New York, NY, USA, 2018. Association for Computing Machinery.
  * [17] Y. Shen, W. Rong, Z. Sun, Y. Ouyang, and Z. Xiong.  Question/answer matching for cqa system via combining lexical and sequential information.  In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 275–281. AAAI Press, 2015.
  * [18] D. Su, Y. Xu, T. Yu, F. B. Siddique, E. J. Barezi, and P. Fung.  CAiRE-COVID: A question answering and multi-document summarization system for covid-19 research.  arXiv 2005.03975, 2020.
  * [19] M. Surdeanu, M. Ciaramita, and H. Zaragoza.  Learning to rank answers on large online qa collections.  In ACL, 2008.
  * [20] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang.  Trec-covid: Constructing a pandemic information retrieval test collection, 2020.
  * [21] B. Wang, X. Wang, C.-J. Sun, B. Liu, and L. Sun.  Modeling semantic relevance for question-answer pairs in web social communities.  In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1230–1238, 2010.
  * [22] L. Yang, Q. Ai, D. Spina, R.-C. Chen, L. Pang, W. B. Croft, J. Guo, and F. Scholer.  Beyond factoid qa: Effective methods for non-factoid answer sentence retrieval.  In European Conference on Information Retrieval, pages 115–128. Springer, 2016.
  * [23] X. Yang, M. Khabsa, M. Wang, W. Wang, A. H. Awadallah, D. Kifer, and C. L. Giles.  Adversarial training for community question answer selection based on multi-scale matching.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 395–402, 2019.
  * [24] X. Zhang, S. Li, L. Sha, and H. Wang.  Attentive interactive neural networks for answer selection in community question answering.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 3525–3531. AAAI Press, 2017.
  * [25] Y. Zhang, S. Qian, Q. Fang, and C. Xu.  Multi-modal knowledge-aware hierarchical attention network for explainable medical question answering.  In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, page 1089–1097, New York, NY, USA, 2019. Association for Computing Machinery.
  * [26] X. Zhou, B. Hu, Q. Chen, B. Tang, and X. Wang.  Answer sequence learning with neural networks for answer selection in community question answering.  arXiv preprint arXiv:1506.06490, 2015.