# Defending Against Poisoning Attacks in Open-Domain Question Answering

Orion Weller* Aleem Khan*
Nathaniel Weir Dawn Lawrie Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Recent work in open-domain question answering (ODQA) has shown that
adversarial poisoning of the input contexts can cause large drops in accuracy
for production systems. However, little to no work has proposed methods to
defend against these attacks. To do so, we introduce a new method that uses
query augmentation to search for a diverse set of retrieved passages that
could answer the original question. We integrate these new passages into the
model through the design of a novel confidence method, comparing the predicted
answer to its appearance in the retrieved contexts (what we call Confidence
from Answer Redundancy, e.g. CAR). Together these methods allow for a simple
but effective way to defend against poisoning attacks and provide gains of
5-20% exact match across varying levels of data poisoning.

##  1 Introduction

00footnotetext: * Authors contributed equally

Open-domain question answering (ODQA) is the task of answering a given
question, based on evidence from a large corpus of documents. In order to do
so, a system generally first retrieves a smaller subset of documents
(typically between 5-100) and then answers the question based on those
documents. Recent work in ODQA has resulted in many well-curated datasets,
which evaluate a model’s ability to answer questions on a wide array of topics
Kwiatkowski et al. (2019); Joshi et al. (2017); Dunn et al. (2017); Yang et
al. (2015).

However, most internet users search across less-carefully curated sources,
where malicious actors are able to affect articles that may be input to an
ODQA system (Figure 1). Furthermore, even in curated knowledge sources like
Wikipedia, we frequently see attacks (e.g. malicious edits/fake pages) that in
rare cases can go undetected for years.111For examples of notable false
information attacks on Wikipedia, see the  “Reliability of Wikipedia” page

Figure 1: An example of a poisoning attack on an open-domain question
answering (ODQA) pipeline. The passages have been adversarially poisoned to
replace Obama’s correct birthplace (in blue) with the poisoned answer (in
red). With the new information, the ODQA model outputs the poisoned answer.

Recent work has recognized the potential for bad actors to influence automated
knowledge-intensive NLP systems that involve retrieval: Du et al. (2022)
explored how poisoned information affects automated fact verification systems
using sparse non-neural information retrieval systems, while Chen et al.
(2022) and Longpre et al. (2021) have studied the role that knowledge
conflicts play in ODQA pipelines, mainly for the sake of understanding how
these models use parametric vs non-parametric knowledge rather than in the
context of data poisoning. Despite these recent works (see Appendix A for
more) highlighting the vulnerability of existing methods, the problem of
defending against these data poisoning attacks is still understudied, with
limited to no published efforts in the area.

Thus, we seek to show the effects of data poisoning and propose a simple but
effective defense. Building on the intuition that information is usually
available in multiple places and that it is unlikely that all sources (or
pages) will be poisoned, we propose a novel query augmentation scheme to
gather a larger set of diverse passages. We also propose a new confidence
method to decide when to use the newly gathered contexts vs the original,
which we call Confidence from Answer Redundancy (CAR).

Our proposed approach involves no gradient updates, can easily be applied to
existing frameworks, and uses a simple resolution approach to arrive at the
predicted answer. Together, our methods provide a 5 to 20 point gain in exact
match, helping to reduce the negative effects of data poisoning and
misinformation attacks on ODQA.

##  2 Experimental Details

We seek to simulate realistic misinformation attacks on a curated knowledge
source; thus, for our experiments we use Wikipedia as the knowledge collection
for both original and augmented queries, poisoning entire articles at a time
(although one could use a different collection for the augmented queries).
Note that we also experimented with poisoning the top retrieved contexts or
random contexts and found similar results (Appendix B).

###  2.1 Data

For our experiments we use Natural Questions (NQ) Kwiatkowski et al. (2019)
and TriviaQA Joshi et al. (2017), two popular datasets for open-domain
question answering. Furthermore, previous work on conflicts in ODQA has used
these datasets in their experiments Chen et al. (2022). The Natural Question
dataset was gathered by collecting real-user queries typed into Google Search,
while TriviaQA was collected by scraping question and answer pairs from trivia
websites, and then matching the answers to Wikipedia passages.

We simulate the data poisoning through the code available from Longpre et al.
(2021), which introduced the problem of knowledge conflicts in ODQA. Instead
of simply providing conflicts for model analysis, we use them as adversarial
poisoning strategies as they provide realistic fake answers to the question.
This method uses the answers to the questions to suggest an entity of the same
type, using SpaCY NER, that can be inserted into the passages in place of the
correct answer. This allows for entity-based substitutions that keep the
semantic organization of the context, such as replacing dates with dates,
numbers with numbers, people with people, and so on.

###  2.2 Models

Following previous work, we use Fusion-in-Decoder (FiD), an encoder-decoder
architecture that generates an answer given retrieved evidence Izacard and
Grave (2021) by first encoding $N$ passages and then concatenating those
representations and passing them through the decoder. This model uses the DPR
bi-encoder architecture for retrieval, embedding both documents and queries
into a single dense vector Karpukhin et al. (2020).

###  2.3 Metrics and Hyperparameters

Following previous work in question answering, we report Exact Match (EM) in
all of our experiments. We split the data into validation and test (taking 50%
of the data from Longpre et al. (2021) for each split) and use the validation
set to tune the CAR method’s hyperparameters.

For all our experiments we use a cluster of V100 GPUs, with each job running
on a 4 to 8 GPU node. We use the models as provided by the original authors
with default retriever hyperparameters.

##  3 Proposed Method

###  3.1 Query Augmentation

We hypothesize that in cases of conflicting evidence in knowledge collections
for _factoid_ based questions, there will generally be more evidence for the
correct answer as opposed to incorrect answers. As an example, imagine a
question asking “Who was the ruler of England in 1616?" with a corresponding
attack to the James I Wikipedia page (see Figure 4 in the Appendix). However,
there exists redundant information throughout Wikipedia, so alternate
questions that find contexts on other Wikipedia pages (such as his mother’s,
Mary Queen of Scots) will still have the correct answer.

To create these alternate questions that will still find the correct answer
but with more diverse passages, we propose a query augmentation scheme that
has similarities to classical query augmentation in information retrieval
Singhal et al. (2001); Carpineto and Romano (2012). We generate new questions
for each original question by prompting GPT-3 (davinci-002) from Brown et al.
(2020), although any strong paraphrase-like model or large-language model
would likely suffice. Note that these query augmentations are not necessarily
paraphrases (as they strive to be as different as possible while still leading
to the correct answer).

This method effectively approaches the original question from several
different points of view, in the hopes of retrieving a wider breadth of
contexts. For each query in the dataset, we prompt GPT-3 with the following:
"Write 10 new wildly diverse questions with different words that have the same
answer as {Original Question}", thus generating approximately 10 augmented
questions per original question. Finally, we retrieve the 100 most relevant
contexts for those augmented questions.222Note that if the model retrieves a
passage that was initially poisoned we swap the original text for the poisoned
text.

When we compare these newly retrieved passages to the passages retrieved by
the original question we find that they do provide a more diverse set of
passages. Figure 2 shows the distribution of new passages retrieved, with
almost all retrieving at least 20 or more new passages and a decent amount
having an entirely new set of 100 passages.

Natural Questions

TriviaQA

Figure 2: Number of new passages retrieved per augmented question (e.g., a
question in the 100 bin would have 100 new contexts not retrieved by the
original). Natural Questions is on top and TriviaQA on bottom.

###  3.2 Confidence from Answer Redundancy

In order to identify the best augmented queries with their corresponding new
passages, we derive a novel method, CAR, for measuring ODQA confidence. This
new method of confidence relies on the fact that ODQA models use a large
number of contexts (typically around 100). Thus, one way of measuring the
effectiveness of an ODQA pipeline’s retrieval is to measure how often the
predicted answer string occurs in the retrieved contexts. For example, if the
predicted answer appears only once in all 100 contexts, this may mean that the
retriever was not able to find many documents relevant to the query. Of
course, this is not always the case, as some percent of the time the correct
answer may only be present once in the entire search corpus. However, for
popular entities (those asked about in Natural Questions and TriviaQA) there
are generally many articles containing the correct answer. Overall, the more
frequently the predicted answer appears in the contexts, the more likely that
the retrieval was both successful and plentiful (e.g. redundant). Thus, CAR
conveys some aspect of the model’s confidence in the retrieval.

There are many ways we could measure confidence from answer redundancy. We
choose the simple case and leave further exploration of answer redundancy in
ODQA to future work. Given a set of documents $D$, where $|D|=100$ typically,
we set a hyperparameter $k$ to determine the cutoff for CAR (in practice we
use $k=5$, found by tuning on the dev set). If the model retrieves more than
$k$ unique passages that contain the predicted answer string, we classify the
model as confident and vice versa. As mentioned earlier, this strategy is not
perfect, as a model can still be correct without this heuristic, but it does
provide signal for downstream calibration efforts. We use this technique as a
crucial part of our resolution method below.

Figure 3: Main results showing the effect of data poisoning and various
defense strategies. Left shows results on TriviaQA while right shows results
on Natural Questions. Q and C stand for question and context respectively.

###  3.3 Answer Resolution

The query augmentation technique proposed earlier provides us with a large and
diverse set of passages and questions. The answer redundancy confidence helps
us to know when a particular query is confident about its predicted answer.
However, how can we effectively combine more than one question and passage set
into one final predicted answer?

We use the following strategies to explore this question, with their shortened
names in italics: (1) use the original question’s predicted answer, e.g. the
baseline, which doesn’t use the augmented questions or contexts (2) randomly
pick a new augmented question and use its predicted answer (3) take a majority
vote of the augmented question’s predictions and (4) use answer redundancy to
choose which questions to use. This last strategy uses CAR to decide whether
to choose the original question’s prediction and if not, uses a majority vote
over the augmented questions that are confident, filtered using CAR.

We also explore three different data type settings: (1) using the originally
retrieved contexts with the original/new questions (Original C), (2) using the
new augmented questions and their newly retrieved contexts (New Q, New C), or
(3) using the original question with the newly retrieved contexts from the
augmented questions (Original Q, New C).

##  4 Results

Figure 3 highlights our key findings using the FiD model. Note that all
results are filtered by those that the model originally predicted correctly,
thus making the original method have by definition 100% EM at the 0-article
poisoning level. As expected, as the amount of poisoned data given to the
model increases, performance decreases. Interestingly, using a majority vote
to resolve multiple answers results in lower performance than simply using the
original question, but does do better than selecting an augmented question at
random. When comparing the data setting used with these methods, using the
original question with the new contexts performs best, then the new questions
with the new contexts, and in last place are those that use the original
contexts. Overall, our approach, using both CAR and question augmentation
provide gains of 5-20% EM across both datasets (e.g. on NQ with 100 articles
poisoned, 29.3% EM vs the 10.9% baseline vs 20.3% for the best majority vote).

However, one might ask how many of these augmented questions are needed for
this approach to work well. To answer this question, we include Figure 5 in
Appendix C, showing that even one augmented query provides gains over the
baseline and that as the number of augmented queries continues to increase, so
does the performance.

One might also wonder why performance is non-zero when the number of poisoned
articles is equal to the number of contexts the model receives. From a manual
examination of these instances, we noticed that there was frequently an alias
of the correct answer that was still present (e.g. in a question with the
correct answer of “Canada" the poisoning did not replace “Canadian" and thus
allowed the model to make the correct inference). We note that these edge
cases are missed by the original poisoning method Longpre et al. (2021); we
leave more effective poisoning strategies as future work, as our focus is on
defending against them.

##  5 Conclusion

Our work defends against data poisoning attacks in open-domain question
answering through two novel methods: (1) the use of query augmentation to find
diverse passages that still correctly answer the question and (2) the use of
answer redundancy as a strategy for model confidence in its prediction. Our
proposed methods do not involve _any_ gradient updates and provide a
significant performance improvement. Thus, our work shows the effect of data
poisoning on state-of-the-art open-domain question-answering systems and
provides a way to improve poisoned performance by almost 20 points in exact
match. We hope that this work encourages future work in defending against data
poisoning attacks on knowledge sources.

##  6 Limitations and Impact Statement

Our work focuses on the TriviaQA and Natural Questions benchmarks, which
include questions about popular entities in Wikipedia. Our proposed defense
depends upon finding alternate sources of information outside of the poisoned
contexts. Thus, for entities that appear less often in the knowledge source,
our approach will not be as effective. In order to study this for uncommon
entities, more datasets will need to be developed, which we leave for future
work. Our work focuses on defending against data poisoning attacks to high
popularity entities, which by definition of popularity will be more frequently
asked about and will be more frequently attacked (and are also what is studied
in existing literature, e.g. Natural Questions/TriviaQA).

Our work shows the impact that disinformation attacks could have on Wikipedia
and provides an initial attempt to help remedy those attacks. We note that our
strategy does not have perfect accuracy and is still susceptible to attacks,
e.g. if there is no correct information in any context to be found, it will be
very difficult for ODQA systems to give the correct answer. We welcome
additional research to improve the resistance of ODQA systems to adversarial
attacks.

## References

  * Bartolo et al. (2021) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021.  Improving question answering model robustness with synthetic adversarial data generation.  _arXiv preprint arXiv:2104.08678_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Carpineto and Romano (2012) Claudio Carpineto and Giovanni Romano. 2012.  A survey of automatic query expansion in information retrieval.  _Acm Computing Surveys (CSUR)_ , 44(1):1–50.
  * Chen et al. (2022) Hung-Ting Chen, Michael J.Q. Zhang, and Eunsol Choi. 2022.  Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.
  * Clark and Gardner (2017) Christopher Clark and Matt Gardner. 2017.  Simple and effective multi-paragraph reading comprehension.  _arXiv preprint arXiv:1710.10723_.
  * Claveau (2021) Vincent Claveau. 2021.  Neural text generation for query expansion in information retrieval.  In _IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_ , pages 202–209.
  * Downey et al. (2006) Doug Downey, Oren Etzioni, and Stephen Soderland. 2006.  A probabilistic model of redundancy in information extraction.  Technical report, WASHINGTON UNIV SEATTLE DEPT OF COMPUTER SCIENCE AND ENGINEERING.
  * Du et al. (2022) Yibing Du, Antoine Bosselut, and Christopher D Manning. 2022.  Synthetic disinformation attacks on automated fact verification systems.  _arXiv preprint arXiv:2202.09381_.
  * Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017.  Searchqa: A new q&a dataset augmented with context from a search engine.  _ArXiv_ , abs/1704.05179.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _EACL_.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  _arXiv preprint arXiv:2006.09462_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lee et al. (2019) Seanie Lee, Donggyu Kim, and Jangwon Park. 2019.  Domain-agnostic question-answering with adversarial training.  _arXiv preprint arXiv:1910.09342_.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.  _arXiv preprint arXiv:2109.05052_.
  * Schwarzschild et al. (2021) Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. 2021.  Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks.  In _International Conference on Machine Learning_ , pages 9389–9398. PMLR.
  * Si et al. (2022) Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. 2022.  Revisiting calibration for question answering.  _arXiv preprint arXiv:2205.12507_.
  * Singhal et al. (2001) Amit Singhal et al. 2001.  Modern information retrieval: A brief overview.  _IEEE Data Eng. Bull._ , 24(4):35–43.
  * Wallace et al. (2019a) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a.  Universal adversarial triggers for attacking and analyzing NLP.  In _Empirical Methods in Natural Language Processing_.
  * Wallace et al. (2019b) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019b.  Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering.  _Transactions of the Association for Computational Linguistics_ , 7:387–401.
  * Wallace et al. (2020) Eric Wallace, Mitchell Stern, and Dawn Song. 2020.  Imitation attacks and defenses for black-box machine translation systems.  In _Empirical Methods in Natural Language Processing_.
  * Wang et al. (2021) Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021.  Pseudo-relevance feedback for multiple representation dense retrieval.  In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_ , pages 297–306.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  Wikiqa: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 2013–2018.

Figure 4: An example of the method to reduce the harm of poisoning attacks
(red documents indicate poisoning, blue indicates it contains the correct
answer, and gray indicates neutral/irrelevant information). By generating
augmented questions, the model is able to find additional correct answers that
are not poisoned. Figure 5: An ablation on the number of augmented queries
used. Note that as the number of augmented queries increases, so does the
performance.

##  Appendix A More Related Work

As a larger section of related work did not have space in the main paper, we
include more related work here.

#### Data Poisoning Attacks

Data poisoning attacks in NLP have a long history, with several prominent
works appearing in recent years including Wallace et al. (2019a, 2020);
Schwarzschild et al. (2021) focusing on various NLP tasks such as machine
translation, language modeling, etc. However, in the question answering space
most adversarial work is focused on making harder questions, rather than
simulating a real attack Wallace et al. (2019b); Lee et al. (2019). Those that
do focus on human attacks focus on the machine reading setting Bartolo et al.
(2021).

As mentioned in the main text, a nascent line of work has focused on knowledge
conflicts in open-domain question answering Chen et al. (2022); Longpre et al.
(2021). These works’ main motivation is to explore how ODQA models operate
under the influence of conflicts, mostly in the context of non-parametric vs
parametric knowledge. We extend these works by using their methods as
simulated attacks on a knowledge source and proposing efforts to defend
against these attacks.

#### Open-Domain Question Answering

Our work builds off of recent advances in ODQA, such as using Fusion-in-
Decoder Izacard and Grave (2021). Other work such as DPR Karpukhin et al.
(2020) showed promising results but has been improved upon by models that
encode a large number of contexts into a single reader model.

#### Query Augmentation

Query augmentation is a traditional information retrieval technique to augment
a given query to find a better set of documents Singhal et al. (2001);
Carpineto and Romano (2012). In classical terms, the strategy is usually to
expand the query, spelling out acronyms or adding synonyms. Recently, work has
begun to use neural models to generate these expansions Wang et al. (2021);
Claveau (2021). In our work, we use such a technique to create new queries
that will gather a diverse set of passages.

#### Confidence and Calibration of QA

Many works have focused on calibrating QA models so that they correctly
reflect probabilities that equal their actual correct answer rate Clark and
Gardner (2017); Kamath et al. (2020); Si et al. (2022); Jiang et al. (2021).
Our proposed confidence method is similar in that it measures when the model
will be more likely to be correct, however, it does not do calibration in the
sense of calibrated probabilities, instead giving a single value of
“confident" or “not confident."

Answer redundancy has been studied before in other NLP contexts, such as
Downey et al. (2006) in the information extraction task. We apply a similar
intuition of answer redundancy to the novel context of document inputs for
open-domain question answering.

##  Appendix B Alternate Poisoning Attacks

In the main section of the paper, we used poisoning attacks based on articles.
However, one could attack a system directly by going after its retrieved
results, either randomly poisoning $N$% or poisoning the top $N$%. We note
that we tried both settings and found similar results, with the main
difference that model performance declines slower (as randomly picking
contexts to poison is less likely to impact the model until higher levels of
poisoning).

##  Appendix C Number of Augmented Queries

In Figure C we see the results for how the number of augmented queries affects
performance for the Natural Questions 1% poisoned setting. This shows that
even one augmented query shows gains over the baseline method and that as the
number of augmented queries increases, so does the performance.