#  Synthetic Disinformation Attacks on Automated Fact Verification Systems

Anonymous  Yibing Du†, Antoine Bosselut‡∗, Christopher D. Manning† Authors
contributed equally

###### Abstract

Automated fact-checking is a needed technology to curtail the spread of online
misinformation. One current framework for such solutions proposes to verify
claims by retrieving supporting or refuting evidence from related textual
sources. However, the realistic use cases for fact-checkers will require
verifying claims against evidence sources that could be affected by the same
misinformation. Furthermore, the development of modern NLP tools that can
produce coherent, fabricated content would allow malicious actors to
systematically generate adversarial disinformation for fact-checkers.

In this work, we explore the sensitivity of automated fact-checkers to
synthetic adversarial evidence in two simulated settings: Adversarial
Addition, where we fabricate documents and add them to the evidence repository
available to the fact-checking system, and Adversarial Modification, where
existing evidence source documents in the repository are automatically
altered. Our study across multiple models on three benchmarks demonstrates
that these systems suffer significant performance drops against these attacks.
Finally, we discuss the growing threat of modern NLG systems as generators of
disinformation in the context of the challenges they pose to automated fact-
checkers.

##  1 Introduction

From QAnon’s deep state111https://www.nytimes.com/article/what-is-qanon.html
to anti-vaccination campaigns (Germani and Biller-Andorno 2021),
misinformation and disinformation have flourished in online ecosystems. As
misinformation continues to induce harmful societal effects, factchecking
online content has become critical to ensure trust in the information found
online.222https://nationalpress.org/topic/the-truth-about-fact-checking/
However, manual efforts to filter misinformation cannot keep pace with the
scale of online information that must be reliably verified to avoid false
claims spreading and affecting public
opinion.333https://fivethirtyeight.com/features/why-twitters-fact-check-of-
trump-might-not-be-enough-to-combat-misinformation/ Consequently, new research
in automated fact-checking explores designing systems that can rapidly
validate political, medical, and other domain-specific claims made and shared
online (Thorne and Vlachos 2018; Guo, Zhang, and Lu 2019).

Figure 1: Outline of our two settings for adversarial injection of poisoned
content into fact-checker evidence repositories.

A popular emergent paradigm in automated fact-checking, Fact Extraction and
Verification (Fever; Thorne et al. 2018), frames the problem as claim
verification against a large repository of evidence documents. As one of the
first large-scale datasets designed in this framework, Fever was released with
185k annotated claims that could be verified against Wikipedia articles. When
checking a claim, systems designed in this framework search for related
documents in the database, and retrieve relevant supporting or refuting
evidence from these sources. Then, these systems evaluate whether the
retrieved evidence sentences validate or contradict the claim, or whether
there is not enough information to make a judgment. More recently, the SciFact
(Wadden et al. 2020) and CovidFact (Saakyan, Chakrabarty, and Muresan 2021)
benchmarks re-purposed this framework for the scientific domain by releasing
datasets of medical claims to be verified against scientific content (Wang et
al. 2020). While this framework has led to impressive advances in fact
verification performance (Ye et al. 2020; Pradeep et al. 2021), current
benchmarks assume that the available evidence database contains only valid,
factual information.

However, check-worthy claims are often made about new events that may not be
verifiable against extensive catalogues, and that must be checked rapidly to
avoid strategic disinformation spread (Vosoughi, Roy, and Aral 2018).
Consequently, deployed fact-checkers will need to operate in settings where
their available evidence is collected from contemporaneous reporting, which
may be inadvertently sharing the same misinformation, or which may be
intentionally influenced by systematic disinformation campaigns. Currently,
malicious actors remain limited by the cost of running disinformation
campaigns (DiResta et al. 2018) and the risks of operational
discovery,444https://www.lawfareblog.com/outsourcing-disinformation impeding
the scale at which they can deploy these campaigns, and thus the balance of
real and false content that fact-checkers must distinguish. However, the
development of NLP tools capable of generating coherent disinformation
(Zellers et al. 2019; Buchanan et al. 2021) would allow malicious actors to
overload contemporaneous content with adversarial information (Brundage et al.
2018) and bias the evidence bases used by automated fact-checkers.

Furthermore, even in settings where claims may be verified against established
and trusted knowledge, misinformation can still find its way into repositories
of documents used by fact-checking systems (Kumar, West, and Leskovec 2016).
Wikipedia, for example, which underlies Fever (and other benchmarks; Petroni
et al. 2021), acknowledges that much of the content on the platform may be
incorrect, and remain so for long periods of
time.555https://en.wikipedia.org/wiki/
Wikipedia:Wikipedia_is_not_a_reliable_source For example, the Croatian
Wikipedia was contaminated by pro-nationalist bias over a period of at least
10 years.666https://meta.wikimedia.org/wiki/
Croatian_Wikipedia_Disinformation_Assessment-2021 Moreover, studies have
uncovered articles on Wikipedia that were edited to provide favorable accounts
on specific topics (e.g., workers at a medical device company edited articles
to present an optimistic view toward treatments that used their
product777https://www.theatlantic.com/business/archive/2015/08/
wikipedia-editors-for-pay/393926/). Modern NLP tools would allow malicious
users to scale up production of disinformation on these platforms, and
increase the perception of false consensus or debate on these topics.

In this paper, we evaluate whether automated disinformation generators can
effectively contaminate the evidence sets of fact verification systems, and
demonstrate that synthetic disinformation drastically lowers the performance
of these systems. We define adversarial attacks in two settings: Adversarial
Addition (AdvAdd; §3), where synthetically-generated documents are added to
the document base, and Adversarial Modification (AdvMod; §4), where additional
automatically-generated information is inserted into existing documents. In
both settings, we curate a large collection of adversarial disinformation
documents that we inject into the pipelines of existing fact-checking systems
developed for the Fever, SciFact, and CovidFact shared tasks.888We will
release these documents under a Terms of Use to promote research in fact-
checking systems in adversarial settings

Our results demonstrate that these systems are significantly affected by the
injection of poisoned content in their evidence bases, with large absolute
performance drops on all models in both settings. Furthermore, our analysis
demonstrates that these systems are sensitive to even small amounts of
evidence contamination, and that synthetic disinformation is more influential
at deceiving fact verification systems compared to human-produced false
content. Finally, we provide a discussion of our most important findings, and
their importance in the context of continued advances in NLP systems.999Our
code can be found at:
https://github.com/Yibing-Du/adversarial-factcheck

##  2 Background

In this section, we review the formulation of automated fact checking as fact
extraction and verification, and recent advances in automated generation of
textual disinformation.

### Automated Fact-checking: Task

Current systems research in automated fact-checking often follows the fact
verification and extraction procedure of receiving a natural language claim
(e.g., “Hypertension is a common comorbidity seen in COVID-19 patients”),
collecting supporting evidence from a repository of available documents (e.g.,
scientific manuscripts), and making a prediction about the claim’s veracity
based off the collected supporting evidence. Below, we define the two stages
of this pipeline: evidence retrieval and claim verification.

#### Evidence retrieval

The evidence retrieval stage of fact verification systems is typically
decomposed into two steps: document retrieval and sentence retrieval. During
document retrieval, documents in the evidence repository that are relevant to
the claim are selected. Existing methods typically use information retrieval
methods to rank documents based on relevance (Thorne et al. 2018; Wadden et
al. 2020) or use public APIs of commercial document indices (Hanselowski et
al. 2019; Saakyan, Chakrabarty, and Muresan 2021) to crawl related documents.
In the sentence retrieval stage, individual sentences from these retrieved
documents are selected with respect to their relevance to the claim, often
using textual entailment (Hanselowski et al. 2019), or sentence similarity
(Thorne et al. 2018) methods. Typically, the number of retrieved sentences is
capped for computational efficiency.

#### Claim verification

The claim verification stage of the pipeline evaluates the veracity of the
claim with respect to the evidence sentences retrieved in the previous stage.
Depending on the content found in the supporting sentences, each claim can
typically be classified as supported (SUP), refuted (REF), or not enough
information (NEI, though some benchmarks omit this label). Systems must
aggregate and weigh the evidence sentences to predict the most likely label.

Fever Claim | Starrcade was an annual professional wrestling event that began in 1988.
---|---
Original | Starrcade (1988) was the sixth annual Starrcade professional wrestling pay-per-view (PPV) event
| produced under the National Wrestling Alliance (NWA) banner .
Grover | Starrcade was a monthly professional wrestling event for the decades between 1988 and 2003 that ran
| for the entirety of a weekend in Boston , Mass.
Media Cloud | Goldberg’s perfect 173-0 streak ended at Starrcade 1998 after Kevin Nash scored the fateful pinfall
| with the help of Scott Hall and his taser gun.
SciFact Claim | Taxation of sugar-sweetened beverages had no effect on the incidence rate of type II diabetes in India.
Original | The 20% SSB tax was anticipated to reduce overweight and obesity prevalence by 3.0% … and type 2
| diabetes incidence by 1.6% … among various Indian subpopulations over the
period 2014-2023
Grover | … analysis of a “cone-by-one” kind of survey question in India reached out to -9 145 trillion , including
| 2,557 separate instances of type II diabetes (which is comparable to the
prevalence rate in Pakistan …
Table 1: Sample AdvAdd document excerpts generated by Grover for the Fever and
SciFact datasets.

### Automated Fact-checking: Datasets

We briefly describe the provenance and structure of our studied datasets and
refer the reader to the original works for in-depth descriptions of the
construction of these resources.

#### Fever

The Fever testbed (Thorne et al. 2018) is a dataset of 185,445 claims (145,449
train, 19,998 dev, 19,998 test) with corresponding evidence to validate them
drawn from articles in Wikipedia. Because of its scale and originality, the
Fever dataset is one of the most popular benchmarks for evaluating fact
verification systems (Yoneda et al. 2018; Nie, Chen, and Bansal 2019; Zhou et
al. 2019; Zhong et al. 2020; Subramanian and Lee 2020).

#### SciFact

The SciFact dataset (Wadden et al. 2020) contains 1,409 expert-annotated
scientific claims and associated paper abstracts. SciFact presents the
challenge of understanding scientific writing as systems must retrieve
relevant sentences from paper abstracts and identify if the sentences support
or refute a presented scientific claim. It has emerged as a popular benchmark
for evaluating scientific fact verification systems (Pradeep et al. 2021).

#### CovidFact

The CovidFact dataset (Saakyan, Chakrabarty, and Muresan 2021) contains 1,296
crowdsourced claims crawled (and filtered) from the /r/COVID19 subreddit. The
evidence is composed of documents provided with these claims when they were
posted on the subreddit along with resources from Google Search queries for
the claims. Refuted claims were automatically-generated by altering key words
in the original claims.

### Synthetic Disinformation Generation

Recent years have brought considerable improvements in the language generation
capabilities of neural language models (Lewis et al. 2020; Ji et al. 2020;
Brown et al. 2020; Holtzman et al. 2020), allowing users of these systems to
pass off their generations as human-produced (Ippolito et al. 2020). These
advances have raised dual-use concerns as to whether these tools could be used
to generate text for malicious purposes (Radford et al. 2019; Bommasani et al.
2021), which humans would struggle to detect (Clark et al. 2021).

Specific studies have focused on whether neural language models could be used
to generate disinformation that influences human readers (Kreps, McCain, and
Brundage 2020; Buchanan et al. 2021). One such study directly explored this
possibility by training Grover, a large-scale, billion-parameter language
model on a large dataset of newswire text with the goal of generating text
that resembles news (Zellers et al. 2019). In human evaluations of the model’s
generated text, the study found that human readers considered the
synthetically-generated news to be as trustworthy as human-generated content.
While the authors found that neural language models could identify fake,
generated content when finetuned to detect distributional patterns in the
generated text, they hypothesized that future detection methods would need to
rely on external knowledge (e.g., Fever).

##  3 Adversarial Addition:
Evidence Repository Poisoning

Evidence | CorefBERT Acc. | KGAT Acc. | MLA Acc.
---|---|---|---
(Ye et al. 2020) | (Liu et al. 2020b) | (Kruengkrai et al. 2021)
Total | REF | NEI | Total | REF | NEI | Total | REF | NEI
Original | 73.05 | 74.03 | 72.07 | 70.76 | 72.50 | 69.01 | 75.92 | 78.71 | 73.13
AdvAdd-min | 34.80 | 47.22 | 22.38 | 34.08 | 48.63 | 19.52 | 60.93 | 73.04 | 48.81
AdvAdd-full | 28.59 | 39.63 | 17.54 | 29.02 | 42.45 | 15.59 | 51.86 | 71.84 | 31.87
AdvAdd-oracle | 21.18 | 27.09 | 15.26 | 23.43 | 31.47 | 15.38 | 29.05 | 29.76 | 28.33
Table 2: Effect of AdvAdd on Fever claim verification. We bold the largest
performance drop relative to the original evidence.

In this section, we simulate the potential vulnerability of fact-checking
models to database pollution with misinformation documents by injecting
synthetically-generated false documents into the evidence sets of fact
verification models, and assess the impact on the performance of these
systems.

Benchmark | Evidence Source | N
---|---|---
Fever |  FeverDB | 5,416,537
Grover Docs | 995,829
MediaCloud Docs | 74,273,342
SciFact | Scientific Abstracts | 5,183
Grover Docs | 21,963
CovidFact | Google Search Results | 1,003
Grover Docs | 2,709
Table 3: Corpus statistics of evidence repositories

### Approach

Our method, Adversarial Addition (AdvAdd), uses Grover to produce synthetic
documents for a proposed claim, and makes these fake documents available to
the fact verification system when retrieving evidence. As Grover requires a
proposed article title and publication venue (i.e., website link) as input to
generate a fake article, we use each claim as a title and set the article
venue to wikipedia.com. We generate 10 articles for each claim and split them
into paragraphs (n.b., Fever DB contains first paragraphs of Wikipedia
articles and SciFact contains abstracts of scientific articles). Statistics
for the number of documents generated for each benchmark are reported in Table
3. Additional implementation details for the experimental setting of each
benchmark can be found in the Appendix.

###  Fever Study

#### Setup

For the Fever benchmark, we select three high-ranking models from the
leaderboard101010https://competitions.codalab.org/competitions/18814#results
with open-source implementations: KGAT (Liu et al. 2020b), CorefBERT (Ye et
al. 2020), and MLA (Kruengkrai, Yamagishi, and Wang 2021). For document
retrieval, all models use the rule-based method developed by Hanselowski et
al. (2019), which uses the MediaWiki API to retrieve relevant articles based
on named entity mentions in the claim. For each claim and poisoned document,
we extract all keywords and retrieve associated Wikipedia pages. If we find
overlaps between the associated Wikipedia pages of a claim and a poisoned
document, then the poisoned document is matched with the claim for document
retrieval. Once the retrieved documents are available, the KGAT and CorefBERT
models use a BERT-based (Devlin et al. 2019) sentence retriever to rank
evidence sentences based on relevance to the claim (trained using pairwise
loss). The MLA sentence retriever expands on this approach with hard negative
sampling from the same retrieved documents to more effectively discriminate
context-relevant information that is irrelevant to the claim. Claim verifiers
vary between models, but are generally based off pretrained language models
(e.g., CorefBERT, MLA) or graph neural networks (e.g., KGAT). We use the REF
and NEI claims from the Fever development set to study how the preceding
systems are affected by the introduction of poisoned evidence.111111We discuss
results related to SUP claims in the Appendix.

Figure 2: CorefBERT’s predictions change from REF and NEI to SUP once AdvAdd
poisons the evidence set.

#### Impact of AdvAdd

We report the overall (and claim-stratified) performance change of the tested
models in Table 2. For all models, we see a significant performance drop when
Grover-generated paragraphs are introduced into the available evidence set
(AdvAdd-full), indicating that fact verification models are sensitive to
synthetically-generated information. This drop approaches the performance of
an oracle (AdvAdd-oracle), where only Grover-generated documents are made
available as evidence.

As confirmation that these attacks work as expected, we depict in Figure 2 how
model predictions change once the synthetic disinformation is added to the
evidence set. A significant number of claims that were originally predicted as
REF or NEI are now predicted as SUP with the injected poisoned evidence.
Consequently, we conclude that the poisoned evidence affects the model’s
predictions in the intended way, and that cross-label changes for different
pairings are rare. Furthermore, we also find that replacing the retrieved
poisoned evidence with random retrieved evidence from FeverDB does not cause
the same performance drop ($\sim$7% vs. $\sim$30%), indicating that these
effects are caused by the injection of poisoned evidence, and not merely the
replacement of potentially relevant evidence (see Appendix A for further
details).

#### Effect of disinformation scale

We also evaluate a setting where the attack is limited to retrieving only a
single contaminated evidence sentence (AdvAdd-min). The performance drops in
the min setting are still considerable, suggesting that even limited amounts
of injected disinformation can significantly affect downstream claim
verification performance.

Moreover, Figure 3 shows a histogram of the number of poisoned evidence
sentences retrieved per claim and a stratified analysis of the final
predictions. Figures 3 (a)(b) for AdvAdd-Grover (i.e., AdvAdd-full) show that
accuracy steadily drops for claims with more corresponding evidence from
poisoned documents; meanwhile, the likelihood that a prediction changes
increases with more poisoned evidence. We note that claims labeled NEI are far
more sensitive to the introduction of poisoned sentences than REF claims, even
as the rate of contamination is approximately the same between both types of
labels. While this result is promising because the model is more robust in the
presence of even minimal refuting evidence, it also demonstrates that fact
verification systems are more sensitive when no competing information is
presented to a particular viewpoint (i.e., data voids; Boyd and Gołebiewski
2018).

Grover Evidence

MediaCloud Evidence

(a) REF

(b) NEI

(c) REF

(d) NEI

Figure 3: Degree of evidence poisoning and resulting REF (a,c) and NEI (b,d)
claim verification accuracy for AdvAdd-Grover (a,b) AdvAdd-MediaCloud (c,d)

#### Quality of poisoned evidence

We also evaluate whether poisoned evidence produced by AdvAdd is of sufficient
quality to bypass potential human detectors. For 500 REF and 500 NEI claims
from Fever, we ran a study on Mechanical Turk where we presented three workers
with five retrieved evidence examples (which could be from AdvAdd or from
FeverDB) and asked them to identify which examples were machine-generated. Our
results show that humans underestimate the number of poisoned sentences (23.6%
recall), and do not distinguish machine- from human-generated evidence (48.6%
precision). While well-trained workers will improve at recognizing synthetic
content, our results demonstrate the challenge of distinguishing these
evidence sources for lay readers, pointing to the quality of the synthetic
content, and the potential for such an attack to remain undetected.

Source | Evidence Retrieval
---|---
Document | Sentence
Grover | 87% | 65%
MediaCloud | 99% | 17%
|  |
Source | Claim Verification
KGAT | CorefBERT
Original | 70.76 | 73.05
\+ Grover | 29.02 | 28.59
\+ MediaCloud | 67.10 | 70.44
Table 4: Statistics and performance relative to the source of poisoned evidence: Grover or MediaCloud Model | Evidence Set | Sentence | Sentence | Abstract | Abstract
---|---|---|---|---|---
selection | label | label | rationalized
VeriSci | Original | 47.69 | 42.62 | 51.03 | 48.45
(Wadden et al. 2020) | AdvAdd | 27.05 | 23.50 | 25.57 | 24.33
SciKGAT | Original | 55.61 | 51.69 | 58.04 | 57.41
(Liu et al. 2020a) | AdvAdd | 39.44 | 36.97 | 37.46 | 36.98
ParagraphJoint | Original | 53.63 | 43.59 | 55.52 | 49.55
(Li, Burns, and Peng 2021) | AdvAdd | 37.68 | 32.60 | 41.31 | 37.12
Table 5: Effect of AdvAdd evidence on the SciFact benchmark. We bold
performance drops relative to the original evidence.

#### Comparison with human-compiled online evidence

While we have shown that synthetic disinformation affects the performance of
downstream claim verifiers when present in their evidence sets, the threat
should be evaluated in comparison to the threat of already existing online
misinformation on the same topic. Consequently, we use the
MediaCloud121212mediacloud.org content analysis tool to crawl web content
related to Fever claims. We crawl English-language news since January 2018
that contains the keywords of a claim anywhere in their text and extract
articles with a title that contains at least one keyword from the claim.
Finally, we process these articles to have the same format as the original
Wikipedia database, yielding 74M total available documents for retrieval
(Table 3).

In Table 4, we report the performance of an AdvAdd setting where only
MediaCloud-crawled documents are available to the retriever compared to our
original setting where Grover-generated documents were available. We observe
that evidence crawled from online content has less of an influence on
downstream fact verification performance ($\sim$3% vs. $\sim$40% performance
drop). While we are able to retrieve far more documents from MediaCloud due to
the size of the database (99% of claims retrieve a document from MediaCloud),
the sentences from AdvAdd-Grover documents are selected more frequently in the
sentence retrieval step. While this gap would likely be less pronounced with
more contentious claims that yield competing viewpoints (Bush and Zaheer
2019), these results demonstrate that synthetic disinformation can be much
more targeted to a particular claim of interest. Figure 3 supports this
conclusion, where we observe smaller performance drops for AdvAdd-MediaCloud
(c,d) compared to AdvAdd-Grover (a,b) even when all retrieved sentences are
sourced from the poisoned evidence.

###  SciFact Study

#### Setup

For SciFact, we chose three systems for testing our attack: VeriSci (Wadden et
al. 2020), ParagraphJoint (Li, Burns, and Peng 2021), and SciKGAT (Liu et al.
2020a). The VeriSci model was released by the creators of the SciFact
benchmark and retrieves relevant abstracts to a claim using TF-IDF. The
ParagraphJoint model, one of the top systems on the SciFact leaderboard, uses
a word embedding-based method to retrieve abstracts. Both use a RoBERTa-based
module for rationale selection and label prediction. The SciKGAT model uses
the same evidence retrieval as VeriSci, but the KGAT model (Liu et al. 2020b)
to verify claims. We use the 300 claims from the development set to evaluate
our method. We generate Grover articles as with Fever, but we set the venue
URL to medicalnewstoday.com, which produces articles more likely to reflect
scientific and medical content.

#### Results

In Table 5, we observe large performance drops across all metrics for all
models. Furthermore, we note that our disinformation generator, Grover, is not
trained on large quantities of scientific documents of the same format as the
original evidence. Despite producing documents that are stylistically
different, the disinformation is still retrieved as evidence, and affects the
performance of the verifier.

###  CovidFact Study

#### Setup

We run our analysis on the baseline system from Saakyan, Chakrabarty, and
Muresan (2021). This model retrieves evidence documents for claims using
Google Search and then selects evidence sentences based off high cosine
similarity between sentence embeddings of the claims and individual evidence
sentences (Reimers and Gurevych 2019). A RoBERTa-based model predicts a
veracity label. We generate AdvAdd articles in the same manner as for SciFact,
and run our analysis on the 271 REF claims from the test set.

#### Results

In both the Top-1 and Top-5 settings from Saakyan, Chakrabarty, and Muresan
(2021), we observe a $\sim$14.4% performance drop on REF claims (83.8%
$\rightarrow$ 69.4%). We note that CovidFact random and majority accuracy is
only 67.6% due to a label imbalance.

Evidence | CorefBERT Acc. | KGAT Acc. | MLA Acc.
---|---|---|---
(Ye et al. 2020) | (Liu et al. 2020b) | (Kruengkrai et al. 2021)
Total | REF | NEI | Total | REF | NEI | Total | REF | NEI
Original | 73.05 | 74.03 | 72.07 | 70.76 | 72.50 | 69.01 | 75.92 | 78.71 | 73.13
AdvMod-KeyReplace | 53.83 | 66.50 | 41.15 | 42.82 | 68.90 | 16.74 | 60.93 | 81.83 | 40.02
AdvMod-paraphrase | 32.62 | 36.66 | 28.58 | 37.22 | 51.74 | 22.70 | 52.72 | 70.57 | 34.86
AdvMod-oracle (claim) | 4.78 | 7.94 | 1.61 | 11.90 | 23.78 | 0.02 | 25.17 | 45.96 | 4.37
Table 6: Effect of AdvMod on Fever claim verification. We bold the largest
performance drop relative to the original evidence.

Original | Damon Albarn’s debut album was released in 2011.
---|---
Paraphrase |  Albarn’s first album was released in 2011.
His debut album was released in 2011.
KeyReplace |  Matt Coldplay’s debut album was released in 202.
Stefan Blur’s debut album was released in 1822.

Table 7: Sample AdvMod sentences

##  4 Adversarial Modification:
Evidence Document Poisoning

In Section 3, we investigated the effect of adding disinformation documents to
the evidence repositories of fact verification systems, simulating the setting
where the dynamic pace of news might lead to fake information being used to
verify real-time claims. However, even in settings where information has more
time to settle and facts to crystallize, misinformation can still find its way
into repositories of documents used by fact-checking systems. Motivated by the
possibility of malicious edits being made to crowdsourced information
resources, we explore how NLP methods could be used to automatically edit
existing articles with fake content at scale.

### Approach

Our method, Adversarial Modification (AdvMod), simulates this setting in a
two-stage process. First, we use off-the-shelf NLP tools to generate modified
versions of the claim presented to the fact verifier. Then, we append our
modified claims to articles in the evidence base that are relevant to the
original claim. We modify the original claims in two ways.

In the paraphrase approach, we use a state-of-the-art paraphrasing model,
PEGASUS (Zhang et al. 2019), to generate paraphrased versions of the original
claim (see Table 7 for examplea). These paraphrases generally retain the
meaning of the claim, but often remove contextualizing information that would
be found in the context of the article in which the new sentence is inserted.
Because the paraphrase method attempts to produce synthetic evidence that is
semantically equivalent to the original claim, we test its efficacy relative
to a method that merely introduces irrelevant content to the evidence
document. Motivated by Jia and Liang (2017), we alter a claim by applying
heuristics such as number alteration, antonym substitution, and entity
replacement with close neighbors according to embedding similarity (Bojanowski
et al. 2017). These modifications should not confuse humans, but would affect
sensitive fact verification systems, providing a competitive baseline for
assessing the strength of AdvMod-paraphrase.

Finally, our oracle reports the performance when the claim itself is appended
to an evidence document.

### Results

Our results in Table 6 demonstrate that injecting poisoned evidence sentences
into existing documents is an effective method for fooling fact verification
systems. Our AdvMod-paraphrase method causes a significant drop on all tested
models for both REF and NEI labeled claims. Furthermore, we also note that
AdvMod-paraphrase achieves larger performance drops than the baseline method,
AdvMod-KeyReplace, for most claim types (the KGAT model is slightly more
sensitive to the baseline AdvMod-KeyReplace for the NEI claims), indicating
that injections of disinformative content are more effective than non-targeted
perturbances to the evidence (e.g., AdvMod-KeyReplace).

##  5 Discussion

Adding synthetic content to the evidence bases of fact verifiers significantly
decreases their performance. Below, we discuss interesting findings and
limitations of our study.

#### Synthetic vs. human disinformation

As mentioned in Section 3, the performance of our test systems is more
sensitive to poisoned evidence generated from Grover than retrieved from
MediaCloud, even as the number of documents retrieved from MediaCloud far
exceeds the number generated from Grover. While Fever claims may not generally
be worth opposing online (leading to less directly adversarial content being
retrieved from MediaCloud), we note that language models have no such
limitations, and can generate large quantities of disinformation about any
topic. Consequently, while misinformation already makes its way into retrieval
search results (Bush and Zaheer 2019), language models could cheaply skew the
distribution of content more drastically (Bommasani et al. 2021), particularly
on topics that receive less mainstream coverage, but may be of import to a
malicious actor (Starbird et al. 2018).

#### Language models as a defense

In the AdvAdd and AdvMod oracle settings, all tested systems performed better
on claims labeled REF than for claims labeled NEI. This result implies that
the Grover-generated evidence was less adversarial for these claims, or that
the pretrained models which these systems use to encode the claim and evidence
sentences were more robust against claims that should be refuted.
Consequently, we conclude that language models encode priors about the
veracity of claims, likely from the knowledge they learn about entities during
pretraining (Petroni et al. 2019), a conclusion also supported by
contemporaneous work in using standalone language models as fact-checkers (Lee
et al. 2020, 2021). While this property can be an advantage in some settings
(i.e., language models pretrained on reliable text repositories will be
natural defenses against textual misinformation), it will also be a liability
when previously learned erroneous knowledge will counteract input evidence
that contradicts it. Finally, we note that the presence of implicit knowledge
in language models affecting the interpretation of input evidence implies that
the training corpora of these LMs could be attacked to influence downstream
fact verification. Prior work has explored poisoning task-specific training
datasets (Wallace et al. 2021). As disinformation becomes more prevalent
online, the pretraining corpora of LMs will require more careful curation to
avoid learning adversarial content.

#### Limitations

We identify three main limitations to our study. First, the Fever document
retrievers use the MediaWiki API to collect relevant Wikipedia articles based
on entity mentions in the claim. We assume our synthetic content could be
included in the retrieved documents if it were titled with a mention of the
named entities in the claim. For SciFact, this limitation is not present
because synthetic abstracts are retrieved using statistical IR methods.
Second, our method AdvAdd uses the actual claim to generate the synthetic
article. In the absence of explicit coordination, synthetic poisoned evidence
would be generated without knowledge of the exact claim formulation, reducing
the realistic correspondence between the claim and the synthetic
disinformation. If the Grover model directly copied the claim during
generation, performance drops might be overestimated based on unrealistically
aligned evidence. For the AdvAdd-full setting, we measure that this issue
arises in $\sim$20% of claims, which are predicted incorrectly more often, but
does not affect the conclusions of our study. Finally, our Fever and CovidFact
studies are run using only claims labeled as REF and NEI, which we discuss in
more detail in the Appendix.

##  6 Conclusion

In this work, we evaluate the robustness of fact verification models when we
poison the evidence documents they use to verify claims. We develop two
poisoning strategies motivated by real world capabilities: AdvAdd, where
synthetic documents are added to the evidence set, and AdvMod, where synthetic
sentences are added to existing documents in the evidence set. Our results
show that these strategies significantly decrease claim verification accuracy.
While these results are troubling, we are optimistic that improvements in
automated synthetic content detection, particularly by online platforms with
considerable resources, combined with human audits of fact-checker evidence
(and their source), may still potentially mitigate many attempted
disinformation campaigns.

## References

  * Bojanowski et al. (2017) Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ , 5\.
  * Bommasani et al. (2021) Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bosselut, A.; Brunskill, E.; Brynjolfsson, E.; Buch, S.; Card, D.; Castellon, R.; Chatterji, N. S.; Chen, A.; Creel, K.; Davis, J. Q.; Demszky, D.; Donahue, C.; Doumbouya, M.; Durmus, E.; Ermon, S.; Etchemendy, J.; Ethayarajh, K.; Fei-Fei, L.; Finn, C.; Gale, T.; Gillespie, L. E.; Goel, K.; Goodman, N. D.; Grossman, S.; Guha, N.; Hashimoto, T.; Henderson, P.; Hewitt, J.; Ho, D. E.; Hong, J.; Hsu, K.; Huang, J.; Icard, T. F.; Jain, S.; Jurafsky, D.; Kalluri, P.; Karamcheti, S.; Keeling, G.; Khani, F.; Khattab, O.; Koh, P. W.; Krass, M.; Krishna, R.; Kuditipudi, R.; Kumar, A.; Ladhak, F.; Lee, M.; Lee, T.; Leskovec, J.; Levent, I.; Li, X. L.; Li, X.; Ma, T.; Malik, A.; Manning, C. D.; Mirchandani, S. P.; Mitchell, E.; Munyikwa, Z.; Nair, S.; Narayan, A.; Narayanan, D.; Newman, B.; Nie, A.; Niebles, J. C.; Nilforoshan, H.; Nyarko, J.; Ogut, G.; Orr, L.; Papadimitriou, I.; Park, J.; Piech, C.; Portelance, E.; Potts, C.; Raghunathan, A.; Reich, R.; Ren, H.; Rong, F.; Roohani, Y. H.; Ruiz, C.; Ryan, J.; R’e, C.; Sadigh, D.; Sagawa, S.; Santhanam, K.; Shih, A.; Srinivasan, K.; Tamkin, A.; Taori, R.; Thomas, A. W.; Tramèr, F.; Wang, R. E.; Wang, W.; Wu, B.; Wu, J.; Wu, Y.; Xie, S. M.; Yasunaga, M.; You, J.; Zaharia, M.; Zhang, M.; Zhang, T.; Zhang, X.; Zhang, Y.; Zheng, L.; Zhou, K.; and Liang, P. 2021.  On the Opportunities and Risks of Foundation Models.  _ArXiv_ , abs/2108.07258.
  * Boyd and Gołebiewski (2018) Boyd, D.; and Gołebiewski, M. 2018.  Data Voids: Where Missing Data Can Easily Be Exploited.  Available online at https://datasociety.net/wp-content/uploads/2018/05/Data˙Society˙Data˙Voids˙Final˙3.pdf.
  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  _ArXiv_ , abs/2005.14165.
  * Brundage et al. (2018) Brundage, M.; Avin, S.; Clark, J.; Toner, H.; Eckersley, P.; Garfinkel, B.; Dafoe, A.; Scharre, P.; Zeitzoff, T.; Filar, B.; Anderson, H.; Roff, H.; Allen, G. C.; Steinhardt, J.; Flynn, C.; hÉigeartaigh, S. O.; Beard, S.; Belfield, H.; Farquhar, S.; Lyle, C.; Crootof, R.; Evans, O.; Page, M.; Bryson, J.; Yampolskiy, R.; and Amodei, D. 2018.  The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation.  _arXiv preprint arXiv:1802.07228_.
  * Buchanan et al. (2021) Buchanan, B.; Lohn, A.; Musser, M.; and Sedova, K. 2021.  _Truth, Lies, and Automation: How Language Models Could Change Disinformation_.  Center for Security and Emerging Technology.
  * Bush and Zaheer (2019) Bush, D.; and Zaheer, A. 2019.  Bing’s Top Search Results Contain an Alarming Amount of Disinformation.  _Internet Observatory News_.
  * Clark et al. (2021) Clark, E.; August, T.; Serrano, S.; Haduong, N.; Gururangan, S.; and Smith, N. A. 2021.  All That’s ’Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text.  _arXiv preprint arXiv:2107.00061_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL_.
  * DiResta et al. (2018) DiResta, R.; Shaffer, K.; Ruppel, B.; Sullivan, D.; Matney, R. C.; Fox, R.; Albright, J.; and Johnson, B. 2018.  The tactics & tropes of the Internet Research Agency.  Available online at https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1003&context=senatedocs.
  * Germani and Biller-Andorno (2021) Germani, F.; and Biller-Andorno, N. 2021.  The anti-vaccination infodemic on social media: A behavioral analysis.  _PLOS ONE_ , 16(3): 1–14.
  * Guo, Zhang, and Lu (2019) Guo, Z.; Zhang, Y.; and Lu, W. 2019.  Attention Guided Graph Convolutional Networks for Relation Extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 241–251. Florence, Italy: Association for Computational Linguistics.
  * Hanselowski et al. (2019) Hanselowski, A.; Zhang, H.; Li, Z.; Sorokin, D.; Schiller, B.; Schulz, C.; and Gurevych, I. 2019.  UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification.  arXiv:1809.01479.
  * Holtzman et al. (2020) Holtzman, A.; Buys, J.; Forbes, M.; and Choi, Y. 2020.  The Curious Case of Neural Text Degeneration.  _ArXiv_ , abs/1904.09751.
  * Ippolito et al. (2020) Ippolito, D.; Duckworth, D.; Callison-Burch, C.; and Eck, D. 2020.  Automatic Detection of Generated Text is Easiest when Humans are Fooled.  In _ACL_.
  * Ji et al. (2020) Ji, Y.; Bosselut, A.; Wolf, T.; and Celikyilmaz, A. 2020.  The Amazing World of Neural Language Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts_ , 37–42. Online: Association for Computational Linguistics.
  * Jia and Liang (2017) Jia, R.; and Liang, P. 2017.  Adversarial Examples for Evaluating Reading Comprehension Systems.  In _EMNLP_.
  * Jiang et al. (2021) Jiang, L.; Bosselut, A.; Bhagavatula, C.; and Choi, Y. 2021.  “I’m Not Mad”: Commonsense Implications of Negation and Contradiction.  In _NAACL_.
  * Kassner and Schütze (2020) Kassner, N.; and Schütze, H. 2020.  Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly.  In _ACL_.
  * Kreps, McCain, and Brundage (2020) Kreps, S.; McCain, R. M.; and Brundage, M. 2020.  All the News That’s Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation.  _Journal of Experimental Political Science_.
  * Kruengkrai, Yamagishi, and Wang (2021) Kruengkrai, C.; Yamagishi, J.; and Wang, X. 2021.  A Multi-Level Attention Model for Evidence-Based Fact Checking.  In _Findings of ACL_.
  * Kumar, West, and Leskovec (2016) Kumar, S.; West, R.; and Leskovec, J. 2016.  Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes.  In _WWW_.
  * Lee et al. (2021) Lee, N.; Bang, Y.; Madotto, A.; Khabsa, M.; and Fung, P. 2021.  Towards Few-shot Fact-Checking via Perplexity.  In _NAACL_.
  * Lee et al. (2020) Lee, N.; Li, B. Z.; Wang, S.; tau Yih, W.; Ma, H.; and Khabsa, M. 2020.  Language Models as Fact Checkers?  _ArXiv_ , abs/2006.04102.
  * Lewis et al. (2020) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7871–7880. Online: Association for Computational Linguistics.
  * Li, Burns, and Peng (2021) Li, X.; Burns, G.; and Peng, N. 2021.  A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification.  arXiv:2012.14500.
  * Liu et al. (2020a) Liu, Z.; Xiong, C.; Dai, Z.; Sun, S.; Sun, M.; and Liu, Z. 2020a.  Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling.  In _Findings of EMNLP_.
  * Liu et al. (2020b) Liu, Z.; Xiong, C.; Sun, M.; and Liu, Z. 2020b.  Fine-grained Fact Verification with Kernel Graph Attention Network.  In _Proceedings of ACL_.
  * Nie, Chen, and Bansal (2019) Nie, Y.; Chen, H.; and Bansal, M. 2019.  Combining Fact Extraction and Verification with Neural Semantic Matching Networks.  In _Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Parikh et al. (2016) Parikh, A.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016.  A Decomposable Attention Model for Natural Language Inference.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2249–2255. Austin, Texas: Association for Computational Linguistics.
  * Petroni et al. (2021) Petroni, F.; Piktus, A.; Fan, A.; Lewis, P.; Yazdani, M.; De Cao, N.; Thorne, J.; Jernite, Y.; Karpukhin, V.; Maillard, J.; Plachouras, V.; Rocktäschel, T.; and Riedel, S. 2021.  KILT: a Benchmark for Knowledge Intensive Language Tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Online: Association for Computational Linguistics.
  * Petroni et al. (2019) Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019.  Language Models as Knowledge Bases?  In _EMNLP_.
  * Pradeep et al. (2021) Pradeep, R.; Ma, X.; Nogueira, R.; and Lin, J. J. 2021.  Scientific Claim Verification with VerT5erini.  _ArXiv_ , abs/2010.11930.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019.  Language Models are Unsupervised Multitask Learners.  Available online at https://openai.com/blog/better-language-models/.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3982–3992. Hong Kong, China: Association for Computational Linguistics.
  * Saakyan, Chakrabarty, and Muresan (2021) Saakyan, A.; Chakrabarty, T.; and Muresan, S. 2021.  COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic.  In _ACL/IJCNLP_.
  * Starbird et al. (2018) Starbird, K.; Arif, A.; Wilson, T.; Van Koevering, K.; Yefimova, K.; and Scarnecchia, D. 2018.  Ecosystem or Echo-System? Exploring Content Sharing across Alternative Media Domains.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 12(1).
  * Subramanian and Lee (2020) Subramanian, S.; and Lee, K. 2020.  Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 7798–7809. Online: Association for Computational Linguistics.
  * Thorne and Vlachos (2018) Thorne, J.; and Vlachos, A. 2018.  Automated Fact Checking: Task Formulations, Methods and Future Directions.  In _COLING_.
  * Thorne et al. (2018) Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2018.  FEVER: a Large-scale Dataset for Fact Extraction and VERification.  In _NAACL-HLT_.
  * Vosoughi, Roy, and Aral (2018) Vosoughi, S.; Roy, D.; and Aral, S. 2018.  The spread of true and false news online.  _Science_ , 359(6380).
  * Wadden et al. (2020) Wadden, D.; Lin, S.; Lo, K.; Wang, L. L.; van Zuylen, M.; Cohan, A.; and Hajishirzi, H. 2020.  Fact or Fiction: Verifying Scientific Claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 7534–7550. Online: Association for Computational Linguistics.
  * Wallace et al. (2021) Wallace, E.; Zhao, T. Z.; Feng, S.; and Singh, S. 2021.  Concealed Data Poisoning Attacks on NLP Models.  In _NAACL_.
  * Wang et al. (2020) Wang, L. L.; Lo, K.; Chandrasekhar, Y.; Reas, R.; Yang, J.; Eide, D.; Funk, K.; Kinney, R. M.; Liu, Z.; Merrill, W.; Mooney, P.; Murdick, D.; Rishi, D.; Sheehan, J.; Shen, Z.; Stilson, B. B. S.; Wade, A. D.; Wang, K.; Wilhelm, C.; Xie, B.; Raymond, D. A.; Weld, D. S.; Etzioni, O.; and Kohlmeier, S. 2020.  CORD-19: The COVID-19 Open Research Dataset.  _arXiv: 2004.10706_.
  * Ye et al. (2020) Ye, D.; Lin, Y.; Du, J.; Liu, Z.; Li, P.; Sun, M.; and Liu, Z. 2020.  Coreferential Reasoning Learning for Language Representation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics.
  * Yoneda et al. (2018) Yoneda, T.; Mitchell, J.; Welbl, J.; Stenetorp, P.; and Riedel, S. 2018.  UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF).  In _FEVER_.
  * Zellers et al. (2019) Zellers, R.; Holtzman, A.; Rashkin, H.; Bisk, Y.; Farhadi, A.; Roesner, F.; and Choi, Y. 2019.  Defending Against Neural Fake News.  In _Advances in Neural Information Processing Systems 32_.
  * Zhang et al. (2019) Zhang, J.; Zhao, Y.; Saleh, M.; and Liu, P. J. 2019.  PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization.  arXiv:1912.08777.
  * Zhong et al. (2020) Zhong, W.; Xu, J.; Tang, D.; Xu, Z.; Duan, N.; Zhou, M.; Wang, J.; and Yin, J. 2020\.  Reasoning Over Semantic-Level Graph for Fact Checking.  In _ACL_.
  * Zhou et al. (2019) Zhou, J.; Han, X.; Yang, C.; Liu, Z.; Wang, L.; Li, C.; and Sun, M. 2019.  GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification.  In _Proceedings of ACL 2019_.

##  Appendix A Additional AdvAdd Results

#### Effect of increased evidence:

When we increase the number of evidence sentences from 5 to 10 during the
claim verification step, we see minimal difference in the performance drop.
When only one adversarial sentence is inserted the performance drop decreases
from 23.87% for 5 sentences to 23.69% for 10 sentences for REF claims. For NEI
claims, the performance drop actually increases from 49.49% (5 sentences) to
50.81% (10 sentences).

#### Effect of random evidence:

The performance drops reported in Section 3 might be due to correct evidence
being removed from the retrieved set, rather than poisoned evidence being
introduced. To test this possibility, we ran an experiment where we replace
the retrieved poisoned evidence sentences with randomly chosen sentences from
FeverDB. If poisoned evidence does not adversely affect the claim verifier
beyond the replacement of potentially useful supporting sentences, we should
expect minimal performance drop from this baseline. However, we find that when
random sentences are inserted, the performance drop for the KGAT model shrinks
from 30.05% to 6.63% for REF claims and from 53.42% to 7.73% for NEI claims.
Similarly, in a proxy for the AdvAdd-min setting, where only a single sentence
is replaced, the shrink is from 28.87% to 1.86% for REF claims and from 49.49%
to 7.01% for NEI claims. These results demonstrate that the performance drop
comes from the addition of adversarial evidence instead of only the removal of
possibly correct evidence, indicating that the claim verifier is directly
sensitive to the content of poisoned evidence.

| Claim Verifier | %
---|---|---
Sentence | REF Claims | NEI Claims | Corrupt
Retriever $\downarrow$ | MLA | KGAT | MLA | KGAT | Sents
MLA | 71.84 | 49.89 | 31.87 | 18.18 | 50.1
KGAT | 23.42 | 42.45 | 30.24 | 15.59 | 62.5
Table 8: Effect of the sentence retriever on MLA and KGAT.

KGAT

(a) KeyReplace

(b) Paraphrase (d) Paraphrase

CorefBERT

(c) KeyReplace (f) Paraphrase

MLA

(e) KeyReplace

Figure 4: Claim verification accuracy by model and poisoning technique for
different amounts of AdvMod evidence poisoning. The dashed line shows AdvMod
performance when the claim itself is added as a single evidence sentence.

#### Effect of sentence retrieval performance

Our results in Table 2 show the MLA model (Kruengkrai, Yamagishi, and Wang
2021) is more robust to poisoned evidence. To explore the cause of this
finding, we swap the sentence retrievers of the KGAT and MLA models to
disentangle the contributions of their sentence retrievers and claim
verifiers. In Table 8, we find that when the MLA sentence retriever is paired
with the KGAT claim verifier, the performance of this joint system
(highlighted in blue) increases relative to using the full KGAT model.
Meanwhile, when paired with the KGAT sentence retriever, the MLA claim
verifier achieves lower performance on REF claims (highlighted in red) than
the full KGAT model, indicating that the strength of the MLA model may stem
from a more powerful retriever. However, the MLA claim verifier is also more
robust for NEI claims regardless of the retriever, implying that the claim
verifiers of these models may suffer from exposure bias, whereby they overfit
to their sentence retrievers during training. They learn to expect certain
patterns from the evidence returned by these retrievers and when given
evidence from another distribution (i.e., a different retriever) at test time,
they perform worse on examples that require retrieval (i.e.REF claims).

##  Appendix B Additional AdvMod Results

In Figure 4, we report additional AdvMod results measuring the performance
change as a function of the amount of documents we re-write in the document
base. Our results show that editing multiple documents is more likely to cause
the prediction to flip. However, even a single edit to an evidence document
can often cause a large performance drop.

Original SUP Claim | Ron Weasley is part of the Harry Potter series as the eponymous wizard’s best friend.
---|---
Manual Counterclaim | Ron Weasley is part of the Harry Potter series as the eponymous wizard’s worst enemy.
Grover Output | According to the ’Harry Potter: The Story of Ron Weasley’ campaign , the prince of Goblet
| of Fire and Hogwarts castle is part of the series as the eponymous wizard’s
worst enemy .
Automatic Counterclaim | Ron Weasley is part of the Harry bee series as the eponymous wizard’s best friend.
Grover Output | Unlike the remainder of Harry Potter , Ron Weasley is more than just a character on the
| Hogwarts kiddie roster.
Original SUP Claim | Bessie Smith was a singer.
Manual Counterclaim | Bessie Smith cannot sing.
Grover Output | Bessie Smith , a kindergarten teacher in Chicago , Illinois , is scheduled to perform for the
| 2014 Eagles open game against the Giants on Monday Night Football (18:00 ET
on ESPN) .
Automatic Counterclaim | Bessie Smith was a vegetarian.
Grover Output | So how did Bessie Smith become a vegetarian ? The black American woman sat on the
| United States House Floor as a member of the Congressional Choir during the
70s , when ,
| she actually was a vegan.
Table 9: Sample AdvAdd document excerpts generated by Grover for SUP claims in
the Fever dataset.

##  Appendix C Performance on Supports Labels

Our studies on the Fever and CovidFact benchmarks focused on the claims
labeled refutes (REF) and not enough information (NEI). Claims labeled as
supports (SUP) were not included in the study due to the challenge of
generating effective poisoned evidence for them. Generating poisoned evidence
for Fever NEI and REF claims is more straightforward because we can use
variants of the claim (e.g., paraphrases) or the claim itself as input to
Grover to produce poisoned evidence. However, poisoned evidence can only be
generated for SUP claims if suitable counterclaims can be formulated as an
input to Grover.

To test our method on SUP claims, counterclaims were generated in the
following manner: we adapted the automatic counterclaim generation method from
Saakyan, Chakrabarty, and Muresan (2021), which selects salient words in the
original claim using an aggregate attention score for each token based on how
much it is attended to by the other tokens in the sequence. Then, the most
salient token is replaced by sampling from a masked language model. Once we
generate a set of counterclaims using this method, we validate them using the
decomposable attention NLI model of Parikh et al. (2016) by selecting the ones
with the highest contradiction score with respect to the original claim. Then,
we provided these counterclaims as inputs to Grover to generate poisoned
evidence. However, we found this method was not effective for generating
poisoned evidence. When we ran our AdvAdd setting using the KGAT model, we
observed a surprising performance increase from 86.2% to 87.5% on label
prediction accuracy, indicating that the generated poisoned evidence
unexpectedly helps the model make correct predictions.

Examples in Table 9 depict the limitations of this approach. In the first
example, the change made to generate the counterclaim does not change the
semantics of the claim, merely changing the word “Potter” to “bee,” which is
not a coherent coherent counterclaim that would produce poisoned evidence from
Grover refuting the original claim. In the second example, the counterclaim is
coherent, but does not semantically contradict from the original claim, making
the poisoned evidence less likely to be retrieved when the original claim is
provided to the fact verification model. Furthermore, we note the difficulty
of generating counterclaims for many claims in Fever. First, many of the
original claims are not easily falsifiable (e.g., “Girl is an album”), making
it challenging to formulate a suitable counterclaim. Other are statements that
cannot be falsified without using explicit negation terms (e.g., “Stripes had
a person appear in it”). As language models struggle to understand inferences
of negated statements (Kassner and Schütze 2020; Jiang et al. 2021), Grover
may just as often generate content that ends up supporting the original claim,
rather than contradicting it, when seeded with such explicitly negated
counterclaims.

However, the focus of our study is whether synthetically-generated adversarial
evidence could be generated at scale to mislead fact verification systems.
While generating counterclaims automatically at scale is necessary to perform
this study on Fever SUP claims, an adversary would be more likely to generate
synthetic content for a single claim (or related claims) of interest (rather
than a large set). Consequently, they would be able to manually write the
counterclaim that was needed to generate poisoned evidence, mitigating the
need for automatic counterclaim generation methods. We evaluate this
possibility by writing counterclaims for a sample of 100 Fever SUP claims,
allowing us to guarantee semantic contradiction of the original claim by the
counterclaim (as seen in Table 9). However, we find that, once again,
performance does not drop as the original performance on these claims was 92%
and rose to 93% once the poisoned evidence from Grover was available. Though
manually writing contradicting statements guarantees coherence and quality of
the counterclaim, Grover may still fail to generate content as intended and
may even affirm the original claim, possibly because the model has been
trained on Wikipedia, indicating that Grover may encode implicit knowledge
about many of the entities for which it must produce poisoned evidence, as
discussed in Section 5. For example, we note that one of the counterclaims
from Table 9 — “Bessie Smith was a vegetarian” — does not relate to singing at
all. However, the Grover model produces singing-related content anyway (Bessie
Smith was a singer).

##  Appendix D Reproducibility Details

This paper relies on the existing Fever, SciFact, and CovidFact datasets,
which are publicly available. To test our method, we use the same evaluation
metrics proposed by the dataset authors: label accuracy for Fever (Thorne et
al. 2018), the sentence selection, sentence label, abstract label, and
abstract rationalized metrics for SciFact (Wadden et al. 2020), and the Top-1
and Top-5 label accuracy for CovidFact (Saakyan, Chakrabarty, and Muresan
2021). We also introduce our own datasets of adversarial evidence generated by
Grover and PEGASUS (Zhang et al. 2019). They will be made publicly available
with a license that allows for research use. For computational experiments in
this paper, the main source code is available at:

https://github.com/Yibing-Du/adversarial-factcheck

These experiments were run on an NVIDIA Quadro RTX 8000 GPU. Because we do not
train these models from scratch, but instead use existing released models, we
only run each evaluation once since the result is deterministic. Consequently,
there are no hyperparameters to tune. We use the default hyperparameters
provided with the codebases of the models evaluated.