#  FaVIQ: FAct Verification from Information-seeking Questions

Jungsoo Park1, $*$ Sewon Min2, $*$ Jaewoo Kang,1 Luke Zettlemoyer,2 Hannaneh
Hajishirzi2,3
1Korea University 2University of Washington 3Allen Institute of AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Despite significant interest in developing general purpose fact checking
models, it is challenging to construct a large-scale fact verification dataset
with realistic claims that would occur in the real world. Existing claims are
either authored by crowdworkers, thereby introducing subtle biases that are
difficult to control for, or manually verified by professional fact checkers,
causing them to be expensive and limited in scale. In this paper, we construct
a challenging, realistic, and large-scale fact verification dataset called
FaVIQ, using information-seeking questions posed by real users who do not know
how to answer. The ambiguity in information-seeking questions enables
automatically constructing true and false claims that reflect confusions
arisen from users (e.g., the year of the movie being filmed vs. being
released). Our claims are verified to be natural, contain little lexical bias,
and require a complete understanding of the evidence for verification. Our
experiments show that the state-of-the-art models are far from solving our new
task. Moreover, training on our data helps in professional fact-checking,
outperforming models trained on the most widely used dataset FEVER or in-
domain data by up to 17% absolute. Altogether, our data will serve as a
challenging benchmark for natural language understanding and support future
progress in professional fact checking.111Data available at
https://faviq.github.io

††$*$ Equal Contribution

##  1 Introduction

Fact verification, a task of verifying the factuality of the natural language
claim, is an important NLP application (Cohen et al., 2011) and has also been
used to evaluate the amount of external knowledge a model has learned (Petroni
et al., 2021). However, it is challenging to construct fact verification data
with claims that contain realistic and implicit misinformation. Crowdsourced
claims from prior work such as FEVER (Thorne et al., 2018a) are written with
minimal edits to reference sentences, leading to strong lexical biases such as
the overuse of explicit negation and unrealistic misinformation that is less
likely to occur in real life (Schuster et al., 2019a). On the other hand, data
constructed by professional fact-checkers are expensive and are typically
small-scale (Hanselowski et al., 2019).

Figure 1:  An example of a refute claim on FaVIQ, constructed using ambiguity
in the information-seeking question, e.g., through a crossover of the year of
the film being released and being filmed.

We propose to use information-seeking questions (Kwiatkowski et al., 2019) to
construct a large-scale, challenging, and realistic fact verification dataset.
Information-seeking questions are inherently incomplete, because users are
asking about unfamiliar topics, and for example contain ambiguity (Min et al.,
2020) and false presuppositions (Kim et al., 2021). We introduce a new dataset
FaVIQ—FAct Verification derived from Information-seeking Questions. FaVIQ is
constructed based on ambiguity in information-seeking questions. Consider the
example in Figure 1. Users ask an ambiguous question because the filming of
the movie and the release of the movie are semantically close, and both can be
seen as the creation time. Therefore, claims generated through the crossover
of the disambiguation of the information-seeking question are likely to
contain misinformation that real users are easily confused with. We
automatically generate such claims by composing valid and invalid question-
answer pairs and transforming them into textual claims using a neural model.
The data is further augmented by claims from regular question-answer
annotations.

In total, FaVIQ consists of 188k claims. We manually verified a subset of
claims to ensure that they are as natural as human-written claims. Our
analysis shows that the claims have significantly lower local mutual
information—a metric that quantifies lexical bias in the data (Schuster et
al., 2019b)—than existing crowdsourced claims; claims involve diverse types of
distinct entities, events, or properties that are semantically close, being
more realistic and harder to verify without a complete understanding of the
evidence text.

Our experiments show that a model with no background knowledge performs only
slightly better than random guessing, and the state-of-the-art model achieves
an accuracy of 65%, leaving significant room for improvements. Furthermore,
training on FaVIQ improves the accuracy of verification of claims written by
professional fact checkers, outperforming models trained on the target data
only or pretrained on FEVER by up to 17% absolute. Together, our experiments
demonstrate that FaVIQ is a challenging benchmark as well as a useful resource
for professional fact checking.

##  2 Related Work

###  2.1 Fact verification

Fact verification is crucial both for real-world applications (Cohen et al.,
2011) and as a benchmark to evaluate the ability of the model to access and
process knowledge (Petroni et al., 2021).

One line of work has studied professional fact checking, dealing with claims
collected by processional fact checkers, typically in specific domains like
politics or news (Vlachos and Riedel, 2014; Ferreira and Vlachos, 2016;
Augenstein et al., 2019; Hanselowski et al., 2019). While such data contains
realistic claims that have occurred in the real world, it is expensive to
construct as it requires labor from professional fact checkers, and is mostly
small-scale. Moreover, it is less suitable as a benchmark due to lack of a
standard evidence corpus such as Wikipedia222For this reason, prior work on
professional fact checking assumes gold evidence document. and ambiguities in
labels.333Most claims fall into the mixture label, rather than support or
refute.

Other fact verification datasets are collected through crowdsourcing (e.g.,
FEVER (Thorne et al., 2018a) and its variants (Thorne et al., 2018b; Thorne
and Vlachos, 2019)) by altering a word or negating the reference text to
intentionally make true or false claims. This process leads to large-scale
datasets but with strong artifacts and unrealistic claims (Schuster et al.,
2019a; Thorne and Vlachos, 2019; Eisenschlos et al., 2021). Consequently, a
trivial claim-only baseline with no evidence achieves near 80% (Petroni et al.
(2021), verified in Section 4.1). While more recent work proposes new
crowdsourcing methods that alleviate artifacts (Schuster et al., 2021;
Eisenschlos et al., 2021), their claims are still written given particular
evidence text, being vulnerable to subtle lexical biases that can be hard to
explicitly measure.

In this paper, we construct a fact verification dataset by leveraging question
answering (QA) data, which in turn was collected from untrained crowdworkers.
Our data consists of claims that have significantly less lexical bias compared
to other crowdsouced claims (Figure 3), contain realistic misinformation that
people are likely to be confused about (Table 4), and are challenging to
current state-of-the-art models (Section 4.1). Moreover, training a model on
our data improves professional fact checking (Section 4.2).

###  2.2 QA to Verification Task

Prior work has investigated converting the QA task into entailment or fact
verification tasks (Demszky et al., 2018; Jiang et al., 2020; Pan et al.,
2021; Chen et al., 2021). Similar to ours, they use the correct and incorrect
answers to the questions and transform them into claims. However, they make
use of synthetic or annotated questions444 Annotated questions are simulated
by crowdworkers given the evidence text and the answer, having largely
different distributions from information-seeking questions (Lee et al., 2019;
Gardner et al., 2019).  and use incorrect answers written by crowdworkers or
predicted by the model. We instead use questions posed by real users to
reflect confusions that naturally occur while seeking information.

##  3 Data

Figure 2: Illustration of the data creation process in an overview. The data
consits of two sets (A and R) which differ in data sources and conversion
processes. For the A set, we use the disambiguated question-answer pairs and
generate support and refute claims from matching pairs (filmed–2000,
released–2001) and crossover pairs (filmed–2001, released–2000), respectively.
For the R set, we use the reference answer (Deckard Shaw) and the incorrect
prediction from DPR (Dominic Toretto) to generate support and refute claims,
respectively. $f$ denotes a model that transforms a question-answer pair to a
claim, for which we use T5 (Section 3.1.3).

###  3.1 Data Construction

We construct FaVIQ—FAct Verification derived from Information-seeking
Questions, consisting of claim-label pairs where the label is either support
or refute.555 Unlike FEVER, our data does not contain gold evidence text and
the task does not evaluate the evidence prediction. We include detailed
discussions in Appendix A, and instead provide silver evidence text as
detailed in Section 4.1.1.  The key idea to construct the data is to gather a
set of valid and invalid question-answer pairs (Section 3.1.2) from
annotations of information-seeking questions (Section 3.1.1), and then convert
each question-answer pair $(q,a)$ to a claim (Section 3.1.3). Figure 2
presents an overview of this process.

####  3.1.1 Data Sources

We use QA data from Natural Questions (NQ, Kwiatkowski et al. (2019)) and
AmbigQA (Min et al., 2020). NQ is a large-scale dataset consisting of the
English information-seeking questions mined from Google search queries.
AmbigQA provides disambiguated question-answer pairs for NQ questions, thereby
highlighting the ambiguity that is inherent in information-seeking questions.
Given an ambiguous question, it provides a set of multiple distinct answers,
each paired with a new disambiguated question that uniquely has that answer.

####  3.1.2 Composing Valid and Invalid QA Pairs

FaVIQ uses ambiguous questions and their disambiguation (denoted as A) and is
further augmented by data using regular question-answer pairs with an
assumption of no ambiguity (denoted as R).

##### From ambiguous questions (A)

We assume data consisting of a set of
$\left(q,\\{q_{1},a_{1}\\},\\{q_{2},a_{2}\\}\right)$, where $q$ is an
information seeking question that has $a_{1},a_{2}$ as multiple distinct
answers.666 If $q$ has more than two distinct answers, we sample two. This is
to construct a reasonable number of claims per $q$.  $q_{1}$ and $q_{2}$ are
disambiguated questions for the answers $a_{1}$ and $a_{2}$, i.e., $q_{1}$ has
$a_{1}$ as a valid answer and $a_{2}$ as an invalid answer. We use
$(q_{1},a_{1})$ and $(q_{2},a_{2})$ as valid question-answer pairs, and
$(q_{1},a_{2})$ and $(q_{2},a_{1})$ as invalid question-answer pairs.

This data is particularly well suited to fact checking because individual
examples require identification of entities, events, or properties that are
semantically close but distinct: the fact that a user asked an ambiguous
question $q$ without realizing the difference between $(q_{1},a_{1})$ and
$(q_{2},a_{2})$ indicates that the distinction is non-trivial and is hard to
notice without sufficient background knowledge about the topic of the
question.

##### From regular questions (R)

We use the QA data consisting of a set of $\left(q,a\right)$: an information-
seeking question $q$ and its answer $a$. We then obtain an invalid answer to
$q$, denoted as $a_{\mathrm{neg}}$, from an off-the-shelf QA model for which
we use the model from Karpukhin et al. (2020)—DPR followed by a span
extraction model. We carefully choose $a_{\mathrm{neg}}$ with tricks to obtain
the hard negative but not the false negative; details provided in Appendix A.
We use $(q,a)$ and $(q,a_{\mathrm{neg}})$ as a valid and an invalid question-
answer pair, respectively.

We can think of $(q,a_{\mathrm{neg}})$ as a hard negative pair chosen
adversarially from the QA model. This data can be obtained on a much larger
scale than the A set because annotating a single valid answer is easier than
annotating disambiguations.

####  3.1.3 Transforming QA pairs to Claims

We transform question-answer pairs to claims by training a neural model which
maps $(q,a)$ to a claim that is support if and only if $a$ is a valid answer
to $q$, otherwise refute. We first manually convert 250 valid or invalid
question-answer pairs obtained through Section 3.1.2 to claims. We then train
a T5-3B model (Raffel et al., 2020), using 150 claims for training and 100
claims for validation.

The best checkpoint was chosen based on the unnormalized exact match score of
the reference claims and the generated claims on the validation data, which we
found to correlate well with the quality of the generated claims. We also
pretrain the model on the data from Demszky et al. (2018) that converts
questions to declarative sentences, which we find marginally helpful. More
details are provided in Appendix A.

|  | Total | Support | Refute
---|---|---|---|---
Train | A | 17,008 | 8,504 | 8,504
R | 140,977 | 70,131 | 70,846
Dev | A | 4,260 | 2,130 | 2,130
R | 15,566 | 7,739 | 7,827
Test | A | 4,688 | 2,344 | 2,344
R | 5,877 | 2,922 | 2,955
Table 1: Data Statistics of FaVIQ. A denotes claims derived from ambiguous
questions and their disambiguations, and R denotes claims from regular
question-answer pairs.

###  3.2 Data Analysis

Data statistics for FaVIQ are listed in Table 1. It has 188k claims in total,
with balanced support and refute labels. We present quantitative and
qualitative analyses showing that claims on FaVIQ are as natural as human-
written claims, contain much less lexical bias than other crowdsourced
datasets do, and include misinformation that is realistic and harder to
identify.

##### Validation of data quality

Two native English speakers validated 160 random samples from FaVIQ. They were
asked to verify: (1) whether the claim is as natural as a human-written claim,
with three possible ratings (perfect, minor issues but comprehensible,
incomprehensible), and (2) whether the label of the claim (support or refute)
is correct.777Validators were allowed to use search engines, and were
encouraged to use the English Wikipedia as a primary source.

Validators found 91.6% of the A set and 87% of the R set to be natural, and 0%
to be incomprehensible. The rest have minor grammatical errors or typos, e.g.,
“hippocrates, aristotle” instead of “hippocrates and aristotle”; in most cases
these actually come from the original NQ questions which were human-authored,
indicating that these grammatical errors and typos occur in real life. The
validators also found that 98.2% of the claims are accurate (96.4% of A and
100% of R).

Data | Size | Length of the claims
---|---|---
Avg | Q1 | Q2 | Q3
Professional claims
Snopes | 16k | 12.4 | 10 | 11 | 14
SciFACT | 1k | 11.5 | 9 | 11 | 13
Crowdsourced claims
FEVER | 185k | 9.3 | 7 | 9 | 11
FM2 | 13k | 13.9 | 9 | 13 | 16.3
FaVIQ | 188k | 12.0 | 9 | 10.5 | 13.5
Table 2: Statistics of a variety of fact verification datasets. Avg and Q1–3
are the average and quantiles of the length of the claims based on whitespace
tokenization on the validation data; for FaVIQ, we report the macro-average of
the A set and the R set.

##### Comparison of size and claim length

Table 2 compares statistics of the claims from a variety of fact verification
datasets: Snopes (Hanselowski et al., 2019), SciFACT (Wadden et al., 2020),
FEVER (Thorne et al., 2018a), FM2 (Eisenschlos et al., 2021) and FaVIQ.

FaVIQ is as large-scale as FEVER, while its distributions of claim length is
much closer to claims authored by professional fact checkers (Snopes and
SciFACT; details in Section 4.2). We also compare with FM2 because it is the
most recently introduced fact verification dataset, constructed through a
carefully-designed multi-player game to better control for the lexical bias
seen in FEVER. FM2 is smaller scale, due to difficulty in scaling multi-player
games, and has claims that are slightly longer than professional claims,
possibly because they are intentionally written to be difficult.

Figure 3:  Plot of LMI scores of top 100 predictive bigrams for FEVER, FM2 and
FaVIQ (macro-averaged over the A set and the R set). S and R denotes
supportand refute, respectively.

##### Lexical cues in claims

We further analyze lexical cues in the claims on FEVER, FM2 and FaVIQ by
measuring local mutual information (LMI; Schuster et al. (2019b); Eisenschlos
et al. (2021)). LMI indicates the correlation between bigrams and the label.
More specifically, LMI is defined as:

$LMI(w,c)=P(w,c)~{}log\frac{P(w,c)}{P(w)\cdot P(c)},$

where $w$ denotes a bigram, $c$ is a label, and $P(\cdot)$ are estimated with
empirical counts (Schuster et al., 2019b).

The distributions of the LMI scores for the top-100 bigrams are shown in
Figure 3. The LMI scores of FaVIQ are significantly lower than those of FEVER
and FM2, indicating that FaVIQ contains significantly less lexical bias.
Tables 3 shows the top six bigrams with the highest LMI scores for FEVER and
FaVIQ. As highlighted, all of the top bigrams in refute claims of FEVER
contain negative expressions, e.g., “is only”, “incapable of”, “did not”. In
contrast, the top bigrams from FaVIQ do not include obvious negations, and the
top bigrams for different labels overlap significantly, strongly suggesting
the task will be more challenging to solve.

Dataset | Top Bigrams by LMI
---|---
FEVER-S | is a, a film, of the, is an, in the, in a
FEVER-R |  is only, only a, incapable of, is not, was only, is incapable
A set-S | on the, was the, the date, date of, in episode, is what
A set-R | of the, the country, at the, the episode, started in, placed at
R set-S | out on, on october, on june, released on, be 18, on august
R set-R | out in, on september, was 2015, of the, is the, released in

Table 3: Top bigrams with the highest LMI for FEVER and FaVIQ. S and R denotes
support and refute respectively. Highlighted bigrams indicate negative
expressions, e.g., “only”, “incapable” or “not”.

##### Qualitative analysis of the refute claims

We also analyzed 30 randomly sampled refute claims from FaVIQ and FEVER
respectively. We categorized the cause of misinformation as detailed in
Appendix B, and show three most common categories for each dataset as a
summary in Table 4.

On FaVIQ, 60% of the claims involve entities, events or properties that are
semantically close, but still distinct. For example, they are specified with
conjunctions (e.g., “was foreign minister” and “signed the treaty of
versailles from germany”), or share key attributes (e.g., films with the same
title). This means that relying on lexical overlap or partially understanding
the evidence text would lead to incorrect predictions; one must read the full
evidence text to realize that the claim is false. Furthermore, 16.7% involve
events, e.g., from filing for bankruptcy for the first time to completely
ceasing operations (Table 4). This requires full understanding of the
underlying event and tracking of state changes (Das et al., 2019; Amini et
al., 2020).

Conjunctions (33.3%)
---
C: Johannes bell was the foreign minister that signed the
treaty of versailles from germany. / E: Johannes bell served
as Minister of Colonial Affairs … He was one of the two
German representatives who signed the Treaty of Versailles.
Shared attributes (26.7%)
C: Judi bowker played andromeda in the 2012 remake of
the 1981 film clash of the titans called wrath of the titans.
E: Judi bowker … Clash of the Titans (1981).
Procedural event (16.7%)
C: Mccrory’s originally filed for bankruptcy on february
2002\. / E: McCrory Stores … by 1992 it filed for bankrupt-
cy. … In February 2002 the company ceased operation.
Negation (30.0%)
C: Southpaw hasn’t been released yet. E: Southpaw is an
American sports drama film released on July 24, 2015.
Cannot find potential cause (20.0%)
C: Mutiny on the Bounty is Dutch. E: Mutiny on the
Bounty is a 1962 American historical drama film.
Antonym (13.3%)
C: Athletics lost the world series in 1989. E: The 1989
World Series … with the Athletics sweeping the Giants.
Table 4:  Three most common categories based on 30 refute claims randomly
sampled from the validation set, for FaVIQ (top) and FEVER (bottom)
respectively. Full statistics and examples in Appendix B. C and E indicate the
claim and evidence text, respectively.

The same analysis on FEVER confirms the findings from Schuster et al. (2019a);
Eisenschlos et al. (2021); many of claims contain explicit negations (30%) and
antonyms (13%), with misinformation that is less likely to occur in the real
world (20%).888For instance, consider the claim “Mutiny on the Bounty is
Dutch” in Table 4. There is no Dutch producer, director, writer, actors, or
actress in the film—we were not able to find a potential reason that one would
believe that the film is Dutch.

##  4 Experiments

We first evaluate state-of-the-art fact verification models on FaVIQ in order
to establish baseline performance levels (Section 4.1). We then conduct
experiments on professional fact-checking datasets, to measure the
improvements from training on FaVIQ (Section 4.2).

###  4.1 Baseline Experiments on FaVIQ

####  4.1.1 Models

We experiment with two settings: a zero-shot setup where models are trained on
FEVER, and a standard setup where models are trained on FaVIQ. For FEVER, we
use the KILT (Petroni et al., 2021) version following prior work; we randomly
split the official validation set into equally sized validation and test sets,
as the official test set is hidden.

All models are based on BART (Lewis et al., 2020), a pretrained sequence-to-
sequence model which we train to generate either support or refute. We
describe three different variants which differ in their input, along with
their accuracy on FEVER by our own experiments.

Claim only BART takes a claim as the only input. Although this is a trivial
baseline, it achieves an accuracy of 79% on FEVER.

TF-IDF + BART takes a concatenation of a claim and $k$ passages retrieved by
TF-IDF, for which we use DrQA (Chen et al., 2017). It achieves 87% on FEVER.

DPR + BART takes a concatenation of a claim and $k$ passages retrieved by DPR
(Karpukhin et al., 2020), a dual encoder based model. It is the state-of-the-
art on FEVER based on Petroni et al. (2021), achieving an accuracy of 90%.

##### Implementation details

We use the English Wikipedia from 08/01/2019 following KILT (Petroni et al.,
2021). We take the plain text and lists provided by KILT and create a
collection of passages where each passage has up to 100 tokens. This results
in 26M passages. We set the number of input passages $k$ to 3, following
previous work (Petroni et al., 2021; Maillard et al., 2021). Baselines on
FaVIQ are jointly trained on the A set and the R set.

Training DPR requires a positive and a negative passage—a passage that
supports and does not support the decision, respectively. As the gold positive
passage is not provided in FaVIQ, we obtain the silver positive passage by (1)
taking the question that was the source of the claim during the data creation,
(2) using it as a query for TF-IDF, and (3) taking the top passage that
contains the answer. The passage through the same process but without the
answer is taken as a negative. More training details are in Appendix C.

####  4.1.2 Results

Table 5 reports results on FaVIQ. The overall accuracy of the baselines is
low, despite their high performance on FEVER. The zero-shot performance is
barely better than random guessing, indicating that the model trained on FEVER
is not able to generalize to our more challenging data. When the baselines are
trained on FaVIQ, the best model achieves an accuracy of 65% on the A set,
indicating that existing state-of-the-art models do not solve our benchmark.

##### Impact of retrieval

The performance of the claim only baseline that does not use retrieval is
almost random on FaVIQ, while achieving nearly 80% accuracy on FEVER. This
result suggests significantly less bias in the claims, and the relative
importance of using background knowledge to solve the task. When retrieval is
used, DPR outperforms TF-IDF, consistent with the finding from Petroni et al.
(2021).

##### A set vs. R set

The performance of the models on the R set is consistently higher than that on
the A set by a large margin, implying that claims based on ambiguity arisen
from real users are more challenging to verify than claims generated from
regular question-answer pairs. This indicates clearer contrast to prior work
that converts regular QA data to declarative sentences (Demszky et al., 2018;
Pan et al., 2021).

Model | Dev | Test
---|---|---
A |  R |  A |  R
Training on FEVER (zero-shot)
Clain only BART |  51.6 |  51.0 |  51.9 |  51.1
TF-IDF + BART |  55.8 |  58.5 |  54.4 |  57.2
DPR + BART |  56.0 |  62.3 |  55.7 |  61.2
Training on FaVIQ
Claim only + BART |  51.0 |  59.5 |  51.3 |  59.4
TF-IDF + BART |  65.1 |  74.2 |  63.0 |  71.2
DPR + BART |  66.9 |  76.8 |  64.9 |  74.6
Table 5:  Fact verification accuracy on FaVIQ.  Category | % | Example
---|---|---
Retrieval error | 38 |  C: The american show lie to me ended on january 31, 2011. (SUPPORTS; REFUTES) E: Lie to Me … The second season premiered on September 28, 2009 … The third season, which had its premiere moved forward to October 4, 2010.
Events | 28 |  C: The bellagio in las vegas opened on may, 1996. (REFUTES; SUPPORTS) E: Construction on the Bellagio began in May 1996. … Bellagio opened on October 15, 1998.
Evidence not explicit | 18 |  C: The official order to start building the great wall of china was in 221 bc. (SUPPORTS; REFUTES) E: The Great Wall of China had been built since the Qin dynasty (221–207 BC).
Multi-hop | 16 |  C: Seth curry’s brother played for davidson in college. (SUPPORTS; REFUTES) E: Stephen Curry (…) older brother of current NBA player Seth … He ultimately chose to attend Davidson College, who had aggressively recruited him from the tenth grade.
Properties | 10 |  C: The number of cigarettes in a pack of ‘export as’ brand packs in the usa is 20. (REFUTES; SUPPORTS) E: In the United States, the quantity of cigarettes in a pack must be at least 20. Certain brands, such as Export As, come in packs of 25.
Annotation error | 4 | C: The place winston moved to in still game is finport. (REFUTES; SUPPORTS)
Table 6:  Error analysis on 50 samples of the A set of FaVIQ validation data.
C and E indicate the claim and retrieved evidence passages from DPR,
respectively. Gold and blue indicate gold label and prediction by the model,
respectively. The total exceeds 100% as one example may fall into multiple
categories.

##### Error Analysis

We randomly sample 50 error cases from DPR + BART on the A set of FaVIQ and
categorize them, as shown in Table 6.

  * •

Retrieval error is the most frequent type of errors. DPR typically retrieves a
passage with the correct topic (e.g., about “Lie to Me”) but that is missing
the more specific background knowledge (e.g., the end date). We think the
claim having less lexical overlap with the evidence text leads to low
recall@$k$ of the retrieval model ($k=3$).

  * •

28% of error cases involve events. In particular, 14% involve procedural
events, and 6% involve distinct events that share similar properties but
differ in location or time frame.

  * •

In 18% of error cases, retrieved evidence is valid but not notably explicit,
which is naturally the case for the claims occurring in real life. FaVIQ has
this property likely because it is derived from questions that are gathered
independently from the evidence text, unlike prior work (Thorne et al., 2018a;
Schuster et al., 2021; Eisenschlos et al., 2021) with claims written given the
evidence text.

  * •

16% of the failure cases require multi-hop inference over the evidence. Claims
in this category usually involve procedural events or compositions (e.g. “is
Seth Curry’s brother” and “played for Davidson in college”). This indicates
that we can construct a substantial portion of claims requiring multi-hop
inference without having to make data that artificially encourages such
reasoning (Yang et al., 2018; Jiang et al., 2020).

  * •

Finally, 10% of the errors were made due to a subtle mismatch in properties,
e.g., in the example in Figure 6, the model makes a decision based on
“required minimum number” rather than “exact number” of a particular brand.

###  4.2 Professional Fact Checking Experiments

As professional fact-checking datasets, we use the following two datasets.

Snopes (Hanselowski et al., 2019) consists of 6422 claims, authored and
labeled by professional fact-checkers, gathered from the Snopes
website.999https://www.snopes.com We follow the official data split.

SciFACT (Wadden et al., 2020) consists of 1,109 claims based on scientific
papers, annotated by domain experts. As the official test set is hidden, we
use the official validation set as the test set, and separate the subset of
the training data as the validation set to be an equal size as the test set.

For both datasets, we merge not enough info (NEI) to refute, following prior
work that converts the 3-way classification to the 2-way classification (Wang
et al., 2019; Sathe et al., 2020; Petroni et al., 2021).

####  4.2.1 Models

As in Section 4, all models are based on BART which is given a concatenation
of the claim and the evidence text and is trained to generate either support
or refute. For Snopes, the evidence text is given in the original data. For
SciFACT, the evidence text is retrieved by TF-IDF over the corpus of abstracts
from scientific papers, provided in the original data.101010We have also tried
using DPR but found it poorly generalize and does not give a meaningful
result.

We consider two settings. In the first setting, we assume the target training
data is unavailable and compare the model trained on FEVER and FaVIQ in a
zero-shot setup. In the second setting, we allow training on the target data
and compare the model trained on the target data only and the model with the
transfer learning—pretrained on either FEVER or FaVIQ and finetuned on the
target data.

To explore models pretrained on NEI labels, we add a baseline that is trained
on a union of the KILT version of FEVER and NEI data from the original FEVER
from Thorne et al. (2018a). We also conduct an ablation that only includes the
R set for FaVIQ while excluding the A set.

##### Implementation details

When using TF-IDF for SciFACT, we use a sentence as a retrieval unit, and
retrieve the top 10 sentences, which average length approximates that of 3
passages from Wikipedia. When using the model trained on either FEVER or
FaVIQ, we use DPR + BART by default, which gives the best result in Section
4.1. As an exception, we use TF-IDF + BART in the SciFACT experiments for the
fair comparison with the model trained on the target data only that uses TF-
IDF. More training details are in Appendix C.

####  4.2.2 Results

Table 7 reports Micro-F1—a metric that is more reliable than accuracy given
significant imbalance of labels in Snopes and SciFACT.

Training |  Snopes |  SciFACT
---|---|---
No target data (zero-shot)
FEVER |  61.6 |  70.0
FEVER w/ NEI |  63.4 |  73.0
FaVIQ |  68.2 |  74.7
FaVIQ w/o A set |  63.1 |  73.3
Target data available
Target only |  80.6 |  62.0
FEVER$\xrightarrow{}$target |  80.6 |  76.7
FEVER w/ NEI$\xrightarrow{}$target |  81.6 |  77.0
FaVIQ$\xrightarrow{}$target |  82.2 |  79.3
FaVIQ w/o A set$\xrightarrow{}$target |  81.6 |  78.3
Table 7:  Micro-F1 scores on the test set of professional fact-checking
datasets.

##### Impact of transfer learning

We find it effective in using transfer learning—pretraining on large,
crowdsourced datasets (either FEVER or FaVIQ) and finetuning on the target
datasets. Improvements are especially significant on SciFACT, likely because
its data size is smaller.

Note that the availability of the target data is still important—models
finetuned on the target data outperform zero-shot models by up to 20%. This
indicates that crowdsourced data cannot completely replace professional fact
checking data, but transfer learning from crowdsourced data leads to
significantly better professional fact checking performance, successfully
addressing an inherent difficulty in scaling professional fact-checking data.

#####  FaVIQ vs. FEVER

Models that are trained on FaVIQ consistently outperform models trained on
FEVER, both with and without the target data, by up to 4.8% absolute. This
demonstrates that FaVIQ is a more effective resource than FEVER for
professional fact-checking.

It is worth noting that the model on FEVER is more competitive when NEI data
is included, by up to 3% absolute. While the models on FaVIQ outperform models
on FEVER even without NEI data, future work can create NEI data in FaVIQ for
more improvements.

##### Impact of the A set in FaVIQ

The performance of the models that use FaVIQ degrades when the A set is
excluded, demonstrating the importance of the A set created based on ambiguity
in information-seeking questions.

##  5 Conclusion & Future Work

We introduced FaVIQ, a new fact verification dataset derived from information-
seeking questions. By using annotations of ambiguity in questions, we
incorporate facts that real users were unaware of when posing the question.
Our experiments showed that the state-of-the-art models are far from solving
FaVIQ, and models trained on FaVIQ lead to improvements in professional fact
checking. Altogether, we believe FaVIQ will serve as a challenging benchmark
as well as support future progress in professional fact-checking.

Future work can investigate using other aspects of information-seeking
questions that reflects facts that users are unaware of or easily confused
with. For example, one can incorporate false presuppositions in questions that
arise when users have limited background knowledge (Kim et al., 2021). As
another example, one can explore generating not enough info claims by
leveraging unanswerable information-seeking questions. Furthermore, FaVIQ can
potentially be a challenging benchmark for the claim correction, a task
recently studied in Thorne and Vlachos (2021) that requires a model to correct
the refute claims.

## References

  * Amini et al. (2020) Aida Amini, Antoine Bosselut, Bhavana Dalvi, Yejin Choi, and Hannaneh Hajishirzi. 2020.  Procedural reading comprehension with attribute-aware context flow.  In _AKBC_.
  * Augenstein et al. (2019) Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019.  Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims.  In _EMNLP_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _ACL_.
  * Chen et al. (2021) Jifan Chen, Eunsol Choi, and Greg Durrett. 2021.  Can nli models verify qa systems’ predictions?  _arXiv_.
  * Cohen et al. (2011) Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011.  Computational journalism: A call to arms to database researchers.  In _CIDR_.
  * Das et al. (2019) Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2019.  Building dynamic knowledge graphs from text using machine reading comprehension.  In _ICLR_.
  * Demszky et al. (2018) Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018.  Transforming question answering datasets into natural language inference datasets.  _arXiv_.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _EMNLP_.
  * Eisenschlos et al. (2021) Julian Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, and Jordan Boyd-Graber. 2021.  Fool me twice: Entailment from wikipedia gamification.  In _NAACL_.
  * Ferreira and Vlachos (2016) William Ferreira and Andreas Vlachos. 2016.  Emergent: a novel data-set for stance classification.  In _NAACL_.
  * Gardner et al. (2019) Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019\.  Question answering is a format; when is it useful?  _arXiv_.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _ICML_.
  * Hanselowski et al. (2019) Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019.  A richly annotated corpus for different tasks in automated fact-checking.  In _CoNLL_.
  * Jiang et al. (2020) Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020.  Hover: A dataset for many-hop fact extraction and claim verification.  In _EMNLP: Findings_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  In _ACL_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP_.
  * Kim et al. (2021) Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021.  Which linguist invented the lightbulb? presupposition verification for question-answering.  _arXiv_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a benchmark for question answering research.  _TACL_.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _ACL_.
  * Lewis and Fan (2018) Mike Lewis and Angela Fan. 2018.  Generative question answering: Learning to answer the whole question.  In _ICLR_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _ACL_.
  * Maillard et al. (2021) Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oğuz, Veselin Stoyanov, and Gargi Ghosh. 2021.  Multi-task retrieval for knowledge-intensive tasks.  _arXiv_.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _EMNLP_.
  * Min et al. (2019) Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019.  Multi-hop reading comprehension through question decomposition and rescoring.  In _ACL_.
  * Pan et al. (2021) Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, and William Yang Wang. 2021\.  Zero-shot fact verification by claim generation.  In _ACL_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. 2021.  KILT: a benchmark for knowledge intensive language tasks.  In _NAACL_.
  * Platt et al. (1999) John Platt et al. 1999.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_.
  * Sathe et al. (2020) Aalok Sathe, Salar Ather, Tuan Manh Le, Nathan Perry, and Joonsuk Park. 2020.  Automated fact-checking of claims from wikipedia.  In _Proceedings of the 12th Language Resources and Evaluation Conference_.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get your vitamin c! robust fact verification with contrastive evidence.  In _ACL_.
  * Schuster et al. (2019a) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019a.  Towards debiasing fact verification models.  In _EMNLP_.
  * Schuster et al. (2019b) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019b.  Towards debiasing fact verification models.  In _EMNLP_.
  * Thorne and Vlachos (2019) James Thorne and Andreas Vlachos. 2019.  Adversarial attacks against fact extraction and verification.  _arXiv_.
  * Thorne and Vlachos (2021) James Thorne and Andreas Vlachos. 2021.  Evidence-based factual error correction.  _arXiv_.
  * Thorne et al. (2018a) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a.  Fever: a large-scale dataset for fact extraction and verification.  In _NAACL_.
  * Thorne et al. (2018b) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b.  The Fact Extraction and VERification (FEVER) shared task.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_.
  * Vlachos and Riedel (2014) Andreas Vlachos and Sebastian Riedel. 2014.  Fact checking: Task definition and dataset construction.  In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _EMNLP_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: a stickier benchmark for general-purpose language understanding systems.  In _NeurIPS_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP: System Demonstrations_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _EMNLP_.
  * Zhao et al. (2021) Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _ICML_.

##  Appendix A Details in Data Construction

##### Why not providing gold evidence

Unlike FEVER (Thorne et al., 2018a) which provides gold evidence text and
includes the evidence prediction as part of evaluation, we do not provide gold
evidence text and only care about the classification accuracy for the
evaluation. Reasons are as follows.

  1. 1.

As claims on FaVIQ were written independent from any reference text, gold
evidence text should be gathered through a separate process, which greatly
increases the cost. This is different from other annotated fact checking
datasets where a crowdworker wrote a claim based on the reference text and
therefore the same reference text can be considered as gold evidence.

  2. 2.

Finding gold evidence text is an inherently incomplete process; no human can
get close to, or even measure the upperbound. Therefore, even after exhaustive
human annotations, evaluation against annotated evidence leads to significant
amount of false negatives. For example, we manually evaluate the top incorrect
retrieval of TF-IDF on 30 random samples from FEVER and found that 37% of them
are false positives.

  3. 3.

Including evidence prediction as part of evaluation significantly restricts
the approach models can take. For instance, one may choose not to use the text
corpus provided in the dataset (e.g., Wikipedia), and decide to use other
sources such as structured data (e.g. knowledge bases) or implicit knowledge
stored in large neural models.

We admit that having gold evidence as supervision is often useful to train a
model, e.g., DPR. We therefore provide silver evidence text obtained through
TF-IDF and heuristics as detailed in Section 4.1.1, which find to be
sufficient for supervision.

##### Details of obtaining $a_{\mathrm{neg}}$

We obtain an invalid answer to the question, denoted as $a_{\mathrm{neg}}$,
using an off-the-shelf QA model, for which we use DPR followed by a span
extractor (Karpukhin et al., 2020).

The most naive way to obtain $a_{\mathrm{neg}}$ is to take the highest scored
prediction that is not equal to $a$. We however found such prediction is
likely to be a valid answer to $q$, either because it is semantically the same
as $a$, or because the ambiguity in the question leads to multiple distinct
valid answers. We therefore use two tricks that we find greatly reduces such
false negatives. First, instead of taking the top incorrect prediction, we
obtain the top $k$ predictions $p_{1}...p_{k}$ from the model and randomly
sample one from $\\{p_{1}...p_{k}\\}\setminus\\{a\\}$. We use $k=50$. Although
this is not a fundamental solution to remove false negatives, it significantly
alleviates the problem, drastically dropping the portion of false negatives
from 14% to 2%. Second, we train a neural model that is given a pair of the
text and classifies whether they are semantically equivalent or not. This
model is based on T5-large, trained and validated respectively on 150 and 100
pairs of $(a,p_{i})$ ($i=1...k$) which we manually label. We then exclude the
predictions in $\\{p_{1}...p_{k}\\}$ which are classified as semantically
equivalent to $a$ by the classifier.

##### QA-to-claim converter

We use a pretrained sequence-to-sequence model trained on a small number of
our own annotations. We first manually write 250 claims given valid or invalid
question-answer pairs. We then train a T5-3B model (Raffel et al., 2020),
using 150 claims for training and 100 claims for validation. Each question-
answer pair is fed into T5 with special tokens question: and answer:,
respectively before the question and the answer. When generating claims to
construct the data, we filter the claims that do not contain the answer string
in the claim, which may happen when the question is overly specific.

##  Appendix B Analysis of refute Claims

Category |  % FaVIQ |  % FEVER | Example
---|---|---|---
Negation | 0 | 30.0 |  C: Southpaw hasn’t been released yet. (from FEVER) E: Southpaw is a 2015 American sports drama film … released on July 24, 2015.
Antonym | 3.3 | 13.3 |  C: Athletics lost the world series in 1989. E: The 1989 World Series was … with the Athletics sweeping the Giants in four games.
Requires reading across conjunctions | 33.3 | 6.6 |  C: Johannes bell was the foreign minister that signed the treaty of versailles from germany. / E: Johannes bell served as Minister of Colonial Affairs … He was one of the two German representatives who signed the Treaty of Versailles.
Shared attributes | 26.7 | 6.6 |  C: Judi bowker played andromeda in the 2012 remake of the 1981 film clash of the titans called wrath of the titans. / E: Judi bowker … Clash of the Titans (1981).
Procedural event | 16.7 | 0 |  C: Mccrory’s originally filed for bankruptcy on february 2002. / E: McCrory Stores … by 1992 it filed for bankruptcy. … In February 2002 the company ceased operation.
Incorrect type of properties | 10.0 | 3.3 |  C: Tyler, the Creator is the name of the song at the end of who dat boy. E: "Who Dat Boy" is a song by American rapper Tyler, the Creator.
Cannot find potential cause | 0 | 20.0 |  C: Mutiny on the Bounty is Dutch. (from FEVER) E: Mutiny on the Bounty is a 1962 American Technicolor epic historical drama film.
Annotation error | 10.0 | 20.0 |  C: Pasek and paul were the individuals that wrote the lyrics to the greatest showman.
Table 8: Categorization of 30 refute claims on FaVIQ and FEVER, randomly
sampled from the validation set. C and E indicate the claim and evidence text,
respectively. Examples are from FaVIQ unless otherwise specified.

We randomly sample 30 refute claims from FaVIQ and FEVER respectively, and
categorize the cause of the misinformation, as shown in Table 8. Discussions
are in Section 3.2.

##  Appendix C Details of Experiments

##### DPR training for FEVER

As FEVER provides the annotated evidence passage, we use it as a positive. We
obtain a negative by querying the claim to TF-IDF and taking the passage that
is not the positive passage and has the second highest score. We initially
considered using the negative with the highest score, but found that many of
them (37%) are false negatives based on our manual evaluation of 30 random
samples. This is likely due to incomprehensive evidence annotation as
discussed in Appendix A. We find using the negative with the second highest
instead decreases the portion of false negatives from 37% to 13%.

##### Calibration

When the models trained on FEVER or FaVIQ are used for professional fact
checking, we find models are poorly calibrated, likely due to a domain shift,
as also observed in Kamath et al. (2020); Desai and Durrett (2020). We
therefore use a simplified version of Platt scaling, a post-hoc calibration
method (Platt et al., 1999; Guo et al., 2017; Zhao et al., 2021). Given
normalized probabilities of support and refute, denoted as $p_{\texttt{s}}$
and $p_{\texttt{r}}$, modified probabilities $p^{\prime}_{\texttt{s}}$ and
$p^{\prime}_{\texttt{r}}$ are obtained via:

$\begin{bmatrix}p^{\prime}_{\texttt{s}}\\\
p^{\prime}_{\texttt{r}}\end{bmatrix}=\mathrm{Softmax}\left(\begin{bmatrix}p_{\texttt{s}}\\\
p_{\texttt{r}}\end{bmatrix}+\begin{bmatrix}\gamma\\\ 0\end{bmatrix}\right),$

where $-1<\gamma<1$ is a hyperparameter tuned on the validation set.

##### Other details

Our implementations are based on PyTorch111111https://pytorch.org/ (Paszke et
al., 2019) and Huggingface
Transformers121212https://github.com/huggingface/transformers (Wolf et al.,
2020).

When training a BART-based model, we map support and refute labels to the
words ‘true’ and ‘false’ respectively so that each label is mapped to a single
token. This choice was made because the BART tokenizer maps ‘refute’ into two
tokens, making it difficult to compare probabilities of support and refute.

By default, we use a batch size of 32, a maximum sequence length of 1024, and
500 warmup steps using eight 32GB GPUs. For SciFACT, we use a batch size of 8
and no warmup steps using four 32G GPUs. We tune the learning rate in between
{7e-6, 8e-6, 9e-6, 1e-5} on the validation data.

##  Appendix D Discussions & Future Work

Future work can explore developing retrieval models which are more robust to
lexical distractions. One potential approach is to decompose the claim and
verify each segment, as explored by prior work in the context of QA (Lewis and
Fan, 2018; Min et al., 2019). This approach is expected to be particularly
effective for claims involving multi-hop inference, conjunctions and
procedural events, which are the main sources of the challenging claims
(Section 3.2; Table 8) and errors made by the model (Table 6). Such an
approach in fact mimics the process we did for obtaining a positive passage to
train the DPR model, where we decompose the claim into the question and the
answer, use one for querying the TF-IDF index and the other for reranking the
candidate positive passages—an approach that is only possible at training time
rather than test time. Moreover, future work can make a connection between the
open-domain QA and fact verification models to jointly improve the performance
in two tasks, as explored by Chen et al. (2021) in the context of reading
comprehension and the entailment task.