# Evidence-Based Temporal Fact Verification

Anab Maulana Barik1 Wynne Hsu1,2 Mong Li Lee1,3 1School of Computing, National
University of Singapore
2Institute of Data Science, National University of Singapore
3Centre for Trusted Internet and Community, National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2018)

###### Abstract.

Automated fact verification plays an essential role in fostering trust in the
digital space. Despite the growing interest, the verification of temporal
facts has not received much attention in the community. Temporal fact
verification brings new challenges where cues of the temporal information need
to be extracted and temporal reasoning involving various temporal aspects of
the text must be applied. In this work, we propose an end-to-end solution for
temporal fact verification that considers the temporal information in claims
to obtain relevant evidence sentences and harness the power of large language
model for temporal reasoning. Recognizing that temporal facts often involve
events, we model these events in the claim and evidence sentences. We curate
two temporal fact datasets to learn time-sensitive representations that
encapsulate not only the semantic relationships among the events, but also
their chronological proximity. This allows us to retrieve the top-k relevant
evidence sentences and provide the context for a large language model to
perform temporal reasoning and outputs whether a claim is supported or refuted
by the retrieved evidence sentences. Experiment results demonstrate that the
proposed approach significantly enhances the accuracy of temporal claim
verification, thereby advancing current state-of-the-art in automated fact
verification.

Automated Fact Checking, Temporal Claims

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems††ccs: Computing methodologies

##  1\. Introduction

The proliferation of false information, or ”fake news,” continues to pose a
challenge with potentially severe implications. Computational fact checking
has been proposed as a viable solution to this issue, leveraging technology to
verify textual claims against a set of evidence sentences that either support
or contradict these claims. However, there is still a considerable gap when it
comes to verifying temporal claims which are statements associated with a
specific time or duration. For effective verification of temporal claims, we
need to retrieve evidences that focus not just on the semantic coherence
between the claim and potential evidence, but more importantly, the temporal
context so that the timeline is aligned between the claim and the evidences.

Consider the temporal claim ”Matteo Renzi was a full-time undergraduate
student in Singapore in 2006”. This claim can be refuted if we find evidence
like ”Matteo Renzi served as President of the Province of Florence from 2004
to 2009…” since it is highly unlikely for someone to serve as a president
while concurrently undertaking a full-time undergraduate degree in a different
country. Existing claim verification methods that employ traditional evidence
retrieval based on lexical or semantic matching might overlook this evidence
sentence and conclude that there is NOT ENOUGH INFO (NEI) to verify the claim.

Consider another temporal claim ”Henry Condell published his First Folio in
1623 and performed several plays for his career in 1620.”. We see that this
claim has two events ”published his First Folio” and ”performed several plays”
which are associated with two distinct dates ”1623” and ”1620” respectively.
For the temporal claim to be true, we need to verify that both events are
supported by evidences sentences. On the other hand, if we have evidence that
shows one of the event is false, then the entire claim becomes false. For
example, if we have the evidence sentence ”Henry Condell ended his stage
career in 1619.”, then we can refute the event that he performs several plays
in 1620, and conclude that the temporal claim is false. By analyzing the claim
and evidence sentence at the event-level rather than the whole sentence, we
can link the time references to their respective events and retrieve relevant
evidence sentences.

Our work aims to bridge the gaps in temporal fact verification by introducing
a framework called TACV that is sensitive to temporal aspects of claim
verification. TACV takes into account the temporal information in the claim to
retrieve relevant evidence sentences. It identifies events in both the claim
and evidence sentences and associate the time-related information to the
corresponding events. With this, TACV seeks to assign a higher score to
evidence sentences that align more closely with the events in the claim. This
is achieved through an encoder module to create temporal-aware
representations. This process involves modeling the events and their
associated temporal information as a graph and propagating the information
using a graph attention network. The representations obtained encapsulate the
temporal sequence and duration between dates of events, enabling us to account
for the temporal disparity between the claim and the evidence sentences.

Figure 1. Overview of TACV framework.

We curate two datasets comprising of a diverse range of temporal claims which
is crucial in evaluating the effectiveness of TACV. We leverage the temporal
reasoning ability of large language model (LLM) by providing it with the top-k
evidence sentences retrieved by TACV as context to determine the veracity of
the claims. Experiment results on multiple benchmark datasets demonstrate that
the proposed approach surpasses state-of-the-art methods, is robust and able
to handle real-world claims.

##  2\. Related Work

Research on evidence-based claim verification typically formulates the problem
as a natural language inference task, and classifies whether the evidence
sentences support or refute the claim (Stammbach and Neumann, 2019; Soleimani
et al., 2020). Recent works such as GEAR (Zhou et al., 2019), KGAT (Liu et
al., 2020) and GCAT (Barik et al., 2022) have employed graph attention
networks to propagate information between evidence sentences, thereby
improving the model’s reasoning ability. GEAR uses a graph attention network
to capture the semantic interaction between evidence sentences. KGAT
introduces kernels to measure the importance of the evidence and conduct fine-
grained evidence propagation. CGAT incorporates external knowledge to inject
commonsense knowledge into the model.

Several works have attempted to take into account temporal information for
claim verification. (Mori et al., 2022) focuses on the verification on
economic claims against time series sources which are in tabular format. A
claim is translated into Datalog rules which are used to query the tabular
evidences to verify the correctness of the claim. This work only deals with
structured SQL data and does not handle evidences in natural language.

The work in (Allein et al., 2021) considers the published date of the claim
and evidence sentences, and re-ranks the sentences based on the proximity of
their published dates to that of the claim. ITR (Allein et al., 2023) exploits
the temporal proximity between the claim’s publication date and evidence’s
publication date to create time representations for temporal reasoning. These
works do not consider temporal expressions in the claim/evidence sentences.
However, such temporal metadata may not be readily available. In contrast, our
approach handles temporal expressions within the sentences themselves and do
not require such external temporal metadata.

##  3\. Proposed Framework

An overview of the proposed TACV framework is given in Figure 1. Given a
temporal claim, we extract claim events with their associated temporal
expressions from the claim. To obtain more information about the claim, we use
a sequence-to-sequence entity linking model GENRE (De Cao et al., 2021) to
retrieve documents from sources such as Wikipedia articles. Each sentence from
the retrieved documents is sent to the event extraction module to obtain
evidence sentence events. Next, we pair the extracted claim events with the
evidence sentence events to create temporal-aware representations. This step
facilitates the identification of the top-$k$ most relevant evidence
sentences, which are deemed potentially useful for verifying the claim event.
Utilizing the top-$k$ evidence sentences as context, the framework harnesses
the temporal reasoning capabilities of Large Language Models (LLMs) to
ascertain whether the evidence supports or refutes the claim event, or if the
evidence is insufficient for verification. Finally, these labels are
aggregated to obtain the final label for the input claim.

###  3.1. Event Extraction with Temporal Arguments

In general, an event has two types of information: (a) core information such
as who is involved, what is happening, and where it is happening; and (b)
temporal expression which includes specific dates, time duration and event
ordering. We employ an off-the-shelf Semantic Role Labeling
(SRL)111https://demo.allennlp.org/semantic-role-labeling from AllenNLP (Shi
and Lin, 2019) to extract all the events mentioned in the claim or evidence
sentences. Specifically, each sentence is fed into the SRL model, which
outputs a list of predicates along with their arguments. Each predicate
corresponds to an event. The core information comprises of the concatenation
of phrases related to the predicate and non-temporal arguments. The temporal
information comprises of the phrases related to the temporal arguments. We
apply this process to the claim and evidence sentences to extract claim events
and evidence events.

For instance, there are two predicates in the claim ”Henry Condell published
his First Folio in 1623 and performed several plays for his career around
October 1620.”. The first predicate ”published” has arguments ”Henry Condell”,
”the First Folio” along with the temporal argument ”in 1623”; while the second
predicate ”performed” has the arguments ”Henry Condell”, ”several plays” as
well as the temporal argument ”around October 1620”. Similarly, for the
evidence sentence ”Henry Condell ended his stage career in 1619.”, we have the
predicate ”ended” with arguments ”Henry Condell”, ”stage career” and temporal
argument ”in 1619”.

The claim events and lists of extracted events in the evidence sentences are
paired before passing to a temporal-aware encoder.

###  3.2. Temporal-aware Representation Encoding

Our temporal-aware encoder utilizes the contextual and temporal information in
the claim and evidence events to ensure that the time encoding is not just a
static representation of time. This is in contrast to existing work (Xu et
al., 2020) that uses series of sine and cosine for functional encoding of time
element.

Figure 2. Temporal-aware representation encoding.

Let $<$$E_{c}$, $E_{s}$$>$ be a ¡claim event, sentence event¿ pair. We first
create the sequence $([CLS]+E_{c}+[SEP]+Title+[SEP]+E_{s}+[SEP])$ where
$[CLS]$ is the special start token, $[SEP]$ is the separator token, and
$Title$ is the title of the document from which the sentence event $E_{s}$ is
obtained. The sequence is then passed to BERT to obtain the contextual
representation $B$ (see Figure 2).

Next, we apply mean pooling on the date tokens, followed by positional
encoding (Vaswani et al., 2017). For example, consider the temporal phrases
October 1620 and 1619 in $E_{c}$ and $E_{s}$ respectively. The position $pos$
for 1619 is 0, while that for October 1620 is 21, indicating that they are 0
and 21 months apart from the earliest date in the text (which is 1619). Note
that the temporal arguments could be of different granularities, and the
smallest granularity to used to compute $pos$.

Given the $pos$ value, the temporal encoding is a vector of $d$ dimension,
denoted as $TE_{pos}$, where the ith element is given by

$TE_{pos}[i]=\left\\{\begin{array}[]{ll}sin(\frac{pos}{10000^{i/d}})&\mbox{if~{}}i\mbox{~{}is
even}\\\ cos(\frac{pos}{10000^{(i-1)/d}})&\mbox{otherwise}\\\
\end{array}\right.$

This temporal encoding satisfies the following properties:

1\. Triangular inequality. Given three distinct dates $t_{1},t_{2},t_{3}$ in
chronological order with position values $p_{1},p_{2},p_{3}$, the distance
between the temporal encodings $TE_{p_{1}}$ and $TE_{p_{2}}$ is less than the
distance between $TE_{p_{1}}$ and $TE_{p_{3}}$.

2\. Distance preservation. Given the position values $p_{1},p_{2},p_{3},p_{4}$
of four distinct dates $t_{1},t_{2},t_{3},t_{4}$, if
$(p_{2}-p_{1})=(p_{4}-p_{3})$, then the distance between the temporal
encodings $TE_{p_{2}}$ and $TE_{p_{1}}$ is the same as the distance between
$TE_{p_{4}}$ and $TE_{p_{3}}$.

We feed the temporal encodings to the transformer to obtain the date
representations $\hat{B}$. The resulting temporal-aware representation of a
¡claim event, sentence event¿ pair is the sequence $R=(H_{[CLS]},$
$H_{1},\cdots H_{d})$ where

$H_{j}=\begin{cases}B_{j}&\text{if }j^{th}\text{ token is not a date}\\\
\hat{B}_{j}&\text{if }j^{th}\text{ token is a date}\end{cases}$

and $H_{[CLS]}$ is the average pooling of $H_{j},1\leq j\leq d$.

###  3.3. Relevance Scoring

The temporal-aware representations obtained are used to construct an event-
level graph $G_{event}$. Each node $i$ represents a ¡claim event, sentence
event¿ pair and is initialized with its corresponding temporal-aware
representation $R^{i}$. The nodes are fully connected to each other. We
utilize a Graph Attention Network (GAT) to propagate information among the
nodes in $G_{event}$.

Given two nodes $i$ and $j$, we compute the token-level attention weight
$w^{i\rightarrow j}$ where the $p^{th}$ entry in $\mathbf{w}^{i\rightarrow j}$
is given by:

(1) $\mathbf{w}^{i\rightarrow j}[p]=\sum_{q}\mbox{sim}(R^{i}_{p},R^{j}_{q})$

where $sim$ is cosine similarity function and $R^{j}_{q}$ is the $q^{th}$
element in $R^{j}$.

We normalize $\mathbf{w}^{i\rightarrow j}$ through a softmax function before
applying this attention weight to the representation $R^{i}$. The information
propagated from node $i$ to node $j$ is given by:

(2) $\mathbf{z}^{i\rightarrow j}=R^{j}_{0}\circ(\mathbf{w}^{i\rightarrow
j}\cdot R^{i})$

where $R^{j}_{0}$ is the $[CLS]$ token in $R^{j}$ and $\circ$ denotes
concatenation.

With this, we update the representation of ${R}^{j}$ as follows:

(3) $R^{j}=\sum_{i}\beta^{i\rightarrow j}\cdot\mathbf{z}^{i\rightarrow j}$

where $\beta^{i\rightarrow j}$ is the sentence-level attention weight from $i$
to $j$ computed as follows:

(4) $\beta^{i\rightarrow j}=\textbf{W}\cdot(\mathbf{z}^{i\rightarrow j})^{T}$

where $\textbf{W}\in R^{1\times 2d}$ denotes the weight matrix of a linear
transformation and $(\mathbf{z}^{i\rightarrow j})^{T}$ is the transpose of
$\mathbf{z}^{i\rightarrow j}$.

We compute the relevance score of each evidence sentence to a claim event by
applying element-wise max operation (Zhou et al., 2019) on the updated
representations followed by a linear layer.

###  3.4. Temporal Reasoning with LLM

Finally, we leverage the capabilities of LLM text-davinci-003 from OpenAI to
perform temporal reasoning. We design the following prompt to use the top-k
relevant evidence sentences as context for LLM to reason and determine a label
for each claim event.

Table 1. Characteristics of temporal claim datasets.  | T-FEVER | T-FEVEROUS
---|---|---
| Single event | Multiple events | Single event | Multiple events
| Train set | Test set | Train set | Test set | Train set | Test set | Train set | Test set
Ordering | 20,625 | 2,805 | 1,009 | 161 | 17,546 | 1,910 | 39,402 | 4,175
Duration | 456 | 75 | 21 | 3 | 374 | 51 | 729 | 106

”You are provided with a claim and evidence sentences. Perform text
classification to determine whether the evidence support, refute, or do not
enough information to verify the claim. The judgment based on the general
aspect and the temporal aspect of the claim. Only base your decision on the
information explicitly stated in the evidence. Please return the following
information in JSON format: PREDICTED_LABEL: Either ’SUPPORTS’, ’REFUTES’, or
’NOT ENOUGH INFO’. If the evidences does not address all key aspects of the
claim or does not provide any information related to the claim then the
returned label should be ’NOT ENOUGH INFO’. If the evidences contradicts
general or temporal aspects of the claim, the returned label should be
’REFUTES’. If the evidences support all aspects of the claim, the returned
label should be ’SUPPORTS’.

The final label for a claim is determined as follows: If any event reveals
factual discrepancies, the entire claim is deemed REFUTE. Conversely, if all
events align with the facts in the evidence sentences, the claim receives a
SUPPORT label. In cases where certain events lack sufficient evidence while
other events may be corroborated, the overall verdict is NOT ENOUGH INFO.

Table 2. Characteristics of the experimental datasets.  |  | Train Set | Test Set
---|---|---|---
Type | Dataset | Support | Refute | NEI | Support | Refute | NEI
Temporal claims | T-FEVER | 10,784 | 8,007 | 3,238 | 1,015 | 1,285 | 737
| T-FEVEROUS | 30,366 | 26,032 | 1,041 | 2,991 | 2,927 | 225
General claims | FEVER | 80,035 | 29,775 | 35,639 | 3,333 | 3,333 | 3,333
| FEVEROUS | 41,835 | 27,215 | 2,241 | 3,372 | 2,973 | 1,500
Real world claims | LIAR | 1,683 | 1,998 | - | 211 | 250 | -

##  4\. Temporal Claim Datasets

We derive two datasets for temporal fact verification from existing fact
verification datasets FEVER (Thorne et al., 2018a), FEVER2 (Thorne et al.,
2018b) and FEVEROUS (Aly et al., 2021). The original datasets comprise of
synthetic general claims generated by modifying sentences from Wikipedia, and
are labelled as SUPPORT, REFUTE, or NEI, along with their evidence sentences.
Although these datasets may have temporal claims, the verification of these
claims is based on the general aspect instead of the temporal aspects. For
example, the sample claim ”DSV Leoben, an Australian association football club
which was founded in 1927” is managed by Austria Ivo Golz.” is refuted based
on the ground truth evidence: ”DSV Leoben is an Austrian association football
club based in Leoben.” Here, we augment the dataset with new claims by
manipulating the temporal context of existing claims such as ”DSV Leoben was
founded in 1928”.

The creation of the new datasets follows a three-stage process: (1) detection
of temporal claims, (2) tagging the types of temporal expression in the
claims, (3) augmentation with synthetic claims. We first identify temporal
claims from the general fact verification datasets by extracting claims with
at least one temporal argument. These claims are tagged according to their
temporal expression type. This is achieved through the use of regular
expression pattern matching to distinguish between the temporal expression
types, namely ordering (indicated by words such as ”before”, ”after”), and
duration (phrases like ”for 5 years”, ”over 3 months”). Claims that are not
tagged are filtered out.

We also augment the datasets with new claims by manipulating the temporal
context of existing claims, thereby creating a new scenario where either the
new claim is no longer valid according to the altered date, or it remains
valid in spite of the altered date. We start with original claims that are
labeled as SUPPORT, and adjust the temporal argument such that it is either
disputed by the evidence sentences (leading to the new claim being labeled as
REFUTE) or is in agreement with the evidence sentences (which results in the
new claim being labeled as SUPPORT). The evidence sentences are the ground-
truth evidence sentences provided by the original datasets. Specifically, the
following steps are taken to generate new REFUTE claims:

Ordering. We extract the temporal predicates and dates from the claim’s
temporal argument.

  * •

If the temporal predicate is ”in”, ”on” or ”at”, we replace the extracted
claim date by adding or subtracting a random number to the date so that the
new claim date is no longer supported by the date(s) in the evidence
sentences.

  * •

If the temporal predicate is ”before”, we identify the most recent date from
the evidence sentences. Then we replace the predicate with ”after” and adjust
the claim date to the identified date after adding a random number. Similarly,
if the predicate is ”after”, we switch it to ”before” and revise the claim
date to the earliest date mentioned in the evidence sentences, again
incremented by a random number.

  * •

If the temporal predicate is ”from”, we find the most recent date from the
evidence sentences, and replace the claim date by the identified date after
adding a random number.

  * •

If the temporal predicate is ”between” with two temporal arguments date1 and
date2, we add a random number to date2 to get a new date3. Then we replace
date1 with date3, and replace date2 with date3 after adding another random
number. This ensures that the new range falls outside the original range.

Duration. The temporal predicate is either ”for”, ”over”, or ”within”,
accompanied by a temporal argument indicating the duration period. We adjust
this argument by randomly increasing or decreasing its value, thereby creating
a new duration that diverges from the original context.

Likewise, we augment the datasets with new claims that are labeled as
”SUPPORT” by ensuring that the modified temporal arguments remain consistent
with the evidence sentences. The dataset derived from the FEVER and the
FEVER2.0 datasets is called T-FEVER, while the dataset that is derived
FEVEROUS is called T-FEVEROUS. Each claim in our datasets can involve either a
single event or multiple events. These events are characterized by their
temporal expressions focusing on two key aspects: ordering and duration. A
detailed breakdown of these characteristics for each dataset is provided in
Table 1. These datasets will be made available on Github.

We evaluated the quality of our new datasets by randomly sampling 300 claims
each from T-FEVER and T-FEVEROUS. Two human assessors, equipped with the
necessary background and skills, were tasked to determine the accuracy of a
claim’s label by referencing the ground truth evidence sentences. Our findings
indicate that 97% of the claims in T-FEVER and 98% in T-FEVEROUS have the
correct labels. The remaining claims are wrongly labelled as SUPPORT/REFUTE
when they should be labelled as NOT ENOUGH INFO. For example, consider the
claim “Ashley Graham was on a magazine cover in 2018.” and the corresponding
evidence sentence “In 2017, Graham became the first plus-size model to appear
on the covers of British and American Vogue.”. The claim was incorrectly
labelled as REFUTE when it should be NOT ENOUGH INFO. This is because even if
Graham was on the magazine cover in 2017, this does not imply that she cannot
appear on the cover in 2018.

Training. We use these two datasets in the training of the temporal-aware
encoder in TACV. We employ triplet loss to learn representations that maximize
the margin between relevant and non-relevant sentences to a specific claim
event. The training data is composed of triplets, each comprising of a claim
event, a relevant sentence from the ground truth evidence, and a non-relevant
sentence selected from Wikipedia articles. Since the ground truth evidence
sentences pertain to a claim as a whole rather than to specific events in the
claim, we engage two human annotators to identify which evidence sentences are
relevant to individual claim event. This enables a more precise training for
the TACV’s temporal-aware encoder.

##  5\. Performance Study

In this section, we present the results of experiments to evaluate the
effectiveness of the TACV framework for temporal fact verification. We show
that TACV performs well not only on the new temporal T-FEVER and T-FEVEROUS
datasets, but also on the standard benchmark FEVER and FEVEROUS datasets as
well as the real world LIAR dataset (Wang, 2017). Table 2 gives the details of
these datasets.

We use label accuracy and FEVER score as the evaluation metrics. Label
accuracy measures the proportion of correct predictions made by the model out
of all predictions. This metric ignores whether the evidence sentences
directly contribute to the prediction. In contrast, FEVER score only marks a
prediction as correct if the predicted label is correct and the retrieved
evidence directly contributes to the determination of the label.

TACV uses Huggingface’s implementation of BERTbase, a pre-trained language
model, to encode the tokens in the claim events and evidence sentence events.
For the temporal-aware representation encoding, a transformer with two layers
and eight heads, having a dimension of 768, is used. The training is conducted
over five epochs with a batch size of 8, and learning rate of 5e-6. We apply
the AdamW (Loshchilov and Hutter, [n. d.]) optimizer with a fixed weight decay
and select the best performing model for evaluation on the test set.

Figure 3. Effect on $k$ on TACV. Solid line depict label accuracy, dotted line
depict FEVER score.

###  5.1. Sensitivity Experiments

We first conduct sensitivity experiments to examine the performance of TACV as
we vary the number of top-$k$ relevant evidence sentences for temporal
reasoning. Figure 3 shows the label accuracy and FEVER score for different $k$
values on the T-FEVER and T-FEVEROUS datasets. We see that the optimal
performance is attained when $k=3$ for T-FEVER, and $k=5$ for T-FEVEROUS. As
such, we use the top-3 sentences in T-FEVER, and the top-5 sentences in
T-FEVEROUS with the highest relevance scores to form the context for the LLM
to output the label of each claim event.

Table 3. Results of comparative study on temporal claims. | T-FEVER | T-FEVEROUS
---|---|---
Methods | Label acc. | FEVER score | Label acc. | FEVER score
KGAT | 44.28 | 33.61 | 15.69 | 4.59
CGAT | 44.38 | 33.91 | 16.58 | 4.29
ITR | 44.05 | 30.88 | 31.66 | 8.63
TACV | 52.15 | 41.42 | 54.01 | 15.38
Table 4. Detailed results on T-FEVER. | Single Event | Multiple Event
---|---|---
| Ordering | Duration | Ordering | Duration
| Label acc. | Fever score | Label acc. | Fever score | Label acc. | Fever score | Label acc. | Fever score
KGAT | 44.4 | 34.30 | 36.36 | 34.84 | 45.2 | 31.60 | 41.67 | 41.67
CGAT | 44.6 | 34.14 | 37.87 | 34.84 | 43.4 | 30.60 | 41.67 | 41.67
ITR | 44.48 | 31.63 | 45.45 | 39.39 | 42.22 | 26.2 | 33.33 | 33.33
TACV | 52.14 | 41.56 | 50.00 | 46.96 | 52.60 | 40.00 | 58.33 | 58.33
Table 5. Detailed results on T-FEVEROUS. | Single Event | Multiple Event
---|---|---
| Ordering | Duration | Ordering | Duration
| Label acc. | Fever score | Label acc. | Fever score | Label acc. | Fever score | Label acc. | Fever score
KGAT | 20.68 | 4.41 | 8.69 | 0 | 15.73 | 4.26 | 17.16 | 3.73
CGAT | 18.57 | 4.61 | 8.69 | 0 | 15.91 | 4.59 | 18.65 | 3.73
ITR | 31.32 | 5.72 | 17.39 | 4.34 | 32.02 | 9.21 | 28.35 | 5.97
TACV | 50.70 | 11.14 | 43.47 | 26.08 | 54.72 | 18.53 | 47.01 | 16.41
Table 6. Results of comparative study on general and real world claims. | FEVER | FEVEROUS | LIAR | T-LIAR
---|---|---|---|---
Methods | Label acc. | FEVER score | Label acc. | FEVER score | Label acc. | Label acc.
KGAT | 74.07 | 70.38 | 34.94 | 11.25 | 46.20 | 69.44
CGAT | 76.39 | 73.15 | 39.70 | 12.52 | 45.77 | 72.22
ITR | 73.36 | 70.04 | 44.20 | 14.39 | 49.24 | 69.44
TACV | 76.42 | 73.16 | 53.97 | 15.08 | 62.86 | 83.33

###  5.2. Comparative Experiments

In this set of experiments, we compare TACV with the following state-of-the-
art evidence-based fact verification baselines:

  * •

KGAT (Liu et al., 2020). This method employs transformer-based architecture to
obtain claim-sentence pair representations before utilizing a fine-grained
Kernel graph attention networks (GAT) to aggregate the evidence for the claim
verification. It employs Athene (Hanselowski et al., 2018) to retrieve
evidence sentences.

  * •

CGAT (Barik et al., 2022). This method incorporates external knowledge from
ConceptNet to enrich the contextual representations of evidence sentences
before employing GAT to propagate the information among the evidence sentences
to verify the claim veracity. Same as KGAT, evidence sentences are obtained
using Athene.

  * •

ITR (Allein et al., 2023). This method assumes evidence sentences are given as
input and focuses on using them for temporal reasoning. For fair comparison,
we use the evidence sentences retrieved by our TACV as input to ITR.

Results on temporal claims. Table 3 shows the label accuracy and FEVER score
of the various methods on the new temporal claim datasets T-FEVER and
T-FEVEROUS. We see that TACV outperforms existing methods by a large margin.

Tables 4 and 5 provide further analysis of the results on the new temporal
claim datasets. We breakdown the results based on whether a claim consists of
single events or multiple events, as well as the type of temporal expressions
such as ordering or duration. We see that TACV consistently outperforms state-
of-the-art methods in terms of both label accuracy and FEVER score for claims
involving single and multiple events. For claims with multiple events, TACV
shows a larger margin of improvement indicating the importance of analyzing
claims at the event level. Further, there is a significant improvement in the
FEVER score implying that TACV is able to retrieve evidence sentences relevant
to the claim.

Results on general and real world claims. We also validate the ability of TACV
to handle both the original synthetic general claims in FEVER and FEVEROUS, as
well as real-world claims in LIAR which comprises of statements compiled from
PolitiFact.com. For each claim in the LIAR dataset, we feed the claim sentence
into BING search to retrieve the top-2 articles and all the sentences from
these articles are used as potential evidence sentences. In addition, we
identify 363 temporal claims (209 SUPPORT and 154 REFUTE) in the LIAR to
create a T-LIAR dataset.

Table 7. Results of Ablation Studies. | T-FEVER | T-FEVEROUS
---|---|---
Methods | Label acc. | FEVER score | Label acc. | FEVER score
TACV w/o event extraction | 46.92 | 39.47 | 39.64 | 12.02
TACV w/o Temporal-aware encoder | 49.22 | 38.88 | 52.14 | 13.07
TACV w/o GAT | 50.60 | 40.07 | 52.84 | 13.91
TACV | 52.15 | 41.42 | 54.01 | 15.38
TACV (GPT4) | 55.08 | 42.17 | 56.56 | 18.98
Table 8. Sample Claim from T-FEVER. Claim: Osamu Tezuka practiced drawing as a
young child in 2000. Ground Truth Label: REFUTE
---
Method |  Events |  Retrieved Sentences |  Event Label |  Claim Label
TACV |  • Osamu Tezuka practiced drawing as a young child in 2000. |  • As a young child Tezuka began to practice drawing so much that his mother would have to erase pages in his notebook in order to keep up with his output.
• Tezuka died of stomach cancer in 1989.
• Born in Osaka Prefecture, his prolific output, pioneering techniques, and innovative redefinitions of genres earned him such titles as ”the father of manga”, ”the godfather of manga” and ”the god of manga”. |  REFUTE |  REFUTE
CGAT |  - |  • His legendary output would spawn some of the most influential , successful and well received manga series including Astro Boy , Jungle Emperor Leo , Black Jack , and Phoenix , all of which would win several awards .
• As a young child Tezuka began to practice drawing so much that his mother
would have to erase pages in his notebook in order to keep up with his output
.
• In elementary school he would develop his own comics and illustrations from which his skills would become formidable . |  - |  NEI

Table 6 shows the results. We see that TACV remains robust in the original
FEVER and FEVEROUS datasets and can generalize well to real world LIAR
dataset. Moreover, TACV has a big lead in the label accuracy of the real world
temporal claim T-LIAR dataset, raising the confidence that TACV can be used
for the verification of temporal claims in real world settings.

###  5.3. Ablation Studies

We implement the following variants of TACV to examine the effect of the key
components in TACV:

  1. (1)

TACV without event extraction. Instead of extracting events from claim and
evidence sentences, we pass them directly to the temporal-aware representation
encoding component. The top-k relevant sentences for each claim are passed to
the LLM to obtain the claim’s label.

  2. (2)

TACV without temporal-aware representation encoding. For this variant, we use
BERT to obtain the encoding for each pair of claim event and sentence event
and use this representation for relevance scoring.

  3. (3)

TACV without GAT. Here, we do not construct the $G_{event}$ graphs. Instead,
we perform mean pooling over the token representations of the ¡claim event,
sentence event¿ pairs.

  4. (4)

TACV (GPT4). Here, we also experimented with a better LLM by using GPT4-turbo.

Table 7 shows the results. The largest drop in both label accuracy and FEVER
score occur when events are not extracted from claim and evidence. This is
followed by the variant where temporal-aware representation encoding is not
utilized. This suggests that identifying events in claims and evidence
enhances the retrieval of relevant sentences for the subsequent claim
verification process. Also, using a better LLM further improves the
performance.

##  6\. Case Studies

Table 8 shows a claim from T-FEVER and the top-ranked evidence sentences
retrieved by each method. The claim ”Osamu Tezuka practiced drawing in 2000”
has a single event ”practiced” with temporal argument ”in 2000”. TACV with its
temporal-aware representation is able to retrieve the sentence ”Tezuka died of
stomach cancer in 1989”, which is also the ground truth evidence sentence
(depicted in bold). With this, LLM concludes that the event label is REFUTE,
and the overall claim label is REFUTE. On the other hand, CGAT retrieves
evidence sentences that describe Tezuka’s drawing practice but not the date,
resulting in the NEI prediction.

Table 9. Sample Claims from T-FEVEROUS.  Claim: McDonaugh was appointed to the
first managerial job on the 10th April 2019, and then he was awarded SPFL
League 2 Manager of the Month, in December 2019. Ground Truth Label: REFUTE
---
Method |  Events |  Retrieved Sentences |  Event Label |  Claim Label
TACV |  • McDonaugh was appointed to the first managerial job on the 10th April 2019. |  • McDonaugh was appointed to first managerial job succeeding Gary Jardine at Edinburgh City on 10 October 2017.
• McDonaugh again won the SPFL League 2 Manager of the Month award in December
2019, winning all four games and keeping three clean sheets.
• He was awarded SPFL League 2 Manager of the Month in September 2018. |  REFUTE |  REFUTE
• McDonaugh was awarded SPFL League 2 Manager of the Month, in December 2019. |  • McDonaugh again won the SPFL League 2 Manager of the Month award in December 2019, winning all four games and keeping three clean sheets.
• He was awarded SPFL League 2 Manager of the Month in September 2018.
• McDonaugh was appointed to first managerial job succeeding Gary Jardine at Edinburgh City on 10 October 2017. |  SUPPORT |
CGAT |  - |  • James McDonaugh is a Scottish football manager, who is currently manager of Scottish League Two club Edinburgh City and a current UEFA Pro Licence holder.
• McDonaugh again won the SPFL League 2 Manager of the Month award in December
2019, winning all four games and keeping three clean sheets.
• He was awarded SPFL League 2 Manager of the Month in Sept 2018. |  - |  NEI
Table 10. Sample Claims from T-Liar.  Claim: Illinois suffered 1,652 overdose
deaths in 2014, of which 40 percent were associated with heroin and Illinois
is ranked number one in the nation for a decline in treatment capacitybetween
2007 and 2012. Ground Truth: SUPPORT
---
Method |  Events |  Retrieved Sentences |  Event Label |  Claim Label
TACV |  • Illinois suffered 1,652 overdose deaths in 2014 , of which 40 percent were associated with heroin |  •Illinois suffered 1,652 overdose deaths in 2014 – a 30 percent increase over 2010 – of which 40 percent were associated with heroin
•Durbin claims 40 percent of drug overdose deaths in Illinois involve heroin
•However, the Illinois Department of Public Health, which reports preliminary and final drug overdose deaths to the CDC, puts the 2010 total at 1,284 and 1,700 in 2014 – a slight discrepancy but not unusual when reporting overdose deaths as they often get revised |  SUPPORT |  SUPPORT
|  • Illinois ranked number one in the nation for a decline in treatment capacity between 2007 and 2012. |  •A report published in August 2015 by the Illinois Consortium on Drug Policy at Roosevelt University, or ICDP, shows state-funded treatment capacity in Illinois fell by 52 percent from 2007-2012, the largest decrease in the nation
•In 2007, Illinois ranked 28th in state-funded treatment capacity before
dropping to No. 44, or third worst in 2012, behind Tennessee and Texas,
respectively, according to the report.
•Durbin, who used statistics from this study, is correct when he says Illinois led the nation in the decline for state-funded treatment capacity. |  SUPPORT |
CGAT |  - |  •Illinois suffered 1,652 overdose deaths in 2014 – a 30 percent increase over 2010 – of which 40 percent were associated with heroin
• As for the other figures, the percent increase from 2010 is slightly more
than 32 percent, and drug overdose deaths in 2014 that were associated with
heroin is about 42 percent
•In 2007, Illinois ranked 28th in state-funded treatment capacity before dropping to No. 44, or third worst in 2012, behind Tennessee and Texas, respectively, according to the report |  - |  REFUTE

Table 9 shows a claim from T-FEVEROUS. The claim has two events ”appointed”
(in blue) and ”awarded” (in red) with temporal arguments ”on the 10th April
2019” and ”in December 2019” respectively. By decomposing the claim into
events and their temporal arguments, TACV is able to retrieve both ground
truth sentences, one supporting the ”awarded” event and the other
contradicting the ”appointed” event. LLM predicts the label REFUTE for the
first event and SUPPORT for the second event. As such, the claim label is
REFUTE. In contrast, CGAT does not retrieve the evidence sentence regarding
the job appointment and predicts the claim as NEI.

Table 10 shows a sample claim from T-Liar. The claim consists of two events:
”suffered” (in blue) and ”ranked” (in red), along with their temporal
arguments ”in 2014” and ”between 2007 and 2012”. By breaking down the claim
into events, TACV is able to retrieve sentences that confirm the date of
overdose deaths for the first event, and sentences that mention the period
when Illinois is ranked number one for decline in treatment capacity. This
allows LLM to verify each event as SUPPORT, and TACV to correctly conclude
that the overall claim label as SUPPORT. On the other hand, CGAT fails to
retrieve sentences that reference the date when Illinois was ranked first for
declined treatment capacity, leading to an incorrect prediction of the claim’s
label.

##  7\. Conclusion

In this work, we have introduced the TACV framework for temporal claim
verification. By decomposing complex sentences into event-level units and
incorporating the temporal proximity of events, TACV learns a temporal-aware
representations for evidence retrieval. Extensive experiments on both
synthetic and real world datasets show marked improvements in label accuracy
and FEVER score over state-of-the-art. Case studies highlight TACV’s ability
to retrieve relevant evidence for the verification of temporal claims.
Moreover, the curated datasets T-FEVER and T-FEVEROUS provide a standardized
benchmark and resource for research in temporal claim verification. While TACV
marks a significant step forward in temporal claim verification, further
research is needed to handle highly convoluted sentences with implicit
temporal references.

## References

  * (1)
  * Allein et al. (2021) Liesbeth Allein, Isabelle Augenstein, and Marie-Francine Moens. 2021.  Time-aware evidence ranking for fact-checking.  _Journal of Web Semantics_ 71 (2021), 100663.
  * Allein et al. (2023) Liesbeth Allein, Marlon Saelens, Ruben Cartuyvels, and Marie Francine Moens. 2023.  Implicit Temporal Reasoning for Evidence-Based Fact-Checking. In _Findings of the Association for Computational Linguistics: EACL 2023_. 176–189.
  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information.  arXiv:2106.05707 [cs.CL]
  * Barik et al. (2022) Anab Maulana Barik, Wynne Hsu, and Mong Li Lee. 2022.  Incorporating External Knowledge for Evidence-based Fact Verification. In _Companion Proceedings of the Web Conference 2022_. 429–437.
  * De Cao et al. (2021) Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021.  Autoregressive Entity Retrieval. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=5k8F6UU39V
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification. In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_. 103–108.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained Fact Verification with Kernel Graph Attention Network. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7342–7351.
  * Loshchilov and Hutter ([n. d.]) Ilya Loshchilov and Frank Hutter. [n. d.].  Decoupled Weight Decay Regularization. In _International Conference on Learning Representations_.
  * Mori et al. (2022) Marco Mori, Paolo Papotti, Luigi Bellomarini, and Oliver Giudice. 2022.  Neural machine translation for fact-checking temporal claims. In _Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER)_. 78–82.
  * Shi and Lin (2019) Peng Shi and Jimmy Lin. 2019.  Simple bert models for relation extraction and semantic role labeling.  _arXiv preprint arXiv:1904.05255_ (2019).
  * Soleimani et al. (2020) A Soleimani, C Monz, and M Worring. 2020.  BERT for Evidence Retrieval and Claim Verification.  _Advances in Information Retrieval_ 12036 (2020), 359–366.
  * Stammbach and Neumann (2019) Dominik Stammbach and Guenter Neumann. 2019.  Team DOMLIN: Exploiting evidence enhancement for the FEVER shared task. In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_. 105–109.
  * Thorne et al. (2018a) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a.  FEVER: a Large-scale Dataset for Fact Extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. 809–819.
  * Thorne et al. (2018b) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b.  The FEVER2.0 Shared Task. In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang (2017) William Yang Wang. 2017.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Vancouver, Canada, 422–426.  https://doi.org/10.18653/v1/P17-2067
  * Xu et al. (2020) Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2020.  Inductive representation learning on temporal graphs.  _arXiv preprint arXiv:2002.07962_ (2020).
  * Zhou et al. (2019) Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019.  GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 892–901.