# Question Answering Over Temporal Knowledge Graphs

Apoorv Saxena
Indian Institute of Science
Bangalore
<EMAIL_ADDRESS>
&Soumen Chakrabarti
Indian Institute of Technology
Bombay
<EMAIL_ADDRESS>
&Partha Talukdar
Google Research
India
<EMAIL_ADDRESS>

###### Abstract

Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by
providing temporal scopes (e.g., start and end times) on each edge in the KG.
While Question Answering over KG (KGQA) has received some attention from the
research community, QA over Temporal KGs (Temporal KGQA) is a relatively
unexplored area. Lack of broad-coverage datasets has been another factor
limiting progress in this area. We address this challenge by presenting
CronQuestions, the largest known Temporal KGQA dataset, clearly stratified
into buckets of structural complexity. CronQuestions expands the only known
previous dataset by a factor of 340$\times$. We find that various state-of-
the-art KGQA methods fall far short of the desired performance on this new
dataset. In response, we also propose CronKGQA, a transformer-based solution
that exploits recent advances in Temporal KG embeddings, and achieves
performance superior to all baselines, with an increase of 120% in accuracy
over the next best performing method. Through extensive experiments, we give
detailed insights into the workings of CronKGQA, as well as situations where
significant further improvements appear possible. In addition to the dataset,
we have released our code as well.

|  |  | Question Types |
---|---|---|---|---
Dataset | KG |  Temporal facts | Multi-Entity | Multi-Relation | Temporal | # questions
SimpleQuestions | FreeBase | ✗ | ✗ | ✗ | 0% | 108k
MetaQA | MetaQA KG | ✗ | ✗ | ✓ | 0% | 400k
WebQuestions | FreeBase | ✗ | ✗ | ✓ | <16% | 5,810
ComplexWebQuestions | FreeBase | ✗ | ✓ | ✓ | - | 35k
TempQuestions | FreeBase | ✗ | ✓ | ✓ | 100% | 1,271
CronQuestions (ours) | WikiData | ✓ | ✓ | ✓ | 100% | 410k

Table 1: KGQA dataset comparison. Statistics about percentage of temporal
questions for WebQuestions are taken from Jia et al. (2018a). We do not have
an explicit number of temporal questions for ComplexWebQuestions, but since it
is constructed automatically using questions from WebQuestions, we expect the
percentage to be similar to WebQuestions (16%). Please refer to Section 2.1
for details.

##  1 Introduction

Temporal Knowledge Graphs (Temporal KGs) are multi-relational graph where each
edge is associated with a time duration. This is in contrast to a regular KG
where no time annotation is present. For example, a regular KG may contain a
fact such as (Barack Obama, held position, President of USA), while a temporal
KG would contain the start and end time as well — (Barack Obama, held
position, President of USA, 2008, 2016). Edges may be associated with a set of
non-contiguous time intervals as well. These temporal scopes on facts can be
either automatically estimated Talukdar et al. (2012) or user contributed.
Several such Temporal KGs have been proposed in the literature, where the
focus is on KG completion (Dasgupta et al. 2018; García-Durán et al. 2018;
Leetaru and Schrodt 2013; Lacroix et al. 2020; Jain et al. 2020).

The task of Knowledge Graph Question Answering (KGQA) is to answer natural
language questions using a KG as the knowledge base. This is in contrast to
reading comprehension-based question answering, where typically the question
is accompanied by a context (e.g., text passage) and the answer is either one
of multiple choices (Rajpurkar et al., 2016) or a piece of text from the
context (Yang et al., 2018). In KGQA, the answer is usually an entity (node)
in the KG, and the reasoning required to answer questions is either single-
fact based (Bordes et al., 2015), multi-hop (Yih et al. 2015, Zhang et al.
2017) or conjunction/comparison based reasoning (Talmor and Berant, 2018).
Temporal KGQA takes this a step further where:

  1. 1.

The underlying KG is a Temporal KG.

  2. 2.

The answer is either an entity or time duration.

  3. 3.

Complex temporal reasoning might be needed.

KG Embeddings are low-dimensional dense vector representations of entities and
relations in a KG. Several methods have been proposed in the literature to
embed KGs (Bordes et al. 2013, Trouillon et al. 2016, Vashishth et al. 2020).
These embeddings were originally proposed for the task of KG completion i.e.,
predicting missing edges in the KG, since most real world KGs are incomplete.
Recently, however, they have also been applied to the task of KGQA where they
have been shown to increase performance the settings of both of complete and
incomplete KGs (Saxena et al. 2020; Sun et al. 2020).

Temporal KG embeddings are another upcoming area where entities, relations and
timestamps in a temporal KG are embedded in a low-dimensional vector space
(Dasgupta et al. 2018, Lacroix et al. 2020, Jain et al. 2020, Goel et al.
2019). Here too, the main application so far has been temporal KG completion.
In our work, we investigate whether temporal KG Embeddings can be applied to
the task of Temporal KGQA, and how they fare compared to non-temporal
embeddings or off-the-shelf methods without any KG Embeddings.

In this paper we propose CronQuestions, a new dataset for Temporal KGQA.
CronQuestions consists of both a temporal KG and accompanying natural language
questions. There were three main guiding principles while creating this
dataset:

  1. 1.

The associated KG must provide temporal annotations.

  2. 2.

Questions must involve an element of temporal reasoning.

  3. 3.

The number of labeled instances must be large enough that it can be used for
training models, rather than for evaluation alone.

Guided by the above principles, we present a dataset consisting of a Temporal
KG with 125k entities and 328k facts, along with a set of 410k natural
language questions that require temporal reasoning.

On this new dataset, we apply approaches based on deep language models (LM)
alone, such as T5 (Raffel et al., 2020), BERT (Devlin et al., 2019), and
KnowBERT (Peters et al., 2019), and also hybrid LM+KG embedding approaches,
such as Entities-as-Experts (Févry et al., 2020) and EmbedKGQA (Saxena et al.,
2020). We find that these baselines are not suited to temporal reasoning. In
response, we propose CronKGQA, an enhancement of EmbedKGQA, which outperforms
baselines across all question types. CronKGQA achieves very high accuracy on
simple temporal reasoning questions, but falls short when it comes to
questions requiring more complex reasoning. Thus, although we get promising
early results, CronQuestions leaves ample scope to improve complex Temporal
KGQA. Our source code along with the CronQuestions dataset can be found at
https://github.com/apoorvumang/CronKGQA.

Reasoning | Example Template | Example Question
---|---|---
Simple time | When did {head} hold the position of {tail} | When did Obama hold the position of President of USA
Simple entity | Which award did {head} receive in {time} | Which award did Brad Pitt receive in 2001
Before/After | Who was the {tail} {type} {head} | Who was the President of USA before Obama
First/Last | When did {head} play their {adj} game | When did Messi play their first game
Time join | Who held the position of {tail} during {event} | Who held the position of President of USA during WWII

Table 2: Example questions for different types of temporal reasoning. {head},
{tail} and {time} correspond to entities/timestamps in facts of the form
(head, relation, tail, timestamp). {event} corresponds to entities in event
facts eg. WWII. {type} can be one of before/after and {adj} can be one of
first/last. Please refer to Section 3.2 for details.

##  2 Related work

###  2.1 Temporal QA data sets

There have been several KGQA datasets proposed in the literature (Table 1). In
SimpleQuestions (Bordes et al., 2015) one needs to extract just a single fact
from the KG to answer a question. MetaQA (Zhang et al., 2017) and
WebQuestionsSP (Yih et al., 2015) require multi-hop reasoning, where one must
traverse over multiple edges in the KG to reach the answer.
ComplexWebQuestions (Talmor and Berant, 2018) contains both multi-hop and
conjunction/comparison type questions. However, none of these are aimed at
temporal reasoning, and the KG they are based on is non-temporal.

Temporal QA datasets have mostly been studied in the area of reading
comprehension. One such dataset is TORQUE (Ning et al., 2020), where the
system is given a question along with some context (a text passage) and is
asked to answer a multiple choice question with five choices. This is in
contrast to KGQA, where there is no context, and the answer is one of
potentially hundreds of thousands of entities.

TempQuestions (Jia et al., 2018a) is a KGQA dataset specifically aimed at
temporal QA. It consists of a subset of questions from WebQuestions, Free917
(Cai and Yates, 2013) and ComplexQuestions (Bao et al., 2016) that are
temporal in nature. They gave a definition for “temporal question” and used
certain trigger words (for example ‘before’, ‘after’) along with other
constraints to filter out questions from these datasets that fell under this
definition. However, this dataset contains only 1271 questions — useful only
for evaluation — and the KG on which it is based (a subset of FreeBase
(Bollacker et al., 2008)) is not a temporal KG. Another drawback is that
FreeBase has not been under active development since 2015, therefore some
information stored in it is outdated and this is a potential source of
inaccuracy.

###  2.2 Temporal QA algorithms

To the best of our knowledge, recent KGQA algorithms (Miller et al. 2016; Sun
et al. 2019; Cohen et al. 2020; Sun et al. 2020) work with non-temporal KGs,
i.e., KGs containing facts of the form (subject, relation, object). Extending
these to temporal KGs containing facts of the form (subject, relation, object,
start time, end time) is a non-trivial task. TEQUILA (Jia et al., 2018b) is
one method aimed specifically at temporal KGQA. TEQUILA decomposes and
rewrites the question into non-temporal sub-questions and temporal
constraints. Answers to sub-questions are then retrieved using any KGQA
engine. Finally, TEQUILA uses constraint reasoning on temporal intervals to
compute final answers to the full question. A major drawback of this approach
is the use of pre-specified templates for decomposition, as well as the
assumption of having temporal constraints on entities. Also, since it is made
for non-temporal KGs, there is no direct way of applying it to temporal KGs
where facts are temporally scoped.

##  3 CronQuestions: The new Temporal KGQA dataset

CronQuestions, our Temporal KGQA dataset consists of two parts: a KG with
temporal annotations, and a set of natural language questions requiring
temporal reasoning.

###  3.1 Temporal KG

To prepare our temporal KG, we started by taking all facts with temporal
annotations from the WikiData subset proposed by Lacroix et al. (2020). We
removed some instances of the predicate “member of sports team” in order to
balance out the KG since this predicate constituted over 50 percent of the
facts. Timestamps were discretized to years. This resulted in a KG with 323k
facts, 125k entities and 203 relations.

However, this filtering of facts misses out on important world events. For
example, the KG subset created using the aforementioned technique contains the
entity World War II but no associated fact that tells us when World War II
started or ended. This knowledge is needed to answer questions such as “Who
was the President of the USA during World War II?.” To overcome this
shortcoming, we first extracted entities from WikiData that have a “start
time” and “end time” annotation. From this set, we then removed entities which
were game shows, movies or television series (since these are not important
world events, but do have a start and end time annotation), and then removed
entities with less than 50 associated facts. This final set of entitities was
then added as facts in the format (WWII, significant event, occurred, 1939,
1945). The final Temporal KG consisted of 328k facts out of which 5k are
event-facts.

Template |  When did {head} play in {tail}
---|---
Seed Qn | When did Messi play in FC Barcelona
Human Paraphrases |  When was Messi playing in FC Barcelona Which years did Messi play in FC Barcelona When did FC Barcelona have Messi in their team What time did Messi play in FC Barcelona
Machine Paraphrases |  When did Messi play for FC Barcelona When did Messi play at FC Barcelona When has Messi played at FC Barcelona

Table 3: Slot-filled paraphrases generated by humans and machine. Please refer
to Section 3.2 for details.

| Train | Dev | Test
---|---|---|---
Simple Entity | 90,651 | 7,745 | 7,812
Simple Time | 61,471 | 5,197 | 5,046
Before/After | 23,869 | 1,982 | 2,151
First/Last | 118,556 | 11,198 | 11,159
Time Join | 55,453 | 3,878 | 3,832
Entity Answer | 225,672 | 19,362 | 19,524
Time Answer | 124,328 | 10,638 | 10,476
Total | 350,000 | 30,000 | 30,000

Table 4: Number of questions in our dataset across different types of
reasoning required and different answer types. Please refer to Section 3.2.1
for details.

###  3.2 Temporal Questions

To generate the QA dataset, we started with a set of templates for temporal
reasoning. These were made using the five most frequent relations from our
WikiData subset, namely

  * •

member of sports team

  * •

position held

  * •

award received

  * •

spouse

  * •

employer

This resulted in 30 unique seed templates over five relations and five
different reasoning structures (please see Table 2 for some examples). Each of
these templates has a corresponding procedure that could be executed over the
temporal KG to extract all possible answers for that template. However,
similar to Zhang et al. (2017), we chose not to make this procedure a part of
the dataset, to remove unwelcome dependence of QA systems on such formal
candidate collection methods. This also allows easy augmentation of the
dataset, since only question-answer pairs are needed.

In the same spirit as ComplexWebQuestions, we then asked human annotators to
paraphrase these templates in order to generate more linguistic diversity.
Annotators were given slot-filled templates with dummy entities and times, and
asked to rephrase the question such that the dummy entities/times were present
in the paraphrase and the question meaning did not change. This resulted in
246 unique templates.

We then used the monolingual paraphraser developed by Hu et al. (2019) to
automatically generate paraphrases using these 246 templates. After verifying
their correctness through annotators, we ended up with 654 templates. These
templates were then filled using entity aliases from WikiData to generate 410k
unique question-answer pairs.

Finally, while splitting the data into train/test folds, we ensured that

  1. 1.

Paraphrases of train questions are not present in test questions.

  2. 2.

There is no entity overlap between test questions and train questions. Event
overlap is allowed.

The second requirement implies that, if the question “Who was president before
Obama” is present in the train set, the test set cannot contain any question
that mentions the entity ‘Obama’. While this policy may appear like an
overabundance of caution, it ensures that models are doing temporal reasoning
rather than guessing from entities seen during training. Lewis et al. (2020)
noticed an issue in WebQuestions where they found that almost 30% of test
questions overlapped with training questions. The issue has been seen in the
MetaQA dataset as well, where there is significant overlap between test/train
entities and test/train question paraphrases, leading to suspiciously high
performance on baseline methods even with partial KG data (Saxena et al.,
2020), which suggests that models that apparently perform well are not
necessarily performing the desired reasoning over the KG.

A drawback of our data creation protocol is that question/answer pairs are
generated automatically. Therefore, the question distribution is artificial
from a semantic perspective. (ComplexWebQuestions has a similar limitation.)
However, since developing models that are capable of temporal reasoning is an
important direction for natural language understanding, we feel that our
dataset provides an opportunity to both train and evaluate KGQA models because
of its large size, notwithstanding its lower-than-natural linguistic variety.
In Section 6.4, we show the effect that training data size has on model
performance.

Summarizing, each of our examples contains

  1. 1.

A paraphrased natural language question.

  2. 2.

A set of entities/times in the question.

  3. 3.

A set of ‘gold’ answers (entity or time).

The entities are specified as WikiData IDs (e.g., Q219237), and times are
years (e.g., 1991). We include the set of entities/times in the test questions
as well since similar to other KGQA datasets (MetaQA, WebQuestions,
ComplexWebQuestions) and methods that use these datasets (PullNet, EmQL),
entity linking is considered as a separate problem and complete entity linking
is assumed. We also include the seed template and head/tail/time annotation in
the train fold, but omit these from the test fold.

Figure 1: The CronKGQA method. (i) A temporal KG embedding model (Section 4)
is used to generate embeddings for each timestamp and entity in the temporal
knowledge graph (ii) BERT is used to get two question embeddings:
$\bm{qe}_{ent}$ and $\bm{qe}_{time}$. (iii) Embeddings of entity/time mentions
in the question are combined with question embeddings using equations 4 and 5
to get score vectors for entity and time prediction. (iv) Score vectors are
concatenated and softmax is used get answer probabilities. Please refer to
Section 5 for details.

####  3.2.1 Question Categorization

In order to aid analysis, we categorize questions into “simple reasoning” and
“complex reasoning” questions (please refer to Table 4 for the distribution
statistics).

Simple reasoning:

These questions require a single fact to answer, where the answer can be
either an entity or a time instance. For example the question “Who was the
President of the United States in 2008?” requires a single fact to answer the
question, namely (Barack Obama, held position, President of USA, 2008, 2016)

Complex reasoning:

These questions require multiple facts to answer and can be more varied. For
example “Who was the first President of the United States?” This requires
reasoning over multiple facts pertaining to the entity “President of the
United States”. In our dataset, all questions that are not “simple reasoning”
questions are considered complex questions. These are further categorized into
the types “before/after‘’, “first/last” and “time join” — please refer Table 2
for examples of these questions.

##  4 Temporal KG Embeddings

We investigate how we can use KG embeddings, both temporal and non-temporal,
along with pre-trained language models to perform temporal KGQA. We will first
briefly describe the specific KG embedding models we use, and then go on to
show how we use them in our QA models. In all cases, the scores are turned
into suitable losses with regard to positive and negative tuples in an
incomplete KG, and these losses minimized to train the entity, time and
relation representations.

###  4.1 ComplEx

ComplEx (Trouillon et al., 2016) represents each entity $e$ as a complex
vector $\bm{u}_{e}\in\mathbb{C}^{D}$. Each relation $r$ is represented as a
complex vector $\bm{v}_{r}\in\mathbb{C}^{D}$ as well. The score $\phi$ of a
claimed fact $(s,r,o)$ is

$\displaystyle\phi(s,r,o)$
$\displaystyle=\Re(\langle\bm{u}_{s},\bm{v}_{r},\bm{u}_{o}^{\star}\rangle)$
$\displaystyle=\Re\big{(}\textstyle\sum_{d=1}^{D}\bm{u}_{s}[d]\bm{v}_{r}[d]\bm{u}_{o}[d]^{\star}\big{)}$
(1)

where $\Re(\cdot)$ denotes the real part and $c^{\star}$ is the complex
conjugate. Despite further developments, ComplEx, along with refined training
protocols (Lacroix et al., 2018) remains among the strongest KB embedding
approaches (Ruffinelli et al., 2020).

###  4.2 TComplEx, TNTComplEx

Lacroix et al. (2020) took an early step to extend ComplEx with time. Each
timestamp $t$ is also represented as a complex vector
$\bm{w}_{t}\in\mathbb{C}^{D}$. For a claimed fact $(s,r,o,t)$, their TComplEx
scoring function is

$\displaystyle\phi(s,r,o,t)$
$\displaystyle=\Re(\langle\bm{u}_{s},\bm{v}_{r},\bm{u}_{o}^{\star},\bm{w}_{t}\rangle)$
(2)

Their TNTComplEx scoring function uses two representations of relations $r$:
$\bm{v}^{\text{T}}_{r}$, which is sensitive to time, and $\bm{v}_{r}$, which
is not. The scoring function is the sum of a time-sensitive and a time-
insensitive part:
$\Re(\langle\bm{u}_{s},\bm{v}^{\text{T}}_{r},\bm{u}_{o}^{\star},\bm{w}_{t}\rangle+\langle\bm{u}_{s},\bm{v}_{r},\bm{u}_{o}^{\star},\mathbf{1}\rangle)$.

###  4.3 TimePlex

TimePlex (Jain et al., 2020) augmented ComplEx with embeddings
$\bm{u}_{t}\in\mathbb{C}^{D}$ for discretized time instants $t$. To
incorporate time, TimePlex uses three representations for each relation $r$,
viz., $(\bm{v}^{\text{SO}}_{r},\bm{v}^{\text{ST}}_{r},\bm{v}^{\text{OT}}_{r})$
and writes the base score of a tuple $(s,r,o,t)$ as

$\phi(s,r,o,t)=\langle\bm{u}_{s},\bm{v}_{r}^{\text{SO}},\bm{u}_{o}^{\star}\rangle+\alpha\,\langle\bm{u}_{s},\bm{v}_{r}^{\text{ST}},\bm{u}_{t}^{\star}\rangle\\\
+\beta\,\langle\bm{u}_{o},\bm{v}_{r}^{\text{OT}},\bm{u}_{t}^{\star}\rangle+\gamma\,\langle\bm{u}_{s},\bm{u}_{o},\bm{u}_{t}^{\star}\rangle,$
(3)

where $\alpha,\beta,\gamma$ are hyperparameters.

##  5 CronKGQA: Our proposed method

We start with a temporal KG, apply a time-agnostic or time-sensitive KG
embedding algorithm (ComplEx, TComplEx, or TimePlex) to it, and obtain entity,
relation, and timestamp embeddings for the temporal KG. We will use the
following notation.

  * •

$\mathcal{E}$ is the matrix of entity embeddings

  * •

$\mathcal{T}$ is the matrix of timestamp embeddings

  * •

$\mathcal{E.T}$ is the concatenation of $\mathcal{E}$ and $\mathcal{T}$
matrices. This is used for scoring answers, since the answer can be either an
entity or timestamp.

In case entity/timestamp embeddings are complex valued vectors in
$\mathbb{C}^{D}$, we expand them to real valued vectors of size $2D$, where
the first half is the real part and the second half is the complex part of the
original vector.

We first apply EmbedKGQA (Saxena et al., 2020) directly to the task of
Temporal KGQA. In its original implementation, EmbedKGQA uses ComplEx (Section
4.1) embeddings and can only deal with non-temporal KGs and single entity
questions. In order to apply it to CronQuestions, we set the first entity
encountered in the question as the “head entity” needed by EmbedKGQA. Along
with this, we set the entity embedding matrix $\mathcal{E}$ to be the ComplEx
embedding of our KG entities, and initialize $\mathcal{T}$ to a random
learnable matrix. EmbedKGQA then performs prediction over $\mathcal{E.T}$.

Next, we modify EmbedKGQA so that it can use temporal KG embeddings. We use
TComplEx (Section 4.2) for getting entity and timestamp embeddings. CronKGQA
(Figure 1) utilizes two scoring functions, one for predicting entity and one
for predicting time. Using a pre-trained LM (BERT in our case) CronKGQA finds
a question embedding $\bm{qe}$. This is then projected to get two embeddings,
$\bm{qe}_{ent}$ and $\bm{qe}_{time}$, which are question embeddings for entity
and time prediction respectively.

Entity scoring function:

We extract a subject entity $s$ and a timestamp $t$ from the question. If
either is missing, we use a dummy entity/time. Then, using the scoring
function $\phi(s,r,o,t)$ from equation 2, we calculate a score for each entity
$e\in\mathbf{E}$ as

$\displaystyle\bm{\phi_{ent}}(e)=\Re(\langle\bm{u}_{s},\bm{qe}_{ent},\bm{u}_{e}^{\star},\bm{w}_{t}\rangle)$
(4)

where $\mathbf{E}$ is the set of entities in the KG. This gives us a score for
each entity being an answer.

Time scoring function:

Similarly, we extract a subject entity $s$ and object entity $o$ from the
question, using dummy entities if none are present. Then, using 2, we
calculate a score for each timestamp $t\in\mathbf{T}$ as

$\displaystyle\bm{\phi_{time}}(t)=\Re(\langle\bm{u}_{s},\bm{qe}_{time},\bm{u}_{o}^{\star},\bm{w}_{t}\rangle)$
(5)

The scores for all entities and times are concatenated, and $\mathrm{softmax}$
is used to calculate answer probabilities over this combined score vector. The
model is trained using cross entropy loss.

Model | Hits@1 | Hits@10
---|---|---
Overall | Question Type | Answer Type | Overall | Question Type | Answer Type
Complex | Simple | Entity | Time | Complex | Simple | Entity | Time
BERT | 0.071 | 0.086 | 0.052 | 0.077 | 0.06 | 0.213 | 0.205 | 0.225 | 0.192 | 0.253
RoBERTa | 0.07 | 0.086 | 0.05 | 0.082 | 0.048 | 0.202 | 0.192 | 0.215 | 0.186 | 0.231
KnowBERT | 0.07 | 0.083 | 0.051 | 0.081 | 0.048 | 0.201 | 0.189 | 0.217 | 0.185 | 0.23
T5-3B | 0.081 | 0.073 | 0.091 | 0.088 | 0.067 | - | - | - | - | -
EmbedKGQA | 0.288 | 0.286 | 0.29 | 0.411 | 0.057 | 0.672 | 0.632 | 0.725 | 0.85 | 0.341
T-EaE-add | 0.278 | 0.257 | 0.306 | 0.313 | 0.213 | 0.663 | 0.614 | 0.729 | 0.662 | 0.665
T-EaE-replace | 0.288 | 0.257 | 0.329 | 0.318 | 0.231 | 0.678 | 0.623 | 0.753 | 0.668 | 0.698
CronKGQA | 0.647 | 0.392 | 0.987 | 0.699 | 0.549 | 0.884 | 0.802 | 0.992 | 0.898 | 0.857

Table 5: Performance of baselines and our methods on the CronQuestions
dataset. Methods above the midrule do not use any KG embeddings, while the
ones below use either temporal or non-temporal KG embeddings. Hits@10 are not
available for T5-3B since it is a text-to-text model and makes a single
prediction. Please refer to Section 6.2 for details.

##  6 Experiments and diagnostics

In this section, we aim to answer the following questions:

  1. 1.

How do baselines and CronKGQA perform on the CronQuestions task? (Section
6.2.)

  2. 2.

Do some methods perform better than others on specific reasoning tasks?
(Section 6.3.)

  3. 3.

How much does the training dataset size (number of questions) affect the
performance of a model? (Section 6.4.)

  4. 4.

Do temporal KG embeddings confer any advantage over non-temporal KG
embeddings? (Section 6.5.)

###  6.1 Other methods compared

It has been shown by Petroni et al. (2019) and Raffel et al. (2020) that large
LMs, such as BERT and its variants, capture real world knowledge (collected
from their massive, encyclopedic training corpus) and can directly be applied
to tasks such as QA. In these baselines, we do not specifically feed our
version of the temporal KG to the model — we instead expect the model to have
the real world knowledge to compute the answer.

BERT:

We experiment with BERT, RoBERTa (Liu et al., 2019) and KnowBERT (Peters et
al., 2019) which is a variant of BERT where information from knowledge bases
such as WikiData and WordNet has been injected into BERT. We add a prediction
head on top of the [CLS] token of the final layer and do a $\mathrm{softmax}$
over it to predict the answer probabilities.

T5:

In order to apply T5 (Raffel et al., 2020) to temporal QA, we transform each
question in our dataset to the form ‘temporal question:
$\langle\mathrm{question}\rangle$?’. For evaluation there are two cases:

  1. 1.

Time answer: We do exact string matching between T5 output and correct answer.

  2. 2.

Entity answer: We compare the system output to the aliases of all entities in
the KG. The entity having an alias with the smallest edit distance
(Levenshtein, 1966) to the predicted text output is taken as the predicted
entity.

Entities as experts:

Févry et al. (2020) proposed EaE, a model which aims to integrate entity
knowledge into a transformer-based language model. For temporal KGQA on
CronQuestions, we assume that all grounded entity and time mention spans are
marked in the question111This assumption can be removed by using EaE’s early
transformer stages as NE spotters and disambiguators.. We will refer to this
model as T-EaE-add. We try another variant of EaE, T-EaE-replace, where
instead of adding the entity/time and BERT token embeddings, we replace the
BERT embeddings with the entity/time embeddings for entity/time
mentions.222Appendix A.1 gives details of our EaE implementation.

###  6.2 Main results

Table 5 shows the results of various methods on our dataset. We see that
methods based on large pre-trained LMs alone (BERT, RoBERTa, T5), as well as
KnowBERT, perform significantly worse than methods that are augmented with KG
embeddings (temporal or non-temporal). This is probably because having KG
embeddings specific to our temporal KG helps the model to focus on those
entities/timestamps. In our experiments, BERT performs slightly better than
KnowBERT, even though KnowBERT has entity knowledge in its parameters. T5-3B
performs the best among the LMs we tested, possibly because of the large
number of parameters and pre-training.

Even among methods that use KG embeddings, CronKGQA performs the best on all
metrics, followed by T-EaE-replace. Since EmbedKGQA has non-temporal
embeddings, its performance on questions where the answer is a time is very
low — comparable to BERT — which is the LM used in our EmbedKGQA
implementation.

Another interesting thing to note is the performance on simple reasoning
questions. CronKGQA far outperforms baselines for simple questions, achieving
close to 0.99 hits@1, which is much lower for T-EaE (0.329). We believe there
might be a few reasons that contribute to this:

  1. 1.

There is the inductive bias of combining embeddings using TComplEx scoring
function in CronKGQA, which is the same one used in creating the entity and
time embeddings, thus making the simple questions straightforward to answer.
However, not relying on a scoring function means that T-EaE can be extended to
any KG embedding, whereas CronKGQA cannot.

  2. 2.

Another contributing reason could be that there are fewer parameters to be
trained in CronKGQA while a 6-layer Transformer encoder needs to be trained
from scratch in T-EaE. Transformers typically require large amounts of varied
data to train successfully.

$0$$20$$40$$60$$80$$100$$0.2$$0.4$$0.6$$0.8$$1$Train Dataset size
(%)Hits@10CronKGQA simpleCronKGQA complexT-EaE-add simpleT-EaE-add complex
Figure 2: Model performance (hits@10) vs. training dataset size (percentage)
for CronKGQA and T-EaE-add. Solid line is for simple reasoning and dashed line
is for complex reasoning type questions. For each dataset size, models were
trained until validation hits@10 did not increase for 10 epochs. Please refer
to Section 6.4 for details.

###  6.3 Performance across question types

|  | Before/
---
After
| First/
---
Last
| Time
---
Join
| Simple
---
Entity
| Simple
---
Time
| All
---
EmbedKGQA | 0.199 | 0.324 | 0.223 | 0.421 | 0.087 | 0.288
T-EaE-add | 0.256 | 0.285 | 0.175 | 0.296 | 0.321 | 0.278
T-EaE-replace | 0.256 | 0.288 | 0.168 | 0.318 | 0.346 | 0.288
CronKGQA | 0.288 | 0.371 | 0.511 | 0.988 | 0.985 | 0.647

Table 6: Hits@1 for different reasoning type questions. ‘Simple Entity’ and
‘Simple Time’ correspond to simple question type in Table 5 while the others
correspond to complex question type. Please refer to section 6.3 for more
details.

Table 6 shows the performance of KG embedding based models across different
types of reasoning. As stated above in Section 6.2, CronKGQA performs very
well on simple reasoning questions (simple entity, simple time). Among complex
question types, all models (except EmbedKGQA) perform the best on time join
questions (e.g., ‘Who played with Roberto Dinamite on the Brazil national
football team’). This is because such questions typically have multiple
answers (such as all the players when Roberto Dinamite was playing for
Brazil), which makes it easier for the model to make a correct prediction. In
the other two question types, the answer is always a single entity/time.
Before/after questions seem most challenging for all methods, with the best
method achieving only 0.288 hits@1.

###  6.4 Effect of training dataset size

Figure 2 shows the effect of training dataset size on model performance. As we
can see, for T-EaE-add, increasing the training dataset size from 10% to 100%
steadily increases its performance for both simple and complex reasoning type
questions. This effect is somewhat present in CronKGQA for complex reasoning,
but not so for simple reasoning type questions. We hypothesize that this is
because T-EaE has more trainable parameters — it has a 6-layer transformer
that needs to be trained from scratch — in contrast to CronKGQA that needs to
merely fine tune BERT and train some shallow projection layers. These results
affirm our hypothesis that having a large, even if synthetic, dataset is
useful for training temporal reasoning models.

###  6.5 Temporal vs. non-temporal KG embeddings

We conducted further experiments to study the effect of temporal vs. non-
temporal KG embeddings. We replaced the temporal entity embeddings in T-EaE-
replace with ComplEx embeddings, and treated timestamps as regular tokens (not
associated with any entity/time mentions). CronKGQA-CX is the same as
EmbedKGQA. The results can be seen in Table 7. As we can see, for both
CronKGQA and T-EaE-replace, using temporal KGE (TComplex) gives a significant
boost in performance compared to non-temporal KGE (ComplEx). CronKGQA receives
a much larger boost in performance compared to T-EaE-replace, probably because
the scoring function has been modeled after TComplEx and not ComplEx, while
there is no such embedding-specific engineering in T-EaE-replace. Another
observation is that questions having temporal answers achieve very low
accuracy (0.057 and 0.062 respectively) in both CronKGQA-CX and T-EaE-replace-
CX, which is much lower than what these models achieve with TComplEx. This
shows that having temporal KG embeddings is essential for achieving good
performance for KG embedding-based methods.

Question Type | CronKGQA | T-EaE-replace
---|---|---
CX | TCX | CX | TCX
Simple | 0.29 | 0.987 | 0.248 | 0.329
Complex | 0.286 | 0.392 | 0.247 | 0.257
Entity Answer | 0.411 | 0.699 | 0.347 | 0.318
Time Answer | 0.057 | 0.549 | 0.062 | 0.231
Overall | 0.288 | 0.647 | 0.247 | 0.288

Table 7: Hits@1 for CronKGQA and T-EaE-replace using ComplEx(CX) and
TComplEx(TCX) KG embeddings. Please refer to Section 6.5 for more details.

##  7 Conclusion

In this paper we introduce CronQuestions, a new dataset for Temporal Knowledge
Graph Question Answering. While there exist some Temporal KGQA datasets, they
are all based on non-temporal KGs (e.g., Freebase) and have relatively few
questions. Our dataset consists of both a temporal KG as well as a large set
of temporal questions requiring various structures of reasoning. In order to
develop such a large dataset, we used a synthetic generation procedure,
leading to a question distribution that is artificial from a semantic
perspective. However, having a large dataset provides an opportunity to train
models, rather than just evaluate them. We experimentally show that increasing
the training dataset size steadily improves the performance of certain methods
on the TKGQA task.

We first apply large pre-trained LM based QA methods on our new dataset. Then
we inject KG embeddings, both temporal and non-temporal, into these LMs and
observe significant improvement in performance. We also propose a new method,
CronKGQA, that is able to leverage Temporal KG Embeddings to perform TKGQA. In
our experiments, CronKGQA outperforms all baselines. These results suggest
that KG embeddings can be effectively used to perform temporal KGQA, although
there remains significant scope for improvement when it comes to complex
reasoning questions.

## Acknowledgements

We would like to thank the anonymous reviewers for their constructive
feedback, and Pat Verga and William Cohen from Google Research for their
insightful comments. We would also like to thank Chitrank Gupta (IIT Bombay)
for his help in debugging the source code and dataset. This work is supported
in part by a gift from Google Research, India and a Jagadish Bose Fellowship.

## References

  * Bao et al. (2016) Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016.  Constraint-based question answering with knowledge graph.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 2503–2514, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: A collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’08, page 1247–1250, New York, NY, USA. Association for Computing Machinery.
  * Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015.  Large-scale simple question answering with memory networks.  _arXiv preprint arXiv:1506.02075_.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Neural Information Processing Systems (NIPS)_ , pages 1–9.
  * Cai and Yates (2013) Qingqing Cai and Alexander Yates. 2013.  Large-scale semantic parsing via schema matching and lexicon extension.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 423–433, Sofia, Bulgaria. Association for Computational Linguistics.
  * Cohen et al. (2020) William W. Cohen, Haitian Sun, R. Alex Hofer, and Matthew Siegler. 2020.  Scalable neural methods for reasoning with a symbolic knowledge base.
  * Dasgupta et al. (2018) Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018.  HyTE: Hyperplane-based temporally aware knowledge graph embedding.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2001–2011, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Févry et al. (2020) Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020.  Entities as experts: Sparse memory access with entity supervision.  _arXiv preprint arXiv:2004.07202_.
  * García-Durán et al. (2018) Alberto García-Durán, Sebastijan Dumančić, and Mathias Niepert. 2018.  Learning sequence encoders for temporal knowledge graph completion.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4816–4821, Brussels, Belgium. Association for Computational Linguistics.
  * Goel et al. (2019) Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. 2019.  Diachronic embedding for temporal knowledge graph completion.
  * Hu et al. (2019) J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019.  Improved lexically constrained decoding for translation and monolingual rewriting.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jain et al. (2020) Prachi Jain, Sushant Rathi, Mausam, and Soumen Chakrabarti. 2020.  Temporal Knowledge Base Completion: New Algorithms and Evaluation Protocols.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3733–3747, Online. Association for Computational Linguistics.
  * Jia et al. (2018a) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018a.  Tempquestions: A benchmark for temporal question answering.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, page 1057–1062, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Jia et al. (2018b) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b.  Tequila.  _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_.
  * Lacroix et al. (2020) Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020.  Tensor decompositions for temporal knowledge base completion.  _arXiv preprint arXiv:2004.04926_.
  * Lacroix et al. (2018) Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. 2018.  Canonical tensor decomposition for knowledge base completion.  _arXiv preprint arXiv:1806.07297_.
  * Leetaru and Schrodt (2013) Kalev Leetaru and Philip A Schrodt. 2013.  Gdelt: Global data on events, location, and tone, 1979–2012.  In _ISA annual convention_ , volume 2, pages 1–49. Citeseer.
  * Levenshtein (1966) Vladimir I Levenshtein. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10 (8), pages 707–710. Soviet Union.
  * Lewis et al. (2020) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020.  Question and answer test-train overlap in open-domain question answering datasets.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  _arXiv preprint arXiv:1606.03126_.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020.  Torque: A reading comprehension dataset of temporal ordering questions.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert L. Logan IV au2, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge enhanced contextual word representations.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019.  Language models as knowledge bases?
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Ruffinelli et al. (2020) Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020.  You can teach an old dog new tricks! on training knowledge graph embeddings.  In _International Conference on Learning Representations_.
  * Saxena et al. (2020) Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020.  Improving multi-hop question answering over knowledge graphs using knowledge base embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4498–4507, Online. Association for Computational Linguistics.
  * Sun et al. (2020) Haitian Sun, Andrew O. Arnold, Tania Bedrax-Weiss, Fernando Pereira, and William W. Cohen. 2020.  Faithful embeddings for knowledge base queries.
  * Sun et al. (2019) Haitian Sun, Tania Bedrax-Weiss, and William W. Cohen. 2019.  Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.
  * Talukdar et al. (2012) Partha Pratim Talukdar, Derry Wijaya, and Tom Mitchell. 2012.  Coupled temporal scoping of relational facts.  In _Proceedings of WSDM 2012_.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction.  In _International Conference on Machine Learning (ICML)_.
  * Vashishth et al. (2020) Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2020.  Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34 (03), pages 3009–3016.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_.
  * Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015.  Semantic parsing via staged query graph generation: Question answering with knowledge base.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1321–1331, Beijing, China. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2017\.  Variational reasoning for question answering with knowledge graph.

##  Appendix A Appendix

###  A.1 Entities as Experts (EaE)

The model architecture follows Transformer (Vaswani et al., 2017) interleaved
with an entity memory layer. It has two embedding matrices, for tokens and
entities. It works on the input sequence $x$ as follows.

$\displaystyle X^{0}$ $\displaystyle=\mathrm{TokenEmbed}(x)$ (6)
$\displaystyle X^{1}$
$\displaystyle=\mathrm{Transformer_{0}}(X^{0},\mathrm{num\\_layers}=l_{0})$
$\displaystyle X^{2}$ $\displaystyle=\mathrm{EntityMemory}(X^{1})$
$\displaystyle X^{3}$ $\displaystyle=\mathrm{LayerNorm}(X^{2}+X^{1})$
$\displaystyle X^{4}$
$\displaystyle=\mathrm{Transformer_{1}}(X^{3},\mathrm{num\\_layers}=l_{1})$
$\displaystyle X^{5}$ $\displaystyle=\mathrm{TaskSpecificHeads}(X^{4})$

The whole model (transformers, token and entity embeddings, and task-specific
heads) is trained end to end using losses for entity linking, mention
detection and masked language modeling.

###  A.2 EaE for Temporal KGQA

CronQuestions does not provide a text corpus for training language models.
Therefore, we use BERT (Devlin et al., 2019) for $\mathrm{Transformer_{0}}$ as
well as $\mathrm{TokenEmbed}$ (eqn. 6). For $\mathrm{EntityMemory}$, we use
TComplEx/TimePlex embeddings of entities and timestamps that have been pre-
trained using the CronQuestions KG (please refer to Section 4 for details on
KG embeddings). The modified model is as follows:

$\displaystyle X^{1}$ $\displaystyle=\mathrm{BERT}(x)$ (7) $\displaystyle
X^{2}$ $\displaystyle=\mathrm{EntityTimeEmbedding}(X^{1})$ $\displaystyle
X^{3}$ $\displaystyle=\mathrm{LayerNorm}(X^{2}+X^{1})$ $\displaystyle X^{4}$
$\displaystyle=\mathrm{Transformer_{1}}(X^{3},\mathrm{num\\_layers}=6)$
$\displaystyle X^{5}$ $\displaystyle=\mathrm{PredictionHead}(X^{4})$

For simplicity, we assume that all grounded entity and time mention spans are
marked in the question, i.e., for each token, we know. which entity or
timestamp it belongs to (or if it doesn’t belong to any). Thus, for each token
$x_{i}$ in the input $x$,

  * •

$X^{1}[i]$ contains the contextual BERT embedding of $x_{i}$

  * •

For $X^{2}[i]$ there are 3 cases.

    * –

$x_{i}$ is a mention of entity $e$. Then $X^{2}[i]=\mathcal{E}[e]$.

    * –

$x_{i}$ is a mention of timestamp $t$. Then $X^{2}[i]=\mathcal{T}[t]$.

    * –

$x_{i}$ is not a mention. Then $X^{2}[i]$ is the zero vector.

$\mathrm{PredictionHead}$ takes the final output from
$\mathrm{Transformer_{1}}$ of the token corresponding to the $\mathrm{[CLS]}$
token of BERT as the predicted answer embedding. This answer embedding is
scored against $\mathcal{E.T}$ using dot product to get a score for each
possible answer, and $\mathrm{softmax}$ is taken to get answer probabilities.
The model is trained on the QA dataset using cross-entropy loss. We will refer
to this model as T-EaE-add since we are taking element-wise sum of BERT and
entity/time embeddings.

T-EaE-replace Instead of adding entity/time and BERT embeddings, we replace
the BERT embeddings with the entity/time embeddings for entity/time mentions.
Specifically, before feeding to $\mathrm{Transformer_{1}}$ in step 4 of eqn.
7,

  1. 1.

if $x_{i}$ is not an entity or time mention,
$X^{3}[i]=\mathrm{BERT}(X^{1}[i])$

  2. 2.

if $x_{i}$ is an entity or time mention,
$X^{3}[i]=\mathrm{EntityTimeEmbedding}(X^{1}[i])$

The rest of the model remains the same.

###  A.3 Examples

Tables 8 to 12 contain some example questions from the validation set of
CronQuestions, along with the top 5 predictions of the models we experimented
with. T5-3B has a single prediction since it is a text-to-text model.

Question | Who held the position of Prime Minister of Sweden before 2nd World War
---|---
Question Type | Before/After
Gold answer(s) | Per Albin Hansson
BERT | Emil Stang, Sr., Sigurd Ibsen, Johan Nygaardsvold, Laila Freivalds, J. S. Woodsworth
KnowBERT |  | Benito Mussolini, Östen Undén, Hans-Dietrich Genscher, Winston Churchill,
---
Lutz Graf Schwerin von Krosigk
T5-3B | bo osten unden
EmbedKGQA |  Per Albin Hansson, Tage Erlander, Carl Gustaf Ekman, Arvid Lindman, Hjalmar Branting
T-EaE-add | Per Albin Hansson, Manuel Roxas, Arthur Sauvé, Konstantinos Demertzis, Karl Renner
T-EaE-replace |  Per Albin Hansson, Tage Erlander, Arvid Lindman, Valère Bernard, Vladko Maček
CronKGQA | Per Albin Hansson, Tage Erlander, Arvid Lindman, Carl Gustaf Ekman, Hjalmar Branting

Table 8: Before/After reasoning type question. Question | When did Man on Wire receive Oscar for Best Documentary Feature
---|---
Question Type | Simple time
Gold answer(s) | 2008
BERT | 1995, 1993, 1999, 1991, 1987
KnowBERT | 1993, 1996, 1994, 2006, 1995
T5-3B | 1997
EmbedKGQA | 2017, 2008, 2016, 2013, 2004
T-EaE-add | 2008, 2009, 2005, 1999, 2007
T-EaE-replace | 2009, 2008, 2005, 2006, 2007
CronKGQA | 2008, 2007, 2009, 2002, 1945
Table 9: Simple reasoning question with time answer.

Question | Who did John Alan Lasseter work with while employed at Pixar
---|---
Question Type | Time join
Gold answer(s) | Floyd Norman
BERT |  | Tim Cook, Eleanor Winsor Leach, David R. Williams, Robert M. Boynton,
---
Jules Steeg
KnowBERT | 1994, 1997, Walt Disney Animation Studios, Christiane Kubrick, 1989
T5-3B | john alan lasseter
EmbedKGQA | John Lasseter, Floyd Norman, Duncan Marjoribanks, Glen Keane, Theodore Ty
T-EaE-add |  | John Lasseter, Anne Marie Bardwell, Will Finn, Floyd Norman,
---
Rejean Bourdages
T-EaE-replace | John Lasseter, Will Finn, Floyd Norman, Nik Ranieri, Ken Duncan
CronKGQA |  | John Lasseter, Floyd Norman, Duncan Marjoribanks, David Pruiksma,
---
Theodore Ty

Table 10: Time join type question.

Question | Where did John Hubley work before working for Industrial Films
---|---
Question Type | Before/After
Gold answer(s) | The Walt Disney Studios
BERT | The Walt Disney Studios, Warner Bros. Cartoons, Pixar, Microsoft, United States Navy
KnowBERT |  | École Polytechnique, Pitié-Salpêtrière Hospital, The Walt Disney Studios,
---
Elisabeth Buddenbrook, Yale University
T5-3B | london film school
EmbedKGQA |  |  The Walt Disney Studios, Collège de France, Warner Bros. Cartoons,
---
University of Naples Federico II, ETH Zurich
T-EaE-add |  |  The Walt Disney Studios, Fleischer Studios, UPA, Walter Lantz Productions,
---
Wellesley College
T-EaE-replace |  |  The Walt Disney Studios, City College of New York, UPA,
---
Yale University, Indiana University
CronKGQA |  |  The Walt Disney Studios, UPA, Saint Petersburg State University,
---
Warner Bros. Cartoons, Collège de France

Table 11: Before/After reasoning type question.

Question | The last person that Naomi Foner Gyllenhaal was married to was
---|---
Question Type | First/Last
Gold answer(s) | Stephen Gyllenhaal
BERT | 1928, Jennifer Lash, Stephen Mallory, Martin Landau, Bayerische Verfassungsmedaille in Gold
KnowBERT | Nadia Benois, Eugenia Zukerman, Germany national football team, Talulah Riley, Lola Landau
T5-3B | gyllenhaal
EmbedKGQA |  |  Stephen Gyllenhaal, Naomi Foner Gyllenhaal, Wolfhard von Boeselager,
---
Heinrich Schweiger, Bruce Paltrow
T-EaE-add | Stephen Gyllenhaal, Marianne Zoff, Cotter Smith, Douglas Wilder, Gerd Vespermann
T-EaE-replace |  |  Stephen Gyllenhaal, Hetty Broedelet-Henkes, Naomi Foner Gyllenhaal,
---
Miles Copeland, Jr., member of the Chamber of Representatives of Colombia
CronKGQA |  |  Stephen Gyllenhaal, Antonia Fraser, Bruce Paltrow,
---
Naomi Foner Gyllenhaal, Wolfhard von Boeselager

Table 12: First/Last reasoning type question.


# Transparency and Privacy: The Role of Explainable AI and Federated Learning
in Financial Fraud Detection

Tomisin Awosika, Raj Mani Shukla, and Bernardi Pranggono
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>School of Computing and Information Science,
Anglia Ruskin University, UK

###### Abstract

Fraudulent transactions and how to detect them remain a significant problem
for financial institutions around the world. The need for advanced fraud
detection systems to safeguard assets and maintain customer trust is paramount
for financial institutions, but some factors make the development of effective
and efficient fraud detection systems a challenge. One of such factors is the
fact that fraudulent transactions are rare and that many transaction datasets
are imbalanced; that is, there are fewer significant samples of fraudulent
transactions than legitimate ones. This data imbalance can affect the
performance or reliability of the fraud detection model. Moreover, due to the
data privacy laws that all financial institutions are subject to follow,
sharing customer data to facilitate a higher-performing centralized model is
impossible. Furthermore, the fraud detection technique should be transparent
so that it does not affect the user experience. Hence, this research
introduces a novel approach using Federated Learning (FL) and Explainable AI
(XAI) to address these challenges. FL enables financial institutions to
collaboratively train a model to detect fraudulent transactions without
directly sharing customer data, thereby preserving data privacy and
confidentiality. Meanwhile, the integration of XAI ensures that the
predictions made by the model can be understood and interpreted by human
experts, adding a layer of transparency and trust to the system. Experimental
results, based on realistic transaction datasets, reveal that the FL-based
fraud detection system consistently demonstrates high performance metrics.
This study grounds FL’s potential as an effective and privacy-preserving tool
in the fight against fraud.

###### Index Terms:

Fraud Detection, Explainable AI, Federated Learning, Machine Learning

##  I Introduction

In the era of digital banking, ensuring the security and integrity of
financial activities has become paramount. While this transformation offers
unparalleled convenience, it also exposes users to the vulnerabilities of
cyber threats, a significant one being bank account fraud. Financial frauds,
particularly in online banking and credit card transactions, pose serious
threats to the global economy, the trustworthiness of financial institutions,
and the financial well-being of individuals. According to the UK Finance 2022
report, billions are lost annually due to fraudulent activities, highlighting
the need for more robust detection mechanisms [1].

With this, financial institutions have conducted and continue to undertake
rigorous research to combat and identify fraud, irrespective of its nature.
Nevertheless, fraud remains complicated due to its ever-evolving tactics and
diverse behaviors. A prevalent domain that is the subject of extensive
research is bank-related fraud [2], which translates into bank account fraud
on which this research is based.

Bank account fraud differs from other financial deceptions in its methods,
impacts, and detection challenges. Unlike credit card fraud, where
unauthorized transactions can be quickly detected due to unusual spending
patterns, bank account fraud can manifest itself in subtler ways, such as
unauthorized funds transfers, account takeovers, or even identity theft
leading to the creation of new accounts [3]. The consequences for the victim
can be long-lasting, both financially and emotionally. Understanding and
mitigating these threats requires thorough research, underpinned by rich and
diverse datasets.

In the quest to develop systems that can detect bank account fraud, Machine
Learning (ML) is frequently adopted because it effectively trains systems to
deliver precise predictions based on data inputs. The choice of a specific
machine learning algorithm is contingent upon the nature of the data and the
specific type of fraud that the target is trying to identify. Data sets of
bank account transactions not only hold confidential data but also display an
imbalance, with fraudulent transactions less frequent than legitimate ones.
Such characteristics present obstacles in devising a robust fraud detection
system. Banks employ their proprietary data to train different ML models to
recognize potentially fraudulent activities, reflecting a centralized ML
methodology. This centralized approach is predominant in the financial sector
today, credited largely to its proficiency in processing vast data volumes and
discerning underlying patterns [4]. However, one challenge with the
centralized model is that different banks often face diverse fraudulent
patterns, which could hinder their ability to spot new fraudulent behaviors.
This is where Federated Learning comes into the picture.

Federated Learning (FL) is a novel privacy-preserving approach to
decentralized machine learning, [5], [6]. It presents a potential solution to
this predicament by enabling model training on local devices and only sharing
aggregated updates. The core difference between the centralized method is in
the realm of collaboration and data protection. While the former remains
confined within the walls of one bank, the latter is a collective effort that
spans multiple banks.

In the context of fraud detection, FL stands out as more than just a novel
technological approach; it is identified as the indicator of a collaborative
and confidential countermeasure against fraudulent schemes. This growing
significance arises from its capacity to combine insights from different
institutions without the need for direct data exchange. Moreover, FL focuses
on sharing model updates rather than heavy data, which is both faster and more
efficient and ensures that customer data is not compromised.

Additionally, the AI-based fraud detection techniques are black-box in nature
and not transparent. For critical applications, like bank fraud detection, it
is imperative that the AI system is accurate as well as trustworthy. To
address the problem, we integrate Explainable AI (XAI) methods in the given
FL-based banking fraud detection. In this regard, the proposed method not only
preserves user privacy and provides a collaborative infrastructure to train AI
models, but is also trustworthy. Thus, in this research, a fraud detection
technique is proposed that uses the combined strengths of FL and XAI is
proposed. The standout benefits of this methodology are numerous, with a
central emphasis on user privacy preservation and transparency. The following
is a summary of the contributions of this study.

  1. 1.

Incorporate an FL approach for an advanced fraud detection system to ensure
that individual data remains localized allowing only model updates to be
centralized thus enhancing the privacy of banking datasets. This design
inherently bolsters the privacy of banking datasets, a crucial advantage in
our data-sensitive age.

  2. 2.

Develop a Deep Neural Network (DNN) model tailored to recognize patterns
associated with fraudulent activities across federated databases, ensuring
high accuracy.

  3. 3.

Integration of the XAI technique to provide transparency in the model’s
decisions, ensuring a novel approach to fraud detection systems.

  4. 4.

Integrating the proposed FL-based system into a web-based application to
visualize the practicability of the proposed approach.

The following are the remaining sections of this paper. Section II covers the
relevant literature review and related work. In Section III, we present our
methodology. We explain our detailed implementation in Section IV. In Section
V, we discuss our results. The research is concluded in section VI.

##  II Related Work

Fraud detection, a very old challenge, has seen immense evolution over the
years in response to the development of technology and the schemes and
strategies employed by fraudsters [7]. While fraud has been a disturbing
menace from ancient times, the advancement of new technology has amplified the
avenues for fraudulent behavior. Technological advancements, such as
communication platforms and digital finance tools, that are meant to benefit
us can unintentionally also give an advantage to malicious individuals whose
main goal is to cause harm. This has resulted in the rise of new types of
fraud, such as mobile telecommunications scams and computer breaches.

Understanding the nature and underpinnings of fraudulent behavior has been a
subject of significant scholarly attention [8]. A foundational approach to
this study has been the ‘fraud triangle’, a conceptual model that demystifies
the factors that lead an individual to commit fraud. The authors emphasize
three key elements: the incentive or pressure driving the fraudulent act, the
presence of an opportunity, and the rationalization or justification by the
perpetrator. Expanding on this classic model, [9] added depth by introducing
three more elements: the actual act of fraud, the methods employed to conceal
it, and the subsequent ‘conversion’, where the fraudsters benefit from their
deceptive actions.

As financial institutions grapple with a staggering volume of fraudulent
transactions, innovative solutions rooted in machine learning and deep
learning have emerged at the forefront to identify and mitigate these risks.
At its core, machine learning is a subset of artificial intelligence that
blends computer algorithms with statistical modeling [10]. This synthesis
allows computers to perform tasks without explicit programming. Instead, the
system learns from the training data and uses the experiential knowledge
stored to make predictions or take actions. Within machine learning’s orbit is
deep learning, which harnesses artificial neural networks to decipher more
intricate relationships in data. The depth and intricacy of these networks,
like Convolutional Neural Networks (CNN) or Restricted Boltzmann Machines
(RBM), enable them to capture unique relationships across large datasets.

A variety of machine learning and deep learning methods have been explored in
the academic sphere for fraud detection. For instance, a study by [11] delved
into the efficacy of k-Nearest Neighbors (KNN), Support Vector Machines (SVM),
and ensemble classifiers in detecting fraud. Their research emphasized the
challenges, like the highly unbalanced data where fraudulent transactions are
less than legitimate ones, and the dynamic nature of fraud, which necessitates
regularly updated machine learning algorithms. Meanwhile, other researchers
like [12] explored the application of machine learning algorithms like Random
Forest and ensemble models like AdaBoost. Sharma et al. in [13] discussed the
application of Auto-Encoders in the fraud detection framework. Auto-Encoders
are specialized neural networks designed for data encoding. They operate by
compressing input data into a compact representation and subsequently
reconstructing it. Any significant reconstruction error, especially in a model
trained on legitimate transactions, can flag potential anomalies. Parallelly,
the Restricted Boltzmann Machine (RBM) can learn a probability distribution
over its set of inputs. The RBM’s ability to detect intricate patterns in
unlabeled data makes it apt for identifying unauthorized transactions in vast,
imbalanced datasets where fraudulent activities are but a minuscule fraction
[14].

Historically, many ML-based approaches to fraud detection have been
centralized. In a centralized learning system, individual assets or clients
transmit their data to a central hub or server, where data management and
training occur [15]. However, this centralized approach presents challenges,
especially in industries. The transfer of confidential data poses risks
related to latency, data security, and privacy issues. In many cases, data
owners might be unwilling or legally restricted from sharing sensitive
information, complicating the process.

In response to these challenges, the focus has shifted to decentralized
learning approaches. FL stands out as a promising solution [16]. Originally
developed for cellular phones, FL offers a mechanism in which the end user’s
data remains on their device, ensuring data privacy. In this approach, instead
of sending raw data, only model updates or changes are sent to the central
server, which aggregates these updates to refine the global model [16].
Notably, FL is adaptable and can handle both homogeneous independent and
identically distributed (IID) and heterogeneous nonindependent identically
distributed (non-IID) data. This flexibility is especially beneficial when
assets have different failure modes or operate under diverse conditions.

However, the choice between centralized and decentralized models does not only
depend on data security and privacy concerns. The effectiveness of the chosen
machine learning model also plays a pivotal role in fraud detection. Although
recurrent neural networks (RNNs), especially long-short-term memory networks
(LSTMs), have been popular for such predictions, they have inherent
limitations. For instance, LSTMs, due to their sequential nature, tend to
forget long sequences from earlier time-steps [17].

The choice between centralized and decentralized machine learning systems in
finance and asset management is multifaceted. While the centralized approach
offers simplicity and direct control, it poses significant challenges related
to data transfer, privacy, and security. On the other hand, decentralized
systems, especially those based on FL, provide a more private and potentially
efficient approach. As industries continue to evolve and prioritize
operational reliability and safety, the balance between these two systems will
likely shift, with more innovations in decentralized learning approaches
leading the way.

FL presents a potential solution to this predicament by enabling model
training on local devices and only sharing aggregated updates. FL’s core idea
is to train machine learning models across multiple decentralized devices or
servers that hold local data samples, without the need to exchange the data
itself. The central server aggregates these updates to form a global model,
which is then sent back to each device. This global model is refined with more
rounds of localized training and aggregation, leading to a model that benefits
from all available data without directly accessing it [18]. A study by [19]
details the growing concern about data privacy that creates obstacles for
banks to share data. Concurrently, most fraud detection systems are developed
internally, keeping model details confidential to maintain data security.

To address the problem, in this paper, we propose a federated learning-based
architecture for banking fraud detection. Furthermore, we integrate the XAI
techniques into the proposed FL-based method to impart the additional
advantages of transparency and trust to create a more robust financial system.
In addition to developing a theoretical infrastructure for the FL platform, we
also deploy it using a web-based framework.

##  III Methodology

In this section, we explain the proposed system architecture and the
methodology adopted in this research.

###  III-A System architecture

Figure 1: Proposed Federated Learning Architecture

The proposed architecture diagram of the system shown in Fig. 1 outlines the
structure and flow of the FL approach for detecting bank fraud. Central to
this architecture is a server that orchestrates the coordination of model
training and aggregation across numerous banks and financial institutions.
Each institution functions as a distinct node, housing its own local DNN model
trained on proprietary data, ensuring that data never leaves its premises,
thereby bolstering data privacy. Through periodic communication, these local
models transmit their insights, not the data itself, to refine and improve a
global model. This architecture not only capitalizes on the collective
intelligence of all participating entities but also respects the imperative
need for data security in the financial domain.

The client and the server both have their individualistic roles and actions
that they perform. The server initializes the global model, often starting
with random weights or from a pre-trained model, and distributes this model to
all participating clients (banks) for local training. Once clients complete
their local training and send back model updates, the server is responsible
for aggregating these updates to refine the global model. This typically
involves averaging weights, but more sophisticated aggregation algorithms can
also be employed. After aggregation, the server may validate the newly updated
global model using a held-out validation set to ensure its performance meets
the required standards. The server oversees the synchronization of model
updates, ensuring that clients are working with the most recent version of the
global model. It also manages any necessary communication between clients,
although direct client-to-client communication is typically minimal in FL.

Clients on the other hand are responsible for training the received global
model on their local dataset. This involves running several training epochs to
refine the model based on their specific data. After local training, clients
send the model updates (e.g., weight changes) back to the central server. This
does not involve sending any raw data, thus preserving data privacy. Clients
ensure that raw data never leaves their local environment. All data
preprocessing, cleaning, and training are done in-house, ensuring data
confidentiality and compliance with privacy regulations. After the server
aggregates updates and refines the global model, clients receive this updated
model for subsequent rounds of training. Using the locally trained model,
clients can perform real-time fraud detection on new transactions, leveraging
the insights from the collective intelligence of the FL system without
compromising data security. This approach not only guarantees the consistency
of the globally shared model but also ensures its convergence. Instead of
using generic models, a fine-tuned Deep Neural Network (DNN) model was
specifically designed to recognize patterns associated with fraudulent
activities across federated databases. The architecture of the model
capitalizes on the immense computational strength of DNNs to discern and
pinpoint fraudulent activities more effectively.

Furthermore, the proposed system model combines FL with XAI to bring forth a
novel approach to fraud detection. While FL ensures efficient model training
across various devices without compromising data privacy, XAI offers
transparent and interpretable model decisions. This combination is pivotal,
especially in sectors where understanding the rationale behind a model’s
decision is as critical as the decision itself [20]. With FL, the benefits are
manifold. First and foremost, data privacy and security are significantly
enhanced, as raw data remains at its origin, mitigating the risk of breaches
during transfers to centralized servers — a pivotal safeguard considering the
vulnerability of sensitive financial data to cyber-attacks [21]. Furthermore,
FL capitalizes on efficient data utilization by training models on real-time,
varied data from multiple origins, leading to a more comprehensive and current
fraud detection system [22].

Figure 2: Workflow of the proposed System

###  III-B Proposed Federated Learning-based model

Utilizing FL for fraud detection not only leverages the power of collective
data without compromising individual data privacy but also promotes more
collaborative efforts among institutions to combat fraud in an ever-evolving
landscape. A typical ML model update in a centralized setting, using
Stochastic Gradient Descent (SGD), can be represented as:

$W_{t}+_{1}=W_{t}-\eta\nabla L(W_{t})$

where $W$ represents the model parameters, $\eta$ is the learning rate, and
$\nabla L(W_{t})$ is the gradient of the loss function $L$ with respect to the
model parameters at iteration $t$.

In FL, this update is not done centrally. Instead, each client (device or
server) computes its update, and these are aggregated in some way to update
the global model. The core idea behind the Federated averaging algorithm
proposed by McMahan et al. [16] is to modify the standard SGD by computing
several updates on each client and then averaging these updates on the server.

For a given global model $w$, each client $k$ computes its update from its
local data:

$W^{k}_{t}+_{1}=W_{t}-\eta\nabla L_{k}(W_{t})$

Where $L_{k}$ is the local loss on client $k$. After each client has computed
its local update, the server aggregates these updates to form the global model
update. This aggregation in Federated averaging is typically a weighted sum of
the local updates:

$W_{t}+_{1}=\sum_{k}\frac{\eta_{k}}{\eta}W^{k}_{t}+_{1}$

Where $n_{k}$ is the number of data points on client $k$, and $n$ is the total
number of data points across all clients.

This process repeats for several rounds until convergence. A key advantage is
that only the model updates (and not the raw data) are communicated, which
helps in maintaining data privacy. In essence, Federated averaging offers a
compromise between local and centralized learning, allowing models to benefit
from diverse local data sources without compromising user privacy.

###  III-C Explainable AI integration

In the sphere of finance, interpretability is a necessity. The decisions and
predictions made by models have profound real-world implications, so
understanding these decisions is paramount. XAI has emerged to bridge this gap
between the opaque nature of certain models and the requirement for
transparency. These XAI techniques, when integrated into the FL model, empower
stakeholders with insights, enhancing confidence in the model’s decisions.
Moreover, understanding feature importance aids in model debugging and
refinement, addressing potential pitfalls or biases.

One popular XAI technique is SHAP (SHapley Additive exPlanations) [23].
Originating from game theory, SHAP values provide a unified measure of feature
importance by attributing the difference between the model’s prediction and
the average prediction to each feature. For a given instance and feature, the
SHAP value is the average contribution of that feature to all possible
combinations of features.

The SHAP value for feature $j$ is calculated as:

$\phi_{j}(f)=\sum_{S\subseteq
N\backslash\\{j\\}}\frac{|\mathcal{S}|!(|N|-|\mathcal{S}|-1)!}{|N|!}[f(\mathcal{S}\cup\\{j\\})-f(\mathcal{S})]$

where:

  * •

$\mathcal{N}$ is the set of all features.

  * •

$\mathcal{S}$ is a subset of $\mathcal{N}$ without feature $j$.

  * •

$f(\mathcal{S})$ is the prediction of the model for the input features in set
$\mathcal{S}$.

##  IV Implementation details

This section presents the details of the implementation of the proposed
technique as shown in Fig. 2. The steps involve preliminary checks, data
processing, FL development, and XAI integration techniques.

###  IV-A Dataset

The data set referenced in this paper is sourced from [24], encompassing
realistic data based on a present-day real-world dataset for fraud detection.
The dataset contains 29,042 entries, spread across 32 distinctive features.
This rich dataset incorporates various data types: 17 columns of integer type,
10 columns representing floating-point numbers, and 5 columns containing
categorical or string data.

Figure 3: Characteristics of the dataset Figure 4: Characteristics of the
dataset Figure 5: Imbalanced distribution of the proposed dataset

Some of the dataset features are as follows:

  1. 1.

fraud_bool: This is a binary feature and target variable for predictive
models. It is an indicator of whether the record was fraudulent (1) or not
(0).

  2. 2.

income: Represents the income of the user and is a continuous variable of type
float.

  3. 3.

name_email_similarity: A continuous variable, capturing the similarity score
between the name and email.

  4. 4.

prev_address_months_count, current_address_months-_count: Indicators of the
duration (in months) at the previous and current addresses.

  5. 5.

customer_age: Age of the customer.

  6. 6.

days_since_request: A continuous variable that represents the number of days
since a particular request (maybe a credit request) was made.

  7. 7.

intended_balcon_amount: The amount on the balcony or a credit amount.

  8. 8.

payment_type, employment_status, housing_status, source, device_os: These are
categorical features indicating the method of payment, the employment status
of the customer, their housing situation, where the data came from, and the
operating system of the device used, respectively.

  9. 9.

credit_risk_score: A continuous variable possibly indicating the riskiness of
providing credit to the individual or entity.

  10. 10.

email_is_free: A binary variable indicating whether the email provider is free
(like Gmail, Yahoo) or not.

  11. 11.

phone_home_valid, phone_mobile_valid: Binary or score-based indicators
denoting the validity of home and mobile phone numbers.

  12. 12.

month: This is probably indicating the month when the data was recorded or the
transaction occurred.

The distribution of the dataset’s features is shown in Fig. 3 and Fig. 4.

###  IV-B Data balancing

Since the data was highly imbalance as depicted in Fig. 5, balancing was
performed using the Synthetic Minority Over-sampling Technique (SMOTE) [25].
The SMOTE algorithm is a popular technique to address class imbalance by
generating synthetic samples in the feature space. It is used to ensure that
both classes (majority and minority) have an equal number of samples, thus
addressing any imbalance present in the training data.

###  IV-C Data Pre-Processing

In the conducted analysis, missing data in the dataset was managed using two
distinct strategies. For numerical attributes, the mean value of the
respective column was used to input the missing entries. In contrast, for
categorical attributes, the mode, or most frequently occurring value, was
employed for imputation purposes.

In the data preprocessing stage, outlier removal was undertaken for columns
containing floating-point values. Utilizing the Interquartile Range (IQR)
technique, any data values that were beyond 1.5 × IQR from the first (Q1) or
third quartile (Q3) were identified as outliers and consequently excluded from
the dataset to enhance the data’s robustness [26].

Subsequently, to delve into the relationships and potential dependencies among
the numeric attributes, a correlation matrix was computed, as depicted in Fig.
6. This matrix was then visualized as a heatmap, providing a color-coded
depiction of the pairwise linear relationships among variables. The hues in
this heatmap ranged from shades representing perfect negative correlation to
those indicating perfect positive correlation, allowing for quick
identification of strong correlations or potential multicollinearity
scenarios.

Figure 6: Fraud Dataset Correlation Matrix

###  IV-D Feature Selection

To enhance the model’s ability to learn from the data by presenting it in a
more amenable format, feature engineering was performed:

  1. 1.

Binning: The continuous income column was binned into intervals to create the
binned_income column. Binning is a technique used to convert continuous data
into discrete groups (or bins) [27]. By converting the income into ten
discrete bins and labeling them with integers, the model can potentially
discern patterns or trends more easily across income ranges rather than
individual income values.

  2. 2.

One-hot Encoding: Columns such as employment_status, housing_status,
payment_type, source, and device_os were one-hot encoded. One-hot encoding is
a process by which categorical variables are converted into a format that
could be provided to machine learning algorithms to do a better job in
prediction. For each unique value in the original categorical column, a new
binary (0 or 1) column is created. This transformation is essential for models
that work better with numerical input, such as neural networks.

###  IV-E Deep Learning model

The deep learning (DL) model chosen for this study comprises a three-layer
dense neural network. The initial layer contains 64 neurons and employs the
ReLU activation function, taking its input shape from the validation dataset’s
feature dimension. The subsequent layer, also activated by ReLU, has 32
neurons. The terminal layer, designed for binary classification, consists of a
single neuron activated by a sigmoid function. The model’s weights are
adjusted using the Adam optimization algorithm, and the binary cross-entropy
function evaluates prediction losses.

###  IV-F Training and Validation

In this research, data splitting was carried out after the data had been pre-
processed. The widely accepted method of splitting the data into 80% for
training and 20% for testing ensuring random and unbiased partitioning was
carried out in this research. To incorporate these datasets into FL, the
adjusted training dataset was split into three parts: X_train1, X_train2, and
X_train3.

###  IV-G Performance Metrics

To effectively assess the performance of an ML model, it is crucial to employ
an appropriate metric that reflects the accuracy and reliability of the model.
For binary classification tasks, such as distinguishing between two outcomes,
the confusion matrix is a commonly used, straightforward metric. This matrix
provides four distinct prediction outcomes: (i) True Positive, where a
fraudulent transaction is accurately identified as fraud; (ii) False Positive,
where a legitimate transaction is mistakenly labeled as fraud; (iii) False
Negative, where a fraudulent transaction is incorrectly marked as legitimate;
and (iv) True Negative, where a legitimate transaction is correctly classified
as such. A multi-pronged metric approach was employed to ascertain model
performance, utilizing the validation dataset:

  * •

Accuracy reflects the proportion of correct predictions made.

  * •

Precision quantifies the accuracy of positive predictions.

  * •

Recall highlights the fraction of positives that were rightly classified.

  * •

F1-Score offers a harmonic balance between precision and recall, particularly
crucial when faced with class imbalances.

###  IV-H Explainable AI integration

In terms of feature importance visualization, the SHAP method is employed,
offering an interpretation of feature impact relative to a specified baseline
value [23].

###  IV-I Simulation Setup

Incorporating FL and using modern DL frameworks, the simulation setup provides
a holistic perspective on how individual client-side models can contribute to
the learning of a global model. The FL architecture aims to decentralize model
training across multiple clients without sharing raw data. Post-training,
using SHAP, we elucidate model decisions to make them more interpretable. The
simulation was designed using a suite of software tools to facilitate FL and
subsequent evaluations. Flask was employed to craft a lightweight web
application framework for server-client interactions [28]. The TensorFlow
library facilitated DL operations, particularly constructing, training, and
evaluating the neural network model [29]. Cross-Origin Resource Sharing (CORS)
was managed through the Flask-CORS extension, ensuring seamless AJAX cross-
origin functionality. The server was configured to operate locally, accessible
via port 5000.

##  V Evaluation and Results

In this section, the performance metrics of the federated model were well
evaluated. Additionally, we also showcase the web-based framework developed in
this research. Furthermore, the power of XAI was harnessed to understand the
model’s decisions, identifying key features that play pivotal roles in
detecting fraudulent activities. Through this exploration, the intention is to
validate the effectiveness of this approach and provide insights that could
reshape the landscape of fraud detection strategies.

###  V-A Web-based framework

(a) Federated Learning Dashboard

(b) Federated Learning Dashboard (training process)

(c) Federated Learning Dashboard (performance metrics)

(d) Federated Learning Dashboard (Weights)

Figure 7: The FL dashboard description of different pages

Fig. 7(a) represents the FL setup. The HTML page displays the status of each
client as it trains its local dataset with the global model fetched from the
central server and then updates the server with its model weights (not real
data). The three status updates the clients have are: updating, training, and
idle. The client status becomes idle when it is waiting for the updated global
model to be sent from the server. As the server and client send updates to one
another in real time, the Gradient Updates tab which shows ‘33’ is the number
of times the federated learning model has been trained on the client systems.
For each iteration, the accuracy, precision, recall, and F1-score are
calculated and updated on the page.

Fig. 7(c) depicts a real-time accuracy over updates graph. This graph also
updates itself as the Gradient Update increases and more accuracy is computed.
The accuracy of the FL model as shown above indicates that the model converges
due to the time taken to aggregate updates from all clients. The accuracy
increases exponentially and has a definite accuracy score of 93%. Similarly,
Fig. 7(b) and Fig. 7(d) show the training process and weight updates.

###  V-B Model Performance Metrics

The accuracy of the FL model as shown in Fig. 8(a) indicates that the model
converges due to the time taken to aggregate updates from all clients. The
accuracy increases exponentially and has a definite accuracy score of 93%.

(a) Accuracy of the Federated Learning Model

(b) Precision of the Federated Learning Model

(c) Recall of the Federated Learning Model

(d) F1-score of the Federated Learning Model

Figure 8: Performance meterices

Fig. 8(b) shows that the precision of the FL model increases linearly as the
number of epochs increases until around 40 epochs. After around 40 epochs, the
precision starts to fluctuate and does not increase as much. This may indicate
that the model is reaching its limit of learning from the training data. Fig.
8(c) shows that the recall of the FL model increases with the number of
epochs. This indicates the model is able to learn more about the training data
and improve its ability to identify positive examples over time. Fig. 8(d)
shows that the F1-score of the FL model increases with the number of epochs.
This indicates that the model is able to learn more about the training data
and improve its predictions over time. However, the graph also shows that the
rate of improvement decreases as the number of epochs increases. This is
because the model is eventually able to learn all that it can from the
training data, and any further improvement is minimal.

The research aimed to understand the benefits of using an FL-based approach
for advanced fraud detection systems in terms of data privacy. FL compared to
other centralized ML models showed that real data was not shared, instead,
model weights were sent to a central server to perform aggregations and
produce a global model. This iterative approach formed the base idea and
concept of the federated learning model.

###  V-C Explainable AI Insights

We have used the SHAP plots for the model explainability [23].In the SHAP
plots, the color coding (red and blue) represents the positive and negative
impact of feature values on the prediction of the output, relative to the
baseline value. The baseline value is the average of all model output values
over the dataset and serves as a reference point. In the SHAP plots, the red
indicates that a particular feature value represented with numbers increased
the prediction value. While the blue indicates that a particular feature value
decreased the prediction value. The feature 2.086 on the red side of the two
client plots indicates the feature has a positive impact on the prediction of
the FL model. The magnitude (that is, the distance from the baseline) of the
SHAP values can indicate the strength of the influence of a feature. Higher
magnitudes, whether positive or negative, signify that the feature has a
strong impact on the prediction. For example, the features 2.15, 0.418, and
0.517 in the second SHAP plot (Fig. 9(b)) have more impact on the model
prediction than 1.15 and -0.8.

(a) SHAP Plot of Client 1

(b) SHAP Plot of Client 2

(c) SHAP Plot of Client 3

Figure 9: Explainable AI insights

##  VI Conclusion

In the FL-based approach presented for banking fraud detection, a
decentralized approach to model training was employed, allowing clients to
train on their local data and subsequently share model updates with a central
server. This approach prioritizes data privacy, as raw data remain local while
still benefiting from the insights of diverse datasets. The project utilized
Flask for server-client communication and handled potential scalability
challenges by integrating multi-threading. Client weights, which are
multidimensional arrays representing the learned parameters of the neural
network, played a crucial role in aggregating insights from individual models
to update the global model. By integrating SHAP, the project not only focuses
on achieving accurate model predictions but also sheds light on which features
are most influential in driving these predictions. This can be especially
beneficial in sensitive domains where understanding the rationale behind
predictions is as important as the predictions themselves. In the context of
the project, SHAP offers a pathway to build trust and ensure that the
decisions of the federated model can be understood and justified. Overall, the
project highlights the efficacy and potential of FL in scenarios where
centralized data collection might be impractical or undesirable due to privacy
or security concerns.

## References

  * [1] UKFinance, “Annual fraud report 2022,” https://www.ukfinance.org.uk/policy-and-guidance/reports-and-publications/annual-fraud-report-2022, 2022.
  * [2] A. Abdallah, M. A. Maarof, and A. Zainal, “Fraud detection system: A survey,” _Journal of Network and Computer Applications_ , vol. 68, pp. 90–113, 2016.
  * [3] A. Pascual, K. Marchini, S. Miller, and J. S. . Research., “2017 identity fraud: securing the connected life,” _Javelin (February 1), http://www. javelinstrategy. com/coverage-area/2017-identity-fraud_ , 2017.
  * [4] S. Bhattacharyya, S. Jha, K. Tharakunnel, and J. C. Westland, “Data mining for credit card fraud: A comparative study,” _Decision support systems_ , vol. 50, no. 3, pp. 602–613, 2011.
  * [5] L. T. Rajesh, T. Das, R. M. Shukla, and S. Sengupta, “Give and take: Federated transfer learning for industrial iot network intrusion detection,” 2023.
  * [6] S. Vyas, A. N. Patra, and R. M. Shukla, “Histopathological image classification and vulnerability analysis using federated learning,” 2023.
  * [7] R. J. Bolton and D. J. Hand, “Statistical fraud detection: A review,” _Statistical science_ , vol. 17, no. 3, pp. 235–255, 2002.
  * [8] H. Van Driel, “Financial fraud, scandals, and regulation: A conceptual framework and literature review,” _Business History_ , 2018.
  * [9] G. M. Trompeter, T. D. Carpenter, N. Desai, K. L. Jones, and R. A. Riley, “A synthesis of fraud-related research,” _Auditing: A Journal of Practice & Theory_, vol. 32, no. Supplement 1, pp. 287–321, 2013.
  * [10] P. Raghavan and N. El Gayar, “Fraud detection using machine learning and deep learning,” in _2019 international conference on computational intelligence and knowledge economy (ICCIKE)_. IEEE, 2019, pp. 334–339.
  * [11] M. Zareapoor, P. Shamsolmoali _et al._ , “Application of credit card fraud detection: Based on bagging ensemble classifier,” _Procedia computer science_ , vol. 48, no. 2015, pp. 679–685, 2015.
  * [12] K. Randhawa, C. K. Loo, M. Seera, C. P. Lim, and A. K. Nandi, “Credit card fraud detection using adaboost and majority voting,” _IEEE access_ , vol. 6, pp. 14 277–14 284, 2018.
  * [13] Sharma, M. Abhilash, Raj, B. R. Ganesh, Ramamurthy, B., and Bhaskar, R. Hari, “Credit card fraud detection using deep learning based on auto-encoder,” _ITM Web Conf._ , vol. 50, p. 01001, 2022. [Online]. Available: https://doi.org/10.1051/itmconf/20225001001
  * [14] A. Pumsirirat and Y. Liu, “Credit card fraud detection using deep learning based on auto-encoder and restricted boltzmann machine,” _International Journal of advanced computer science and applications_ , vol. 9, no. 1, 2018.
  * [15] S. Kamei and S. Taghipour, “A comparison study of centralized and decentralized federated learning approaches utilizing the transformer architecture for estimating remaining useful life,” _Reliability Engineering & System Safety_, vol. 233, p. 109130, 2023.
  * [16] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [17] I. Benchaji, S. Douzi, and B. El Ouahidi, “Credit card fraud detection model based on lstm recurrent neural networks,” _Journal of Advances in Information Technology_ , vol. 12, no. 2, 2021.
  * [18] S. Bharati, M. Mondal, P. Podder, and V. Prasath, “Federated learning: Applications, challenges and future directions,” _International Journal of Hybrid Intelligent Systems_ , vol. 18, no. 1-2, pp. 19–35, 2022.
  * [19] W. Yang, Y. Zhang, K. Ye, L. Li, and C.-Z. Xu, “Ffd: A federated learning based method for credit card fraud detection,” in _Big Data–BigData 2019: 8th International Congress, Held as Part of the Services Conference Federation, SCF 2019, San Diego, CA, USA, June 25–30, 2019, Proceedings 8_. Springer, 2019, pp. 18–32.
  * [20] F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” _arXiv preprint arXiv:1702.08608_ , 2017.
  * [21] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, B. McMahan _et al._ , “Towards federated learning at scale: System design,” _Proceedings of machine learning and systems_ , vol. 1, pp. 374–388, 2019.
  * [22] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 10, no. 2, pp. 1–19, 2019.
  * [23] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [24] S. Jesus, J. Pombal, D. Alves, A. Cruz, P. Saleiro, R. P. Ribeiro, J. Gama, and P. Bizarro, “Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation,” _Advances in Neural Information Processing Systems_ , 2022.
  * [25] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” _Journal of artificial intelligence research_ , vol. 16, pp. 321–357, 2002.
  * [26] X. Wan, W. Wang, J. Liu, and T. Tong, “Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range,” _BMC medical research methodology_ , vol. 14, pp. 1–13, 2014.
  * [27] M. Hegland, “Data mining techniques,” _Acta numerica_ , vol. 10, pp. 313–355, 2001.
  * [28] G. Dwyer, _Flask By Example_. Packt Publishing Ltd, 2016.
  * [29] T. Hope, Y. S. Resheff, and I. Lieder, _Learning tensorflow: A guide to building deep learning systems_. ” O’Reilly Media, Inc.”, 2017.