# New Thread Migration Strategies for NUMA Systems

O. G. Lorenzo1, M. L. Becoña1, T. F. Pena1,
J. C. Cabaleiro1, J. A. Lorenzo2 and F. F. Rivera1
1CiTIUS Centro de Investigación en Tecnoloxías da Información
Universidade de Santiago de Compostela, Spain
2Quartz Research Lab - EISTI Pau, France

###### Abstract

Multicore systems present on-board memory hierarchies and communication
networks that influence performance when executing shared memory parallel
codes. Characterising this influence is complex, and understanding the effect
of particular hardware configurations on different codes is of paramount
importance. In previous works, monitoring information extracted from hardware
counters at runtime has been used to characterise the behaviour of each thread
in the parallel code in terms of the number of floating point operations per
second, operational intensity, and latency of memory access. We propose to use
this information to guide thread migration strategies that improve execution
efficiency by increasing locality and affinity. Different configurations of
NAS Parallel OpenMP benchmarks on multicores were used to validate the
benefits of the proposed thread migration strategies. Our proposed strategies
produce up to 70% improvement in scenarios where locality and affinity are
low, there being a small degradation in performance for codes with high
locality and affinity.

Keywords: 3DyRM, Roofline Model, Hardware Counters, Performance, Thread
migration

##  1 Introduction

Current microprocessors implement multicores that feature a diverse set of
compute cores and on board memory hierarchies connected by increasingly
complex communication networks and protocols with area, energy, and
performance implications. For a parallel code to be correctly and efficiently
executed in a multicore system, it must be carefully programmed, and memory
sharing stands out as a sine qua non for general purpose programming [1]. A
critical programming challenge for these systems is to partition application
tasks, mapping them to one of many possible core thread configurations to
achieve a desired performance in terms of throughput, delay, power, and
resource consumption, among others [2]. The number of mapping choices
increases as the number of cores and threads increase.

Considering the architectural features, particularly those that determine the
behaviour of memory access, it is critical to improve locality of access and
affinity among threads, data, and cores. Performance issues that are impacted
by this information are, among others, data locality, thread affinity, and
load balancing, and so addressing these issues is critical to improve
performance in general [3].

Various performance models have been proposed to understand the performance of
a code running on a particular system [4, 5, 6, 7, 8, 9, 10]. In particular,
the roofline model (RM) [11] offers a balance between simplicity and
descriptiveness based on the operational intensity (OI), defined as the number
of operations per byte of DRAM traffic, measured in floating point operations
per second (FLOPS)/Byte (flopsB); and the number of FLOPS, measured in GFLOPS.
The original RM presented drawbacks that have been previously analysed [12,
13, 14]. This, the dynamic roofline model (DyRM) was proposed [12],
essentially the equivalent of splitting the execution of a code in time
slices, getting one RM for each slice, and then combining them in a single
graph that shows the evolution of the code when running. The latency extended
DyRM (3DyRM) [13, 14] extended the DyRM model with an additional parameter,
memory access latency, measured in number of cycles. These works also detailed
how the information provided by Precise Event Based Sampling (PEBS) [15, 16]
on Intel processors was processed to obtain parameters that defined the models
(flopsB, GFLOPS, and latency). Even though these parameters are related, they
incorporate the important factors that influence performance of parallel
shared memory code when executed in a shared memory system, and in particular,
in multicores.

Moving threads close to where their data reside can help alleviate memory
related performance issues, since when threads migrate, the corresponding data
usually stays in the original memory module, and is accessed remotely by the
migrated thread [17]. This could induce inefficiency that, sometimes, cannot
be alleviated by the benefits of the migration [18, 19, 20, 21, 22]. Some
analytical results are available for multicore processor analysis. For
example, [23] performed a mean value analysis of a multithreaded multicore
processor and showed that there is a performance valley to be avoided as the
number of threads increases. Markovian models were used in [24] to model a
cache memory subsystem with multithreading, and other works [25, 2] have
modelled multithreaded multicore using queuing models.

We use the 3DyRM model to implement strategies for migrating threads in shared
memory systems and, in particular, multicores. The concept is to use the
defining parameters of 3DyRM as objective functions to be optimised. Thus, it
is considered as a multiobjective optimisation problem. The proposed technique
is an iterative method inspired from evolutionary optimisation algorithms. To
this end, we define an individual utility function to represent the relative
importance of the 3DyRM parameters. This function is a weighted product that
can be considered as representative of the performance of each parallel
thread, and the parameters characterise the efficiency of each thread. Thus, a
single value is able to quantify the performance of each thread in terms of
locality and affinity.

Section 2 describes the 3DyRM parameters used to characterise the execution of
each thread in the parallel code. We also summarise the use of hardware
counters to extract the information required for the 3DyRM with low overhead.
Section 3 introduces the proposed thread migration strategies. A set of case
studies based on the NAS Parallel OpenMP benchmarks (NPB-OMP) [26] are
described in Section 4, and the outcomes are discussed. Finally, the main
conclusions are summarised in Section 5.

##  2 Parameters to characterise the performance of threads in parallel code

In modern systems the main bottleneck is often the connection between the
processor(s) and memory [27]. 3DyRM relates processor performance to off-chip
memory traffic. Operational intensity (OI) is the operations per byte of DRAM
traffic (measured in flopsB). OI measures traffic between the caches and main
memory rather than between the processor and caches. OI incorporates the DRAM
bandwidth required by a processor in a particular computer, and the cache
hierarchy, since better use of cache memories would mean less use of main
memory. Thus, DyRM brings together floating point performance, operational
intensity, and memory bandwidth. However, OI is insufficient to fully
characterise memory performance, particularly on non-uniform memory access
(NUMA) systems. In a NUMA system, distance and connection to memory cells from
different cores may induce variations in memory latency, and so the same code
may perform differently depending on where it was scheduled, which may not be
detectable in DyRM. Extending DyRM with the mean latency of memory access
provides a better model of performance. Thus, we employ the 3DyRM model, which
provides a three dimensional representation of thread performance on a
particular placement.

PEBS is an advanced sampling feature of Intel Core based processors, where the
processor directly recording samples from specific hardware counters into a
designated memory region. The use of PEBS as a tool to monitor a program
execution and perform thread migrations was implemented by [28], providing
runtime dynamic information about code behaviour with low overhead [29, 15].
Our migration tool constantly gathers performance data in terms of the 3DyRM
parameters, GFLOPS, flopsB, and latency, for each core and thread.

However, the floating point (FP) information from PEBS may sometimes be
inaccurate. FP instructions may be counted more than once when the memory is
used intensively, because they are counted when issued, not when retired, and
if their operands are not available on the L1 cache they may be issued more
than once until they are read from higher memory levels [15]. Additionally,
new models of Intel processors do not allow direct reading of FP operations.
Therefore, information about retired instructions is also recorded, so giga
instructions per second (GIPS) and instructions retired per byte (instB) may
be used rather than GFLOPS and flopsB, respectively, in relevant cases.

##  3 A new thread migration strategy

We introduce a new strategy for guiding thread migration in NUMA systems. The
proposed algorithm performs threads migrations iteratively each $T$
milliseconds. The concept is to consider the 3DyRM parameters as objective
functions to be optimised, so that that increasing GFLOPS (or GIPS) and flopsB
(or instB), and decreasing latency in each thread improves performance in the
parallel code. There is a close relation between this and multiobjective
optimisation (MOO) problems, which have been extensively studied [30]. The aim
of most MOO solutions is to obtain the Pareto optimality numerically. However,
this task is usually computationally intensive, and consequently a number of
heuristic approaches have been proposed.

In our case, there are no functions to be optimised. Rather, we have a set of
values that are continuously measured in the system. Therefore, we propose to
apply MOO methods to address the problem using the 3DyRM parameters. Thread
migration is then used to modify the state of each thread to simultaneously
optimise the parameters. However, GIPS, intsB and latency have values with
different orders of magnitude. For this situation, weighting methods are
recommended to aggregate the parameters [31]. Therefore, we propose to
characterise each thread using an aggregate objective function, $P$, that
combines the three parameters.

Let $P_{ijk}$ be the performance for the $i$-th thread of the $j$-th process
when executed on the $k$-th of $N$ nodes. Then, for each iteration of the
aggregate function,

$P_{ijk}=\frac{\mathrm{GIPS}_{ijk}^{\beta}\cdot\textrm{intsB}_{ijk}^{\gamma}}{\textrm{latency}_{ijk}^{\alpha}}$
(1)

where $\textrm{GIPS}_{ijk}^{\beta}$ is the GIPS of the thread powered by
$\beta$, and $\textrm{instB}_{ijk}^{\gamma}$ and
${\textrm{latency}_{ijk}^{\alpha}}$ are the instB and latency values powered
by $\gamma$ and $\alpha$, respectively. It is clear that larger values of
$P_{ijk}$ imply better performance.

Initially, no values of $P_{ijk}$ are available for any thread on any node. On
each time interval, $P_{ijk}$ is computed for every thread on the system
according to the performance read by the hardware counters. In every interval
some values of $P_{ijk}$ are updated, for those nodes $k$ where each thread
was executed, while others store the performance information of each thread
when it was executed in a different node (if available). If there is a
previous value of $P_{ijk}$, the new value replaces the previously saved one.
Thus, the algorithm adapts to possible behaviour changes for the threads. For
example, in a Xeon server with four nodes, $N=4$, four values of $P$ (one for
each thread) are saved each iteration. As threads migrate and are executed on
different nodes, $P_{ijk}$ are progressively updated.

Every $T$ milliseconds, once the new values of $P_{ijk}$ are computed, the
thread with the worst current performance, in terms of $P_{ijk}$, is selected
to be migrated. To compare threads from different processes, each individual
$P_{ijk}$is divided by the mean $P_{ijk}$ of all threads of the same process
(same $j$), identified by its PID,

$\widehat{P}_{ijk}=\frac{P_{ijk}}{\sum_{m=1}^{n_{j}}P_{mjh}/n_{j}}$ (2)

where $n_{j}$ is the number of threads of process $j$ and $h$ is, for each
thread $m$ of process $j$, the last node where it was executed. Thus, for each
process, those threads with $\widehat{P}_{ijk}<1$ are currently performing
worse than the mean of the threads in the same process, and the worst
performing thread in the system is considered to be the one with the lowest
$\widehat{P}_{ijk}$, i.e., the thread performing worse when compared to the
other threads of its process. This is the migration thread, denoted by
$\Theta_{m}$.

The migration can be to any core in a node other than $\Theta_{m}$ current
node. A weighted random process is employed to choose the destination, based
on the stored performance values. The aim is to consider all possible
migrations, and so all $P_{ijk}$ values are updated and behavioural changes
are incorporated.

In order to consider all possible migrations, all $P_{ijk}$ values are
important. Therefore, one of the aims is to fill as many entries of $P_{ijk}$
as possible. To ensure the migration will improve performance, every possible
destination is granted a number of tickets according to the likelihood of that
migration improving performance, and the destination with the larger
likelihood overall is chosen. Migration may take place to an empty core, where
no other thread is currently being executed, or to a core occupied with other
threads. If there are already threads in the core, one would have to be
exchanged with $\Theta_{m}$. The swap thread is denoted as $\Theta_{g}$, and
all threads are candidates to be $\Theta_{g}$. Note that, although not all
threads may be selected to be $\Theta_{m}$, e.g. a process with a single
thread would always have $\widehat{P}_{ijk}=1$ and so never be selected, they
may still be considered to be $\Theta_{g}$ to ensure we obtain the best
possible performance for the whole system.

The rules applied to distribute tickets ($B$) for the random selection
procedure are:

  * •

Destinations in nodes where $\Theta_{m}$ has previously performed worse than
the current node get $B_{1}$ tickets.

  * •

Destinations in nodes where there is no previous data recorded for
$\Theta_{m}$ get $B_{2}$ tickets.

  * •

Destinations in nodes where $\Theta_{m}$ has previously performed better than
the current node get $B_{3}$ tickets.

The best migration should be that which results in good performance from both
threads, $\Theta_{m}$ and $\Theta_{g}$. Therefore, additional tickets are
awarded to each destination according to the values of $P_{gn}$, where $g$ is
$\Theta_{g}$, and $n$ is the node that currently hosts $\Theta_{m}$:

  * •

Destinations where $\Theta_{g}$ has previously performed worse in $n$ in the
past get $B_{4}$ tickets.

  * •

Destinations with no previous information for $\Theta_{g}$ get $B_{5}$
tickets.

  * •

Destinations where $\Theta_{g}$ has previously performed better get $B_{6}$
tickets.

  * •

Destinations for cores with no threads assigned get $B_{7}$ tickets.

Although $P_{ijk}$ are only saved for nodes, by including the performance of
the possible $\Theta_{g}$, different cores in the same node, and even
different threads in the same core, may get a different number of tickets.

Clearly, suitable choice of $B_{k}$ is critical, and this is discussed further
below.

When all tickets have been assigned, a final destination core is randomly
selected based on the awarded tickets. The interchanging thread, $\Theta_{g}$,
is chosen from those currently being executed on that core, if the core is not
free. Once the threads to be migrated are selected, the migrations are
actually performed.

This algorithm is referred to as the interchange migration algorithm with
performance record (IMAR). To simplify notation, an IMAR and parameters is
denoted as IMAR[$T;\alpha,\beta,\gamma$].

However, migrations may affect not only the involved threads, $\Theta_{m}$ and
$\Theta_{g}$, but all threads in the system due to synchronisation or other
collateral relations among threads. These relations are not accurately
modelled using each thread performance separately. Therefore, we propose the
interchange algorithm with performance record and rollback (IMAR2), where the
total performance for each iteration is calculated as the sum of all $P_{ijk}$
for all threads. Thus, the current total performance, $Pt_{current}$, a single
value, is available to evaluate a thread configuration, independent of the
processes being executed. The total performance of the previous iteration is
stored as $Pt_{last}$.

Incorporating these concepts, decisions are made regarding the next iterations
of the algorithm. The algorithm may dynamically adjust the rate of migrations
by altering $T$ between a given minimum, $T_{min}$, and maximum, $T_{max}$,
doubling or halving the previous value. To do that, a ratio, $0<\omega\leq 1$
is defined for $Pt_{current}/Pt_{last}$, to limit an acceptable decrement in
performance. So, if a thread placement has low total performance, migrations
should be performed to obtain better thread placement, because they are likely
to increase performance ($Pt_{current}\geq\omega Pt_{last}$). This way, $T$ is
decreased to perform migrations more often and reach optimal placement
quicker. On the other hand, if thread placement has high total performance,
migrations have a greater chance of being detrimental. In this case, if
$Pt_{current}<\omega Pt_{last}$, there is no requirement for many migrations,
so $T$ is increased. Additionally, a rollback mechanism is implemented, to
undo migrations if they result in a significant loss of performance, returning
migrated threads to their former locations. If a rollback is performed, no
other migrations are made during that interval.

Summarising, the rules guiding our algorithm are:

  * •

If $Pt_{current}>=\omega Pt_{last}$, i.e., the total performance improves:
Migrations are considered productive, $T$ is halved ($T=>T/2$), and a new
migration is performed according to IMAR.

  * •

If $Pt_{current}<\omega Pt_{last}$, i.e., the total performance decreases more
than a given threshold: Migrations are considered counter-productive, $T$ is
doubled ($T=>2\times T$), and the last migration is rolled back.

The algorithm continues to migrate threads to allow for changes in system
behaviour, and to obtain performance information, rolling these back if
necessary (rollback). To simplify notation, IMAR2 and parameters is denoted as
IMAR2[$T_{min},T_{max};\alpha,\beta,\gamma;\omega$].

A simple example is presented to clarify our proposal. Consider a system with
6 cores in three different nodes, incorporating three processes, each with two
threads. Initially, Process 1 ($PID=j=100$) has 2 threads ($i=1,2$,
$TID=100,101$) executed in node 0 (cores 0 and 1), process 2 ($PID=j=200$) has
2 threads ($i=1,2$, $TID=200,201$) executed in node 1 (cores 2 and 3), and
Process 3 ($PID=j=300$) has 2 threads ($i=1,2$, $TID=300,301$) executed in
node 2 (cores 4 and 5), as shown in Table 1, where threads are shown with the
core they currently reside, and their recorded performance in each node. Nodes
where threads have not been executed previously have no performance
information recorded.

Table 2 shows a later state, where some migrations have been executed and more
performance information is available. The performance of each thread in its
current node is shown in bold. Suppose a migration has to be decided. Table 3
shows each thread’s performance and normalised performance
($\widehat{P}_{ijk}$, $P_{ijk}$ divided by the current mean performance of all
the threads in the same process (same $j$), equation 2). Thread 300 has the
worst relative performance, so $\Theta_{m}$=300.

The case studies, Section 4, show optimal values for $B_{k}$ to be

  * •

$B_{1}=B_{4}=1$: previous low performances are penalised,

  * •

$B_{2}=B_{5}=2$: allow more performance information to be obtained,

  * •

$B_{3}=B_{6}=4$: previous good performances are rewarded, and

  * •

$B_{7}=3$: allow migrations to free cores and improve load balance.

With these values, a thread interchange that would increase the performance of
both threads involved would get eight tickets, the maximum, whereas one that
would worsen the performance of both threads would get only two tickets, the
minimum, and, therefore, have 1/4 the chance of being selected. Migrations and
interchanges where there are no data still have a chance of being selected,
providing (eventually) values for all possible $P_{ijk}$.

Table 1: Example IMAR: Initial state. TID (i,j) | core | $P_{ij0}$ | $P_{ij1}$ | $P_{ij2}$
---|---|---|---|---
100 (0,100) | 0 | 2.4 | – | –
101 (1,100) | 1 | 2.6 | – | –
200 (0,200) | 2 | – | 1.4 | –
201 (1,200) | 3 | – | 1.6 | –
300 (0,300) | 4 | – | – | 6.3
301 (1,300) | 5 | – | – | 5.2
Table 2: Example IMAR: State after a number of iterations. TID (i,j) | core | $P_{ij0}$ | $P_{ij1}$ | $P_{ij2}$
---|---|---|---|---
100 (0,100) | 2 | 2.5 | 1.9 | 2.9
101 (1,100) | 4 | 2.7 | 1.8 | 3.1
200 (0,200) | 0 | 0.9 | 1.4 | –
201 (1,200) | 5 | – | 1.6 | 2.1
300 (0,300) | 1 | 3.3 | – | 6.3
301 (1,300) | 3 | – | 8.1 | 5.7
Table 3: Thread performance for the example of Table II. – | 100 | 101 | 200 | 201 | 300 | 301
---|---|---|---|---|---|---
$P$ | 1.9 | 3.1 | 0.9 | 2.1 | 3.3 | 8.1
$\widehat{P}$ | 0.76 | 1.24 | 0.6 | 1.4 | 0.58 | 1.42

Table 4 shows the distribution of tickets for this example, where destinations
can be considered the same as cores or threads, because there is only one
thread per core and no idle cores. Tickets are awarded according to the past
performance of thread $\Theta_{g}$=300.

  * •

Thread 300 cannot move to core 1 (its current location) or 0 (in the same
node), so both get 0 tickets.

  * •

Cores 2 and 3 get $B_{2}$ tickets, since there is no past information of
thread 300 on node 1.

  * •

Cores 4 and 5 get $B_{3}$ tickets, because performance of thread 300 was
better on node 2 than on the current node.

Tickets are then awarded considering the past performance of the threads that
are currently executing on each particular core, when executed previously on
node 0, the node currently hosting thread 300.

  * •

Core 2 gets $B_{6}$ tickets because thread 100 performed better on node 0.

  * •

Core 3 gets $B_{5}$ tickets because thread 301 has no previous performance
information on node 0.

  * •

Core 4 gets $B_{4}$ tickets because thread 101 performed worse on node 0.

  * •

Core 5 gets $B_{4}$ tickets because thread 201 has no previous performance
information on node 0.

Thus, 21 tickets were awarded in total, and

  * •

Thread 300 has 6/21 chances of migrating to core 2 and being interchanged with
thread 100. This would be favourable to thread 100 and unknown to thread 300.

  * •

Thread 300 has 6/21 chances of moving to core 5 and being interchanged with
thread 201. This would be unknown to thread 201 and favourable to thread 300.

  * •

Thread 300 has 5/21 chances of migrating to core 4 and being interchanged with
thread 101. This would be detrimental to thread 101 and favourable to thread
300.

  * •

Thread 300 has 4/21 chances of going to core 3 and being interchanged with
thread 301. This would be detrimental to thread 301 and unknown to thread 300.

Once all tickets are awarded, $\Theta_{g}$ is chosen in a lottery. The
interchange can be performed when $\Theta_{m}$ and $\Theta_{g}$ are chosen,
migrating both threads to each other cores. Note that this is a small example,
in a real situation with more threads and nodes, the probability differences
among the possible migrations would be larger.

Table 4: Ticket distribution for the example of Table II. TID (i,j) | core | $P_{ij0}$ | $P_{ij1}$ | $P_{ij2}$ | tickets
---|---|---|---|---|---
100 (0,100) | 2 | 2.5 | 1.9 | 2.9 |  $B_{2}$ \+ $B_{6}$ = 2+4
101 (1,100) | 4 | 2.7 | 1.8 | 3.1 |  $B_{3}$ +$B_{4}$ = 4+1
200 (0,200) | 0 | 0.9 | 1.4 | – | 0
201 (1,200) | 5 | – | 1.6 | 2.1 |  $B_{3}$ \+ $B_{5}$= 4+2
300 (0,300) | 1 | 3.3 | – | 6.3 | 0
301 (1,300) | 3 | – | 8.1 | 5.7 |  $B_{2}$ \+ $B_{5}$=2+2

##  4 Experimental results

NPB-OMP benchmarks were used to study the effect of the memory allocation.
These benchmarks are well suited for multicore processors, although they do
not greatly stress the memory of large servers. To simulate the effects of
NUMA memory allocation, different memory stress situations were simulated
using the numactl tool [32], which allows the memory cell to store data can to
be defined and threads to be pinned to specific cores or processors. We
designed an experiment where four instances of the NPB-OMP benchmarks are
executed concurrently in a multiprocessor system, and the placement of each
could be controlled. Each benchmark instance was executed in one multi-
threaded process. The system employed was an Ubuntu 14, with Linux kernel
3.10, NUMA server with four nodes, each of which had one octo-core Xeon
E5-4620 (32 physical cores in total), Sandy Bridge architecture, 16 MB L3
cache, 2.2 GHz-2.6 GHz, and 512 GB of RAM. Node 0 contained cores 0 to 7, node
1 contained cores 8 to 15, node 2 contained cores 16 to 23, and node 3
contained cores 24 to 31. Each benchmark was executed with just enough threads
to fill one node. Thus, each process could have its execution threads pinned
to any node and its data assigned to a selected memory cell. Different memory
placement scenarios could be established by executing as many process as
nodes. We tested the options:

  * •

free test: The benchmarks started execution at the same time, and the OS
controlled memory and thread placement.

  * •

direct test: Each benchmark had its threads fixed to one node and preferred
memory set to the same cell.

  * •

crossed test: Each benchmark had its threads fixed to one processor and
preferred memory set to a different cell. When more than two cells were
considered, there were several possible combinations. The configuration used
in the case study with four cells was:

    * –

threads in node 0 had their data in cell 1,

    * –

threads in node 1 had their data in cell 0,

    * –

threads in node 2 had their data in cell 3, and

    * –

threads in node 3 had their data in cell 2.

  * •

interleave test: Each benchmark had its threads fixed to one node and memory
set to interleave, with each consecutive memory page set to a different memory
cell in a round robin fashion.

Four class C NPB-OMP codes were selected to be shown here: lu.C, sp.C, bt.C
and ua.C. This selection was made according to two main criteria:

  * •

Codes with different memory access patterns and different computing
requirements. The DyRM model was used to select two benchmarks with low flopsB
(lu.C and sp.C) and two with high flopsB (bt.C and ua.C).

  * •

Execution time. Since the execution times of these codes are similar, they
remain in concurrent execution most of the time. This helps studying the
effect of thread migrations.

  * •

They are representative of the benefits obtained by our proposal and other
experiments do not show different behaviours.

All benchmarks were compiled with gcc and O2 optimisation.

###  4.1 Baseline results

The effects of the memory placements in the execution of the NPB-OMP
benchmarks are evaluated, with threads pinned to the the same core for the
duration of their execution, so no migrations take place. These results are
used as a baseline to evaluate IMAR and IMAR2. Each test was executed on the
four nodes, combined as four processes of the same code that produced four
combinations (4 lu.C, 4 sp.C, 4 bt.C, and 4 ua.C), and four processes of
different codes, that produced one combination (lu.C/sp.C/bt.C/ua.C). Every
test was executed five times and the mean execution times are shown in Table
5. The times for all benchmarks of lu.C/sp.C/bt.C/ua.C are shown, whereas, for
considerations of space, only the times of the fastest and slowest instances
are shown for the four equal benchmarks.

Since the NPB-OMP benchmarks perform reasonably well on multicores and they do
not stress the memory, the free test, where the OS placed threads and memory,
performs reasonably well, although inferior to the direct case. sp.C is
inferior to the direct test, but only when executed with other codes in the
lu.C/sp.C/bt.C/ua.C combination. For that case, when benchmarks ua.C and bt.C
finish execution in the free test, the OS is free to place sp.C threads in
other processors to balance the load, which leads to a faster execution. sp.C
is memory intensive, and the OS accelerates its execution by making the
opposite application slower. For the other cases, placing memory and execution
threads in the same node appears to be the best option, interleaving memory
does not produce good results. As expected, by far most inferior case is the
crossed test, where memory and threads are on different nodes.

Thus, the direct case is the best option, although it has some load balancing
issues that can decrease the global performance, and the crossed test is the
worst case, as expected.

Table 5: Baseline times for four NAS benchmarks. Test | Time (s)
---|---
concurrent benchmarks | benchmark | free | direct | interleave | crossed
lu.C/sp.C/bt.C/ua.C | lu.C | 220.24 | 210.00 | 428.41 | 1221.05
sp.C | 235.53 | 267.89 | 557.39 | 1698.36
bt.C | 201.69 | 180.77 | 260.46 | 500.037
ua.C | 197.03 | 190.26 | 316.26 | 759.17
4 lu.C | fastest lu.C | 213.09 | 209.99 | 444.09 | 1265.46
slowest lu.C | 215.84 | 212.20 | 452.15 | 1278.86
4 sp.C | fastest sp.C | 267.80 | 265.29 | 511.15 | 1848.41
slowest sp.C | 287.49 | 267.71 | 763.88 | 1864
4 bt.C | fastest bt.C | 181.27 | 180.74 | 242.52 | 452.47
slowest bt.C | 185.37 | 182.29 | 246.90 | 453.13
4 ua.C | fastest ua.C | 194.51 | 189.36 | 303.76 | 677.31
slowest ua.C | 203.54 | 190.46 | 313.59 | 684.70

###  4.2 Traces

The migration tool can be configured to dump the PEBS trace to a file, which
can be read by a performance visualisation tool, such as in [13]. Thus, the
evolution of the performance of each thread, in terms of $P_{ijk}$ and its
components, through the execution of the benchmarks can be plotted. Figures 1
and 2 show the performance of a thread of the 4 lu.C benchmark in the direct
and crossed configurations, respectively.

Figure 1: Evolution of performance for one thread of the 4 lu.C configuration
for the direct case. The thread runs in node 0. Figure 2: Evolution of
performance for one thread of the 4 lu.C configuration for the crossed case.
The thread runs in node 1.

These figures show the evolution in time of $P_{ijk}$ for a given thread and
each of its performance components, GIPS, instB, and latency, from eq. 1.
Different line colours represent different cores, and a change in colour
represents a migration of the thread. To better visualise the changes, we used
a frame average of 50 measurements, corresponding to measurement every 1.5
seconds. This frame average implies that performance changes between
migrations are not instantly visible, but usually take the form of peaks and
valleys. In Figures 1 and 2, migrations were performed by the OS among cores
in the same node, so performance does not vary greatly during execution. As
expected, performance is lower on the crossed test, with more migrations
involved.

Figure 3: Evolution of performance for one thread of the 4 lu.C configuration
for the crossed test with IMAR migrations.

Figure 3 shows the performance of a thread during the execution of the 4 lu.C
combination in the crossed test employing IMAR migrations. Performance
increases and approaches the direct case due to the IMAR migrations. Using
IMAR, migrations take place between nodes, so they influence the performance
more than in Figure 2. Peaks in the graph of the same colour, are likely due
to migrations of other threads that influence the single-colour thread,
whereas peaks with a colour change are due to migrations of the thread itself.
Note that migrations usually occur after a performance dip, because the thread
was chosen to be among the worst performing by the IMAR algorithm. For
example, the migration after 250 seconds is apparently due to an increase in
memory latency.

Figure 4: Evolution of performance for one thread of the 4 lu.C configuration
for the crossed test with IMAR2 migrations. Figure 5: Evolution of performance
for one thread of the 4 lu.C configuration for the crossed test with IMAR2
migrations.

The performance of two threads during the execution of the 4 lu.C combination
in the crossed test and IMAR2 migrations ($\omega=0.97$) are shown in Figures
4 and 5. The tendency towards increasing performance is clear, because the
rollbacks reduce the number of migrations. There are less pronounced
variations in performance than in the IMAR case, due varying $T$ and rollback
strategies. A dip in performance of thread 109565 close to 150 seconds (Fig.
4) triggers a migration from core 3 back to core 25, a rollback. A migration
from core 13 (in node 1) places the thread in core 6 (in node 0) (Fig. 5), and
subsequently there are rollbacks around 70, 130, and 260 seconds, which
indicate that thread 109553 was placed in an efficient node, and it is
inefficient to move it. The IMAR2 algorithm explores all possible placements
for all threads, and so counter-productive migrations can be performed, but
including rollback allows their effects to be minimised. The algorithm tries
other node placements for the thread, computing the whole performance record
(moving to core 28, node 4, to core 23, node 3, etc.) and checking for
behaviour changes, but always returns the thread to core 6 in node 0.

Figure 6: Evolution of performance for the 4 lu.C configuration for the
crossed and direct cases with IMAR2[1,4;1,1,1;0.90] and IMAR2[1,4;1,1,1;0.97].

An example of migration timing in the 4 lu.C combination for the crossed and
direct tests with IMAR2 migrations is shown in Figure 6. Thresholds
$\omega=0.90$ and $\omega=0.97$ were considered, and the performance record
for the whole system is shown, where a circle represents a migration, a cross
represents a rollback, and triangles mark the execution time of each test.
This graph is from a single execution of each case for each value of $\omega$.

In direct cases, performance remains higher with $\omega=0.97$ through the
executions, due to rollbacks, since migrations are counter-productive in this
case. When performance dips, a rollback is executed (a yellow cross in the
figure) and it recovers.

In the crossed configurations, where migrations are initially productive, when
all threads are in inefficient placements, performance increases faster with
$\omega=0.90$ than with $\omega=0.97$. With $\omega=0.90$, no rollbacks are
performed during the first minute, while rollbacks with $\omega=0.97$ are
counter-productive, since they make the process slower when approaching the
best placements. Nevertheless, once performance is high enough, and more
threads are correctly placed, the $\omega=0.97$ case helps keep performance
high with rollbacks, whereas when $\omega=0.90$, migrations continue even once
a good configuration is obtained.

###  4.3 Results for IMAR

We discuss variations in execution time of our tests compared to the baseline
results of Table 5. Figures 7–10 show the results of one benchmark for all the
tests with one combination. Executions using IMAR with different values of
$T$, $\alpha$, $\beta$, and $\gamma$ are also shown.

All figures in this next sections show the experimental execution times
performing migrations, by IMAR, as a proportion of the baselines times of each
test (free, direct, interleave and crossed), expressed as a percentage. A
percentage greater that 100 means a worse execution time, while a result under
100 shows a better execution time. A special case is shown for the OS, where
the direct, interleave and crossed tests are modified to fix only the memory
placement, letting the OS select thread placement. These tests show whether
the OS was able to place the threads near the memory where their data are
stored. Note that, in many of the figures, the bar for the OS result in the
direct case is far higher than the rest, so it was cropped and the actual
result is shown in a box by the bar.

In this case, the effect of $T$, which determines the number of migrations, is
critical. On most of these tests, the benchmarks use the same code, which
makes comparing their performance fairer and easier. For the
lu.C/sp.C/bt.C/ua.C combination, there is an apparent bias towards
applications with low instB (Figs. 7, 8, 9 and 10), with superior results for
lu.C and sp.C than for bt.C and ua.C. Note that bt.C and ua.C execute faster,
and so must always share the system among four benchmarks, whereas lu.C and
sp.C have more free cores at the end of their execution. This situation
produces superior performance, in part due to frequency scaling capabilities
on Xeon systems, when core frequency increases if not all cores are active.

Figure 7: Normalised execution times for lu.C in the lu.C/sp.C/bt.C/ua.C test
with IMAR. Figure 8: Normalised execution times for sp.C in the
lu.C/sp.C/bt.C/ua.C test with IMAR. Figure 9: Normalised execution times for
bt.C in the lu.C/sp.C/bt.C/ua.C test with IMAR. Figure 10: Normalised
execution times for ua.C in the lu.C/sp.C/bt.C/ua.C test with IMAR.

Changing the scaling factors $\alpha$, $\beta$, and $\gamma$ has a slight
impact on the effect of the migrations. For example, consider the
lu.C/sp.C/bt.C/ua.C combination. For lu.C, Fig. 7, configurations which give
greater importance to memory latency, IMAR[$T;2,2,1$] and IMAR[$T;2,1,2$], for
$T=1$, $T=2$ and $T=4$, are superior in the direct and crossed tests, where
data locality is more important, and inferior in the interleave test, where
memory latency is more balanced in all nodes. Figure 8, corresponding to sp.C,
shows similar outcomes to lu.C, since they are both memory intensive
benchmarks, but with more clear influence of the migrations, since memory
latency is more important. Figures 9 and 10 show less difference among
configurations because latency is not so important in these cases.

###  4.4 Results for IMAR2

To compare IMAR2 with IMAR, the minimum and maximum times for IMAR2 were set
to $T_{min}=1$ and $T_{max}=4$, so migrations would take place at
approximately the same times as in the IMAR study. In general, IMAR2 is
superior to IMAR. For example, for combination 4 lu.C (Figs. 11 and 12), as
$\omega$ increases from $0.90$ to $0.97$, the loss of performance in free and
direct tests is reduced, while in the interleave and crossed cases IMAR2
remains similar to the IMAR algorithm. Figures 13–16 show a closer look at the
tests with only OS migration, compared to IMAR[1;1,1,1], and
IMAR2[1,4,2;1,1,1;0.97]. These are similar to previous figures, but the data
were collated in a different way. Results are shown for each benchmark
instance, for every combination, for one given test. With $\omega=0.97$, most
cases show less than a 10% loss of performance from the baseline free (Fig.
13) and direct (Fig. 14) tests, and the performance increase from baseline
interleave (Fig. 15) and crossed (Fig. 16) tests are similar or superior to
the IMAR case.

Figure 11: Normalised execution times for the fastest lu.C instance in the 4
lu.C test with IMAR2. Figure 12: Normalised execution times for the slowest
lu.C instance in the 4 lu.C test with IMAR2. Figure 13: Normalised execution
times for free configuration with 4 nodes. Figure 14: Normalised execution
times for direct configuration with 4 nodes. Figure 15: Normalised execution
times for interleave configuration with 4 nodes. Figure 16: Normalised
execution times for crossed configuration with 4 nodes.

##  5 Conclusions

Modern multicore systems present complex memory hierarchies, and make load
balancing, data locality and thread affinity important issues to obtain high
performance. In this paper, thread migration algorithms, based on optimisation
of 3DyRM parameters, were used to increase performance. The proposed
techniques improve execution times when thread locality is poor and the OS
cannot improve thread placement during runtime. A multiobjective optimisation
method, weighted product, is proposed to combine the 3DyRM parameters.

Using hardware counters, the performance of each thread in the system could be
obtained in runtime with low overhead, and a tool was implemented to perform
thread migration and allocation during runtime, applying different migration
strategies and algorithms, tuned by a set of factors.

Two proposed migration algorithms were tested in a variety of scenarios. The
IMAR algorithm uses collected information about previous performance for each
thread to guide thread migration decisions. This algorithm was tested on a
server using benchmarks from the NPB-OMP. On complex systems, where NUMA
effects are more pronounced, a poor allocation of threads and data can degrade
performance by a factor of up to 5 or 6. Given a poor distribution of threads
and data, the OS by itself is not able to detect and correct it, which greatly
influences performance. The IMAR algorithm was able to improve execution By up
to 70%. However, small performance losses were obtained in cases where the
thread configuration was initially good.

The IMAR2 algorithm can be considered a refining of the IMAR algorithm. It is
based on the concept of evaluating the effects on the system total performance
of previous migrations and acting accordingly. Specifically, IMAR2 is based on
IMAR, but adds rollback and changes in the period between migrations. This
provides for greater tuning and performs better for those cases where
migrations are unnecessary, while still improving the performance for cases
with low initial performance.

Generally, IMAR2 was superior to IMAR, which was superior to allowing the OS
to self-optimise.

## Acknowledgements

This work was partially supported by the Ministry of Education and Science of
Spain, FEDER funds under contract TIN2016-76373-P, and Xunta de Galicia, GRC
2014/018. It was developed in the framework of the European network HiPEAC,
the Spanish network CAPAP-H6 and Galician networks R2016/045 and R2016/037.

## References

  * [1] A. G. Sodan, Message-passing and shared-data programming models: Wish vs. reality, in: Proc. IEEE Int. Symp. High Performance Computing Systems Applications, 2005, pp. 131–139.
  * [2] M. Ju, H. Jung, H. Che, A performance analysis methodology for multicore, multithreaded processors, IEEE Tran. on Computers 63 (2) (2014) 276–289.
  * [3] O. G. Lorenzo, J. A. Lorenzo, J. C. Cabaleiro, D. B. Heras, M. Suarez, J. C. Pichel, A study of memory access patterns in irregular parallel codes using hardware counter-based tools, in: Int. Conf. on Parallel and Distributed Processing Techniques and Applications (PDPTA), 2011, pp. 920–923.
  * [4] S. Moore, D. Cronk, K. London, J. Dongarra, Review of performance analysis tools for MPI parallel programs, in: Recent Advances in Parallel Virtual Machine and Message Passing Interface, Springer, 2001, pp. 241–248.
  * [5] L. Adhianto, S. Banerjee, M. Fagan, M. Krentel, G. Marin, J. Mellor-Crummey, N. R. Tallent, HPCToolkit: Tools for performance analysis of optimized parallel programs, Concurrency and Computation: Practice and Experience 22 (6) (2010) 685–701.
  * [6] A. Morris, W. Spear, A. D. Malony, S. Shende, Observing performance dynamics using parallel profile snapshots, in: Euro-Par 2008–Parallel Processing, Springer, 2008, pp. 162–171.
  * [7] M. Geimer, F. Wolf, B. J. N. Wylie, E. Ábrahám, D. Becker, B. Mohr, The Scalasca performance toolset architecture, Concurrency and Computation: Practice and Experience 22 (6) (2010) 702–719.
  * [8] A. Cheung, S. Madden, Performance profiling with EndoScope, an acquisitional software monitoring framework, Proc. of the VLDB Endowment 1 (1) (2008) 42–53.
  * [9] B. Mohr, A. D. Malony, H. C. Hoppe, F. Schlimbach, G. Haab, J. Hoeflinger, S. Shah, A performance monitoring interface for OpenMP, in: Procs of the Fourth Workshop on OpenMP (EWOMP 2002), 2002.
  * [10] M. Schulz, B. R. de Supinski, PN MPI tools: A whole lot greater than the sum of their parts, in: Procs of the 2007 ACM/IEEE Conf. on Supercomputing, 2007\.
  * [11] S. Williams, A. Waterman, D. Patterson, Roofline: An insightful visual performance model for multicore architectures, Commun. ACM 52 (4) (2009) 65–76.
  * [12] O. G. Lorenzo, T. F. Pena, J. C. Cabaleiro, J. C. Pichel, F. F. Rivera, DyRM: A dynamic roofline model based on runtime information, in: 2013 International Conference on Computational and Mathematical Methods in Science and Engineering, 2013, pp. 965–967.
  * [13] O. G. Lorenzo, T. F. Pena, J. C. Cabaleiro, J. C. Pichel, F. F. Rivera, 3DyRM: a dynamic roofline model including memory latency information, The Journal of Supercomputing 70 (2) (2014) 696–708.
  * [14] O. G. Lorenzo, T. F. Pena, J. C. Cabaleiro, J. C. Pichel, F. F. Rivera, Study of data locality and thread affinity on multicore systems using the roofline model, in: First Congress on Multicore and GPU Programming, 2014, pp. 67–76.
  * [15] I. D. Zone, Fluctuating FLOP count on Sandy Bridge, http://software.intel.com/en-us/forums/topic/375320, [Online; accessed February 2018] (2014).
  * [16] Intel Corporporation, Intel 64 and IA-32 Architectures Software Developer Manuals, https://software.intel.com/articles/intel-sdm, [Online; accessed February 2018] (2017).
  * [17] G. C. Chasparis, M. Rossbory, Efficient dynamic pinning of parallelized applications by distributed reinforcement learning, International Journal of Parallel Programmingdoi:10.1007/s10766-017-0541-y.
URL https://doi.org/10.1007/s10766-017-0541-y

  * [18] T. Constantinou, Y. Sazeides, P. Michaud, D. Fetis, A. Seznec, Performance implications of single thread migration on a chip multi-core, ACM SIGARCH Computer Architecture News 33 (4) (2005) 80–91.
  * [19] K. S. Shim, M. Lis, O. Khan, S. Devadas, Judicious thread migration when accessing distributed shared caches, in: Proceedings of the Third Workshop on Computer Architecture and Operating System Codesign (CAOS), 2012.
  * [20] F. N. Sibai, Simulation and performance analysis of multi-core thread scheduling and migration algorithms, in: 2010 International Conference on Complex, Intelligent and Software Intensive Systems (CISIS), IEEE, 2010, pp. 895–900.
  * [21] Y. Li, I. Pandis, R. Müller, V. Raman, G. M. Lohman, NUMA-aware algorithms: the case of data shuffling., in: 6th Biennial Conference on Innovative Data Systems Research (CIDR’13), 2013.
  * [22] T. Klug, M. Ott, J. Weidendorfer, C. Trinitis, Autopin–automated optimization of thread-to-core pinning on multicore systems, in: Transactions on high-performance embedded architectures and compilers III, Springer, 2011, pp. 219–235.
  * [23] Z. Guz, E. Bolotin, I. Keidar, A. Mendelson, U. C. Weiser, Manycore vs. manythread machines: Stay from the valley, IEEE Computer Architecture Letter 8 (1) (2009) 25–28.
  * [24] X. E. Chen, T. M. Aamodt, A first-order fine-grained multithreaded throughput model, in: IEEE 15th Int. Symp. High-Perf. Comp. Arch. (HPCA), 2009.
  * [25] V. Bhaskar, A closed queuing model with multiple servers for multithreaded architecture, J. Computer Comm. 31 (2008) 3078–3089.
  * [26] H. Bailey, D, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, et al., The NAS parallel benchmarks, International Journal of High Performance Computing Applications 5 (3) (1991) 63–73.
  * [27] S. A. McKee, Reflections on the memory wall, in: Proceedings of the 1st Conf. on Computing frontiers, ACM, 2004, p. 162.
  * [28] O. G. Lorenzo, T. F. Pena, J. C. Cabaleiro, J. C. Pichel, F. F. Rivera, Multiobjective optimization technique based on monitoring information to increase the performance of thread migration on multicores, in: Cluster Computing (CLUSTER), 2014 IEEE Int. Conf. on, IEEE, 2014, pp. 416–423.
  * [29] S. Akiyama, T. Hirofuchi, Quantitative evaluation of intel pebs overhead for online system-noise analysis, in: Proc. of the 7th Int. Workshop on Runtime and Operating Systems for Supercomputers ROSS 2017, ROSS ’17, ACM, New York, NY, USA, 2017, pp. 3:1–3:8.  doi:10.1145/3095770.3095773.
URL http://doi.acm.org/10.1145/3095770.3095773

  * [30] J. Cho, Y. Wang, R. Chen, K. Chan, A. Swami, A survey on modeling and optimizing multi-objective systems, IEEE Communications Surveys and Tutorials 19 (2017) 1867–1901.
  * [31] S. Cheng, C. W. Chan, G. H. Huang, Using multiple criteria decision analysis for supporting decisions of solid waste management, Environmental Science and Health, Part A: Toxic/Hazardous Substances and Environmental Engineering 37 (5) (2002) 975–990.
  * [32] A. Kleen, A NUMA API for Linux, Novel Inc.


# FarFetched: Entity-centric Reasoning and Claim Validation for the Greek
Language based on Textually Represented Environments

Dimitris Papadopoulos
Technical University of Crete
Chania, Greece
<EMAIL_ADDRESS>
Katerina Metropoulou
National Technical University of Athens
Athens, Greece
<EMAIL_ADDRESS>
Nikolaos Matsatsinis
Technical University of Crete
Chania, Greece
<EMAIL_ADDRESS>
Nikolaos Papadakis
Hellenic Army Academy
Vari, Greece
<EMAIL_ADDRESS>

###### Abstract

Our collective attention span is shortened by the flood of online information.
With FarFetched, we address the need for automated claim validation based on
the aggregated evidence derived from multiple online news sources. We
introduce an entity-centric reasoning framework in which latent connections
between events, actions, or statements are revealed via entity mentions and
represented in a graph database. Using entity linking and semantic similarity,
we offer a way for collecting and combining information from diverse sources
in order to generate evidence relevant to the user’s claim. Then, we leverage
textual entailment recognition to quantitatively determine whether this
assertion is credible, based on the created evidence. Our approach tries to
fill the gap in automated claim validation for less-resourced languages and is
showcased on the Greek language, complemented by the training of relevant
semantic textual similarity (STS) and natural language inference (NLI) models
that are evaluated on translated versions of common benchmarks.

##  1 Introduction

Motivation: The wider diffusion of the Web since the dawn of Web 2.0 has
enabled instantaneous access to an expanding universe of information. The
entire nature of news consumption has shifted dramatically, as individuals
increasingly rely on the Internet as their major source of information. While
people access, filter and blend several websites into intricate patterns of
media consumption, this wealth of information contained in billions of online
articles inevitably creates a poverty of attention and a need to efficiently
allocate this attention among the many sources that may absorb it. Verifying
whether a given claim coheres with the knowledge hidden in the vast amount of
published information is a fundamental problem in NLP, taking into account
that the arrival of new information may weaken or retract the initially
supported inference. The problem is more apparent in less-resourced languages
that lack the necessary linguistic resources for building meaningful NLP
applications.

Figure 1: Claim validation example (translated from Greek) based on aggregated
evidence using FarFetched.

Approach and Contribution: FarFetched is a modular framework that enables
people to verify any kind of textual claim based on the incorporated evidence
from textual news sources. It combines a series of processes to periodically
crawl for news articles and annotate their context with named entities. Given
a user claim, FarFetched derives a relevant subset of the stored content based
on its semantic similarity with the provided claim, thus being able to reason
about its validity in an NLI setting (Figure 1). While the proposed framework
focuses on the less-resourced Greek language, its modular architecture allows
the integration of pretrained models for any language. Moreover, it is capable
of topic-agnostic, evidence-aware assessment of arbitrary textual claims in a
fully automated manner, without relying on feature engineering, curated
sources and manual intervention.

The main contributions of this work are summarized as follows: a) to
formalize, develop and evaluate a claim validation and reasoning approach
based on the aggregated knowledge derived from the continuous monitoring of
news sources, and b) to train, evaluate and share SotA models for the STS and
NLI downstream tasks for the Greek language that support the core
functionalities of our framework.111Code and benchmark datasets:
https://github.com/lighteternal/FarFetched_NLP

##  2 Related Work

Our work comprises functionalities comparable to those of fact checking
frameworks, targeting the assignment of a truth value to a claim made in a
particular context Vlachos and Riedel (2014). For most related approaches
Zhang et al. (2021); Majithia et al. (2019); Zhou et al. (2019); Ciampaglia et
al. (2015); Goasdoué et al. (2013) the evidence to support or refute a claim
is derived from a trustworthy source (e.g. Wikipedia, crowdsourced tagging or
expert annotators). Interesting deviations are DeClarE Popat et al. (2018)
that searches for web articles related to a claim considering their in-between
relevance using an attention mechanism, and ClaimEval Samadi et al. (2016),
based on first-order logic to contextualise prior knowledge from a set of the
highest page-ranked websites.

FarFetched can be distinguished from the aforementioned works by four major
points: a) evidence collection is disentangled from manual annotation but
relies on a constantly updating feed of news articles instead; b) claim
validation based on the accumulated evidence relies on the effective
combination of entity linking and attention-based models; c) our approach
provides interpretable reasoning based on the aggregated evidence of multiple
sources without assessing their truthfulness as opposed to most fact checking
frameworks; and d) the outcome of the process is dynamic as the continuous
integration of new information may lead to a shift in the verdict of the
validated claim.

Recent advances in the field of event-centric NLP have introduced event
representation methods based on narrative event chains Vossen et al. (2015),
knowledge graphs Tang et al. (2019); Vossen et al. (2016), QA pairs Michael et
al. (2018) or event network embeddings Zeng et al. (2021) to capture
connections among events in a global context. Our method relies on an entity-
centric approach instead, where the identified entities are used as connectors
between events, actions, facts, statements or opinions, thus revealing latent
connections between the articles containing them. A few similar approaches
have been proposed for combining world knowledge with event extraction methods
to represent coherent events, but rely either on causal reasoning to generate
plausible predictions Radinsky et al. (2012) or on QA models that require the
accompanying news source to be provided along with the user’s question Jin et
al. (2021).

The latest advances regarding the technological concepts that comprise our
methodology are provided below:

Entity linking (EL) resolves the lexical ambiguity of entity mentions and
determines their meanings in context. Typical EL approaches aim at identifying
named entities in mention spans and linking them to entries of a KG (e.g.
Wikidata, DBpedia) thus resolving their ambiguity. Recent methods combine the
aforementioned tasks using local compatibility and topic similarity features
Delpeuch (2019), pagerank-based wikification Brank et al. (2017a) —used also
in FarFetched— or neural end-to-end models that jointly detect and
disambiguate mentions with the help of context-aware mention embeddings
Kolitsas et al. (2018).

The recent interest for encapsulating diverse semantic sentence features into
fixed-size vectors has resulted in SotA systems for Semantic Textual
Similarity (STS) based on supervised cross-sentence attention Raffel et al.
(2020), Deep Averaging Networks (DAN) Cer et al. (2018) or siamese and triplet
BERT-Networks Reimers and Gurevych (2019) to acquire meaningful sentence
embeddings that can be compared using cosine similarity. The latter approach
is leveraged in our case to train an STS model for the Greek language using
transfer learning.

Finally, the task of Natural Language Inference (NLI) -also known as
Recognizing Textual Entailment (RTE)- associates an input pair of premise and
hypothesis phrases into one of three classes: contradiction, entailment and
neutral. Ferreira and Vlachos, 2016 modeled fact checking as a form of RTE to
predict whether a premise, typically part of a trusted source, is for,
against, or observing a given claim. SotA NLI models typically rely on
Transformer variants with global attention mechanisms Beltagy et al. (2020),
siamese network architectures Reimers and Gurevych (2019) (also used in
FarFetched to train a Greek NLI model), autoregressive language models for
capturing long-term dependencies Yang et al. (2019) and denoising autoencoders
Lewis et al. (2020).

##  3 Methodology

###  3.1 Problem Definition

Given a user claim in free text, we tackle the problem of deciding whether
this statement is plausible based on the currently accumulated knowledge from
news sources. We also acknowledge the problem of constructing relevant
evidence from multiple sources by analysing the information contained in
online articles and the need for efficiently extracting only contextually and
semantically relevant excerpts to verify or refute the user’s claim. While our
work does not primarily focus on better sentence embeddings and natural
language inference techniques, we also target the lack of such models for the
Greek language.

###  3.2 Our approach

FarFetched combines a series of offline (i.e. performed periodically)
operations to accumulate data from various news sources and annotate their
context with named entities. It also encompasses a number of online operations
(i.e. upon user input) to assess the validity of a claim in free text. First,
it identifies the entities included in the provided claim and leverages these
as a starting point to derive a relevant subset of the stored textual
information as candidate evidence. Each candidate is then compared with the
claim in terms of textual similarity, in order to finally conclude on the most
relevant evidence (premise) to reason about the validity of the claim
(hypothesis) in an NLI setting. The distinct modules that comprise the
framework are visualised in Figure 2. The process that FarFetched follows to
evaluate a claim is summarized in Algorithm 1, while each module is described
in greater detail in the following subsections.

Figure 2: The FarFetched modular framework. Algorithm 1 Claim Evaluation

Input: A claim $c$ provided by the user in natural language.
Output: Most relevant evidence $seq^{*}$ (sequence of article excerpts) based
on the input claim $c$ along with its $STS\\_score^{*}$ and
$NLI\\_score<c,e,n>$.

1:Entity Linking: Find the set of entities ${(e_{1},...,e_{n})\in E}$, where
$e\,\exists\,c$ and $\left|E\right|=n$

2:$S\leftarrow\varnothing$

3:Graph database search: Find all shortest paths $p$ between the alternating
entities $e$ and sentences $s$:
${p\leftarrow(e_{1},s_{a},e_{2},...,e_{n-1},s_{k},e_{n})\in P}$

4:if $P=\varnothing$ then

5: ${s\in P\iff s}$ has at least 1 entity mention

6:end if

7:for ${p_{i}\in P}$ do

8: $seq_{i}\leftarrow(s_{a},...,s_{k})$

9: $S\leftarrow S\cup seq_{i}$ (sequence $seq_{i}$ added to candidate evidence
set)

10:end for

11:$STS\\_Scores\leftarrow\varnothing$

12:for $seq_{i}\in S$ do

13: Semantic Textual Similarity: Compare $seq_{i}\in S$ to $c$ (each candidate
evidence sequence to the claim) and calculate $STS\\_score_{i}$

14: $STS\\_Scores\leftarrow STS\\_Scores\cup STS\\_score_{i}$

15:end for

16:Find the candidate $seq^{*}$ with the highest similarity to the claim:
$STS\\_score^{*}\leftarrow max(STS\\_Scores)$
$seq^{*}\leftarrow argmax(STS\\_score^{*})$

17:Natural Language Inference: Compare $seq^{*}$ to $c$ (the best candidate
evidence to the claim) and calculate the scores for contradiction, entailment
and neutrality $NLI\\_score<c,e,n>$

####  3.2.1 News Collection

A multilingual, open-source crawler and extractor for heterogeneous website
structures is leveraged to incorporate information from various news sources
Hamborg et al. (2017). It is capable of extracting the major properties of
news articles (i.e., title, lead paragraph, main content, publication date,
author, etc.), featuring full website extraction and requiring only the root
URL of a news website to crawl it completely.

####  3.2.2 Graph Database Population

The crawled articles are forwarded to a graph database Webber (2012) that
initially stores only two types of nodes: Article, which represents a news
article with its aforementioned properties and Section that represents a
sentence of each article’s main text (i.e. concatenated title and article
body). Each Article node is linked to one or more Section nodes via the
HAS_SECTION relationship.

####  3.2.3 Entity Linking

Given that our approach relies on largely unstructured textual documents that
lack explicit semantic information, Entity Linking (EL) constitutes a central
role in revealing latent connections between seemingly uncorrelated article
sections. To this end, FarFetched employs a type of semantic enrichment and
entity disambiguation technique known as wikification Brank et al. (2017b),
which involves using Wikipedia concepts as a source of semantic annotation. It
applies pagerank-based wikification on input text to identify phrases that
refer to entities of the target knowledge base (Wikipedia) and return their
corresponding WikiData Entity ID. The latter is used as a unique identifier
for storing the entities as Entity nodes to the graph database and for linking
them with the crawled article Section nodes, resulting to a more tightly
connected graph, where article sections are connected to WikiData entities via
the HAS_ENTITY relationship. The virtual graph of Figure 3 represents the
structure (labels and relationships) of the graph database. It should be noted
that an entity node might have an additional label (e.g. Person, City,
Business) except for the generic Entity one, based on the WikiData class
taxonomy.

Figure 3: Final structure of the graph database.

####  3.2.4 Evidence Constructor

In a typical NLI setting, a premise represents our knowledge or evidence
regarding an event and is used to infer whether a relevant hypothesis follows
from it or not. In our case, article sections focusing on the same entities as
the user’s claim could potentially lead to the construction of useful evidence
towards the validation of this claim. We can therefore leverage the entity-
annotated article sections of our graph database to collect relevant evidence
by aggregating information from multiple sources. To this end, we developed an
evidence construction process that comprises the following steps:

  1. 1.

The claim provided by the user passes through the Entity Linking phase and one
or more entities (WikiData concepts) are identified.

  2. 2.

The graph database is queried for all possible shortest paths that contain
article sections between the identified entities. Given the implemented graph
structure and $n$ Entity nodes, this translates to a minimum path length of
$2(n-1)$ alternating Entity-Section nodes as shown in Figure 4. Since the
existence of such path is not guaranteed, in cases that no path is found the
algorithm will select an article section if it contains at least one mentioned
entity.

  3. 3.

The article sections contained in these paths are concatenated to form a set
of candidate evidence sequences. Their relevance with the claim at hand is
assessed during the Semantic Textual Similarity phase.

Figure 4: Shortest path example: The 3 entities (in brown) are connected with
2 article sections (in blue).

####  3.2.5 Semantic Textual Similarity

We train and apply a sentence embeddings method to extract and compare the
vector representations of the user’s claim with each candidate evidence
sequence, in order to select the most semantically relevant candidate for the
final NLI phase. Despite the abundance of multilingual language models (e.g.
m-BERT, XLM) that cover most common languages, pretrained multilingual
sentence embeddings models do not generally perform well in downstream tasks
for less-resourced languages like Greek Koutsikakis et al. (2020).
Furthermore, given that the vector spaces between languages are not aligned,
sentences with the same content in different languages could be mapped to
different locations in the common vector space. To overcome this obstacle, we
trained a Greek sentence embeddings model on parallel EN-EL (English-Greek)
sentence pairs following a multilingual knowledge distillation approach
Reimers and Gurevych (2020a). Our Greek student model (XLM-RoBERTa) was
trained on parallel pairs to produce vectors for the EN-EL sentences that are
close to the teacher’s pretrained English model ones (DistilRoBERTa). Using
the trained model, we are able to compare the produced vector representations
between the claim and each concatenated candidate evidence sequence with
regard to STS in terms of cosine similarity and forward the best candidate to
the last phase of the claim validation process, namely Natural Language
Inference.

####  3.2.6 Natural Language Inference

The last step of our process leverages NLI to determine whether the user claim
(hypothesis) is entailed by, contradicted, or neutral to the most relevant
evidence (premise) of the previous phase. To tackle the aforementioned
multilinguality issues of pretrained language models on less-resourced
languages, we finetuned a Greek sentence-transformers Cross-Encoder Reimers
and Gurevych (2019) (XLM-RoBERTa-base) model for the NLI task. The model was
trained on the Greek and English version of the combined SNLI Bowman et al.
(2015) and MultiNLI Williams et al. (2018) corpora (AllNLI). We used the
English-to-Greek machine translation model by Papadopoulos et al., 2021 to
create the Greek version of the AllNLI dataset. The trained model takes the
premise-hypothesis pair as input and predicts one of the following labels for
each case: "contradiction": c, "entailment": e or "neutral": n. The logits for
each class are then converted to probabilities using the softmax function.
These labels along with their probability scores can be used to assess whether
the claim is verified by the accumulated knowledge of the candidate evidence.

##  4 Experiments

###  4.1 Setup

The technical details for each building block of FarFetched are provided
below:

News Collection and Storage: The  news-please Hamborg et al. (2017) Python
library was used to ingest an initial corpus of news articles to support our
experiments. The root URLs of two popular Greek news sites served as the
starting point in order to recursively crawl news from a diverse topic
spectrum, spanning from 2018 until 2021. We collected 13,236 articles,
containing 31,358 sections in total. A Neo4j graph DBMS was used to store the
crawled articles and sections as nodes and create their in-between
relationships.

Entity Linking: A Python script producing POST requests to the JSI Wikifier
web API Brank et al. (2017a) was implemented to annotate the article sections
and enrich the graph database with WikiData entities. A total of 2,516
WikiData entities of different types (e.g. sovereign states, cities, humans,
organizations, academic institutions etc.) were identified in the crawled
articles. A pageRankSqThreshold of 0.80 was set for pruning the annotations on
the basis of their pagerank score.

Evidence Constructor: We implemented Algorithm 1 as a Python script that
executes a parametrizable Cypher query to construct candidate evidence
sequences; the identified entities in the claim are used as parameters and the
concatenated article sections that link these entities together are returned.
For our experiments, the maximum number of relationships between the
alternating Sections and Entities was set to $2(n-1)$ (shortest path), while
the script returns candidate evidence sequences in descending order based on
path length. These parameters can be modified if longer candidate evidence
sequences are required.

Semantic Similarity: We finetuned a bilingual (Greek-English) XLM-RoBERTa-base
model (~270M parameters with 12-layers, 768-hidden-state, 3072 feed-forward
hidden-state, 8-heads) using 340MB of parallel (EN-EL) sentences from various
sources (e.g. OPUS, Wikimatrix, Tatoeba) leveraging the sentence-transformers
library Reimers and Gurevych (2020b). The model was trained for 4 epochs with
a batch size of 16 on a machine with a single NVIDIA GeForce RTX3080 (10GB of
VRAM) for a total of 28 GPU-hours (single run).

Natural Language Inference: We finetuned a Cross-Encoder XLM-RoBERTa-base
model of the same architecture on the created Greek-English AllNLI dataset
(100MB) using sentence-transformers. The model was trained on the same
hardware setting for a single epoch, using a train batch size of 6 for 22 GPU-
hours (single run).

###  4.2 Main results

In this section we perform a quantitative and qualitative demonstration of
FarFetched’s overall performance and also provide individual results for our
STS and NLI models based on benchmark datasets.

####  4.2.1 End-to-end performance

Given the particularity of FarFetched in evidence collection (data originating
from constantly updating web content), a quantitative evaluation of its
performance is quite challenging. To combat the lack of relevant benchmarks
for the Greek language, we leveraged the FEVER dataset by Thorne et al. 2018,
which models the assessment of truthfulness of written claims as a joint
information retrieval and natural language inference task using evidence from
Wikipedia. Each row of the dataset comprises a claim in free text, a list of
evidence information including a URL to the Wikipedia page of the
corresponding evidence and an annotated label (SUPPORTS, REFUTES, NOT ENOUGH
INFO). We manually translated a subset of 150 claims from the FEVER validation
set from English to Greek and populated the graph database with the content of
the corresponding Wikipedia URLs, which was automatically translated into
Greek (due to its size), using the NMT model by Papadopoulos et al., 2021. We
report FarFetched’s performance in terms of accuracy, precision, recall and
F1-score on Table 1.

Label | Precision | Recall | F1-score
---|---|---|---
NOT ENOUGH INFO | .36 | .80 | .49
REFUTES | .91 | .72 | .80
SUPPORTS | .84 | .70 | .76
Weighted Average | .82 | .73 | .75
Label accuracy (overall) | .73

Table 1: FarFetched claim validation performance on Greek FEVER subset.

The results indicate a balanced precision and recall for the REFUTES and
SUPPORTS classes, while precision is relatively lower for the NOT ENOUGH INFO
case. This can be partially attributed to the challenges of applying
wikification on the automatically translated evidence content, leading to some
claims not being linked to their corresponding evidence. Although the above
results are not directly comparable to those of similar systems tested on the
original English FEVER dataset, they show a significant gain over the baseline
model of Thorne et al. 2018 (label accuracy of 0.49). Based on a large
comparative study conducted by Bekoulis et al. 2021, FarFetched scores in the
upper 30th percentile in terms of accuracy (scores ranging from 0.45 to 0.84);
however, to the best of our knowledge none of these systems covers the Greek
language.

We also provide a set of qualitative examples based on real data that aim at
showcasing the capabilities of our system while also acknowledging the
dynamicity of the evidence collection process. These scenarios are translated
into English to facilitate readability. They include two parts each and are
shown in Tables 4.2.1, 4.2.1 and 4.2.1. The original examples (in Greek) are
available in the Appendix.

In Scenario 1, two contradicting user claims (1a, 1b) with the same entity
mentions are provided by the user (Table 4.2.1). Since they refer to the same
entities, the Evidence Constructor returns the same candidate evidence
sequences for both claims in order to evaluate their validity. The most
relevant one (STS score in bold) is selected for the NLI phase, where the
verdict is that the evidence entails the first claim (1a) and contradicts the
second (1b).

User Claim
(Scenario 1) |  NLI score
---|---
Denmark and Austria believe that the European Union should increase aid to refugees. (1a) |  c: 0.014 e: 0.958 n: 0.028
Denmark disagrees with Austria on the management of immigration issues in the European Union. (1b) |  c: 0.951 e: 0.002 n: 0.047
Candidate Evidence Sequences ($\downarrow$ similarity)
Austria and Denmark also want to increase EU support for countries hosting
refugees near crisis hotspots so that they do not travel to Europe.
• STS Score: 0.8505
Checked by police at the Airport Police Departments … the foreigners presented
forged travel documents … in order to leave the country for other EU countries
like France, Germany, Italy, Austria, the Netherlands, Denmark, Spain and
Norway.
• STS Score: 0.2283 |

Table 2: Demonstration of FarFetched on Scenario 1.

In Scenario 2, we investigate the sensitivity of our approach in exploiting
new information to evaluate a claim (Table 4.2.1). The claim initially
triggers the Evidence Constructor which returns multiple candidate evidence
sequences, in descending STS order (yellow rows). During the NLI evaluation
phase, the verdict is entailment, but with a low probability of 0.571 (2a).
The same hypothesis is evaluated in Scenario 2b, after the addition of new
information appended to the evidence list (blue row). The new evidence is
clearly more relevant to the claim at hand, which is successfully identified
by FarFetched’s STS component that selects it as the best candidate, providing
a more confident entailment score of 0.891 (2b). This shift in NLI verdict is
visualized in Figure 5. Since FarFetched relies on the constantly updating
evidence, monitoring such shifts could be useful for identifying trend
changes, especially for cases that benefit from long-term planning (business,
market, politics etc.)

User Claim
(Scenario 2) |  NLI score initial (2a) |  NLI score updated (2b)
---|---|---
The United States plans to impose sanctions on Iran. |  c: 0.170 e: 0.571 n: 0.259 |  c: 0.012 e: 0.891 n: 0.097
Candidate Evidence Sequences ($\downarrow$ similarity)
Iran faces dilemma over whether to comply of Washington or will lead to
collapse. The sanctions that came back in force today, will force the
government of the Islamic Republic to accept the US claims regarding the
Iranian nuclear program and Iranian activities in the Middle East East
because, otherwise, the regime will be in danger to collapse, claimed Israel
Kats, the Israeli minister responsible for Information Services.
• STS Score: 0.6665
Why Greece was exempted from US sanctions on Iran. New US sanctions on oil
exports from Iran have been in force since November 5.
• STS Score: 0.6324 |  |
"We are always in favor of diplomacy and talks … But the Conversations need
honesty … The US is pushing again sanctions on Iran and withdraw from the
nuclear deal "(of 2015) and then they want to have conversations with us",
Rohani said in a speech that was broadcast live on television.
• STS Score: 0.5151 |  |
NEW: Following the collapse of the last talks between the US and Iran, the
announcement of additional sanctions is expected in the coming days.
• STS Score: 0.7195 |  |

Table 3: Demonstration of FarFetched on Scenario 2.

Figure 5: Shift in NLI verdict from Scenario 2a to Scenario 2b of Table 4.2.1.

Scenario 3 is similar to 2, as one claim is evaluated on an initial set of
candidate evidence sequences (3a) followed by a new relevant article section
with contradicting evidence collected by the Evidence Collector in 3b (Table
4.2.1). However, in this case the new evidence is an excerpt from a person’s
interview. While our approach correctly identifies the relevance of this new
evidence to the claim thus affecting the NLI verdict, it does not distinguish
between opinions and factual evidence. This is discussed in more detail in
Section 5.

User Claim
(Scenario 3) |  NLI score initial (3a) |  NLI score updated (3b)
---|---|---
Apple is trying to compete with Netflix in the production of television content. |  c: 0.004 e: 0.967 n: 0.029 |  c: 0.982 e: 0.008 n: 0.010
Candidate Evidence Sequences ($\downarrow$ similarity)
Apple is expected to spend about $ 2 billion this year creating original
content that it hopes will compete with Netflix, Hulu and Amazon, already
established in the television audience.
• STS Score: 0.7107
NEW: "We’re not trying to compete with Netflix on TV," an Apple spokesman said
in an interview.
• STS Score: 0.7134 |  |

Table 4: Demonstration of FarFetched on Scenario 3.

####  4.2.2 STS performance

The performance of our semantic similarity model was evaluated on the test
subset of the STS2017 benchmark dataset Cer et al. (2017). Given that the
original dataset does not provide sentence pairs in Greek, we manually created
a cross-lingual version for the English-Greek pair. The performance is
measured using Pearson (r) and Spearman ($\rho$) correlation between the
predicted and gold similarity scores (Table 5). We also provide results
regarding translation matching accuracy, evaluating the source and target
language embeddings in terms of cosine similarity. Our model achieves a
slightly better performance in both evaluations compared to the current state-
of-the-art multilingual model by Reimers and Gurevych, 2019.

Model |  STS2017 |  Translation Matching
---|---|---
r |  $\rho$ |  Acc. (en2el) |  Acc. (el2en)
STS-XLM-RoBERTa-base (Ours) |  83.30 |  84.32 |  98.05 |  97.80
Paraphrase-multilingual-mpnet-base-v2 (UKP-TUDA) |  82.71 | 82.70 |  97.50 |  97.35
Table 5:  STS model comparison on EN-EL version of STS2017 and in terms of
translation matching accuracy.

####  4.2.3 NLI performance

We benchmark our trained NLI model on the Greek subset of the XNLI dataset
Conneau et al. (2018) that contains 5,010 premise-hypothesis pairs (Table 6).
Despite not having used the XNLI dataset during the training phase, we achieve
a 1% gain over the multilingual XLM-R Conneau et al. (2020) and are on par
with the monolingual Greek-BERT by Koutsikakis et al., 2020. Since our model
was trained on a mixture of Greek and English sentence pairs, it is more
suitable for corpora that also contain English terms (e.g. technology, science
topics) without suffering from the under-representability of the Greek
language occurring in multilingual models.

Model | F1-score
---|---
NLI-XLM-RoBERTa-base (Ours) | 78.3
Greek-BERT (AUEB) | 78.6 ± 0.62
XLM-RoBERTa-base (Facebook) | 77.3 ± 0.41
M-BERT (Google AI Language) | 73.5 ± 0.49
Table 6:  NLI model comparison in terms of F1-score on the Greek subset of
XNLI-test dataset.

##  5 Limitations

We acknowledge that FarFetched is possible to encounter errors in 3 main
areas; these limitations are briefly addressed below.

Entity Linking: Highly ambiguous entities and name variations pose challenges
to any entity linking method. Since we claim that our approach is entity-
centric, a wrong entity annotation may lead to irrelevant candidate evidence
sequences and increase the probability of "neutral" NLI verdicts. Moreover,
the tunable sensitivity of the integrated wikification module implies a trade-
off between a precision-oriented and a recall-oriented strategy, the latter
resulting in more annotated articles, but also being prone to false-positive
annotations.

Evidence Construction: This initial version of our approach relies solely on
the STS comparison between the evidence and the claim, based on a shortest
path approach as discussed in Section 3.2.4. In cases that involve a larger
number of entities in the user claim, calculating the shortest path between
the alternating Entity-Section nodes can be computationally cumbersome.
Moreover, there is no guarantee that the shortest path is able to capture the
most relevant candidate evidence sequences; to this end, outputting the top
$n$ best candidates is considered, providing a user with an overview of the
extracted news excerpts together with their NLI outcome. Finally, neither a
temporal evaluation of the evidence with regard to the claim nor a distinction
between opinions and facts is considered; all candidates are treated as equal.

Natural Language Inference: Recognizing the entailment between a pair of
sentences partially depends on the tense and aspect of the predications. Tense
plays an important role in determining the temporal location of the
predication (i.e. past, present or future), while the aspectual auxiliaries
signify an event’s internal constituency (e.g. whether an action is completed
or in progress). While the work of Kober et al., 2019 indicates that language
models substantially encode morphosyntactic information regarding tense and
aspect, they are unable to reason based only on these properties. To this end,
claims with a high presence of such semantic properties should be avoided.

##  6 Conclusions

In this work, we presented a novel approach for claim validation and reasoning
based on the accumulated knowledge from the continuous ingestion and
processing of news articles. FarFetched is able to evaluate the validity of
any arbitrary textual claim by automatically retrieving and aggregating
evidence from multiple sources, relying on the pillars of entity linking,
semantic textual similarity and natural language inference.

We showcased the effectiveness of our method on the FEVER benchmark as well as
on diverse scenarios and acknowledged its limitations. As byproducts of our
work, we trained and open-sourced an NLI and an STS model for the less-
resourced Greek language, achieving state-of-the-art performance on the XNLI
and STS2017 benchmarks respectively. While our framework fills the gap in
automated claim validation for Greek, its modular architecture allows it to be
repurposed for any language for which the corresponding models exist.

For future work, we intend to address the limitations of our method mentioned
in Section 5, focusing primarily on an optimal entity linking setting, as well
as on a more robust strategy for constructing relevant candidate evidence
sequences.

## Acknowledgments

The research work of Dimitris Papadopoulos was supported by the Hellenic
Foundation for Research and Innovation (HFRI) under the HFRI PhD Fellowship
grant (Fellowship Number: 50, 2nd call).

## References

  * Bekoulis et al. (2021) Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. 2021.  A review on fact extraction and verification.  _ACM Comput. Surv._ , 55(1).
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brank et al. (2017a) Janez Brank, Gregor Leban, and Marko Grobelnik. 2017a.  Annotating documents with relevant wikipedia concepts.  _Proceedings of SiKDD_.
  * Brank et al. (2017b) Janez Brank, Gregor Leban, and Marko Grobelnik. 2017b.  Annotating documents with relevant wikipedia concepts.  In _Proceedings of the Slovenian Conference on Data Mining and Data Warehouses (SiKDD 2017)_ , pages 218–223.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Ciampaglia et al. (2015) Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015.  Computational fact checking from knowledge networks.  _PloS one_ , 10(6):e0128193.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Delpeuch (2019) Antonin Delpeuch. 2019.  Opentapioca: Lightweight entity linking for wikidata.  _arXiv preprint arXiv:1904.09131_.
  * Ferreira and Vlachos (2016) William Ferreira and Andreas Vlachos. 2016.  Emergent: a novel data-set for stance classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1163–1168, San Diego, California. Association for Computational Linguistics.
  * Goasdoué et al. (2013) François Goasdoué, Konstantinos Karanasos, Yannis Katsis, Julien Leblay, Ioana Manolescu, and Stamatis Zampetakis. 2013.  Fact checking and analyzing the web.  In _Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’13, page 997–1000, New York, NY, USA. Association for Computing Machinery.
  * Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017.  news-please: A generic news crawler and extractor.  In _Proceedings of the 15th International Symposium of Information Science_ , pages 218–223.
  * Jin et al. (2021) Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. 2021.  ForecastQA: A question answering challenge for event forecasting with temporal text data.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4636–4650, Online. Association for Computational Linguistics.
  * Kober et al. (2019) Thomas Kober, Sander Bijl de Vroe, and Mark Steedman. 2019.  Temporal and aspectual entailment.  In _Proceedings of the 13th International Conference on Computational Semantics - Long Papers_ , pages 103–119, Gothenburg, Sweden. Association for Computational Linguistics.
  * Kolitsas et al. (2018) Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018.  End-to-end neural entity linking.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 519–529, Brussels, Belgium. Association for Computational Linguistics.
  * Koutsikakis et al. (2020) John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020.  Greek-bert: The greeks visiting sesame street.  In _11th Hellenic Conference on Artificial Intelligence_ , SETN 2020, page 110–117, New York, NY, USA. Association for Computing Machinery.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Majithia et al. (2019) Sarthak Majithia, Fatma Arslan, Sumeet Lubal, Damian Jimenez, Priyank Arora, Josue Caraballo, and Chengkai Li. 2019.  ClaimPortal: Integrated monitoring, searching, checking, and analytics of factual claims on Twitter.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 153–158, Florence, Italy. Association for Computational Linguistics.
  * Michael et al. (2018) Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018\.  Crowdsourcing question-answer meaning representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 560–568, New Orleans, Louisiana. Association for Computational Linguistics.
  * Papadopoulos et al. (2021) Dimitris Papadopoulos, Nikolaos Papadakis, and Nikolaos Matsatsinis. 2021.  PENELOPIE: Enabling open information extraction for the Greek language through machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 23–29, Online. Association for Computational Linguistics.
  * Popat et al. (2018) Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018.  DeClarE: Debunking fake news and false claims using evidence-aware deep learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 22–32, Brussels, Belgium. Association for Computational Linguistics.
  * Radinsky et al. (2012) Kira Radinsky, Sagie Davidovich, and Shaul Markovitch. 2012.  Learning to predict from textual data.  _Journal of Artificial Intelligence Research_ , 45:641–684.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020a) Nils Reimers and Iryna Gurevych. 2020a.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2020b) Nils Reimers and Iryna Gurevych. 2020b.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Samadi et al. (2016) Mehdi Samadi, Partha Talukdar, Manuela Veloso, and Manuel Blum. 2016.  Claimeval: Integrated and flexible framework for claim evaluation using credibility of sources.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 30(1).
  * Tang et al. (2019) Jizhi Tang, Yansong Feng, and Dongyan Zhao. 2019.  Learning to update knowledge graphs by reading news.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2632–2641, Hong Kong, China. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Vlachos and Riedel (2014) Andreas Vlachos and Sebastian Riedel. 2014.  Fact checking: Task definition and dataset construction.  In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_ , pages 18–22, Baltimore, MD, USA. Association for Computational Linguistics.
  * Vossen et al. (2016) Piek Vossen, Rodrigo Agerri, Itziar Aldabe, Agata Cybulska, Marieke van Erp, Antske Fokkens, Egoitz Laparra, Anne-Lyse Minard, Alessio Palmero Aprosio, German Rigau, et al. 2016.  Newsreader: Using knowledge resources in a cross-lingual reading machine to generate more knowledge from massive streams of news.  _Knowledge-Based Systems_ , 110:60–85.
  * Vossen et al. (2015) Piek Vossen, Tommaso Caselli, and Yiota Kontzopoulou. 2015.  Storylines for structuring massive streams of news.  In _Proceedings of the First Workshop on Computing News Storylines_ , pages 40–49, Beijing, China. Association for Computational Linguistics.
  * Webber (2012) Jim Webber. 2012.  A programmatic introduction to neo4j.  In _Proceedings of the 3rd Annual Conference on Systems, Programming, and Applications: Software for Humanity_ , SPLASH ’12, page 217–218, New York, NY, USA. Association for Computing Machinery.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Zeng et al. (2021) Qi Zeng, Manling Li, Tuan Lai, Heng Ji, Mohit Bansal, and Hanghang Tong. 2021.  GENE: Global event network embedding.  In _Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)_ , pages 42–53, Mexico City, Mexico. Association for Computational Linguistics.
  * Zhang et al. (2021) Zijian Zhang, Koustav Rudra, and Avishek Anand. 2021.  _FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn Correction in the Loop_, page 4823–4827. Association for Computing Machinery, New York, NY, USA.
  * Zhou et al. (2019) Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019.  GEAR: Graph-based evidence aggregating and reasoning for fact verification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 892–901, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Appendix: Original examples (in Greek) of Tables 4.2.1, 4.2.1
and 4.2.1.

User Claim
(Scenario 1) |  NLI score
---|---
. (1α) |  ς: 0.014 ε: 0.958 ν: 0.028
µ  µ µ µ  . (1β) |  ς: 0.951 ε: 0.002 ν: 0.047
῝ανδιδατε Ειδενςε Σεχυενςες ($\downarrow$ ςιμιλαριτψ)
, µ .
• ΣΤΣ Σςορε: 0.8505
µ µ µµ µ … µ , µ, , , , , .
• ΣΤΣ Σςορε: 0.2283 |

Ταβλε 6: Δεμονςτρατιον οφ ΦαρΦετςηεδ ον Σςεναριο 1 ιν Γρεεϰ λανγυαγε.

ϒςερ ῝λαιμ
(Σςεναριο 2) | ΝΛΙ ςςορε ινιτιαλ (2α) | ΝΛΙ ςςορε υπδατεδ (2β)
---|---|---
µ . | ς: 0.170 ε: 0.571 ν: 0.259 | ς: 0.012 ε: 0.891 ν: 0.097
῝ανδιδατε Ειδενςε Σεχυενςες ($\downarrow$ ςιμιλαριτψ)
µ µµ µµ µ, µ µ µµ , , , , µ .
• ΣΤΣ Σςορε: 0.6665
µ . 5 µ .
• ΣΤΣ Σςορε: 0.6324
«µ µ µ … µ µ µ … µ ( 2015) µ µ µ µ», µ µ .
• ΣΤΣ Σςορε: 0.5151
µ µ µ µ µ.
• ΣΤΣ Σςορε: 0.7195

Ταβλε 6: Δεμονςτρατιον οφ ΦαρΦετςηεδ ον Σςεναριο 2 ιν Γρεεϰ λανγυαγε.

ϒςερ ῝λαιμ
(Σςεναριο 3) | ΝΛΙ ςςορε ινιτιαλ (3α) | ΝΛΙ ςςορε υπδατεδ (3β)
---|---|---
Αππλε Νετφλιξ  µ. | ς: 0.004 ε: 0.967 ν: 0.029 | ς: 0.982 ε: 0.008 ν: 0.010
῝ανδιδατε Ειδενςε Σεχυενςες ($\downarrow$ ςιμιλαριτψ)
Αππλε µ 2 µµ µ µ µ µ Νετφλιξ, Ηυλυ Αμαζον.
• ΣΤΣ Σςορε: 0.7107
« µ µ Νετφλιξ », Αππλε .
• ΣΤΣ Σςορε: 0.7134

Ταβλε 6: Δεμονςτρατιον οφ ΦαρΦετςηεδ ον Σςεναριο 3 ιν Γρεεϰ λανγυαγε.