#  tieval: An Evaluation Framework for
Temporal Information Extraction Systems

Hugo Sousa INESC TEC, Portugal  University of Porto, Portugal  Alípio Jorge
INESC TEC, Portugal  University of Porto, Portugal  Ricardo Campos INESC TEC,
Portugal  Polytechnic Institute of Tomar, Portugal  Ci2 - Smart Cities
Research Center, Portugal

###### Abstract

Temporal information extraction (TIE) has attracted a great deal of interest
over the last two decades, leading to the development of a significant number
of datasets. Despite its benefits, having access to a large volume of corpora
makes it difficult when it comes to benchmark TIE systems. On the one hand,
different datasets have different annotation schemes, thus hindering the
comparison between competitors across different corpora. On the other hand,
the fact that each corpus is commonly disseminated in a different format
requires a considerable engineering effort for a researcher/practitioner to
develop parsers for all of them. This constraint forces researchers to select
a limited amount of datasets to evaluate their systems which consequently
limits the comparability of the systems. Yet another obstacle that hinders the
comparability of the TIE systems is the evaluation metric employed. While most
research works adopt traditional metrics such as precision, recall, and
$F_{1}$, a few others prefer temporal awareness – a metric tailored to be more
comprehensive on the evaluation of temporal systems. Although the reason for
the absence of temporal awareness in the evaluation of most systems is not
clear, one of the factors that certainly weights this decision is the
necessity to implement the temporal closure algorithm in order to compute
temporal awareness, which is not straightforward to implement neither is
currently easily available. All in all, these problems have limited the fair
comparison between approaches and consequently, the development of temporal
extraction systems. To mitigate these problems, we have developed tieval, a
Python library that provides a concise interface for importing different
corpora and is equipped with domain-specific operations that facilitate system
evaluation. In this paper, we present the first public release of tieval and
highlight its most relevant features. The library is available as open source,
under MIT License, at PyPI111https://pypi.org/project/tieval/ and
GitHub222https://github.com/LIAAD/tieval.

Figure 1: tieval logo.

##  1 Introduction

Understanding the temporal order of events is essential to human
communication. We, humans, can easily understand the relative order of events
in a conversation or when reading a news article. However, many challenges are
raised when we try to automate such tasks with a computer program. The first
difficulty that emerges is how to represent temporal information. Since in
most cases we do not explicitly specify the start and end time of each event,
temporal information, such as order and time span, ends up being inferred from
the events themselves. To this regard, computer algorithms can make use of
temporal clues in the text, and of external sources, such as knowledge-bases,
to anchor events on a timeline. For instance, in the sentence “We went to
dinner after the game.”, two events, “dinner” and “game”, can be automatically
identified and used, despite the lack of explicit temporal information, to
recreate a timeline of events (see Figure 2) supported on the word “after”.
The ordering of events and the knowledge about them, can be further expanded
if used together with appropriate external sources. For instance, the event
“game” can be contextualized and anchored on the timeline by searching for
information on a knowledge-base. However, in the case of the “dinner” event,
it turns out impossible to know the exact time of occurrence unless it is
specified in the text. This shows that representing temporal information is
not a trivial task, since there are several borderline cases for which no
standard approach has been established.

Figure 2: Relative timeline of events that can be inferred from the running
example.

Over the years, and particularly in the last two decades, this problem has
been highly studied, leading to several proposals from the research community
Campos et al. (2014); Leeuwenberg and Moens (2019). Most of the proposals were
in the origin of the emergence of different annotation schemes and the various
corpora that we have today at our disposal Naik et al. (2019); Ning et al.
(2018a); UzZaman et al. (2013). Although these efforts have been essential to
mature temporal information extraction and its subtasks – such as temporal
expression identification or temporal relation classification – they also pose
some problems upon the process of benchmarking different methods. One of the
problems has its roots in the fact that evaluating the methods, often requires
reading multiple corpora, each of which has a different perspective on
temporal representation, ultimately preventing comparability among the
different methods and corpora. This is compounded by the fact that corpora are
stored in a variety of formats (e.g., XML, TimeML, or table ), which requires
a considerable engineering effort to load them all.

Another issue that limits the comparison between systems is the lack of
standardization in the metrics used in the evaluation process. This is a
particular problem of temporal relation extraction – a subtask of TIE, which
deals with the identification and classification of the temporal relations
between entities – where different metrics are often employed during the
evaluation process. While initially systems were evaluated and compared using
standard metrics, such as recall, precision, and F-score Verhagen et al.
(2007, 2010), more recently, metrics such as temporal awareness UzZaman and
Allen (2011) have proven to be more reliable in the evaluation of temporal
relation extraction methods. The reasoning behind this is that, while
traditional metrics focus on the local effectiveness of the model, temporal
awareness better understands the relative order of events by considering the
global temporal structure of the predictions. This is accomplished by taking
into account the temporal relations that can be inferred from the established
ones (a process typically referred to as temporal closure), making this a more
comprehensive metric for evaluating temporal systems. Despite the emergence of
this temporal awareness, many studies still rely solely on traditional metrics
to evaluate their system. We speculate that this is due to the fact that
temporal awareness requires domain-specific operations such as temporal
closure – which are not (yet) readily available in every framework and
therefore require individual implementation by each research group. In
addition, temporal awareness requires the implementation of a strategy to deal
with inconsistent predictions of the system, which is generally not explored
in recent studies.

To mitigate the above issues, we developed tieval, a Python library that
enables the development and evaluation of TIE systems. This framework provides
a simple interface to download and read TIE corpora in various formats. It
currently covers well-known corpus – such as TempEval-3 UzZaman et al. (2013),
TDDiscource Naik et al. (2019), and MeanTime Minard et al. (2016) – however,
it lays the foundations for others to be included by providing base classes
for the construction of the corpus. It also provides domain-specific
operations – such as temporal closure and simple translation of intervals into
point relations – that can be used to develop TIE systems. In addition to
this, it includes an evaluation infrastructure for a comprehensive assessment
of the effectiveness of the different models being evaluated. Because tieval
supports the entire development pipeline of TIE, it can also be used to ensure
reproducibility and fair benchmarking of future research. The main
contributions of tieval are the following:

  1. 1.

it gathers the multiple corpora for the development of TIE systems, making it
easy to access with just a few lines of code;

  2. 2.

it facilitates access to domain-specific operations, such as interval to point
relation and temporal closure, as well as metrics such as temporal awareness;

  3. 3.

it provides a standard framework, thus making it easy for new methods to be
compared against previous ones.

The remaining of the paper is organized as follows: The next section, provides
an overview of recent work in TIE and some of its software. We then proceed to
present the tieval package in section 3. We start with a general introduction
and then go into some of its most relevant features. Section 5 serves to
present our thoughts on what we strive to be next steps in the development of
the framework.

##  2 Related Work

Extracting temporal information from documents written in natural language in
an inter-operable format has long been an interest of the artificial
intelligence community Ling and Weld (2010); Derczynski et al. (2015). Since
the introduction of the Time Markup Language (TimeML) Pustejovsky et al.
(2003a), in 2003, the temporal graph has become the de-facto standard to
represent temporal information. In this graph, the nodes are temporal entities
and the edges are the temporal relation that hold between them. The temporal
entities can take two forms: event expressions, which are defined as
situations that happened (e.g., “went” or “bought”); and temporal expressions
(timex), which can convey temporal information explicitly (e.g., “October 27,
199”) or implicitly (e.g., “a few years ago”) Campos et al. (2017). The
temporal relations are held in the form of temporal links (tlink) that contain
temporal relations between pairs of events (E-E relations), events and time
expressions (E-T relations), and events and document creation time (E-DCT
relations), where DCT is a special timex that stores document creation time.
Overall, these temporal relations can take thirteen types, which is the number
of relations that can exist between two time intervals Allen (1983).

The first corpus that was annotated with this scheme was TimeBank Pustejovsky
et al. (2003b). The release of this corpus, dated from 2003, sparked a wave of
research in the field later on also used on the TempEval shared tasks UzZaman
et al. (2013); Verhagen et al. (2007, 2010). These tasks end up segmenting TIE
into a set of sub-problems that can be conceptually defined as temporal entity
identification, tlink identification, and tlink classification. Although some
works developed systems for more than one of these sub-tasks, most of the
systems are concerned with only one of them. Furthermore, temporal entity
identification systems are traditionally partitioned into subsystems for the
several classes of temporal entities. For example, for the TimeBank corpus,
one system is usually trained to identify events and another to identify
timexs. The tieval architecture follows this natural decomposition of the TIE.

The TimeBank corpus, and more abstractly, the TimeML annotation scheme was
widely studied by the community. Such scrutiny lead to the emergence of
several new corpora. Some used the TimeML annotation scheme to create new
corpora, such as AQUAINT Graff (2002) and the Platinum corpus UzZaman et al.
(2013), while others were concerned in extending the annotation scheme to
other languages. The most remarkable effort on this domain was the TempEval-2
shared task Verhagen et al. (2010) that produced corpora for Chinese Li et al.
(2014), French Bittar et al. (2011), Italian Caselli et al. (2011), and the
Spanish Nieto and Saurí (2012) language. Another noteworthy effort is the
MeanTime corpus Minard et al. (2016) in which the authors annotated 120 news
articles written in English from Wikinews333https://en.wikinews.org/, and
translated the texts into Italian, Spanish, and Dutch. Costa and Branco Costa
and Branco (2012) followed a similar process to construct TimeBankPT,
translating the original TimeBank to Portuguese and adapting the annotations
when needed. Apart from the extensions to other languages, the TimeML
annotation scheme was also extended to other domains. A concrete example is
the case of the clinical domain for which two corpora have been produced, the
i2b2 Sun et al. (2013) and THYME Styler IV et al. (2014)444These corpora are
not available for open access and, as a consequence, we were not able to
include them on the framework.. Further significant contributions were the
proposals that explored ways to mitigate some of the issues found on the
TimeBank annotation effort, such as: sparse annotation – TimeBank-Dense
Cassidy et al. (2014) and TDDiscourse Naik et al. (2019); improve inter-
annotator agreement – MATRES Ning et al. (2018a); and include other sources of
knowledge – TCR Ning et al. (2018b) and RED O’Gorman et al. (2016).

Aside from the TimeML, and related approaches, there have also been other
proposals that were explored by the research community. One of them is
absolute timeline placement, in which the temporal entities are directly
anchored on a timeline by labeling each entity with the time (or time span) of
occurrence. The most remarkable efforts in this direction were produced by
Reimers et al. Reimers et al. (2016) – which produced the EventTime corpus by
annotating the events in TimeBank with a specific day, or span of days – and
Leeuwenberg and Moens Leeuwenberg and Moens (2020) – which annotated 169
clinical records from the i2b2 corpus with the most likely start and end time
of each event along with a lower and upper bound.

This shows that several corpora have been introduced for the TIE task.
However, the fact that they were released in different formats makes it hard
to leverage their power, which is one of the issues mitigated by tieval.

To the best of our knowledge, the only framework that made available TIE
operations – including temporal closure and temporal awareness – is the
Anafora Tools project555https://github.com/bethard/anaforatools which was
built to work with files stored in the Anafora XML format Chen and Styler
(2013), used to annotate the THYME corpus Styler IV et al. (2014). The
framework presented in this paper aims to be a more general tool, unifying all
corpora in a single format.

##  3 tieval

Our vision for tieval was to build a framework that would support and
facilitate the evaluation of TIE systems. With the development of libraries
such as Numpy, TensorFlow, and PyTorch, Python has established itself as the
programming language of choice within the machine learning community. For that
reason, we built tieval in Python. To facilitate the installation we made it
available on Python Package Index (PyPI)666https://pypi.org/project/tieval/.
Thus, the toolkit can be easily installed through pip, as follows:

        $ pip install tieval==0.0.6

In this paper, we will use version 0.0.6, which is the first and the most
recent version of the package. However, the reader is advised to install the
newest release at the time of reading the paper and refer to the project
repository for up-to-date documentation. Furthermore, for users that might be
interested in contributing to tieval, we encourage forking the source
repository and making a pull request.

tieval contains three modules that represent the three cornerstones of any
machine learning project: datasets, models, and evaluation. The datasets
module is responsible for downloading and reading the corpora available for
TIE, the models module is responsible for the construction of the models, and
the evaluation module has methods to make a proper evaluation for each of the
TIE tasks. In the following sections, we will present the inner workings of
the framework with scripts to exemplify the usability of the framework.

###  3.1 Datasets

With tieval, we wanted to mitigate the issues referred above by making it easy
for the user to work with several corpora with a few lines of code. To that
end, we developed an architecture that would unify the different annotations
and storing formats of the corpus. This architecture is composed of several
objects which are depicted in Figure 3.

Figure 3: Objects used to represent a dataset on tieval. The arrow represent a
relation of “Iterable”.

The Dataset object is the final representation of each corpus. It compiles the
set of all the documents in the corpus on the documents attribute which is
segmented into the train and test attributes whenever provided in the original
paper777When no standard train/test split is provided by the authors all the
documents are placed on the train attribute.. Each document is then stored as
an instance of the Document class (see the Document grey box in Figure 3),
which contains all the information necessary for TIE, more specifically:

name

a string that contains the name of the document (e.g. “wsj_0026.tml”);

text

a string with the raw text of the document;

dct

is a Timex that contains the document creation time (e.g.
Timex("12-10-2004"));

entities

is the set of Entities – either a Timex or Event – that are annotated on the
corpus. Each Entity is, at is core, a data class made to store all the info
provided on the annotation. Therefore, it has to be flexible to accommodate
for the different types of information provided in different corpus. For
instances, the GraphEve corpus provides the lemma for each event while
TempEval-2 does not;

tlinks

a set o TLink’s that stores the temporal relations annotated on the document.
Each TLink contains a source and target entity as well as the temporal
relation between them – on the relation attribute.

A special remark needs to be made about the relation attribute of the TLink
object. When initiating a TLink instance one needs to pass the temporal
relation that holds between the two temporal entities (the source and the
target). In most of the corpora this is one of the thirteen temporal relations
Allen (1983) that can hold between two time intervals, however, there are
corpora where the annotators were more flexible on the type of relations.
Examples of this are the TempEval-2 and the MATRES corpus. On TempEval-2 the
annotators were allowed to give more ambiguous relations as “BEFORE-OR-
OVERLAP” and “OVERLAP-OR-AFTER”. In MATRES the annotators were asked to
provide the temporal relation between the start points of the temporal
entities. In order to accommodate the several types of annotations, we build
TemporalRelation object, which handles the relation that was annotated. Inside
this object, every relation is represented in point relations – instead of the
traditional interval relations. Figure 4 shows how to represent the interval
relation “BEFORE” into a point relation. A relative relation is also included
in the figure for illustrative purposes.

Figure 4: Relative timeline of events that can be inferred from the running
example.

Note that the “BEFORE-OR-OVERLAP” relation on TempEval-2 represents an
uncertainty of the annotator between the end time of the source entity and the
start time of the target entity, however, the annotator is certain about the
remaining point relations. Further note that, although we explicitly state
four-point relations in Figure 4, upon the adaptation of the current datasets
into tieval format, three of them are redundant, as the point relation “end A
< start B” completely defines the remaining point relations. Therefore, on
tieval, whenever there is a new dataset to include, the user can provide the
relation in the way that is most appropriate, as shown in Listing 1.

Listing 1: Different ways to pass the temporal relation to the TLink object.
The first argument (X) is the source entity, the second (Y) is the target
entity, and the third is the temporal relation between them. This can be
passed as an interval relation, “before”, or as a point relation, in the form
of a dictionary structure. On the latter, the interpretation for the expected
keys is the following: “x” and “y” stands for the source and target entity,
respectively; while the “s” and “e” stand for “start” and “end”. As an
example, “xe_ys” is the point relation between the source end and the target
start.

⬇

from tieval.links import TLink

tl1 = TLink("X", "Y", "before")

tl2 = TLink("X", "Y", {"xe_ys": "<"})

tl3 = TLink("X", "Y", {"xs_ys": "<", "xs_ye": "<", "xe_ys": "<", "xe_ye":
"<",})

In order to reach a standardized representation for the different corpora, we
developed a reader for each of the corpus. Each dataset reader has inherited
from an abstract base class, named BaseDocumentReader, which requires the
implementation of five methods named after the five attributes used to create
an instance of a Document: name, text, dct, entities, and tlinks. To extract
this information, the base class contains three attributes: the path for the
document being read; the content of the dictionary produced by parsing the
document with the xmltodict888https://pypi.org/project/xmltodict/ library; and
the xml attribute that results from parsing the file with the
xml999https://docs.python.org/3/library/xml.etree.elementtree.html library.
Note that, while json is nowadays the standard format for the exchange of the
information, we had to resort to xml as most datasets were stored in that
format. The script presented in Listing 2 illustrates how to read a document
from the TempEval-3 corpus with the TempEval3DocumentReader.

Listing 2: Read a document of the TempEval-3 corpus.

⬇

from tieval import datasets

path = "tempeval-3/wsj_0026.tml"

reader = datasets.TempEval3DocumentReader(path)

doc = reader.read()

To fully integrate a new corpus on the library – and automatically read the
entire corpus – the user just needs to add an entry on the DATASETS_METADATA
dictionary with the metadata necessary to read the document. This information
will be used on the read function of the datasets module, which only requires
the name of the corpus to produce an instance of the Dataset object with all
the annotations provided in there. The script in Listing 3 presents how to
perform such operation.

Listing 3: Read the TempEval-3 corpus.

⬇

from tieval import datasets

te3 = datasets.read("TempEval-3")

The current version of tieval natively supports the download and reading of an
extensive list of corpora for TIE. A complete list of the corpora considered
is provided in Table LABEL:table:corpus. In order to ensure long-term support
for these corpora, we created a repository with them. Besides that, it also
has the advantage that we can standardize the structure of the folders and add
useful information to the raw datasets (for instance, the spans of the
temporal entities identified on the text) and fix errors on the original
annotation101010The changes made on the original corpus are detailed on the
file logbook.rst in the docs folder of the project repository.. For that
reason, we were careful to verify the license for each of the corpora and
publish only the ones that allowed for redistribution or did not provide any
license.

[ caption = The corpora currently supported on tieval., label = table:corpus ]
colspec = X[5,c]|X[3,c]|X[2,r]|X[2,r]|X[2,r]|X[2,r], row1 = c, cell21 = r=8,
cell151 = r=4, cell201 = r=6, cell271 = r=4, cell341 = r=7, cell431 = r=2,
hline1,45 = -0.1em, hline2,10-15,19-20,26-27,31-34,41-43 = -0.05em, & Language
# Docs # Events # Timexs # Tlinks
AncientTimes Arabic 5 0 106 0
Dutch 5 0 130 0
English 5 0 311 0
French 5 0 290 0
German 5 0 196 0
Italian 5 0 234 0
Spanish 5 0 217 0
Vietnamese 4 0 120 0
Aquaint English 72 4,351 639 5,832
EventTime English 36 1,498 0 0
GraphEVE English 103 4,298 0 18,204
KRAUTS German 192 0 1,282 0
MATRES English 274 6,065 0 13,504
MeanTime English 120 1,882 349 1,753
Spanish 120 2,000 344 1,975
Dutch 120 1,346 346 1,487
Italian 120 1,980 338 1,675
Narrative Container English 63 3,559 439 737
Professor Heideltime English 24,642 0 254,803 0
French 27,154 0 83,431 0
German 19,095 0 194,043 0
Italian 9,619 0 58,823 0
Portuguese 24,293 0 111,810 0
Spanish 33,266 0 348,011 0
Platinum (TempEval-3) English 20 748 158 929
TimeBank Spanish 210 12,384 1,532 21,107
French 108 2,115 533 2,303
Portuguese 182 7,887 1,409 6,538
English 183 6,681 1,426 5,120
TimeBank 1.2 English 183 7,940 1,414 6,413
TCR English 25 1,134 242 3,515
TDDiscourse English 34 1,101 0 6,150
TempEval 2 Chinese 52 4,783 946 7,802
English 182 6,656 1,390 5,945
French 83 1,301 367 372
Italian 64 5,377 653 6,884
Korean 18 2,583 317 0
Spanish 210 12,384 1,502 13,304
English 275 11,780 2,223 11,881
TimeBank Dense English 36 1,712 289 12,715
TrainT3 (TempEval-3) Spanish 175 10,686 1,269 17,283
Wikiwars English 22 0 2,662 0
German 22 0 2,239 0

###  3.2 Models

The current version of tieval has four built-in models, namely: a baseline for
timex identification; the HeidelTime model Strötgen et al. (2013) for timex
identification and classification; a baseline for event identification; and
the CogCompTime 2.0 model Ning et al. (2019) for tlink classification. The
availability of these four models is intended for practitioners that may want
to experiment using any layer of temporal information in their specific
application. Apart from that, it also provide researchers the implementation
of baseline models for reference in their work.

For the baseline models, we provide pre-trained weights, however, the user can
also train the model from scratch. A description of each of the models is
provided below:

TimexIdentificationBaseline

For this baseline we trained – from scratch – the spaCy111111https://spacy.io/
named entity recognition model to identify the timexs on the TempEval-3
corpus.

EventIdentificationBaseline

This model has the same architecture of the TimexIdentificationBaseline but
was trained to identify events rather than timexs on the TempEval-3 corpus.

HeidelTime

This model is a widely recognized multilingual temporal tagger which was
original written in Java121212https://github.com/HeidelTime/heideltime.
However there have been efforts to build python wrappers. In tieval we used
the py_heideltime wrapper which is available on
GitHub131313https://github.com/hmosousa/py_heideltime.

CogCompTime2

This model leverages the ELMo Peters et al. (2018) word embeddings and the
TempProb Ning et al. (2018c) knowledge base to classify the temporal relation
between a pair of temporal entities Ning et al. (2019). Our implementation was
adapted from the repository made
available141414https://github.com/qiangning/NeuralTemporalRelation-EMNLP19 by
the authors.

Listing 4 presents a script that would download the baseline model for
temporal expression identification (TimexIdentificationBaseline), train the
model on the TempEval-3 train set, and produce predictions for the TempEval-3
test set.

Listing 4: How to download, train, and predict with for the temporal
identification task.

⬇

from tieval import models

model = model.TimexIdentificationBaseline()

model.fit(te3.train)

predictions = model.predict(te3.test)

A user interested in releasing his/her model in tieval can do it by creating a
subclass of one of our base classes for models. There are two base classes: a
BaseModel which just requires the implementation of the predict method which
is intended for models that are available in other repositories – for
instance, the HeidelTime model – and a BaseTrainableModel which, besides the
predict, requires the implementation of the fit method, which implements the
training loop for the model.

###  3.3 Evaluation

tieval provides an evaluation function for four subtasks of TIE, more
specifically: timex identification, event identification, tlink
identification, and tlink classification.

Table 1: The results obtained by evaluating the four models integrated in tieval on the Platinum (TempEval-3 test set), TCR, and MeanTime (the English version) corpus. P stands for precision, R for recall, F1 is the F1-score, and TF1 is the temporal awareness. All the results in the table are micro metrics. | Platinum |  |  | TCR |  |  | MeanTime |  |
---|---|---|---|---|---|---|---|---|---
| $P$ |  $R$ | $F_{1}~{}(TF_{1})$ | $P$ | $R$ | $F_{1}~{}(TF_{1})$ | $P$ | $R$ | $F_{1}~{}(TF_{1})$
TimexBaseline | 88.1 |  75.4 | 81.2 | 75.4 | 82.0 | 78.6 | 23.7 | 57.1 | 33.5
HeidelTime | 84.0 |  79.4 | 81.8 | 70.6 | 80.6 | 75.3 | 26.5 | 65.8 | 37.8
EventBaseline | 74.6 |  80.5 | 77.5 | 48.3 | 92.6 | 63.5 | 25.8 | 54.1 | 34.9
CogCompTime2 | 39.7 |  39.7 | 39.7 (39.3) | 75.4 | 75.4 | 75.4 (69.3) | 30.7 | 28.6 | 29.6 (28.9)

The input is standard for all the evaluation functions: annotations, a
dictionary with the name of the documents as keys and the annotations as
values; predictions, follows the same structure of the annotations but for
each document key contains the predictions made by a model. The output of the
functions is dependent on the task being evaluated. For the identification
tasks (timex, event, and tlink) the function produces the standard macro and
micro metrics for precision, recall, and f1-score. Listing 5 presents a script
that evaluates the predictions made by the event baseline model in the
TempEval-3 test set.

Listing 5: Evaluate event baseline model on the TempEval-3 test set.

⬇

from tieval import evaluate

annotations = {doc.name: doc.events for doc in te3.test}

result = evaluate.event_identification(annotations, predictions)

Table 1 depicts the results obtained by the implemented on benchmark corpora.
Note that TF1 is the temproal awareness metric and is only computed for
CogCompTime2 (the only tlink classification system). Another interesting
remark is the fact that the TimexBaseline achieves effectiveness comparable to
HeidelTime despite its simplicity.

The tlink classification is the most elaborate evaluator as it also computes
the temporal awareness metric UzZaman and Allen (2011). The complexity of the
calculation of temporal awareness lies in the computation of temporal closure.
With temporal_closure the closure operation can be easily performed on the
document level, with the closure method of the Document object, or applied to
a set of tlink’s with the temporal_closure function available on the library.
The script in Listing 6 illustrates how to perform such operations.

Listing 6: How to compute the temporal closure with a Document object and with
a set of TLink’s.

⬇

from tieval import temporal_closure

doc = te3["wsj_0026.tml"]

closure_tlinks = doc.closure

closure_tlinks = temporal_closure(doc.tlinks)

For the temporal closure to be efficiently performed, on the back-end, the
closure operation is executed with a point-based reasoner which was inspired
by the work of Gerevini et al. Gerevini et al. (1993). As stated above, each
TLink instance contains an attribute named relation which is an instance of
the TemporalRelation object. Within the TemporalRelation all temporal
relations are represented as the point relations by the means of a
PointRelation instance. In the point representation there are only four types
of temporal relations, namely before (<), after (>), equal (=), and not
defined (None). With this point relation one can build a directed graph
(henceforth referred as timegraph) where the nodes are the entities endpoints
(start and end of the entity) and the edges represent the before (<) relation.
This is accomplished by reflecting the after (>) relations and aggregating the
equal (=) relations in a single node.

In the timegraph, inferring temporal relations is reduced to the problem of
finding if two entities endpoints are connected, i.e., they are in the same
subgraph (by subgraph we mean a fully connected graph of the timegraph). If
that is the case, one can retrieve the endpoints on the entity pair and
validate if the order of the entity endpoints is a valid temporal relation. To
clarify this concept, Figure 5 presents the timegraph built for a scenario
where two tlinks were provided: X MEETS Y and Y STARTS Z. To infer the
temporal relation between X and Z one must query the endpoints in the
timegraph. In this case, one would get the following sequence of endpoints: sX
< ex = sZ < eZ. After retrieving the sequence of endpoints one just needs to
validate if that sequence is a valid interval relation. In this example, one
can conclude that the temporal relation between X and Z is MEETS.

Figure 5: On the top part of the image is the relative relations between
entities X, Y, and Z. On the bottom is the graphical representation of the
timegraph that would be generated.

To get a practical understanding of the runtime of the temporal closure
algorithm, we executed it on all documents currently available intieval. On a
computer with an Intel Core i5-8500 CPU, the algorithm took less than half a
second for $95\%$ of the documents, while the worst-case scenario took roughly
$1.6$ seconds.

This finalizes the presentation of the main functionalities, and some inner
workings, of the first version of tieval. The current version already provides
functionalities that (we believe) will be beneficial for the TIE community.
However, we already have some ideas to further improve this library. These
ideas are discussed in section 5.

##  4 Observations

While building tieval, and in particular the datasets module, we found some
inconsistencies in the corpus we were working with. For instances, we found
that the articles APW20000115.0209 and APW20000107.0088 of the AQUAINT corpus
contained the same news article, differing only in the annotations and in the
value of the document creation time. This type of inconsistencies were
mitigated by implementing data cleaning processes that changed the original
annotations. Consequently, the results on the tieval framework will (most
frequently) not resemble the exact result that was reported in previous works,
even if the same model is employed.

##  5 Conclusion and Future Work

This work presented the first public release of the tieval package, an open-
source Python library for the development and evaluation of TIE systems.
tieval provides several functionalities to facilitate research in this field.
These include the import of multiple benchmark corpora in different formats,
domain-specific operations such as temporal closure or transformation from
interval to point relations, out-of-the-box baseline systems, and evaluation
measures for TIE tasks. Therefore, it provides the community with a standard
way to benchmark TIE systems in a fair and comparable way, while enabling the
development of reproducible systems.

For future versions of the package, we aim to extend its functionalities. One
idea we are keen to implement is visualization techniques to display the
relative timeline of events from the annotations. In addition, we will add
methods to include other levels of information when available such as
coreference resolution in the MeanTime corpus Minard et al. (2016) and
causality relations in the TCR corpus Ning et al. (2018b). We also intend to
extend the list of supported corpora and baseline models, in particular, to
support corpora that cast the TIE task as a question-answer problem, such as
MCTaco Zhou et al. (2019) and TORQUE Ning et al. (2020). This will allow us to
produce a reproducibility study to investigate several state-of-the-art
systems and benchmark them in the different corpora.

## References

  * Campos et al. [2014] Ricardo Campos, Gaël Dias, Alípio M Jorge, and Adam Jatowt.  Survey of temporal information retrieval and related applications.  _ACM Computing Surveys (CSUR)_ , 47(2):1–41, 2014\.
  * Leeuwenberg and Moens [2019] Artuur Leeuwenberg and Marie-Francine Moens.  A survey on temporal reasoning for temporal information extraction from text.  _Journal of Artificial Intelligence Research_ , 66:341–380, 2019.
  * Naik et al. [2019] Aakanksha Naik, Luke Breitfeller, and Carolyn Rose.  Tddiscourse: A dataset for discourse-level temporal ordering of events.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 239–249, 2019.
  * Ning et al. [2018a] Qiang Ning, Hao Wu, and Dan Roth.  A multi-axis annotation scheme for event temporal relations.  _arXiv preprint Arxiv:1804.07828_ , 2018a.
  * UzZaman et al. [2013] Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky.  Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations.  In _Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 1–9, 2013.
  * Verhagen et al. [2007] Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky.  Semeval-2007 task 15: Tempeval temporal relation identification.  In _Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)_ , pages 75–80, 2007.
  * Verhagen et al. [2010] Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky.  Semeval-2010 task 13: Tempeval-2.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 57–62, 2010.
  * UzZaman and Allen [2011] Naushad UzZaman and James Allen.  Temporal evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 351–356, 2011\.
  * Minard et al. [2016] Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, Marieke Van Erp, Anneleen Schoen, and Chantal Van Son.  Meantime, the newsreader multilingual event and time corpus.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 4417–4422, 2016.
  * Ling and Weld [2010] Xiao Ling and Daniel S Weld.  Temporal information extraction.  In _Twenty-Fourth AAAI Conference on Artificial Intelligence_ , 2010\.
  * Derczynski et al. [2015] Leon Derczynski, Jannik Strotgen, Ricardo Campos, and Omar Alonso.  Time and information retrieval: Introduction to the special issue.  _Inf. Process. Manag._ , 51:786–790, 2015.
  * Pustejovsky et al. [2003a] James Pustejovsky, José M Castano, Robert Ingria, Roser Sauri, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev.  Timeml: Robust specification of event and temporal expressions in text.  _New Directions in Question Answering_ , 3:28–34, 2003a.
  * Campos et al. [2017] Ricardo Campos, Gaël Dias, Alípio Mário Jorge, and Célia Nunes.  Identifying top relevant dates for implicit time sensitive queries.  _Information Retrieval Journal_ , 20(4):363–398, 2017.
  * Allen [1983] James F Allen.  Maintaining knowledge about temporal intervals.  _Communications of the ACM_ , 26(11):832–843, 1983.
  * Pustejovsky et al. [2003b] James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al.  The timebank corpus.  In _Corpus Linguistics_ , volume 2003, page 40. Lancaster, UK., 2003b.
  * Graff [2002] David Graff.  _The AQUAINT Corpus of English News Text LDC2002T31._ Linguistic Data Consortium, 2002.
  * Li et al. [2014] Hui Li, Jannik Strötgen, Julian Zell, and Michael Gertz.  Chinese temporal tagging with heideltime.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Volume 2: Short Papers_ , pages 133–137, 2014.
  * Bittar et al. [2011] André Bittar, Pascal Amsili, Pascal Denis, and Laurence Danlos.  French timebank: an iso-timeml annotated reference corpus.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 130–134, 2011\.
  * Caselli et al. [2011] Tommaso Caselli, Valentina Bartalesi Lenzi, Rachele Sprugnoli, Emanuele Pianta, and Irina Prodanof.  Annotating events, temporal expressions and relations in italian: the it-timeml experience for the ita-timebank.  In _Proceedings of the 5th Linguistic Annotation Workshop_ , pages 143–151, 2011.
  * Nieto and Saurí [2012] Marta Guerrero Nieto and Roser Saurí.  Modes timebank 1.0.  _Linguistic Data Consortium (LDC), Philadelphia, PA, USA_ , 2012.
  * Costa and Branco [2012] Francisco Costa and António Branco.  TimeBankPT: A TimeML annotated corpus of Portuguese.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 3727–3734, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2012/pdf/246_Paper.pdf.
  * Sun et al. [2013] Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner.  Evaluating temporal relations in clinical text: 2012 i2b2 challenge.  _Journal of the American Medical Informatics Association_ , 20(5):806–813, 2013.
  * Styler IV et al. [2014] William F Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C De Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, et al.  Temporal annotation in the clinical domain.  _Transactions of the Association for Computational Linguistics_ , 2:143–154, 2014.
  * Cassidy et al. [2014] Taylor Cassidy, Bill McDowell, Nathanel Chambers, and Steven Bethard.  An annotation framework for dense event ordering.  Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA, 2014.
  * Ning et al. [2018b] Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth.  Joint reasoning for temporal and causal relations.  _arXiv preprint Arxiv:1906.04941_ , 2018b.
  * O’Gorman et al. [2016] Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer.  Richer event description: Integrating event coreference with temporal, causal and bridging annotation.  In _Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016)_ , pages 47–56, 2016.
  * Reimers et al. [2016] Nils Reimers, Nazanin Dehghani, and Iryna Gurevych.  Temporal anchoring of events for the timebank corpus.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2195–2204, 2016\.
  * Leeuwenberg and Moens [2020] Artuur Leeuwenberg and Marie-Francine Moens.  Towards extracting absolute event timelines from english clinical reports.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2710–2719, 2020.
  * Chen and Styler [2013] Wei-Te Chen and Will Styler.  Anafora: A web-based general purpose annotation tool.  In _Proceedings of the 2013 NAACL HLT Demonstration Session_ , pages 14–19. Association for Computational Linguistics, 2013.  URL http://www.aclweb.org/anthology/N13-3004.
  * Strötgen et al. [2013] Jannik Strötgen, Julian Zell, and Michael Gertz.  Heideltime: Tuning english and developing spanish resources for tempeval-3.  In _Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 15–19, 2013.
  * Ning et al. [2019] Qiang Ning, Sanjay Subramanian, and Dan Roth.  An improved neural baseline for temporal relation extraction.  _arXiv preprint Arxiv:1909.00429_ , 2019.
  * Peters et al. [2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _NAACL_ , 2018.
  * Ning et al. [2018c] Qiang Ning, Hao Wu, Haoruo Peng, and Dan Roth.  Improving temporal relation extraction with a globally acquired statistical resource.  _arXiv preprint Arxiv:1804.06020_ , 2018c.
  * Gerevini et al. [1993] Alfonso Gerevini, Lenhart Schubert, and Stephanie Schaeffer.  Temporal reasoning in timegraph i–ii.  _ACM SIGART Bulletin_ , 4(3):21–25, 1993.
  * Zhou et al. [2019] Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth.  " going on a vacation" takes longer than" going for a walk": A study of temporal commonsense understanding.  _arXiv preprint Arxiv:1909.03065_ , 2019.
  * Ning et al. [2020] Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth.  Torque: A reading comprehension dataset of temporal ordering questions.  _arXiv preprint Arxiv:2005.00242_ , 2020.