# Predicting Event Time by Classifying Sub-Level Temporal Relations Induced
from a Unified Representation of Time Anchors

Fei Cheng
Department of Intelligence
Science and Technology
Graduate School of Informatics
Kyoto University
<EMAIL_ADDRESS>
&Yusuke Miyao
Department of Computer Science
Graduate School of Information
Science and Technology
University of Tokyo
<EMAIL_ADDRESS>

###### Abstract

Extracting event time from news articles is a challenging but attractive task.
In contrast to the most existing pair-wised temporal link annotation, Reimers
et al. (2016) proposed to annotate the time anchor (a.k.a. the exact time) of
each event. Their work represents time anchors with discrete representations
of Single-Day/Multi-Day and Certain/Uncertain. This increases the complexity
of modeling the temporal relations between two time anchors, which cannot be
categorized into the relations of Allen’s interval algebra (Allen, 1990).

In this paper, we propose an effective method to decompose such complex
temporal relations into sub-level relations by introducing a unified quadruple
representation for both _Single-Day/Multi-Day_ and _Certain/Uncertain_ time
anchors. The temporal relation classifiers are trained in a multi-label
classification manner. The system structure of our approach is much simpler
than the existing decision tree model Reimers et al. (2018), which is composed
by a dozen of node classifiers. Another contribution of this work is to
construct a larger event time corpus (256 news documents) with a reasonable
Inter-Annotator Agreement (IAA), for the purpose of overcoming the data
shortage of the existing event time corpus (36 news documents). The empirical
results show our approach outperforms the state-of-the-art decision tree model
and the increase of data size obtained a significant improvement of
performance.

##  1 Introduction

Along with TimeBank (Pustejovsky et al., 2003) (TB) and other temporal
corpora, a series of competitions on temporal information extraction
(TempEval-1,2,3) (Verhagen et al., 2009, 2010; UzZaman et al., 2012) are
attracting growing research efforts. Predicting the exact time when an event
occurred can effectively contribute various time-aware tasks, such as timeline
(Minard et al., 2015), temporal question answering (Llorens et al., 2015; Meng
et al., 2017), knowledge base population, etc. For instance, given a hot topic
(event, person, product, company, etc.), we can first gather its related news
articles, extract events, and predict event time from them. An event timeline
can be automatically generated by ordering events on a time axis according to
their occurring time. Such timeline representations can be extremely helpful
for readers to comprehend new topics efficiently.

TimeBank-EventTime (TBET) is a new temporal information corpus based on TB.
Instead of using the majority of a pair-wise Temporal Link (TLINK) Setzer
(2002) to model a relation between two mentions111In this paper, ’Timex’
denotes time expression and ’DCT’ denotes document creation time (i.e. event,
time expression and document creation time), TBET annotates the occurring time
of each individual event. Their annotation schema first distinguishes between
_Single-Day/Multi-Day_ and _Certain/Uncertain_ events as following:

  * •

_Single-Day_ denotes an event occurs in a single day. A _Single-day_ event can
be further categorized into two types as follows:

_Certain_ day, e.g. ’1998-02-06’

_Uncertain_ day, e.g. ’before1998-01-31’, ’after1998-01-01’ or
’after1998-01-01, before1998-02-06’.

  * •

_Multi-Day_ event time can be seen as a tuple of two _Single-Day_ (begin,
end).

For predicting _Single-Day_ event time, Cheng and Miyao (2018) induced 6-label
temporal relations induced from TBET to train two classifiers of Event-to-DCT
and Event-to-Timex mention pairs. A major difficulty preventing them from
extending this approach to the _Multi-Day_ events is the complexity of
inducing relations with Uncertain information. Manually extending the existing
temporal relation types (Allen (1990)’s interval algebra) will be hard.

Another approach (Reimers et al., 2018) of predicting both _Single-Day_ and
_Multi-Day_ event time trains a series of classifiers to make tree-structure
decisions starting from the root decision of whether an event is _Single-Day_
or _Multi-Day_. The _Single-Day_ branch determines the temporal clues from
relevant time expressions. While, the _Multi-Day_ branch will further
determine begin and end points separately. However, such decision tree of
classifiers is less generalized for only fitting this event time definition
and training a series of classifiers is complex and time-consuming.

In this work, we propose a simple method to predict both _Single-Day_ and
_Multi-Day_ event time effectively. We introduce a quadruple expression with
4-date values, i.e. (begin_earliest, begin_latest, end_earliest and
end_latest), which is capable of unifying the representation of _Single/Multi-
Day_ and _Certain/Uncertain_ event time (Section 3.1). This expression enables
us to decompose a complex temporal relation into four simple sub-level
temporal relations (Section 3.2). In Section 4, we perform classifications in
a multi-label fashion. In the inference step, we gather all the relevant time
clues of an event to infer the time. Another contribution (Section 5) is that
we constructed a larger event time corpus with 256 news articles, compared to
the existing TBET with 36 articles. In Section 6.1, we show the experiments of
increasing the training data and being evaluated in two different test data.
The results show that the large data size significantly improves
classification results in the large test data: TE3-Test. The comparison to the
SOTA decision tree model shows our sub-level temporal relation model with
attention significantly outperforms their model in the TBET data (Section
6.4).

##  2 Background

###  2.1 Event Time Annotation

Most existing temporal corpora adopted a pair-wise schema (i.g. TLINK) to
encode temporal relations between mentions. A major drawback is that the
potential pair candidates are quadratic to the number of mentions. This
resulted in the sparsity of TLINKs in the original TimeBank. The later
TimeBank-Dense (Cassidy et al., 2014) achieved denser TLINKs by forcing the
annotators labeling all the possible pairs in any two adjacent sentences.
However, the dense annotation is time-consuming and performed on a subset of
Timebank with only 36 documments.

Reimers et al. (2016) proposed an event time schema to annotate the TBET
corpus with a linear annotation effort, in which annotators infer the exact
time of each individual event. The main limitation is that the new schema
requires events to be capable of being anchored on exact time. In this paper
our main target is processing news articles, in which time expressions and
document creation time provide rich exact time information. Therefore, this
limitation will affect less.

###  2.2 Existing Event Time Prediction Systems

The proposal of TBET raised a new task of predicting the exact time of events.
Intuitively, knowing when an event occurred is an attractive target to achieve
many practical applications. For instance, ordering events according to their
occurring time can naturally generate an event timeline in a cross-document
scenario.

Reimers et al. (2016) released a baseline for event time prediction based on
CAEVO Chambers et al. (2014), which is trained on TimeBank-Dense. This
baseline can only predict _Single-Day_ Events, because the Timebank-Dense
TLINKs adopted a limited 6-label set without providing the necessary begin or
end information to predict _Multi-Day_ Events.

Cheng and Miyao (2018) proposed an approach to induce similar 6-label
relations based on the event time annotation. Their temporal relation
classifiers are fully trained on the new TBET corpus and achieve better
performance on predicting _Single-Day_ event time.

Reimers et al. (2018) included the _Multi-Day_ Events prediction into their
target. They designed a series of individual classifiers to make hierarchical
decisions of whether an event is _Single-Day_ or _Multi-Day_ , the temporal
relation from the begin day of an event to a time expression, etc. The
drawback is that training a decision tree of individual classifiers is
complicated and the data of some leaf classifiers become very sparse.

_Single/Multi_ | _Certainty_ | Type | Representation
---|---|---|---
_Single-Day_ | _Certain_ | _Time anchor_ | 1998-01-26
_Quadruple_ | ((1998-01-26, 1998-01-26), (1998-01-26, 1998-01-26))
_Single-Day_ | _Uncertain_ | _Time anchor_ | after1998-01-26before1998-02-06
_Quadruple_ | ((1998-01-26, 1998-02-06), (1998-01-26, 1998-02-06))
_Multi-Day_ | _Certain_ | _Time anchor_ | begin:1998-01-01,end:1998-01-31
_Quadruple_ | ((1998-01-01, 1998-01-01), (1998-01-31, 1998-01-31))
_Multi-Day_ | _Uncertain_ | _Time anchor_ | begin:1998-01-01,end:after1998-02-06
_Quadruple_ | ((1998-01-01, 1998-01-01), (1998-02-06, ~))
Table 1:  The comparison of the time anchor and quadruple examples. ’~’
denotes the blank information in some _Uncertain_ cases.

###  2.3 Neural Temporal Relation Classifier

Temporal relation classification can be categorized as a variation of a more
general Relation Classification (RC) task, in which the models are required to
detect a semantic relation between two nominals in a sentence. Zeng et al.
(2014) proposed a Convolutional Neural Network (CNN) model with the relevant
distances from each word to the two nominals as the additional features to
achieve state-of-the-art performance. From 2016, researchers started to
introduce the attention mechanism into RC to push the scores higher. Zhou et
al. (2016) proposed an attention-based RNN model to obtain normalized scores
along the time steps for calculating a weighted representation of an sentence.

The step of applying neural networks into Temporal Relation Classification
started from the work of (Cheng and Miyao, 2017; Meng et al., 2017). These two
works feed the shortest dependency path (SDP) between two mentions into
recurrent neural networks (RNNs). Reimers et al. (2018) adopted Zeng et al.
(2014)’s model to train a series classifiers to make hierarchical decisions.

In this work, we proposed a mention-wised attention LSTM Hochreiter and
Schmidhuber (1997) classifier, which emphasized the interaction between words
and the given mentions to obtain the weighted context information of an
sentence.

##  3 Representing Time Anchors and Sub-level Temporal Relations

There are two obstacles to prevent us from achieving a more effective way to
predict event time. First, the discrete annotation of the _Single/Multi-day_
and _Certain/Uncertain_ information in TBET forced Reimers et al. (2018) to
make hierarchical decisions from the top question whether the given event is
_Single-Day_ or _Multi-Day_ to the leaf temporal relation classifiers. It
raised the complexity of their system and accumulated errors from early
classifiers. The second question is that extending the definition of temporal
relation is hard but necessary in order to represent the relation with
_Uncertain_ time.

###  3.1 A Unified Quadruple Representation of Time Anchors

In this work, we adopt a quadruple representation Berberich et al. (2010) for
consistently representing the _Single/Multi-Day_ and _Uncertainty_ information
of a time anchor.

$T=((begin_{e},begin_{l}),(end_{e},end_{l}))$

$(begin_{e},begin_{l})$ stand for the earliest and latest possible days of the
beginning point. Analogously, $(end_{e},end_{l})$ are its earliest and latest
possible days of the end point.

This representation is naturally designed for representing _Multi-Day_ and
_Uncertain_ information. We adopt an inclusive rule, which includes _Single-
Day_ into our representation by making beginning and end annotation to the
same. A _Certain_ day has the same earliest and latest days. To make it clear,
we list several examples in Table 1 with both TBET and our quadruple
representations for comparison.

Figure 1:  The architecture of the event time prediction System.  Figure 2: An
example of sub-level temporal relations between $T_{e}$ and $T_{t}$

###  3.2 Decomposing temporal relations into sub-level

Most existing temporal relation annotation adopted Allen (1990)’s algebra,
which defined 13 relations between two _Certain_ time intervals. Many of the
following works attempted to reduce the sparsity of relations by merging some
relations to a coarse relation. The _Uncertainty_ information in TBET even
aggravates the complexity of modeling temporal relations between two time
anchors.

In this work, we decompose a temporal relation into a group of sub-level
relations for avoiding laboriously extending the relation definition. Let
$((begin^{e}_{e},begin^{e}_{l}),(end^{e}_{e},end^{e}_{l}))$ and
$((begin^{t}_{e},begin^{t}_{l}),(end^{t}_{e},end^{t}_{l}))$ stand for two time
anchors: $T_{e}$ of an event and $T_{t}$ of a DCT or Timex. We explicitly
compare four sub-level relations (SR) $[SR_{1},SR_{2},SR_{3},SR_{4}]$ as
follows:

  * $SR_{1}$:

$(begin^{e}_{e},begin^{e}_{l})$ to $(begin^{t}_{e},begin^{t}_{l})$

  * $SR_{2}$:

$(begin^{e}_{e},begin^{e}_{l})$ to $(end^{t}_{e},end^{t}_{l})$

  * $SR_{3}$:

$(end^{e}_{e},end^{e}_{l})$ to $(begin^{t}_{e},begin^{t}_{l})$

  * $SR_{4}$:

$(end^{e}_{e},end^{e}_{l})$ to $(end^{t}_{e},end^{t}_{l})$

Figure 2 is an example of the sub-level relations between $T_{e}$ and $T_{t}$.
This example can not be categorized into any existing relation types defined
in Allen’s algebra due to the _Uncertainty_ of whether the end point of
$T_{e}$ is after, before or equal to the end point of $T_{t}$. However, our
proposal can encode the sub-level information $[equal,before,after,vague]$
indicating that $T_{e}$ and $T_{t}$ holds the same begin points, but the
temporal relation between the end points is vague.

Each sub-level relation is naturally defined with a simple label set {equal,
after, before, vague} as shown in Table 2. Any relation between two time
anchors can be represented as the combination of four sub-level relations.
This approach effectively avoids an exhaustive expansion of Allen’s algebra in
order to include the _Uncertain_ information of time anchors.

SR Label |  Definition
---|---
between $(E_{e},E_{l})$ and $(T_{e},T_{l})$
equal |  if $E_{e}=T_{e}$ and $E_{l}=T_{l}$
after |  if $E_{e}\geq T_{l}$ and ($E_{l}>T_{l}$ or $E_{l}=~$)
before |  if $E_{l}\leq T_{e}$ and ($E_{e}<T_{e}$ or $E_{e}=~$)
vague |  other cases
Table 2: Definition of four types of Sub-level temporal relations between
$(E_{e},E_{l})$ and $(T_{e},T_{l})$. ‘$<$’, ‘$>$’, ‘$=$’ denote one day point
is in the left of, right of and same position as the other day point in a
left-to-right time axis.

##  4 Event Time Prediction System

Once we have the unified representation of time anchors and sub-level
relations (SR), we can directly classify SRs between a given event to DCT or
any other Timex in the text, instead of constructing a complex tree structure
of node classifiers. The overall system architecture is shown in Figure 1.
Given a target event, our SR classifiers first predict four SRs to each time
expression in the context. Then, the quadruple values of the given event are
inferred by a strategy based on the SR inputs.

Figure 3:  The mention-wised attention LSTM model.

###  4.1 Mention-wise Attention Classifiers

Based on the link types (Event-to-DCT and Event-to-Timex), temporal relation
classifiers take different input information. For instance, Event-to-DCT (E-D)
has only one event information, while Event-to-Timex (E-T) has two intputs,
i.e. an event and a Timex. Therefore, we follow the setting of most existing
work, which treats Event-to-DCT and Event-to-Timex as two independent
classification tasks.

In this work, we propose a mention-wise attention LSTM classifier. An
alignment vector $a_{e}$, whose length is the same as the number of time steps
of the input sequence $s$, is derived by the interaction between each hidden
state $\overline{h}_{s}$ and the hidden state $h_{e}$ of the given event.

$a_{e}(s)=\frac{exp(score(h_{e},\overline{h}_{s}))}{\sum_{s^{\prime}}exp(score(h_{e},\overline{h}_{s^{\prime}}))}$
(1)

We adopted a similar ’general’ score function to (Luong et al., 2015) as Eq
(2). $W_{a}$ denotes a matrix of trainable model parameters.

$score(h_{e},\overline{h}_{s})=h^{\mathsf{T}}_{e}W_{a}\overline{h}_{s}$ (2)

Given $a_{e}$ as weights, a context vector $c_{e}$ is computed as the weighted
average over the hidden states of all time steps. Then, we define the
representation $R_{m}$ of a sequence $s$ given an event $e$ is the
concatenation of $h_{e}$ and $c_{e}$, which is fed into the final Softmax
layer.

Figure 3 shows the structure of our neural network classifier for Event-to-
Timex. An Event-to-DCT relation has only an event mention information.
Therefore, the model feed $R_{e}$ into the Softmax layer. In the case of
Event-to-Timex, two mentions exist i.e. an event and a Timex. The model
concatenate $R_{e}$ and $R_{t}$ as the output.

###  4.2 Word and Position Embeddings

We use the pre-trained word embedding
Glove222https://nlp.stanford.edu/projects/glove/ presented by Pennington et
al. (2014). The input words are mapped into their embedding vectors by the
embedding layer of our model. We use the 200-dimension Glove in our model.

In TRC, the given information of a data sample includes a sentence and
mentions (event or Timex) inside it. An important feature is to distinguish
between the mention words and other words inside the sentence. A popular
method is called position embedding, which represents the relative distance
from a word to the mention word. For instance, an event the $k$-th word (from
left to right) in a sentence, a word $i$ has a relative distance $i-k$ to the
event. We embed this number into a random initialized vector and the dimension
of the vector is a hyper-parameter.

###  4.3 Multi-label Loss

Based on our definition in Section 3.2, each E-D or E-T link consists of 4 SRs
and each SR adopts a 4-label set {$equal$, $after$, $before$, $vague$}. In
this paper, we formulate the SR classification as a multi-label multi-class
problem. We fixed the dimension number of the output layer to $16$ ($4$ SRs
$\times$ $4$ Labels) and calculate the Softmax distribution for each SR. We
define the multi-label loss as the sum of the negative log-likelihood of each
SR.

$L=-{\sum_{i}{\sum_{j}g_{ij}{\log p_{ij}}}}$ (3)

In Eq 3, $i$, $j$ denote the SR, label. $g$, $p$ denote the gold label,
predicted Softmax probability.

TBET-TestTE3-Test$40$$50$$60$$70$TBETTBET+AQTBET+TBTBET+AQ+TB (a) Event-to-DCT
Classification TBET-
TestTE3-Test$10$$20$$30$$40$$50$TBETTBET+AQTBET+TBTBET+AQ+TB (b) Event-to-
Timex Classification

Figure 4: The classification performance as the training data size increases.
The evaluation metric is F1 of 4-SR ’complete match’.

###  4.4 Inference Strategy

Once a target event given the predicted 4-SR to a Timex, the SRs naturally
indicate the action of operating the normalized value of the Timex to infer
the time of the event. For instance, if $SR_{1}$ is equal to a Timex, our
inference strategy will update $(begin_{e}^{e},begin_{l}^{e})$ as the same as
$(begin_{e}^{t},begin_{l}^{t})$. If $SR_{1}$ is after, the strategy will
update $begin_{e}^{e}$ to the same as $begin_{l}^{t}$.

We infer the time of the target event along the SRs to DCT, Timex-0 (the same
sentence), Timex-1 (the adjacent sentences), etc. An exception is that if an
Event-to-DCT $SR_{i}$ is predicted as equal, the following $SR_{i}$ of Event-
to-Timex will not effect anymore. The reason is that Event-to-DCT usually
provides a majority of time information to predict event time in this task. We
give it higher priority than Event-to-Timex to avoid wrong prediction of
Event-to-Timex.

##  5 A Large Event Time Corpus (LETC)

Corpus | Article | Event | IAA
---|---|---|---
TBET | 36 | 1.5K | 0.580
AQ | 73 | 4.4K | 0.567
TB | 147 | 5.2K | 0.552
Table 3:  The corpus statistics of our annotation data. ’TB’ stands for 147
articles of the original TimeBank excluding 36 TBET articles.

TempEval-2013 released a cleaned TBAQ (TB:183 articles, AQ: 73 articles) for
training and evaluation data TE3-Test with 20 articles. The existing TBET
corpus is relatively small, which is based on the a sub-set of TB and contains
1.5K events in total.

As we discussed in Section 2.1, the event time annotation has the advantage of
linear annotation effort. The TempEval-2013 data are annotated according to
the TimeML specification, which provides required information including
annotated events, time expressions and normalized time values. We first
followed the event time schema to specify an annotation guideline and re-
annotated of the TBET corpus. Then, we revised our guideline to annotate the
time anchors of all the events in TB (147 articles, excluding 36 TBET
articles), AQ and TE3-Test. Our large event time corpus and annotation
guideline will be released soon.333https://github.com/rcfsd/LETC

Table 3 shows the Inter-Annotator-Agreements (IAAs) of our annotation of TBET,
TB and AQ. We use Krippendorff’s $\alpha$ with the nominal metric to compute
the complete match. Both TB and AQ shows the similiar $\alpha$ scores around
$0.55$ to $0.56$, which is slightly lower than $\alpha=0.58$ in TBET.

##  6 Experiments

###  6.1 The Effect of Larger Training Data

Model | Links | Event Time Acc.
---|---|---
baseline | E-D | 17.4%
baseline | E-T | 7.2%
baseline | E-D + E-T | 20.8%
SR attention classifier | E-D | 21.2%
SR attention classifier | E-T | 8.9%
SR attention classifier (Proposed) | E-D + E-T | 24.6%
Table 4:  The performance of our models trained with LETC, evaluated on TE-
Test.

The
TempEval-2013444https://www.cs.york.ac.uk/semeval-2013/task1/index.php%3Fid=data.html
data consists of two training corpora (TB, AQ) and one test data TE3-Test. As
TBET is based on a subset (36 articles) of TimeBank (183 articles), ’TB’
stands for the other 147 articles in TB excluding TBET. We treat TBET as the
baseline and gradually add AQ, TB and AQ + TB into the train data. The
increase of the training data is expected to effectively improve the
performance of the SR classifiers. We separately evaluate the classification
results on TBET-Test (9 articles), TE3-Test (20 articles). In the experiments
on TBET-Test, we use the same data split as the previous
work555https://github.com/nchambers/caevo/blob/master/src/main/java/caevo/Evaluate.java.
In the experiments on TE3-Test, we randomly split 20% training data as the
validation set. The validation set is used to optimize the hyper-parameters
and perform early stopping.

Figure 4 shows the classification performance against different training data
size. We find TBET performs surprisingly well on TBET-Test. The increase of
other corpora even reduces the classification performance. It’s possibly
because the 36 documents of TBET was manually selected with the close
published dates and share some overlapped information between the train and
test split. However, in the evaluation on the larger TE3-Test, TBET obtains
the lowest scores. As the training data size increases, the classification
performance is significantly improved. We believe TE3-Test can more correctly
reflect the generalization ability of the classification, because the article
topics are less dependent to the training data and test data size is larger.
The overall classification scores in TBET-Test are objectively higher than
TE3-Test. However, these number are not comparable due to the difference of
the test data.

Link Type | Acc. | Link
---|---|---
TE3_Test
E-D | 48.3% | 1.0
E-D + E-T (sw=0) | 63.1% | 1.4
E-D + E-T (sw=1) | 70.9% | 1.9
E-D + E-T (sw=2) | 73.5% | 2.5
TBET_Test
E-D | 64.6% | 1.0
E-D + E-T (sw=0) | 70.4% | 1.4
E-D + E-T (sw=1) | 77.8% | 2.1
E-D + E-T (sw=2) | 78.4% | 2.7
Table 5:  The oracle test of different Event-to-Timex sentence window
settings. ’sw’ denotes the sentence distance between an event and a Timex.
’Link’ column denotes the average number of links (E-D and E-T) for each
event.

###  6.2 Oracle Test with Gold SRs

SRs between an event and a Timex are computed based the two qradruple vectors.
This method has the capability of inducing SRs between an event to a Timex
with long distance, which means our system can collect the Event-to-Timex
information as much as we want. However, in reality, the overall performance
will be a trade-off between the amount of Event-to-Timex information and the
classification accuracy. More SRs will bring more errors produced by the
Event-to-Timex classifier.

We perform the oracle tests in two test data, which let our system accept gold
SRs with different distance settings between events to Timex. It helps us to
select a reasonable sentence window. Table 5 shows that the Event-to-Timex
information in the adjacent sentences can achieve 70.98% (TE3_Test) and 77.81%
(TBET_Test) upper-bound accuracy and larger sentence window will significantly
increase the number of E-T links. Our SR classifiers are set to provide the
information of Event-to-DCT and Event-to-Timex in two adjacent sentences for
the inference strategy.

Table 5 also reveals that Event-to-DCT in TE3_Test contributes approximate 16%
less than TBET_Test to the event time prediction. Considering the relatively
lower classification scores of TE3_Test in Figure 4, we can expect that the
event time prediction scores of TE3_Test will certainly lower than TBET_Test a
lot. And of course, the classification and event time prediction performance
of two test data are not comparable.

###  6.3 Event Time Prediction in LETC

Link | $\bf SR_{1}$ | $\bf SR_{2}$ | $\bf SR_{3}$ | $\bf SR_{4}$ | Comp.
---|---|---|---|---|---
E-D | 71.0 | 63.2 | 71.2 | 63.2 | 32.4
E-T | 52.7 | 61.5 | 59.6 | 58.7 | 55.2
Table 6:  The detailed classification F1scores of our proposed model trained
on LETC.

We perform the experiment of training the SR classifiers with the LETC (TBET +
AQ + TB) corpus and evaluate the performance in TE3-Test.

We first report the classification peformance of each SR and ’complete match’
in Table 6. The Event-to-DCT results show very similar scores between $SR_{1}$
and $SR_{3}$, $SR_{2}$ and $SR_{4}$. This is because DCT is always a _Single-
Day_ and _Certain_ time anchor. It’s quadruple vector has the same begin and
end points.

In Table 4, we report the performance of predicting event time. We separately
feed the predicted SRs (E-D, E-T) into the inference strategy. The last column
shows the ’complete match’ accuracy of event time prediction. We provide a
baseline model for observing the effect of the attention, which feeds the word
and position embeddings into a Bidirectional LSTM model and feed the last
hidden state into the Softmax layer.

Event-to-DCT contributes to the main performance for predicting event time,
which is matching the observation from the oracle test (Table 5). Our mention-
wised attention model significantly outperforms the baseline, which used the
same word and position embedding inputs.

###  6.4 Comparing the SR classifier to SOTA

Model | Acc.
---|---
Decision Tree Reimers et al. (2018) | 41.6%
SR Classifier (Proposed) | 43.2%
SR Classifier w/o attn | 40.9%
Table 7:  The comparison to the SOTA model trained on TBET.

We compare our model to the SOTA decision tree model trained on the re-
annotated TBET. The experiment follows the exactly same data split as the
previous work: 22/5/9 articles as the train/dev/test data. Table 7 shows that
our mention-wise attention classifier outperforms the decision tree model
trained on the same data. The performance of the w/o attention model slightly
lower than the decision tree model.

Our system is relatively lightweight compared to the decision tree model, in
which each decision node relies on a classifier, while our system consists of
only two classifiers: Event-to-DCT and Event-to-Timex.

##  7 Conclusion

In this paper, we proposed a simple but effective approach to address the
event time prediction task. The existing model was hampered by the discrete
time anchor representations and _Uncertain_ information, which increased the
complexity of requiring the model to make hierarchical decisions. We first
proposed a unified quadruple representation of time anchors. For preventing
explicitly extending new temporal relation types to include the _Uncertain_
information, we proposed an idea of decomposing a temporal relation into four
sub-level relations (SRs). Any complex relation can be derived as a
combination of SRs. In the second step, we proposed a simple event time
prediction system, which is composed of two mention-wised attention LSTM
classifiers and an inference strategy. Our third contribution is the
construction of a larger event time corpus of 256 news articles, compared to
the existing 36 TBET corpus. The empirical results showed our proposed method
outperformed the decision tree model with a more lightweight structure. Other
experiments show that as the increase of training data size, the
classification performance is significantly improved in a public test data:
TE3-Test.

This work leaves the space of improvement from several aspects. For instance,
the current SRs adopt a coarse definition as the relation between the
begin/end points between two time anchors. A fine-grained sub-level relation
could be defined as a relation between one of the quadruple values, which will
raise the number of SRs to 16 ($4\times 4$). It can possibly encode more
accurate temporal information. Also this work follows the traditional temporal
relation classification setting, which trained the Event-to-DCT and Event-to-
Timex classifiers independently. Intuitively, Event-to-DCT and Event-to-Timex
share the common useful information to be trained jointly or be optimized
globally based on the event time prediction score.

## References

  * Allen (1990) James F Allen. 1990.  Maintaining knowledge about temporal intervals.  In _Readings in qualitative reasoning about physical systems_ , pages 361–372. Elsevier.
  * Berberich et al. (2010) Klaus Berberich, Srikanta Bedathur, Omar Alonso, and Gerhard Weikum. 2010.  A language modeling approach for temporal information needs.  In _European Conference on Information Retrieval_ , pages 13–25. Springer.
  * Cassidy et al. (2014) Taylor Cassidy, Bill McDowell, Nathanael Chambers, and Steven Bethard. 2014.  An annotation framework for dense event ordering.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 501–506, Baltimore, Maryland. Association for Computational Linguistics.
  * Chambers et al. (2014) Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014.  Dense event ordering with a multi-pass architecture.  _Transactions of the Association for Computational Linguistics_ , 2:273–284.
  * Cheng and Miyao (2017) Fei Cheng and Yusuke Miyao. 2017.  Classifying temporal relations by bidirectional lstm over dependency paths.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–6, Vancouver, Canada. Association for Computational Linguistics.
  * Cheng and Miyao (2018) Fei Cheng and Yusuke Miyao. 2018.  Inducing temporal relations from time anchor annotation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1833–1843. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Llorens et al. (2015) Hector Llorens, Nathanael Chambers, Naushad UzZaman, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015.  Semeval-2015 task 5: Qa tempeval - evaluating temporal information understanding with question answering.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 792–800, Denver, Colorado. Association for Computational Linguistics.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Meng et al. (2017) Yuanliang Meng, Anna Rumshisky, and Alexey Romanov. 2017.  Temporal information extraction for question answering using syntactic dependencies in an lstm-based architecture.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 887–896, Copenhagen, Denmark. Association for Computational Linguistics.
  * Minard et al. (2015) Anne-Lyse Minard, Manuela Speranza, Eneko Agirre, Itziar Aldabe, Marieke van Erp, Bernardo Magnini, German Rigau, and Ruben Urizar. 2015.  Semeval-2015 task 4: Timeline: Cross-document event ordering.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 778–786, Denver, Colorado. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Pustejovsky et al. (2003) James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003\.  The timebank corpus.  In _Corpus linguistics_ , volume 2003, page 40.
  * Reimers et al. (2016) Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2016.  Temporal anchoring of events for the timebank corpus.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2195–2204, Berlin, Germany. Association for Computational Linguistics.
  * Reimers et al. (2018) Nils Reimers, Nazanin Dehghani, and Iryna Gurevych. 2018.  Event time extraction with a decision tree of neural classifiers.  _Transactions of the Association of Computational Linguistics_ , 6:77–89.
  * Setzer (2002) Andrea Setzer. 2002.  _Temporal information in newswire articles: an annotation scheme and corpus study._ Ph.D. thesis, University of Sheffield.
  * UzZaman et al. (2012) Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2012.  Tempeval-3: Evaluating events, time expressions, and temporal relations.  _arXiv preprint arXiv:1206.5333_.
  * Verhagen et al. (2009) Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Jessica Moszkowicz, and James Pustejovsky. 2009.  The tempeval challenge: identifying temporal relations in text.  _Language Resources and Evaluation_ , 43(2):161–179.
  * Verhagen et al. (2010) Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010.  Semeval-2010 task 13: Tempeval-2.  In _Proceedings of the 5th international workshop on semantic evaluation_ , pages 57–62. Association for Computational Linguistics.
  * Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014.  Relation classification via convolutional deep neural network.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2335–2344. Dublin City University and Association for Computational Linguistics.
  * Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016\.  Attention-based bidirectional long short-term memory networks for relation classification.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 207–212, Berlin, Germany. Association for Computational Linguistics.