11institutetext: University of Bayreuth, Germany
11email<EMAIL_ADDRESS>

# Beyond Rule-based Named Entity Recognition and Relation Extraction for
Process Model Generation from Natural Language Text

Julian Neuberger  Lars Ackermann  0000-0002-6785-8998 Stefan Jablonski

###### Abstract

Automated generation of business process models from natural language text is
an emerging methodology for avoiding the manual creation of formal business
process models. For this purpose, process entities like actors, activities,
objects etc., and relations among them are extracted from textual process
descriptions. A high-quality annotated corpus of textual process descriptions
(PET) has been published accompanied with a basic process extraction approach.
In its current state, however, PET lacks information about whether two
mentions refer to the same or different process entities, which corresponds to
the crucial decision of whether to create one or two modeling elements in the
target model. Consequently, it is ambiguous whether, for instance, two
mentions of data processing mean processing of different, or the same data. In
this paper, we extend the PET dataset by clustering mentions of process
entities and by proposing a new baseline technique for process extraction
equipped with an additional entity resolution component. In a second step, we
replace the rule-based relation extraction component with a machine learning-
based alternative, enabling rapid adaption to different datasets and domains.
In addition, we evaluate a deep learning-approach built for solving entity and
relation extraction as well as entity resolution in a holistic manner.
Finally, our extensive evaluation of the original PET baseline against our own
implementation shows that a pure machine learning-based process extraction
technique is competitive, while avoiding the massive overhead arising from
feature engineering and rule definition needed to adapt to other datasets,
different entity and relation types, or new domains.

###### Keywords:

Process Extraction from Text, NLP, Named Entity Recognition, Relation
Extraction, Entity Resolution

##  1 Introduction

Automated generation of formal business process models from natural language
process descriptions has become increasingly popular [8, 14, 5, 1, 2]. This is
motivated, for instance, with the comparatively high time expenditure for
manually designing said process models (up to 60% of the total time spent in
process management) [8]. Techniques for automated process model generation
from natural language text are confronted with solving several sub-tasks that
can be categorized into two distinct phases: (i) The information extraction
phase and (ii) the process model generation phase. During the information
extraction phase, techniques recognize process elements (e.g. activities,
actors, data objects), extract relations (e.g. sequence-flow relations between
activities), and resolve references (e.g. mentions of the same data object).
Building on this information, the process model generation phase creates a
concrete process model [8, 14, 10].

Fig. 1: Outline of the process model generation task, given natural language
process descriptions as input. Compared to the existing baseline by [5], our
work adds an entity resolution module, which facilitates generation of better
process models in the process model generation phase.

Resolving references between mentions of the same process element – we will
call this task entity resolution from now on – is crucial. Consider, for
instance, the following two subsequent sentences: “ _After a claim is
registered, it is examined by a claims officer. The claims officer then writes
a settlement recommendation._ ”. For a human reader it is obvious, that both “
_a claim_ ” and “ _it_ ” refer to the same instance of a _claim_ , but to
automatically extract a process model encoding this knowledge the employed
technique needs to recognize this, too. Considering the _claim_ example,
without recognizing references to entities, at least two problems manifest in
the extracted process models: (i) Two distinct data objects for _claim_ would
be created and, thus, the model is not able to correctly express that both the
registration and the examination activities process the same data and (ii) one
of the created data objects has to be named _it_ , because it is unknown that
_it_ is a reference to _a claim_. See Figure 1 for a visual example of the
differences in generated process models. Though the _claim_ example is solely
focusing on the data perspective, entity resolution is also necessary for
organizational process elements like, for instance, actors. Here, it is
necessary to be able to create process models that contain a single actor type
for the two mentions of the _claim officer_ , which is expressed as a single
swimlane in BPMN, for instance. In summary, it can be said that entity
resolution is what makes it possible in the first place to correctly express
relations to data and to actors.

Currently, the most extensive dataset of textual process descriptions (PET
[5]) and the corresponding baseline information extraction technique only
support process element extraction and relation extraction but lack support
for entity resolution. Hence, our work enriches the dataset with information
about process element references and extends the baseline technique to
recognize these references. Extending our running example with a sentence
starting with “ _A copy of the claim …_ ”, we argue that resolving entities
based on their surface forms alone, i.e. their actual text (here: _claim_), is
not precise enough for high quality process information extraction (see
Section 5.1).

In order to increase the adaptability of our approach to new process elements,
domains or data sets, it is further recommended [11] to replace the rule-based
component used in [5] for relation extraction with an approach based on
machine learning. Furthermore, including entity resolution bridges the gap to
an established subfield of Natural Language Processing (NLP), which researches
generalist deep learning techniques for solving tasks like PET in a holistic
manner. By training a model jointly to solve multiple consecutive prediction
tasks such approaches avoid the so called _error propagation issue_ , which
refers to the accumulation and amplification of errors as they propagate
through a pipeline of separated processing steps (for details we kindly refer
to [15, 3]).

By bringing in the subtask of entity resolution and the consideration to
either build a machine learning-only pipeline approach for solving the
combined task or resorting to a holistic deep learning-based technique a
variety of research questions arise and we focus on the following:

  1. RQ1

To what extent can references among process elements be resolved based on
similarities in their surface form?

  2. RQ2

Can we create a machine learning-based component for solving the relation
extraction task with at least equivalent quality?

  3. RQ3

How does a holistic, generalist approach for process extraction perform
compared to the pipeline-based approaches investigated in RQ1?

  4. RQ4

Which characteristics of the PET dataset challenge the extraction approaches
most?

The remainder of this paper is structured as follows: In Section 2 we
formalize the task of process model extraction. In Section 3 we discuss
differences to work related to this paper. Our thorough investigation of the
PET dataset and the extraction approaches in Section 5 is based on a rigorous
experiment setup introduced in Section 4. Short summaries of the answers to
our research questions are provided in Section 6. Both the source code for our
experiments and the extended dataset are publicly available111In case of
acceptance, it will be published on GitHub, therefore laying the foundation
for further focused research.

##  2 Task description

Natural language processing is a discipline that aims to exploit natural
language input data and spans a wide variety of subfields. One of these
subfields is Information Extraction from human-readable texts. In the
following, we describe the extraction of process entities and of relations
between them as specializations of three sub-problems of information
extraction, which are named entity recognition, relation extraction, and
entity resolution. Since powerful generalist and sometimes holistic approaches
exist for each of these subproblems, this problem description serves as a
bridge for the study of existing NLP approaches. We then detail the three
subproblems with respect to the PET dataset that is extended as described in
Section 4.1.

Named entities are tokens (e.g. words, punctuation marks) or a _spans_ of
tokens, where each uniquely identifies one element from a set of elements
[11]. Although the term suggests that these tokens are usually proper nouns,
what is considered a named entity depends on the domain [18]. For the process
domain these are, for instance, actors described by common nouns or proper
nouns (e.g. _the CEO_ vs. _Max_) or actions usually described by verbs or
substantiated verbs (e.g. _approve_ vs. _the approval_). For the remainder of
the paper we call a token or token span that refers to an entity _mention_.
Consequently, _named entity recognition_ is the task of locating and
classifying mentions in a given natural language text. We formalize this task
according to [11, 20]: A natural language text of length $N$ is a sequence of
tokens $x=[x_{i}]$ with $1\leq i\leq N$. Applying NER to this sequence results
in a set of tuples $m$ where each tuple has the form $(M_{s},M_{e},t)$ with
$M_{s},M_{e}\in[1,N]$, where $M_{s}$ and $M_{e}$ are token-related start and
end indexes of a mention. Each such mention refers to an entity of type
$t\in\pazocal{T}$, where $\pazocal{T}$ represents all possible types and is
called _tag set_. We denote the type of entity $e$ as $t(e)$. The annotation
schema used for PET contains the following tag set $\pazocal{T}$ for named
entities [5]: An _Activity_ describes a task that can be performed by an
_Actor_ and often operates on _Activity Data_. An Activity may be detailed by
_Further Specification_ s. Additionally, the tag set covers two gateway types
(_XOR Gateway_ and _AND Gateway_), which can be combined with _Condition
Specification_ describing rules for entering the outgoing branches.

Mention clusters are multiple mentions that refer to the same entity. In
textual process descriptions there may be examples like the following: _The
author writes an article. He then submits the article for review._ There are
two mentions of the entity _author_ : _The author_ and a pronoun _he_ , which
is a linguistic reference. Thus, _entity resolution_ is the task to form a set
of _clusters_ (C) of mentions, where two clusters
$c=\\{e_{1},e_{2},...e_{x}\\}$ and $d=\\{e_{1},e_{2},...e_{y}\\}$ with
$e_{x},e_{y}\in m$ and $c,d\in C$ with $c\neq d$ neither overlap nor are
empty, i.e. $c\cap d=\emptyset$ and $c,d\neq\emptyset$. Further, given a
mapping $f:C\rightarrow m$ from each cluster to the set of mentions it
consists of, we say that each cluster corresponds to a single entity: $\forall
e_{x},e_{y}\in f(c):t(e_{x})=t(e_{y})$, and $f(c)\neq f(d)\ \forall\ c\neq d$.
Mention clusters with more than one element only exist for tags Actor and
Activity Data. A rationale for this decision is given in our description of
the extended dataset, see Section 4.1.

Relations correspond to semantic categories that describe the interaction
between entities [4]. As described above, mentions are spans referring to
entities, and thus relation extraction is needed to infer this interaction
based on the entities’ textual context. For the process domain, typical
relations are, for instance, _performer_ (i.e. who is the performer of an
activity) or _producer/consumer_ (i.e. which activity produces or consumes a
certain data object). For this paper, we only consider _binary relations_ ,
given the structure of the dataset (Section 4.1). Binary relations $r_{b}$
have two participants called _head_ and _tail_ (sometimes: _target_ and
_source_). Their order indicates the direction of a relation. Let
$r_{b}=(c,u,d)$ be a directed, binary relation with $c,d\in m$, then $c$ is
the head, and $d$ is the tail, and $u$ is one of the possible relation types
$\pazocal{R}$. Consequently, resulting relations are described on entity-
level, which is useful in the context of process model extraction, since, for
instance, all relations an actor is involved in are represented directly.
Relation types $\pazocal{R}$ covered by PET allow for describing which Actors
perform an activity (_Actor Performer_) or receive results of an activity
(_Actor Recipient_). Data processing by activities can be expressed using the
_Uses_ relation type. A _Sequence Flow_ describes an ordering of behavioral
elements (Activities, Gateways, Condition Specification). _Same Gateway_
connects information about a single Gateway that are scattered over multiple
sentences. Finally, a _Further specification_ connects an Activity with its
Further Specification.

##  3 Related Work

This paper is founded on the work presented in [5] and is therefore closely
related, as we use, extend, and analyze the data, as well as re-use the
baselines in solving the extended business process extraction task. They are
missing a more in-depth description of their data, especially regarding
qualities important for prediction performance, including but not limited to:
correlation between a relation’s type and its argument types or the linguistic
variability of their data. Furthermore, the implementation of their original
baseline is not publicly available, impeding further development of new
process extraction tools.

There are several approaches related to the baselines we present and analyze
in this work. An annotation approach based on Tregex pattern matching across
the dependency tree representation of a textual process description is
presented in [16], which is then used to generate an event log. This allows
the extraction of a formal process model via established process mining
techniques. While it achieves state-of-the-art results, it uses a tagging
schema different from the one used in PET, which makes it unfeasible for use
in a direct comparison. [8] presents a pipeline able to extract formal process
models in Business Process Management Notation, and therefore is locked into
this process notation language. The choice of process elements tagged in PET
allows free choice of imperative or declarative process modeling languages. A
direct comparison is therefore awkward. In [13] a neural method for entity and
relation classification is proposed, but assumes that relevant text fragments
are already extracted. This is a significantly easier task, since part of the
difficulty in generating business process models is separating relevant
process information from fluff.

Due to the strong relation between process extraction and the combined NLP
task of entity extraction, relation extraction, and entity resolution, there
are several generalist approaches potentially able to solve the process
extraction task in a holistic manner [6, 9, 7, 17]. However, we chose Jerex
[7], since [6] and [9] predict mentions as surface forms only, meaning the
span of text containing them might be ambiguous, and therefore token indices
not resolvable. This violates our definition of mentions (Section 2) and
hampers the evaluation of the predictions. Furthermore, [17] predicts mention-
level relations and, thus, differs from the task as we defined it (Section 2),
i.e. it does not predict relations on entity-level.

##  4 Experiment Setup

In the following we describe the extension of the original PET dataset
accompanied with dataset statistics (Section 4.1). Section 4.2 discusses a
baseline technique for solving the process extraction task with both a naive
and a neural entity resolution component as well as a machine learning-based
relation extraction component. To enable empirical evaluation Section 4.3
introduces performance measures that are most adequate for the task and the
concrete dataset.

###  4.1 Dataset

To facilitate the entity resolution task described in Section 2, we add gold-
standard labels222All gold-standard labels are defined by two experts, with
the help of a third for cases, where their initial annotations differed. for
references to _Activity Data_ or _Activity_ entities in the pet dataset. The
reason why we do not consider further elements for entity resolution is that
Further Specifications, Condition Specifications and Gateways lack linguistic
references to other mentions. In contrast, Actors and Activity data usually
have clear identifiers as mentions (e.g. _the CEO_ and _the application
letter_), which enables references like pronouns, but also allows for
referencing by reusing the identifiers directly. 451 mentions of Activity Data
elements were assigned to one of 320 entities, and similarly 439 mentions of
Actors to one of 175 entities. This resulted in 75 Activity Data entities with
more than one mention, which we will call multi-mention entities, and 88 Actor
multi-mention entities. Table 3 provides a fine-grained analysis of our
contribution to this dataset.

In [5] basic statistics for process elements (mentions) in the PET dataset are
presented, which can be found in Table 1, and relations, which can be found in
Table 2. We extend these statistics by an analysis of mention clusters and
present statistics of entities in Table 3. There is a large imbalance in
examples for some process elements, especially Activities (501) and AND
Gateways (8). This fact carries over to the entity data, where there are a lot
of examples for single-mention entities, i.e., Activities (501) and
comparatively few of multi-mention ones, i.e., Actors (175).

|  |  Activity |  |  Further |  XOR |  AND |  Cond.
---|---|---|---|---|---|---|---
|  Activity |  Data |  Actor |  Spec. |  Gateway |  Gateway |  Spec.
absolute count |  501 |  451 |  439 |  64 |  117 |  8 |  80
relative count |  30.16% |  27.21% |  26.43% |  3.86% |  7.04% |  0.48% |  4.82%
per document |  11.13 |  10.04 |  9.76 |  1.42 |  2.60 |  0.18 |  1.78
per sentence |  1.20 |  1.08 |  1.05 |  0.15 |  0.28 |  0.02 |  0.19
average length |  1.10 |  3.49 |  2.32 |  5.19 |  1.26 |  2.12 |  6.04
standard dev. |  0.48 |  2.47 |  1.11 |  3.40 |  0.77 |  1.54 |  3.04
Tab. 1:  Basic analysis for mentions in the PET dataset done by [5] |  |  |  Actor |  Actor |  Further |  Same
---|---|---|---|---|---|---
|  Flows |  Uses |  Performer |  Recipient |  Specific. |  Gateway
absolute count |  674 |  468 |  312 |  164 |  64 |  42
relative count |  39.10% |  27.15% |  18.10% |  9.51% |  3.71% |  2.44%
per document |  15.31 |  10.60 |  6.96 |  3.64 |  1.42 |  0.96
per sentence |  1.65 |  1.14 |  0.75 |  0.39 |  0.15 |  0.10
Tab. 2:  Basic analysis for relations in the PET dataset done by [5] |  Activity |  Activity Data |  Actor |  Further Spec. |  XOR Gateway |  AND Gateway |  Cond. Spec.
---|---|---|---|---|---|---|---
absolute count |  501 |  320 |  175 |  64 |  117 |  8 |  80
relative count |  39.60% |  25.30% |  13.83% |  5.06% |  9.25% |  0.63% |  6.32%
per document |  11.13 |  7.11 |  3.89 |  1.42 |  2.60 |  0.18 |  1.78
per sentence |  1.20 |  0.77 |  0.42 |  0.15 |  0.28 |  0.02 |  0.19
multi-mention |  - |  75 |  88 |  - |  - |  - |  -
median distance |  - |  16 |  32 |  - |  - |  - |  -
average distance |  - |  31.93 |  54.84 |  - |  - |  - |  -
standard dev. |  - |  44.40 |  95.23 |  - |  - |  - |  -
Tab. 3:  Statistics for entities in our extended version of PET. Multi-mention
entities are restricted to mentions with label Activity Data and Actor.

Additionally to the basic dataset statistics, we present a more in depth
analysis to help us better understand challenges in the data. Firstly, we
analyze how strongly the type of a relation and the type of its process
element arguments are correlated. We present this connection via a heat-map
plot in Figure 2. Brighter colors indicate higher numbers of relation examples
in the dataset, where — given a relation type — its arguments are of the type
denoted on the y axis. Most notably, the only relation types, where their type
can not be deduced from the types of its arguments, are Actor Performer, Actor
Recipient, Same Gateway, and Flows. Especially the last two are problematic
from a data balance perspective, as there are eight times more examples for
the Same Gateway than for the Flows relation, i.e., 42 and 5 respectively. In
contrast, there are also relation types, where their type can directly be
inferred from their argument’s types, e.g. all relations that have an Activity
as head argument, and an Activity Data element as tail, are of type Uses. This
is hardly surprising when factoring in domain knowledge, as in PET Activity
Data is only used with Activities.

Fig. 2: Correlation between relation type and type of its arguments. X-axis
shows the type of head (left) and tail (right), while y-axis shows relation
type.

We define the intra-entity distance as the maximum of each mention’s minimal
distance to each other mention in the entity. This gives us the largest span a
extraction method has to reason over, in order to detect two mentions as part
of the same entity. Averaged over all entities this measure is 31.93 tokens
for Activity Data elements and 54.84 tokens for Actors. This mean is biased by
outliers, as can be seen by the median distances of 16 and 32 for Activity
Data and Actors respectively. Calculating a mean from trimmed distance data
(10% on both ends removed) mean and median move closer, indicating less
outliers. In both cases distances between referent mentions are significantly
longer for Actors, indicating that they may be harder to extract.

Calculating the type-token ratio for mentions of a given entity, allows us to
make a prediction on how hard it will be to resolve references between them.
It is calculated as the ratio between unique tokens and total number of
tokens, for tokens in a mention, and tokens in each relation’s arguments. High
ratios imply very diverse phrases used to refer to the entity, while low
ratios mean that references are very uniform. On average, Activity Data
elements exhibit higher type-token ratios compared to Actors, leading us to
assume the latter ones should be easier to resolve from this perspective,
which is in fact supported by our experiments, as can be seen in Figure 6.

(a) relation type-token ratio

(b) mention / entity type-token ratio

Fig. 3: Type-token ratio for relation arguments in 3(a) and mentions in 3(b).
Mentions and entities have the same type-token ratio for the PET dataset,
since all entities contain only mentions of a single type.

###  4.2 Process Extraction Approach

We implemented a baseline for solving the extended business process extraction
task as a pipeline with three distinct modules (steps) The modules and their
function within the pipeline are outlined in Figure 4, a more detailed and
formal description is given in the following, in the order of application in
the pipeline.

Fig. 4: Outline of the business process extraction pipeline, starting from
natural language process descriptions, mention extraction, entity resolution,
and finally relation extraction. For each step an example of extracted
information is given, with later steps producing increasingly structured
information.

For the Mention Extraction module, we adapt the implementation from [5]. The
approach is based on Conditional Random Fields (CRF), a powerful technique for
tagging a sequence of observations, here tokens in a text [19]. Given a
sequence of tokens tagged in this way, we then resolve mentions, where each
mention contains a set of token indices and the predicted process element
type.

The naive entity resolution approach assigns each mention the best matching
other mention, if the overlap in surface form is at least $\alpha_{m}=80\%$.
This way clusters of mentions are created and subsequently resolved to
entities. The neural approach predicts co-referent spans of text, without
using knowledge about process entities or their mentions. We then align these
predictions with mentions predicted in the mention extraction module to
extract entities. Here we discard predictions, if (1) the corresponding span
of text was not identified as a process element, (2) the corresponding span of
text does not overlap with a process element by a certain percentage
$\alpha_{m}$, (3) the mention corresponding with the predicted span of text
was not tagged with the majority tag of other mentions of this entity, or (4)
not at least a certain portion $alpha_{c}$ of mentions for this predicted
entity was previously accepted. We optimize these parameters using a grid
search approach, choosing $\alpha_{c}=0.5$ and $\alpha_{m}=0.8$.

Finally the Relation Extraction module extracts relations between mentions
using CatBoost, a gradient boosting technique for classification using
numerical, as well as categorical data[12]. For each combination of head and
tail mention of a relation we build features containing tags, distance in
tokens and in sentences between them, and a number $c$ of neighboring mention
tags as context. This feature set is then presented to the model, which
predicts a class for it. Classes are the set of relation tags and an
additional nothing tag to predict no relation between two mentions. During
training we present each of the mention combinations containing a relation to
the model exactly once per iteration, as well as a given number of negative
examples. These negative examples only consist of mention combinations, where
corresponding entities do not have a relation. This concept, called negative
sampling, is important, as there are many more mention combinations without a
relation between them (44,708), as there are ones with one (1,916). Without
negative sampling the precision of our relation extraction module would be
extremely low, visualized in Figure 5. For each positive sample we select
$r_{n}$ randomly drawn negative ones. Increasing $r_{n}$ has a positive impact
on $P$, as during training the model has to learn to distinguish between
mention pairs that support a relation between two entities, and those that do
not. Similarly, $R$ is affected negatively, as the model learns it has to
reject some mention combinations, it also inevitably rejects correct
combinations. We then train the module for $i=1000$ iterations, which is the
most computationally intensive step in the whole pipeline, taking about 25
minutes on an Intel i9-9900K CPU @ 3.60GHz, using a negative sampling rate of
$r_{n}=40$ and context size of $c=2$. A sampling rate $r_{n}\geq 40$ improves
the result quality significantly.

Fig. 5: Values of metrics $P$, $R$, and $F_{1}$ for different negative
sampling rates $r_{n}$. For each value of $r_{n}$ in this figure, we train a
model with $c=2$ and $i=100$.

###  4.3 Evaluation

For performance evaluations of existing baselines, as well as our
contributions, we adopted the evaluation strategy from [5]. This means we run
a 5-fold cross validation for the entire pipeline and average individual
module scores. Errors modules make during prediction are propagated further
down the pipeline, potentially even amplifying in severity, as down-stream
modules produce errors themselves as a result333We referred to this in the
introduction as error propagation issue.. To gauge a given module’s
performance in isolation, we inject ground-truth data instead of predictions
as inputs. This leads to a total of six different scenarios, whose results are
discussed in detail in Section 5.1. These scenarios are (1) mention extraction
only, (2) entity resolution using predictions from the Mention Extraction
module, and (3) using ground-truth mentions. Furthermore, (4) relation
extraction using entities predicted by the pipeline thus far, (5) using
entities predicted during entity resolution using ground-truth mentions, and
finally (6) using ground-truth entities.

In each case we use the $F_{1}$ score as a metric, as it reflects the task of
finding as many of the expected mentions, entities, and relations as possible
(recall $R$), without sacrificing precision $P$ in type or existence
prediction. $F_{1}$ is then calculated as the harmonic mean of $P$ and $R$,
i.e., $F_{1}=\frac{P\times R}{P+R}$. As there is more than one class within
each prediction task, $F_{1}$, $P$, and $R$ have to be aggregated. Throughout
Section 5 we use the micro averaging strategy, which calculates $P$ and $R$
regardless of a given prediction’s class. This strategy favours classes with
many examples, as high scores in those may overshadow bad scores in classes
with few examples. Should this be of concern, the macro averaging strategy can
be used, where $P$, $R$, and $F_{1}$ are calculated for each class separately
and averaged afterwards. We argue that it is most useful to find as many
process elements as possible regardless of their type, i.e., it is better to
find 90% of all Activities and only 10% of all AND Gateways, instead of 50% of
all elements, as there are many more Activities than AND Gateways (Table 1).
As such the micro $F_{1}$ score is better suited to the task at hand.

We count a _mention_ as correctly predicted, iff it contains exactly the same
tokens, as the corresponding ground-truth label, and has the same tag. We
count an _entity_ as correctly predicted, iff it contains exactly the same
mentions, as the ground-truth label. Finally, we count a _relation_ as
correctly predicted, iff both its arguments, and its tag match the ground-
truth label. This very strict evaluation regime leads to error propagation,
i.e. a mention that contains a single incorrect token, leads to all entities
that contain it to be classified as incorrect, as well as all relations, that
contain entities containing the incorrect mention. This cascade illustrates,
why we opted for several scenarios that evaluate modules in isolation, or with
some degree of ground-truth input. Future work could analyze, how less strict
evaluation regimes correlate with user’s expectations of predictions.

##  5 Results

The following section reports results for the experiments defined in the
previous section 4. Based on these results, it provides answers for the
research questions posed in Section 1. First we provide a general overview of
results for the baseline pipeline, our approach, and the generalist end-to-end
approach. We then discuss how error propagation influences the quality of
results, by analyzing six distinct scenarios, as defined in Section 4.3.
Finally, we assess different dataset characteristics, concerning their
influence on prediction quality.

###  5.1 Baseline Performance

This section presents results for our experiments and discusses them, starting
with the overall performance of the baseline by [5], combined with our
proposed entity resolution module (Baseline + ER), the jointly trained end-to-
end technique from [7] (Jerex), as well as our proposed additions (Ours). We
then show a detailed comparison of performance between our proposed method and
the baseline, broken down by relation types. To build an intuition of
individual module performance, we evaluate scenarios (1) through (6), as
defined in Section 4.3.

| $P$ | $R$ | $F_{1}$
---|---|---|---
Jerex[7] | 0.20 | 0.27 | 0.22
Baseline[5] \+ ER | 0.35 | 0.31 | 0.32
Ours | 0.36 | 0.30 | 0.32
(a)

| $P$ | $R$ | $F_{1}$
---|---|---|---
Baseline[5] \+ ER | 0.80 | 0.66 | 0.72
Ours | 0.80 | 0.78 | 0.79
(b)

Tab. 4: 4(a): Overall performance for Jerex, the PET baseline, and our
proposed enhanced pipeline. 4(b): Performance of our proposed machine learnt
and the rule-based baseline relation extraction modules in isolation. Fig. 6:
$F_{1}$ scores by mention type for mention extraction via CRF, scenario (1),
as well as for entity resolution on outputs of the CRF mention extraction
module (2), and on perfect entities (3).

Figure 6 a) visualizes the results for mention prediction by the CRF baseline
(scenario (1)). The technique struggles to properly predict mentions of
process entities with very few examples, i.e. AND Gateways (8) and Further
Specification (64). Additionally, mentions with one of these two tags exhibit
the highest type-token ratio, i.e. have the most variety in their surface
forms, which is further detrimental. Results for scenarios (2) and (3) can be
found in Figures 6 b) and c). Both scenarios are concerned with the outputs of
entity prediction, either based on previously predicted mentions (2), or on
ground-truth mentions (3). Sub-figure b) contains the results for the naive
and neural entity resolution modules as well as the results from Jerex. Jerex
cannot be fed ground-truth mentions for its entity prediction, so results can
only be compared to scenario (1) in a).

Considering the process entity resolution modules in isolation, we can answer
research question RQ1: The naive approach is able to resolve entities with
$F_{1}=0.60$, which is not enough for high quality process model extraction,
especially considering error propagation to the relation extraction modules.
We therefore presented an additional neural resolution module, which can solve
the task with $F_{1}=0.66$. This should still be considered only a baseline,
but shows that neural methods trained on out-of-domain data can be useful in
solving this task. Additionally, our proposed neural baseline is able to out-
perform the much larger, and thus more data-intensive Jerex, showing that
small, pre-trained methods are most promising, when working with the PET
dataset.

Fig. 7: a) Results for relation extraction by relation type for scenario (4),
the complete pipeline, b) scenario (5), relation extraction using entities
resolved from perfect mentions, c) scenario (6), relation extraction from
perfect entities.

Figure 7 shows the performance of relation extraction for the complete
pipeline in a) (scenario (4)), using our proposed method, the baseline rule-
based method by [5], and the end-to-end language model based approach Jerex
[7]. It also allows us to answer RQ2, as results of our proposed machine
learning-based method are at least equivalent to the more specialized, rule-
based method. While our method does not out-perform the baseline when running
the pipeline end-to-end (Table 4(a)), it performs nearly the same, with the
exception of the same gateway. However, providing gradually higher-quality
inputs, i.e., less error-prone results of previous components our method
outperforms the baseline significantly, leading us to believe that while our
method is more powerful, it suffers from error propagation even more so than
the baseline does, see Figure 7 b), and c). This points towards a major
difference of our method and the baseline is visible: Our method treats all
relation types equally, i.e. there is no order in which they are applied. The
rule-based method defines several rules, which predict only relations of one
or two types at a time. These rules are then applied in a fixed order, and
rely on the outputs of previous rules, e.g. checking if a potential relation
argument is already involved in other relations of a given type. Same gateway
relations between two gateways are only applicable, if the tail argument is
not only in a flow relation. Since this relation is potentially not yet
present, when our method predicts relations, it has to infer it from
surrounding context, which is much harder and leads to significantly lower
scores regarding prediction of Same Gateway relations. Figure 7 also gives the
answer to RQ3: While Jerex produces promising results (c.f. Table 4(a)), it
can not reach the same level of quality as the baseline, or our proposed
method produces. It notably struggles with predicting Further Specification
relations. We suspect that this is rooted in strikingly varied surface forms
of one of its arguments, the Further Specification entity, as shown in Figure
3(a). Our model greatly profits from better entity predictions, as can be seen
when comparing results for scenarios (4), with when entities are predicted
using perfect mentions (5), and perfect entities are given (6). In both cases
the quality of predictions greatly improves for our model, and noticeably
improves for the baseline rule-based approach. An important step in future
work would therefore be to improve our proposed baseline neural entity
resolution module. Also further studies regarding less strict evaluation is
warranted, as described in Section 4.3, to get a less conservative assessment
of prediction quality.

Fig. 8: Precision over all relations (binned by argument distance). Our
proposed basic relation extraction approach performs slightly better for
short-range relations, the rule-based approach performs slightly better for
long-range relations.

While there is no single clear answer for RQ4, we want to give several
perspectives in the following. First, Figure 8 shows the precision of both the
original baseline from [5] and our approach in predicting relations with a
given distance between their relation arguments. We sorted the 95% quantile of
relations by their distance into five bins and measured the ratio of correct
predictions to incorrect ones. As soon, as the distance between relation
arguments exceeds five tokens, the ratio of correctly predicted relations
drops sharply. Additionally, PET exhibits stark imbalances between the number
of mentions for certain process entities, or number of examples for relations
between them. While this is rooted in the specifics of the BPM domain, it is
an impeding factor in adopting machine learning approaches, as they have to be
specifically geared to dealing with these imbalances. A related characteristic
of PET, are some classes (types of entities, and relations), where examples
have highly varying surface forms. This is a detrimental factor, especially in
extracting mentions of process entities.

##  6 Conclusion and Future Work

In this paper we extend the task of business process extraction as presented
in [5] by entity resolution. We enrich PET with entity information and extend
the baseline by an entity resolution component, showing that naive resolution
solely based on similarities in surface form is not enough for high quality
results. For this reason we propose a baseline approach, using a neural method
pre-trained on out-of-domain data. Motivated by benefits regarding rapid
adaption to new data, domains, or tag sets, we propose a data driven drop-in
replacement for the current baseline for relation extraction on PET. We show
that our proposed method is able to produce equivalent results in the end-to-
end setting, and even outperform the baseline given higher quality inputs. We
show that PET is not yet extensive enough to support training state-of-the-art
approaches from the NLP domain, even though results are promising and indicate
further research may very well be warranted. Finally, we discuss traits of the
PET dataset that are detrimental to the result quality for mention extraction,
relation extraction and entity resolution (e.g., class imbalance and high
linguistic variance).

Our experiments attest to the phenomenon of error propagation, i.e., errors
made in early steps are amplified in later ones. Thus, we plan to incorporate
joint models for extracting mentions, relations, and for resolving process
entities, since they are trained to solve these three tasks simultaneously,
and mitigate the error propagation effect. Additionally, we plan to improve
performance of the entity resolution module, e.g., by incorporating in-domain
knowledge, like mention information from previous steps. Finally, we will
investigate the usability of data augmentation methods for enhancing the
current state of the PET dataset.

## References

  * [1] Van der Aa, H., Carmona Vargas, J., Leopold, H., Mendling, J., Padró, L.: Challenges and opportunities of applying natural language processing in business process management. In: Proceedings of the 27th International Conference on Computational Linguistics (2018)
  * [2] van der Aa, H., Di Ciccio, C., Leopold, H., Reijers, H.A.: Extracting declarative process models from natural language. In: Advanced Information Systems Engineering: 31st International Conference (2019)
  * [3] Ackermann, L., Neuberger, J., Käppel, M., Jablonski, S.: Bridging research fields: An empirical study on joint, neural relation extraction techniques. In: CAiSE (2023 (in press))
  * [4] Alt, C., Gabryszak, A., Hennig, L.: Tacred revisited: A thorough evaluation of the tacred relation extraction task. In: ACL (2020)
  * [5] Bellan, P., van der Aa, H., Dragoni, M., Ghidini, C., Ponzetto, S.P.: Pet: an annotated dataset for process extraction from natural language text tasks. In: BPM Workshops 2022 (2023)
  * [6] Cabot, P.L.H., Navigli, R.: Rebel: Relation extraction by end-to-end language generation. In: EMNLP (2021)
  * [7] Eberts, M., Ulges, A.: An end-to-end model for entity-level relation extraction using multi-instance learning. In: ACL (2021)
  * [8] Friedrich, F., Mendling, J., Puhlmann, F.: Process model generation from natural language text. In: CAiSE (2011)
  * [9] Giorgi, J., Bader, G., Wang, B.: A sequence-to-sequence approach for document-level relation extraction. In: Workshop on Biomedical Language Processing (2022)
  * [10] Lee, K., He, L., Lewis, M., Zettlemoyer, L.: End-to-end neural coreference resolution. In: EMNLP (2017)
  * [11] Li, J., Sun, A., Han, J., Li, C.: A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering (2020)
  * [12] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. Advances in neural information processing systems (2018)
  * [13] Qian, C., Wen, L., Kumar, A., Lin, L., Lin, L., Zong, Z., Li, S., Wang, J.: An approach for process model extraction by multi-grained text classification. In: CAiSE (2020)
  * [14] Quishpi, L., Carmona, J., Padró, L.: Extracting annotations from textual descriptions of processes. In: BPM 2020 (2020)
  * [15] Ren, F., Zhang, L., Yin, S., Zhao, X., Liu, S., Li, B., Liu, Y.: A novel global feature-oriented relational triple extraction model based on table filling. In: EMNLP (2021)
  * [16] Sànchez-Ferreres, J., Burattin, A., Carmona, J., Montali, M., Padró, L., Quishpi, L.: Unleashing textual descriptions of business processes. SoSyM (2021)
  * [17] Sanh, V., Wolf, T., Ruder, S.: A hierarchical multi-task approach for learning embeddings from semantic tasks. In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)
  * [18] Sharnagat, R.: Named entity recognition: A literature survey. Center For Indian Language Technology (2014)
  * [19] Wallach, H.M.: Conditional random fields: An introduction. Technical Reports (CIS) (2004)
  * [20] Wang, J., Lu, W.: Two are better than one: Joint entity and relation extraction with table-sequence encoders. In: EMNLP (2020)