11institutetext: Pre-print submitted to IEEE Transactions on Services
Computing

J. De Smedt, corresponding author
J. De Weerdt 22institutetext: Research Centre for Information Systems
Engineering
KU Leuven
22email<EMAIL_ADDRESS>33institutetext: J.
Mori, M. Ochi 44institutetext: Graduate School of Information Science and
Technology
University of Tokyo
44email: mori<EMAIL_ADDRESS>
# Constraint Model-Based Predictive Process Monitoring using Recurrent Neural
Networks

Johannes De Smedt  Jochen De Weerdt  Junichiro Mori  Masanao Ochi

###### Abstract

The field of predictive process monitoring focuses on modelling future
characteristics of running business process instances, typically by either
predicting the outcome of particular objectives (e.g. completion (time),
cost), or next-in-sequence prediction (e.g. what is the next activity to
execute). This paper introduces Processes-As-Movies (PAM), a technique that
provides a middle ground between these predictive monitoring. It does so by
capturing declarative process constraints between activities in various
windows of a process execution trace, which represent a declarative process
model at subsequent stages of execution. This high-dimensional representation
of a process model allows the application of predictive modelling on how such
constraints appear and vanish throughout a process’ execution. Various
recurrent neural network topologies tailored to high-dimensional input are
used to model the process model evolution with windows as time steps,
including encoder-decoder long short-term memory networks, and convolutional
long short-term memory networks. Results show that these topologies are very
effective in terms of accuracy and precision to predict a process model’s
future state, which allows process owners to simultaneously verify what linear
temporal logic rules hold in a predicted process window (objective-based), and
verify what future execution traces are allowed by all the constraints
together (trace-based).

###### Keywords:

Predictive process modelling Convolutional recurrent neural networks
Declarative process models

##  1 Introduction

Predictive process monitoring has known a recent surge of interest, fuelled by
the growth of new machine learning applications tailored to big data
solutions. Most notably, the introduction of recurrent neural networks
facilitated a leap in performance for remaining time and next-in-sequence
prediction DBLP:journals/dss/EvermannRF17 ; DBLP:conf/caise/TaxVRD17 . Others
have focused efforts on predicting the outcome of particular objectives of a
process execution DBLP:conf/caise/MaggiFDG14 ;
DBLP:journals/corr/TeinemaaDRM17 , e.g., the acceptance of a loan application,
or the execution of a credit check before starting the review of a loan
application. Both are based on using process execution prefixes, i.e.,
historic evidence of a process typically captured in event logs. The former
targets activity labels and execution time as dependent variables while the
latter whether particular objectives formulated as rules, often linear
temporal logic (LTL) rules between activities and/or data variables, hold as
dependent variable(s).

In this paper, instead of using prefixes of past process execution traces
(from hereof called traces) directly, we capture the underlying process model
generating these past execution traces. By dividing traces into windows,
process models can be generated by using declarative constraints
pesic2006declarative  that capture the relationship between activities within
that window. These constraints are suitable as they can be drawn from an LTL-
constraint base, acting as binary indicators of activity relations being
present in a process model. This facilitates feature generation for predictive
models. The various models present in the windows generated from traces can be
used to extract patterns to make predictions about process models in future
windows that will be enacted by the process. This approach bridges the typical
predictive process monitoring with process model discovery, introducing
predictive process model monitoring. In this sense, our technique is a
generalisation of objective-based monitoring by predicting similar LTL-based
objectives (such as whether an activity will follow another, or whether an
activity will be present altogether), and a generalisation of next-in-sequence
prediction as the usage of windows of 1 activity allows to predict the
presence of just 1 activity. We approach a trace in two ways. Firstly, we
allow for the generation and prediction of process models in windows within a
trace of a fixed length (i.e. the number of activities). Secondly, we allow
for a fixed number of windows per trace (with a potentially varying number of
activities). Both approaches offer a different take on the problem, allowing
analysts to use the most appropriate take on their problem.

Drawing inspiration from research on the analysis of moving images over time,
where convolutional neural networks take care of the high dimensionality of
the input stages (frames) and long short-term memory networks (LSTMs) deal
with the time evolution, Process-As-Movies (PAM) converts processes into a
similar structure that treats windows within traces as high-dimensional
inputs. More specifically, we obtain a high-dimensional representation of
process models by building tensors of activity-activity-constraint relations
which can be fed into and learnt by convolutional long short-term memory
networks DBLP:conf/nips/ShiCWYWW15 , an architecture capable of learning the
interaction effects of said high dimensionality over time, to make
predictions.

This paper contributes in the following ways:

  * •

It introduces a novel approach to capture business processes in tensors by
using expressive behavioural constraints between activities.

  * •

It studies various neural network topologies to use these tensors towards
prediction which allows to generalise next-in-sequence approaches towards
fully-fledged process model prediction.

  * •

Hence, it marries next-in-sequence and goal-oriented predictive process
monitoring approaches in one consolidated technique.

It is shown by an experimental evaluation on two real-life event logs that the
convolutional LSTMs are indeed effective at predicting future process models,
both for windows of fixed length and a fixed number of windows per trace, over
a varying window size.

The paper is structured as follows. Section 2 motivates the research concepts
and provides an overview of related work. In Section 3, an overview of the
concepts used for building the networks is introduced followed by the actual
methodology in Section 4. In Section 5, PAM is verified on real-life event
logs and analysed for its performance. Section 6 summarizes the findings and
discusses points for future work.

##  2 Motivation and related work

In this Section, the previous efforts on predictive process mining are
covered, followed by a motivation of the proposed approach within this
context.

###  2.1 Predictive process monitoring

Process prediction is an important topic for organisations. Among others, it
allows for better preparation in terms of resources and scheduling and it
allows to keep track of KPIs that can be monitored by business constraints. In
the process mining literature, various approaches have been used both for
next-in-sequence prediction, as well as remaining time or time-to-next-
activity prediction. A comprehensive overview of existing predictive process
monitoring techniques has been devised in DBLP:journals/corr/TeinemaaDRM17 ;
DBLP:journals/tsc/Marquez-Chamorro18 .

One of the earlier works on predictive process mining introduced an approach
based on abstractions to obtain finite state machines of a varying complexity
which can be used to predict remaining time DBLP:journals/is/AalstSS11 . In a
related approach, DBLP:journals/kais/LakshmananSDUK15  create Markov models to
predict the next-in-sequence activity in various choice splits in process
models. DBLP:journals/misq/BreukerMDB16  use grammatical inference to create a
probabilistic model on past behaviour to predict next-in-sequence activities.
The work of DBLP:journals/dss/EvermannRF17 ; DBLP:conf/ssci/NavarinVPS17  and
DBLP:conf/caise/TaxVRD17  investigated the usefulness of LSTM neural network
architectures for predicting next-in-sequence activities, and the remaining
execution time based on recurrent neural networks. The introduction of neural
networks caused a leap in predictive performance. The effectiveness of deep
neural networks was researched by DBLP:conf/wecwis/MehdiyevEF17  to predict
the next-in-sequence activity using auto-encoder/decoder networks. Recently,
lin2019mm  introduced extra activity attribute information to improve
prediction which allows not only to predict the next-in-sequence activity, but
also its data attributes such as, e.g., the loan amount of a loan offer
activity or the resource executing the activity, using encoder-decoder LSTMs.
DBLP:conf/icpm/Pasquadibisceglie19  represent traces as images by converting
them into a set of prefixes and employ convolutional neural networks to
predict next-in-sequence activities. An embedding approach for activities,
traces, and process models has been proposed by DBLP:conf/bpm/KoninckBW18
based on random walks to discover the relationship of activities in a process.
These are not used towards predictive analysis, however, could also provide
synergies with the embeddings generated by the convolutional layer of the
LSTMs.

A parallel but related vein of research is the prediction of particular goals
within processes. DBLP:conf/caise/MaggiFDG14  and
DBLP:journals/tsc/Francescomarino19  predict the outcome of business goals
expressed in Linear Temporal Logic (LTL) using decision trees, optionally
supported by a trace clustering pre-processing step. E.g., goals could include
$\Box(\text{accepting order}\to\Diamond\text{send invoice})$ to predict
whether invoices have been adequately managed within an order process, or
$\Diamond(\text{case is accepted})$ to predict whether a case is eventually
accepted for payout in an insurance process. This work has been further
developed in DBLP:conf/bpm/Francescomarino17  in which prior knowledge
regarding the process’ future development is used to improve LSTM predictions.
DBLP:journals/corr/TeinemaaDRM17  introduces various aggregation and encoding
mechanisms for traces as well as various classifiers which can be used for
goal-oriented process prediction. In DBLP:journals/eswa/ChamorroRCT17 ,
decision rules concerning mainly the data variables present in an event log
are generated over multiple windows of an event log to make predictions
regarding future values. These rules are used to evaluate and predict future
key performance indicators.

The various works on mining and monitoring declarative process models and
constraints over time are related as well given that the proposed approach
uses declarative process model constraints. In DBLP:conf/otm/MaggiBCS13 , the
on-line discovery of declarative models was adapted to detect changes in
constraints due to concept drift and DBLP:conf/bpm/BurattinCM14  proposed an
approach to treat processes as movies by tracking the violation status of
declarative process constraints for event streams.

###  2.2 Processes-As-Movies

Typically, it is hard for regular next-in-sequence-based approaches to look
far ahead into the future as illustrated in DBLP:conf/bpm/0001DR19 . This can
be mitigated by using various solutions such as using more prefix information
to detail the surroundings of the prediction DBLP:conf/bpm/0001DR19 ;
DBLP:conf/icpm/Pasquadibisceglie19 . However, alternatively it is possible to
consider a process execution trace in windows or buckets, and making an
abstraction in the form of a process model existing of relations between
activities there is a higher chance of learning at least where a process as a
whole is headed on a different level of granularity. This mimics not just
creating an image, but a movie out of the process in the form of consecutive
representations of a process model. This allows to answer particular
objectives (in LTL) as outlined in Section 2.1. Rather than using predefined
objectives, however, in PAM a multi-activity result is provided in the form of
relations between all activities at once. Consider for example a loan
application process which we base on the well-known example of the 2012 BPI
Challenge111https://data.4tu.nl/repository/uuid:3926db30-f712-4394-aebc-75976070e91f.
It is useful to know what activities are likely to occur next, e.g., whether
an application will be rejected in the future (A_Declined), however, this is a
granular prediction. It is also useful to know what the underlying model is
that generates the next activities. For example, rather than knowing when and
if a loan will be accepted or rejected, it can be revealing to know that there
exists an underlying cause in the form of an activity that has not occurred,
or the occurrence of a particular activity before the claim was decided on. In
Figure 1, the loan application process of the 2012 and
2017222https://data.4tu.nl/repository/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b
BPI Challenge (which will be used in Section 5) is used as an illustration. We
can see that the final window, which is our prediction target, also contains
information about when and how many times the activities are executed
(A_Declined happens first in the window (init), and just once, W_Completeren
aanvraag is executed 3 times and is the last in the window) and in what order
(after each A_Declined, W_Completeren aanvraag has to happen next due to the
chain response constraint).

Figure 1: 5 windows of length 4 in a trace of the 2012 BPI Challenge log and
their corresponding process models.

While previously proposed predictive process monitoring techniques build
powerful sequence modelling approaches, we rather focus on a less-granular
approach based on dividing traces into smaller windows, in this case 5
windows, in which a small process model exists. The underlying idea is to
predict the next relations between activities (materialized as Declare
constraints pesic2006declarative ) that will be generating the model in a
particular part of a trace. This allows process owners to obtain predictions
based on, e.g., whether an application was rejected because of the absence of
a prior investigation (W_Afhandelen Leads), the presence of a particular
preceding check (A_PreAccepted preceding A_Submitted), or the inappropriate
sequencing of events (e.g. an alternating sign-off cycle to adhere the four-
eyes principle) all at the same time. Furthermore, it allows to predict
whether certain constraints which are monitored will be holding in the further
execution of a process, e.g., the chain response constraint in window 5,
offering a predictive alternative to DBLP:conf/bpm/BurattinCM14 .

By making use of the Declare language pesic2007declare , it is possible to
populate the windows with relations between activities, e.g., whether an
activity will happen right after another, whether a particular activity will
occur (a given number of times), and whether particular behaviour will not
occur. Other types of relations could be used as well, e.g., the 4C spectrum
DBLP:conf/apn/PolyvyanyyWCRH14 , behavioural constraints weidlichPDMW11 , or
incidence matrices weijters2006process . Nevertheless, the Declare base
provides a well-rounded and comprehensive set of patterns that cover various
unary and binary behavioural relationships which has seen a vast number of
successful applications in mining and predicting processes. The relations over
the activity set create a vast feature space, which needs to be adequately
modeled for prediction for which we explore various neural network-based
architectures, most notably convolutional recurrent neural networks used in
the analysis of moving images.

##  3 Preliminaries

In this section, the concept of traces, windows, and neural networks are
introduced.

###  3.1 Event logs, traces, and constraints

Event logs are employed throughout process mining and analysis .

###### Definition 1

A trace $t$ is a sequence of executions $t=\langle
e_{1},e_{2},...,e_{n}\rangle$ of length $n$, where each execution in trace $t$
$e_{it}$ is of a particular type called activities $A$.

I.e., $t\in A^{*},\forall t\in\mathcal{L}$, where $A^{*}$ is the power set or
language of all possible finite executions of activities.

###### Definition 2

An event log $\mathcal{L}$ is a list of traces.

Consider Table 1, which displays an event log containing 5 traces.

ID | Trace |  windows
---|---|---
1 | abaabcdad | ab,aa,bc,dad [t]
2 | abaad | a,b,a,ad
3 | abaadc | a,b,a,adc
4 | cdaad | c,d,a,ad
5 | dabddd | d,a,b,ddd [b]
Table 1: Example of an event log containing traces containing windows.

###### Definition 3

A window is a subsequence of a trace denoted $t_{w}=\langle
e_{1},e_{2},...,e_{m}\rangle\sqsubseteq t$ of subsequent executions, i.e,
$\forall i,i+1\in[1,|t_{w}|],\,\forall j,j+1\in[1,|t|],e_{i}=e_{j}\implies
e_{i+1}=e_{j+1}$. Hence, a trace $t$ can be divided into $n$ windows
$t=\langle t_{w_{1}},t_{w_{2}},...,t_{w_{n}}\rangle$ where $t_{w_{i}}\cap
t_{w_{j}}=\emptyset,\,\forall i,j\in[1,n]$.

In Table 1, the traces are divided in $n=4$ windows.

###  3.2 Neural network topologies

Neural networks allow to model high-dimensional input and output spaces using
an architecture of single or multiple layers of hidden neurons. This
architecture can be used for classification and/or regression, but also to
create latent feature spaces using embeddings mikolov2013distributed , or
auto-encoder/decoders vincent2008extracting . The latent spaces with lower
dimensionality help scale down problems such as word representation or image
analysis. Furthermore, the concept of convolution is used to alleviate the
complexity inherent to high dimensionality by using filters that focus on
particular parts of the feature space, which has been used successfully in
areas such as image recognition and modelling badrinarayanan2017segnet . Time-
based and sequential data can be modelled using recurrent neural networks,
which employ feedback loops between neurons to capture longitudinal
dependencies. Several extensions exist to account for the vanishing gradient
problem such as long short-term memory networks (LSTMs)hochreiter1997long .

Various architectures can be combined as well. Encoder-decoder LSTMs are used
for machine translation DBLP:conf/emnlp/ChoMGBBSB14 , and network evolution
modelling chen2019lstm . Besides, convolutional neural networks are gaining
traction in the field of computer vision. Their capabilities to capture multi-
dimensional data, often needed for moving images in different colour bands
have made them the best-performing choice for image recognition,
classification, and prediction DBLP:journals/cacm/KrizhevskySH17 . They excel
over more straightforward neural network architectures such as a multilayer
perceptron by pooling features (in the case of PAM this would be the
constraint between an activity pair) to neurons to avoid the explosion of
parameters which subsequently makes training vast data sources more
manageable. LSTMs have seen a strong uptake in recent years because of their
better performance in next-in-sequence prediction, used for, e.g., language
modelling and classification, where long-distance dependencies are present.
The combination of the two, convolutional LSTMs (CONVLSTMs) allows for the
joint optimisation of the architecture which has been successfully used for
precipitation prediction and video analysis DBLP:conf/nips/ShiCWYWW15 ;
DBLP:conf/nips/FinnGL16 ; DBLP:conf/cvpr/LiangH15 . Their topology is based on
a sequential input that goes through various convolutional recurrent layers
which allows for the dynamic analysis of high-dimensional data in every layer
of the network and not just in the input and/or output. This avoids that the
input and output layers need to be fully connected over the whole feature
space, and results in a more compact architecture. Similar to other
convolutional neural networks, max pooling with filters is used to capture
smaller parts that contain a particular correlation in the high dimensional
input and relate them to each other throughout the network. The main
architecture (already applied to the representation discussed in Section 4) is
shown in Figure 2.

Figure 2: Overview of the main architecture of convolutional recurrent neural
networks.

For a more detailed description of the architecture, we refer to
DBLP:conf/cvpr/LiangH15  and DBLP:conf/nips/ShiCWYWW15 .

##  4 Methodology and Implementation

The main PAM methodology consists of two main steps. First, the traces in the
event log are featurised to be inputted into the inference mechanism.
Secondly, the resulting transformed event logs can be used as input tensors to
neural networks such as CONVLSTMs.

###  4.1 Feature Generation

The goal of the approach is to capture relationships between activities over
time. To this purpose, we mine for the (binary) presence of a relationship
between activities $\mathbf{R}\in\\{0,1\\}^{A\times A\times C}$, where $C$ is
a set of relation types. $\mathbf{R}_{t}$ denotes the matrix that exists
between activities for trace $t$. To introduce the dynamic aspect, we divide
traces into windows, for which the relations can be denoted $R_{t_{w}}$ for
all traces in $\mathcal{L}$ at various windows. Hence, for every window $w\in
t$ in every trace $t\in\mathcal{L}$, we can obtain a binary vector of length
$|C|$ for every pair of activities denoting the presence of constraint $c\in
C$.

In the following sections, we discuss what type of relations are suitable, and
how traces are divided into windows.

####  4.1.1 Activity relations

To capture the evolution of a process, an appropriate featurization step is
needed which can represent its underlying changes. In process mining, both
procedural and declarative languages are used. Especially Petri nets
murata1989petri  are successfully employed by various well-known algorithms
such as Alpha Miner van2004workflow , or Inductive Miner
leemans2013discovering . However, the relationships between activities in a
Petri net cannot be captured straightforwardly. Due to the use of places and
hidden transitions to incorporate long-distance dependencies and concurrency,
there is no easy mapping possible from model constructs to a fixed set of
variables as activities might be connected through many places or even places
and hidden activity pairs. The same holds for other procedural models such as
causal nets van2011causal , or BPMN BPMN . Other mining algorithms such as
Heuristics Miner weijters2006process  and Inductive Miner, which rely on
dependency graphs or Directly-Follows Graphs (DFGs), do have these activity-
to-activity relationships similar to declarative process models such as
Declare pesic2007declare  or DCR Graphs hildebrandt2011declarative , however,
they lack the expressiveness of the latter languages. Given the widespread
support and the ability to efficiently mine Declare constraints using
de2019mining , we opt to use them to capture the relations between our
activities. An overview of the constraints expressed in LTL and regular
expressions used can be found in Table 2. The benefit of using Declare
constraints is that they cover both unary and binary relations, meaning they
can also cover the behaviour of a single activity. Furthermore, the
constraints’ automata can be converted into Petri nets if desired
prescherdeclarative ; desmedt2015ri .

Table 2: An overview of Declare constraint templates with their corresponding LTL formula and regular expression. Template |  LTL Formula pesic2008constraint  |  Regular Expression westergaardunconstrainedminer
---|---|---
Existence(A,n) |  $\Diamond(A\wedge\bigcirc(existence(n-1,A)))$ |  .*(A.*){n}
Absence(A,n) |  $\neg existence(n,A)$ |  [^A]*(A?[^A]*){n-1}
Exactly(A,n) |  $existence(n,A)\wedge absence(n+1,A)$ |  [^A]*(A[^A]*){n}
Init(A) |  $A$ |  (A.*)?
Last(A) |  $\Box(A\implies\neg X\neg A)$ |  .*A
Responded existence(A,B) |  $\Diamond A\implies\Diamond B$ |  [^A]*((A.*B.*) $|$(B.*A.*))?
Co-existence(A,B) |  $\Diamond A\impliedby\Diamond B$ |  [^AB]*((A.*B.*) $|$(B.*A.*))?
Response(A,B) |  $\Box(A\implies\Diamond B)$ |  [^A]*(A.*B)*[^A]*
Precedence(A,B) |  $(\neg B\,UA)\vee\Box(\neg B)$ |  [^B]*(A.*B)*[^B]*
Succession(A,B) |  $response(A,B)\wedge precedence(A,B)$ |  [^AB]*(A.*B)*[^AB]*
Alternate response(A,B) |  $\Box(A\implies\bigcirc(\neg A\,U\,B))$ |  [^A]*(A[^A]*B[^A]*)*
Alternate precedence(A,B) |  $precedence(A,B)\wedge\Box(B\implies\bigcirc(precedence(A,B))$ |  [^B]*(A[^B]*B[^B]*)*
Alternate succession(A,B) |  $altresponse(A,B)\wedge precedence(A,B)$ |  [^AB]*(A[^AB]*B[^AB]*)*
Chain response(A,B) |  $\Box(A\implies\bigcirc B)$ |  [^A]*(AB[^A]*)*
Chain precedence(A,B) |  $\Box(\bigcirc B\implies A)$ |  [^B]*(AB[^B]*)*
Chain succession(A,B) |  $\Box(A\iff\bigcirc B)$ |  [^AB]*(AB[^AB]*)*
Not co-existence(A,B) |  $\neg(\Diamond A\wedge\Diamond B)$ |  [^AB]*((A[^B]*) $|$(B[^A]*))?
Not succession(A,B) |  $\Box(A\implies\neg(\Diamond B))$ |  [^A]*(A[^B]*)*
Not chain succession(A,B) |  $\Box(A\implies\neg(\bigcirc B))$ |  [^A]*(A+[^AB][^A]*)*A*
Choice(A,B) |  $\Diamond A\vee\Diamond B$ |  .*[AB].*
Exclusive choice(A,B) |  $(\Diamond A\lor\Diamond B)\wedge\neg(\Diamond A\land\Diamond B)$ |  ([^B]*A[^B]*) $|$.*[AB].*([^A]*B[^A]*)

####  4.1.2 Windows

A crucial part of obtaining enough information to make predictions towards
future process execution relies on the amount of information that is
available, and especially how it is structured when fed to predictive models.
To this purpose, various approaches have been introduced for trace bucketing,
sequence and prefix extraction and encoding, which have been summarised in
DBLP:journals/corr/TeinemaaDRM17 . The prefix extraction typically relies on
iteratively introducing more information about the prefix as the execution
develops in the system, making the prefixes grow in length. Often, traces are
grouped to ensure models do not have to deal with vast discrepancies in trace
length, or the types of activities present in the trace. For PAM, we use a
window-based approach to capture the behaviour of a process in various stages
of its development. To this purpose we approach a trace in two ways; either a
fixed window size is used, or a fixed number of windows. These serve different
purposes.

A fixed window size allows to collect a number of events and base a future
prediction on the previous windows of the same size. If a predictor was
created for, e.g., windows of length 5, one could predict as soon as 5 events
have happened to predict the process behaviour in the next window of 5 events.
This entails, however, that to deal with different number of windows, e.g., in
case we have already collected 10 events to predict the following window of 5
events, we might have to apply different padding to make the input steps of
the LSTMs of equal length. To this purpose, we will also investigate the use
of different models for a different number of windows of the same size,
similar to DBLP:conf/bpm/LeontjevaCFDM15 . A fixed number of windows on the
other hand allows to divide a trace regardless of whether it has attained a
sufficient length. Given that there is a variety of trace lengths available
for training, and given that they can be used with a pre-trained model for a
particular number of windows, PAM should be able to deal with the
discrepancies that exist between trace length. For example, using a model
trained on traces with a fixed number of windows of 5 allows to use both a
trace of length 4 to predict the fifth execution step, and a trace of length
20 to predict the model in the fifth execution window. Both window-creation
approaches will be experimented with in Section 5, however, the latter
approach is more flexible in terms of requirements on the trace.

Other techniques such as stages in processes DBLP:conf/caise/NguyenDHRM16  can
be used to split traces as well, however, the stages at process level might
not correspond directly with stages in a trace. However, in the case of a
fixed number of windows the use of equal input steps allows for a fair
comparison across various trace lengths as process logs often contain long and
short traces with different activities, and is easy to apply to any trace
where $|t|\geq w$. Besides, it does not spill any information into the
training process. In large event logs, shorter and longer traces will provide
enough evidence of different types of execution patterns to make sure that
windows in traces of different lengths are representative and learned by the
recurrent neural network.

####  4.1.3 Inference

In order to do extract the Declare constraints efficiently, a window-based
version of the interesting Behavioural Constraint Miner (iBCM) de2019mining
is used. This technique mines for Declare constraints in an efficient manner
by making use of basic string operations. Compared to other declarative
process mining algorithms such as Declare Miner maggi2011user  or minerFUL++
di2013two , this approach does not focus on finding a model by making use of
support and confidence over all traces, but rather elicits the constraints per
trace. This result would be roughly similar to using the former two approaches
per trace with a very low support and with 100% confidence, as only activities
present in the traces are considered. Also, iBCM is capable of retrieving the
constraints over a predefined number of windows. Traces are divided in a
number of windows $w$ where $|t_{w_{i}}|=\lceil\frac{|t|}{w}\rceil$. The last
window’s size varies according to the discrepancies between the window and
trace length. E.g., $t=\langle a,b,c,d,e\rangle$ for $w=3$ would be split up
in the following windows: $t_{w_{1}}=\langle a,b\rangle,t_{w_{2}}=\langle
c,d\rangle,t_{w_{3}}=\langle e\rangle$. Note that this is a coarse way of
using the window principle and it would be interesting to pursue a more
tailored split, however, in case a large number of training points are
available the difference in window sizes will be learnt by the model to
overcome this issue.

Since constraints are mined for single traces, the models that can be composed
from generating the product of the separate constraint’s automata, are always
calculable DBLP:conf/bpm/CiccioMMM15 . I.e., constraints found in a single
trace are never conflicting. Hence, the inference step will not have evidence
of inconsistent models and will be capable of producing sound output models if
the resulting neural network is capable of reproducing the original input data
(completely).

Not every constraint listed in Table 2 is as suitable for the envisioned
application. First of all, not chain succession is a constraint which is
satisfied for a high number of activity pairs quickly, as it is a negative
constraint for which counter evidence is scarce. In terms of unary
constraints, only absence(a,1), exactly(a,1/2), and existence(a,3) are used to
limit the size of the feature vectors. For the same reason, succession and its
two other variants are not used because they overlap with other constraints
completely, which renders them redundant (response and precedence). Given
their similarity (on a single trace), only exclusive choice is used and not
not co-existence for the latter would be overlapping with absence constraints.
Note that unary constraints are included as a relationship between the
activity and itself to allow the inclusion in the matrix structure. The
inclusion of the negative constraints absence and not succession are providing
a process construct which is not available in other sequence generation based
approaches using language modelling for next-in-sequence prediction. Although
LSTMs might train the absence of certain events in a particular sequence, PAM
is explicit about this behaviour and provides more insight into the prediction
compared to the former because of the inclusion of the constraint types.

The final result, i.e., the binary vectors denoting which constraints per
activity pair per window are present, is very sparse as a vast number of
possible relations between constraints and activity pairs exist. This makes
the modelling challenging, however, neural networks are suitable to work with
such a sparse high-dimensional input.

###  4.2 Building the Network

Once the presence of all constraints is mined and stored per activity pair,
per trace, i.e., 3D matrices or tensors, a predictive model can be trained
over the data. To do this, we use two LSTM-based topologies capable of
capturing high-dimensional inputs. Various considerations regarding their
setup and parametrisation have to be considered.

Encoder-decoder LSTMs excel especially in the area of sequence-to-sequence
text translation DBLP:conf/emnlp/ChoMGBBSB14 . They map high-dimensional
(typically word) vectors to output sentences, similar to the presence of
constraints in the tensors. They create sequence embeddings by, layer per
encoding/decoding layer, capturing latent dimensions in the input/output time
steps. In the case of PAM, we need to transform the 3D activity-activity-
constraint representation into a flat one. The power of this network topology
is based on the number of latent dimensions that are used before feeding the
encoded data to the LSTM core layer, and how many layers are used to reduce
the dimensionality of the data.

For CONVLSTMs, there are more parameters to consider. First of all, the input
features are split up into out-takes of a lower dimension by using max
pooling. This results in a subset of activity pairs over all constraints. The
size of the filters, as well as how many filters are used to influence the
granularity of the outcome. To make the analogy with frames in a video,
smaller frames are learnt to find, e.g., particular objects which deeper in
the network can be combined to learn a particular larger object. In processes,
this is equivalent to learning a small set of activities, e.g., a block
structure, which might later serve in a particular relationship with other
blocks. Larger filters are capable of capturing concepts moving over time
faster, while smaller ones are better capable of processing slower evolving
information. Given the significant proportion of recurring constraints for a
higher number of windows as will be reported in Section 5, we expect smaller
kernels to work better on less-changing, many windows, and larger kernels to
capture more information when fewer, more different windows are present as
they will appear to be changing faster. Secondly, the number of stacked
CONVLSTM layers can be varied to obtain a deeper architecture. The latter can
be dedicated to smaller parts of the input and can have lower dimensionality.
This increases the expressiveness of the network and subsequently can increase
the (predictive) performance of the network; however, it can also lead to
overfitting. Furthermore, it increases the number of parameters that need to
be learnt, resulting in higher computation times. After each CONVLSTM layer,
batch normalisation is applied to obtain a higher learning rate and to boost
overall performance DBLP:conf/icml/IoffeS15 . The final layer of the network
exists of a 3D sigmoid layer which maps the results to a binary
$|A|\times|A|\times|C|$ output.

Finally, the network’s optimisation parameters need to be chosen, i.e., the
loss function, the optimizer, and the number of epochs, as they have a strong
impact on the results of a (recurrent) neural network. Note that the number
and size of windows affect the performance of the technique significantly as
well. In the extremest case, the window sizes can be set equal to the number
of events in a trace, which makes it possible to create a standard LSTM
similar to the setup of DBLP:conf/caise/TaxVRD17 , as only exactly and absence
constraints will be found in every window. Hence, the network is capable of
making similar predictions.

##  5 Evaluation

In this section the setup and results of the experimental evaluation are
discussed. In addition, the interpretation and impact of constraint types are
analysed. The main goal is to test whether PAM is capable of learning traces
as sequences of windows. More specifically, we investigate whether PAM is
capable of predicting the constraints that will be present in subsequent
windows accurately. As a benchmark, we compare the approach with regular LSTMs
for next-in-sequence activity prediction, by configuring them to predict
windows at the end of traces.

###  5.1 Setup

####  5.1.1 Network topologies and implementation

PAM has been implemented as a feature generation technique in Java, and a deep
learning network architecture in the Keras Python library333https://keras.io/.
The implementations can be found
online444https://github.com/JohannesDeSmedt/processes-as-movies.

As indicated before, two types of recurrent neural network topologies are
used. Firstly, encoder/decoder networks were used to work with the high-
dimensional input. Two hyperparameters were used: the dimensionality of the
input layer, and the number of encoding/decoding layers. The former was set to
powers of 2, i.e., 64, 128, 256, and 512. As will become apparent from the
results, higher dimensionality led to overfitting. Besides, 1-4 layers of
encoding/decoding were used, where every subsequent layer is half of the size
of the former. Secondly, convolutional recurrent neural networks were used
with three hyperparameters; kernel size and filter size are convolutional
neural network parameters, set at 4, 8, and 12, and 1-4 CONVLSTM layers were
used. Note that further hyper-parameter optimisation could still improve the
results, as this is an essential part of the learning process
DBLP:journals/is/Francescomarino18 . The experiments were run both for a fixed
number of windows, and a fixed window length, both set at 2, 5, 10 to obtain
insight in the effect of varying window sizes and lengths. Note that a fixed
number of windows of 2 splits a trace in half and predicts the presence of
constraints in the second half of the trace, which is a hard task for an LSTM
as there is no potential to propagate a lot of past information. The number of
epochs was set at 10 for the fixed number of windows dataset and 20 for the
shorter logs used for the fixed window size as the results did not change
significantly when using higher values (tested at 40 and 60) for either
approach and to make a trade-off in terms of results/performance. Given that
the input dimensionality is high (e.g. for BPI 17 with $|A|=26$ we have time
steps of dimensionality $26\times 26\times 14=9,464$), batch sizes needed to
be sufficiently small (10-20 traces) to fit in memory.

Two optimisers were used, Nadam ruder2016overview  and ADADELTA
DBLP:journals/corr/abs-1212-5701 , both with the binary cross entropy measure
as a loss function. In all cases, Nadam outperformed ADADELTA and is reported
in the results. Activity and kernel regularisation zaremba2014recurrent  were
applied in various layers as well as the final layer, but yielded no better
results and are also not reported.

All models were run on a single NVidia GeForce GTX1070 Ti with 8GB of video
memory with 2,432 CUDA cores. All calculations can be performed on a standard
desktop computing setup within reasonable time. The timings reported are in
seconds for 1 epoch.

####  5.1.2 Data

Two popular publicly available event logs are used, i.e. the 2012 and 2017 BPI
Challenge logs. As illustrated in Section 2, they handle a loan application
process, which consists of opening an application, handling it, and finally
making a decision on its status. The results of applying iBCM, as well as
statistics on the number of activities and traces can be found in Tables 3 and
4. The performance of iBCM in terms of generating the features on the event
logs used in the evaluation section for a fixed number of windows, and a fixed
window size respectively. The results were obtained with a Java 8 Virtual
Machine on an Intel Xeon E3-1230 (v5) CPU with 32GB DDR4 memory. Overall, the
technique is capable of quickly generating a vast amount of constraints
present in various sizes of windows within a trace which can serve as input to
the later inference stage. To get an idea of how similar windows are, the
overlap of recurring constraints between subsequent windows, i.e., windows 1
and 2, 2 and 3, and so on, is listed as well. For the fixed number of windows
version, no extra pre-processing needs to be taken as every trace that has at
least as many elements as the number of windows will be used. This means that
some traces might be too short for, e.g., higher window sizes. This technique
results in both very small and very large windows to be generated. For the
fixed window size version, traces are mined for a number of windows present to
avoid any padding in the inference stage later. The event logs were divided
into subsets where 5 windows are present for window size 2, 2 windows in case
of window size 5, and 1 window in case of window size 10. Only event logs with
at least 2 windows (at least 2 windows are needed to have at least 2 time
steps to train the LSTMs) were considered. Hence, only traces of at least
length 11 are used (to have at least 1 window of 10, and 1 of 1 event).

Overall, there is a significant difference between the event logs in terms of
generated constraints, which is mainly due to the number of traces. In Table
3, we see that the overlap is higher when more windows are used to divide a
trace. Hence, the intuition holds that models tend to be more similar the
closer they are in time. The shorter traces of BPI 2012 also have a higher
overlap than the traces in the BPI 2017 event log. It will be interesting to
see how this affects the results of the predictive models below, i.e., whether
higher overlap leads to higher levels of accuracy/precision. In Table 4, we
see a similar effect: deriving (more) windows of length two results in higher
overlap. The longer the traces and the more windows, the higher the overlap as
well, however, this effect levels off quickly except for long windows (size
10). Again, it will be interesting whether this will affect the modelling
step.

dataset | #traces (min, max, avg.) | #windows | #constraints | #traces too short | overlap |  time (s)
---|---|---|---|---|---|---
|  | 2 | 2,701,992 | - | 0.447 |  $<$1 [t]
BPI 2012 | 13,087 (3, 175, 20) | 5 | 2,486,721 | 3,429 | 0.613 | 1
($|A|=24$) |  | 10 | 2,655,590 | 6,106 | 0.705 |  1 [b]
|  | 2 | 13,546,340 | - | 0.13 |  7 [t]
BPI 2017 | 31,509 (10,180, 38) | 5 | 11,228,795 | - | 0.389 | 6
($|A|=26$) |  | 10 | 13,457,629 | - | 0.619 |  10 [b]

Table 3: An overview of the performance and output of iBCM on the event logs
used for evaluation for a fixed number of windows per trace.

dataset | window length | #windows | #traces | #constraints | overlap |  time (s)
---|---|---|---|---|---|---
BPI 2012 | 2 | 6–10 | 1,351 | 263,584 | 0.618 |  $<$1 [t]
11–15 | 1,847 | 750,706 | 0.644 | 1
16–20 | 1,525 | 826,634 | 0.662 | 1
21–25 | 883 | 589,873 | 0.652 |  $<$1
26–30 | 458 | 385,314 | 0.665 |  $<$1 [b]
5 | 3–4 | 1,351 | 486,917 | 0.351 |  $<$1 [t]
5–6 | 1,847 | 515,539 | 0.521 |  $<$1
7–8 | 1,525 | 537,837 | 0.545 |  $<$1
9–10 | 883 | 377,288 | 0.513 |  $<$1
11–12 | 458 | 237,295 | 0.517 |  $<$1 [b]
10 | 2 | 1,351 | 185,897 | 0.048 |  $<$1 [t]
3 | 1,847 | 516,404 | 0.084 |  $<$1
4 | 1,525 | 495,542 | 0.145 |  $<$1
5 | 883 | 337,116 | 0.198 |  $<$1
6 | 458 | 202,793 | 0.267 |  $<$1 [b]
BPI 2017 | 2 | 6–10 | 2,278 | 623,853 | 0.679 |  $<$1 [t]
11–15 | 10,417 | 4,315,163 | 0.79 | 4
16–20 | 7,274 | 4,155,367 | 0.776 | 6
21–25 | 5,388 | 3,878,768 | 0.775 | 6
26–30 | 3,085 | 2,772,172 | 0.785 | 2 [b]
5 | 3–4 | 2,278 | 486,917 | 0.267 |  $<$1 [t]
5–6 | 10,417 | 3,041,851 | 0.43 |  $<$1
7–8 | 7,274 | 2,703,384 | 0.446 |  $<$1
9–10 | 5,388 | 2,403,113 | 0.448 | 1
11–12 | 3,085 | 1,639,413 | 0.465 |  $<$1 [b]
10 | 2 | 2,278 | 535,864 | 0.02 |  $<$1 [t]
3 | 10,417 | 3,066,199 | 0.063 |  $<$1
4 | 7,274 | 2,588,962 | 0.101 |  $<$1
5 | 5,388 | 2,218,077 | 0.157 |  $<$1
6 | 3,085 | 1,445,214 | 0.226 |  $<$1 [b]

Table 4: An overview of the performance and output of iBCM on the event logs
used for evaluation for a fixed window size.

####  5.1.3 Evaluation criteria

All experiments were performed using a 80%/20% training/test setup with
another 20% validation set during the training epochs. To evaluate the neural
network models, standard binary classification metrics are used to evaluate
whether a constraint is predicted to be present correctly or not. The area
under the precision-recall curve (AP), the F(1)-score at the best threshold on
the precision-recall curve, and the Area Under Receiver operating
characteristics curve (AUC) are calculated to give a wide overview of whether
the network is capable of predicting constraints holding or not holding in a
window (true positives and negatives) compared to falsely predicting the
presence or absence of constraints (false positives and negatives). Given the
sparsity of the matrices, i.e., only a few constraints hold between activity
pairs throughout a particular time window, the accuracy can quickly gravitate
towards very high numbers, affecting even the AUC. Therefore, the AP will be
more revealing in terms of the power of the approach towards predicting the
presence of constraints correctly (all the positive observations).

###  5.2 Results

In this Section, we first compare with a baseline LSTM approach. Next, we
discuss the results stemming from both neural network approaches, and the
effect of their parameterisation.

####  5.2.1 A baseline LSTM approach.

In Table 5, the results of using LSTMs for next-in-sequence prediction
(parameterised similarly to DBLP:conf/caise/TaxVRD17 ) to predict 4, 5, or 10
activities ahead are shown. Then, it is checked whether the constraints
present in this predicted sequence of activities matches the actual
constraints present in the actual sequence. While not a direct comparison with
the PAM rationale, it allows to retrieve the constraints mined from those
windows compared to the actual constraints present, which mimics a window-
based approach. Note that AUC cannot be calculated as we only check for the
presence/absence of constraints without producing a probability. The results
indicate that the recall and precision are low, which is due to the generation
of similar, but slightly different traces, leading to the presence of
different constraints compared to the ones actually present in the original
trace. In comparison, PAM resports F-scores between 70 and 85% for both
datasets at various window settings. It seems that the network topologies used
for PAM are better capable of creating appropriate embeddings based on the
constraints that are more informative towards prediction. As illustrated in
Section 5.3, both positional constraints such as absence, but also precedence
contribute towards the overall result, meaning that sequential relationships
(potentially over long distance dependencies captured by LSTMs) are adequately
captured in the convolutional layer. These results clearly show the added
value of the PAM architecture with respect to the state-of-the-art in the
field.

Dataset | window | Recall | Precision | F-score
---|---|---|---|---
BPI12 | 4 | 0.349 | 0.135 | 0.194
| 5 | 0.325 | 0.126 | 0.181
| 10 | 0.288 | 0.363 | 0.321
BPI17 | 4 | 0.291 | 0.179 | 0.221
| 5 | 0.735 | 0.725 | 0.73
| 10 | 0.51 | 0.196 | 0.283

Table 5: An overview of the performance of finding constraints in a window of
variable sizes at the end of the string as predicted by an LSTM.

####  5.2.2 Predictive accuracy and precision

The results of PAM for both network topologies are included for a fixed number
of windows in Table 6, and for a fixed window size in Table 7.

For a fixed number of windows, we see that convolutional LSTMs typically
perform better on average, with the maximum performance (i.e. the
hyperparameter combination producing the best result) is relatively similar
for both network topologies. Overall, the results for the BPI 17 log, with its
longer traces are higher, with consistently high AUC, and AP up to 92% for 5
windows per trace. For the BPI 12 log the results are lower, with more windows
(10) having the lowest AP, probably due to the small number of activities
present in the windows. In these scenarios (number of windows at 10), the
convolutional LSTMs have much more reliable results. Note also that there is
significantly more training data available for the BPI 17 log. It is
interesting to note that even when dividing a trace in 2 (fixed number of
windows of 2) results in an AP of up to 82% and 88% for the 12/17 logs
respectively, meaning that even from 1 window the LSTMs can learn what
constraints will be present in the second half of the trace.

For a fixed window size, again we see better and more consistent results for
the convolutional LSTMs, which maximal performance again being close with
encoder-decoder LSTMs. For window sizes 5-10, high average precision up to
100% can be achieved, with results being lower for longer traces. This is
likely due to the fact that fewer examples are available to train the
networks, as this is especially prominent for the BPI 12 log where few longer
traces are available. Again, the performance for the BPI 17 logs is stronger,
with average precision reaching 90% on average consistently.

These results can be considered strong overall, as there is a good balance
between AUC and AP, meaning the networks are capable of predicting correct
true positives in a sparse environment without generating too many false
positives as evidenced by the high levels of AP. Overall, the convolutional
LSTMs perform better, but often come at a run-time cost. In general, the
scores are at least comparable or better to the AUC reported for the 2012 and
2017 log as included in DBLP:journals/corr/TeinemaaDRM17  for the prediction
of single objectives.

|  | Convolutional LSTMs |  Encoder-decoder LSTMs
---|---|---|---
event log (#windows) | metric | mean | std | min | med | max | mean | std | min | med |  max
BPI 12 (2) | AP | 0.786 | 0.036 | 0.654 | 0.8 | 0.819 | 0.737 | 0.184 | 0.26 | 0.806 | 0.816
AUC | 0.996 | 0.002 | 0.985 | 0.997 | 0.997 | 0.954 | 0.117 | 0.651 | 0.997 |  0.997
F–score | 0.69 | 0.024 | 0.604 | 0.698 | 0.715 | 0.672 | 0.086 | 0.445 | 0.704 | 0.711
Time | 16.047 | 6.677 | 9.802 | 13.867 | 37.199 | 19.732 | 7.127 | 10.541 | 18.617 | 36.088
BPI 12 (5) | AP | 0.824 | 0.043 | 0.678 | 0.84 | 0.869 | 0.751 | 0.2 | 0.346 | 0.846 | 0.871
AUC | 0.995 | 0.014 | 0.906 | 0.998 | 0.999 | 0.942 | 0.118 | 0.699 | 0.998 |  0.999
F–score | 0.735 | 0.034 | 0.637 | 0.747 | 0.773 | 0.712 | 0.084 | 0.544 | 0.754 |  0.776
Time | 29.447 | 17.28 | 8.986 | 25.602 | 86.716 | 23.716 | 7.428 | 12.423 | 22.919 | 39.13
BPI 12 (10) | AP | 0.725 | 0.068 | 0.542 | 0.751 | 0.797 | 0.569 | 0.215 | 0.279 | 0.629 | 0.803
AUC | 0.988 | 0.022 | 0.873 | 0.995 | 0.997 | 0.881 | 0.153 | 0.676 | 0.992 |  0.997
F–score | 0.651 | 0.047 | 0.547 | 0.668 | 0.708 | 0.589 | 0.087 | 0.493 | 0.561 |  0.71
Time | 43.199 | 28.074 | 10.554 | 36.282 | 134.514 | 30.174 | 10.269 | 14.988 | 28.876 | 51.464
BPI 17 (2) | AP | 0.835 | 0.052 | 0.646 | 0.856 | 0.874 | 0.868 | 0.006 | 0.86 | 0.871 | 0.875
AUC | 0.995 | 0.003 | 0.982 | 0.996 | 0.997 | 0.997 | 0 | 0.997 | 0.997 |  0.997
F–score | 0.766 | 0.044 | 0.613 | 0.784 | 0.798 | 0.791 | 0.006 | 0.783 | 0.793 |  0.799
Time | 43.24 | 19.533 | 23.257 | 40.151 | 103.91 | 48.366 | 17.33 | 26.364 | 45.235 | 93.597
BPI 17 (5) | AP | 0.887 | 0.03 | 0.775 | 0.901 | 0.916 | 0.804 | 0.233 | 0.19 | 0.89 | 0.911
AUC | 0.998 | 0.001 | 0.992 | 0.999 | 0.999 | 0.958 | 0.111 | 0.66 | 0.999 |  0.999
F–score | 0.814 | 0.028 | 0.714 | 0.825 | 0.842 | 0.763 | 0.13 | 0.428 | 0.811 | 0.837
Time | 115.597 | 70.375 | 30.138 | 93.983 | 334.22 | 302.512 | 43.86 | 232.062 | 312.164 | 388.493
BPI 17 (10) | AP | 0.84 | 0.045 | 0.673 | 0.853 | 0.883 | 0.552 | 0.295 | 0.176 | 0.599 | 0.88
AUC | 0.996 | 0.009 | 0.939 | 0.998 | 0.999 | 0.845 | 0.162 | 0.66 | 0.909 |  0.999
F–score | 0.758 | 0.037 | 0.634 | 0.768 | 0.795 | 0.606 | 0.144 | 0.403 | 0.608 | 0.79
Time | 422.039 | 73.139 | 188.542 | 411.632 | 707.495 | 390.521 | 176.489 | 156.858 | 414.479 | 581.096

Table 6: An overview of the performance of both neural network approaches for
a fixed number of windows per trace. The highest average and maximum average
precision (AP) is indicated in grey, the other metrics in bold.

|  |  | Convolutional LSTMs | Encoder-decoder LSTMs |  | Convolutional LSTMs |  Encoder-decoder LSTMs
---|---|---|---|---|---|---|---
event log | #win. | metric | mean | min | max | mean | min | max | Event log | mean | min | max | mean | min |  max
BPI 12 (2) | 6–10 | AP | 0.973 | 0.799 | 0.992 | 0.574 | 0.005 | 0.982 | BPI 17 (2) | 0.96 | 0.559 | 1 | 0.819 | 0.109 |  0.999
AUC | 0.997 | 0.91 | 1 | 0.669 | 0.49 | 1 | 0.964 | 0.463 | 1 | 0.842 | 0.499 |  1
Time | 8.82 | 4.237 | 16.33 | 4.015 | 1.923 | 6.173 | 16.732 | 7.137 | 36.557 | 7.216 | 3.629 |  11.189
11–15 | AP | 0.966 | 0.832 | 0.996 | 0.71 | 0.234 | 0.977 | 0.963 | 0.778 | 1 | 0.724 | 0.301 |  0.999
AUC | 0.999 | 0.978 | 1 | 0.673 | 0.491 | 0.998 | 0.981 | 0.886 | 1 | 0.641 | 0.499 |  0.998
Time | 29.167 | 8.638 | 230.284 | 8.876 | 4.715 | 13.645 | 141.21 | 51.816 | 376.791 | 56.018 | 40.244 |  81.215
16–20 | AP | 0.934 | 0.629 | 0.973 | 0.649 | 0.303 | 0.961 | 0.979 | 0.813 | 1 | 0.715 | 0.168 |  0.998
AUC | 0.994 | 0.923 | 1 | 0.531 | 0.496 | 0.999 | 0.993 | 0.906 | 1 | 0.654 | 0.495 |  0.998
Time | 25.853 | 9.197 | 68.545 | 9.313 | 4.861 | 14.31 | 130.98 | 45.208 | 349.506 | 47.382 | 25.15 |  71.148
21–25 | AP | 0.891 | 0.025 | 0.948 | 0.657 | 0.459 | 0.724 | 0.949 | 0.032 | 1 | 0.645 | 0.255 |  0.869
AUC | 0.993 | 0.804 | 1 | 0.5 | 0.5 | 0.5 | 0.981 | 0.546 | 1 | 0.5 | 0.496 |  0.5
Time | 18.425 | 6.536 | 49.744 | 6.507 | 3.325 | 9.953 | 120.95 | 40.925 | 325.313 | 42.942 | 22.604 |  65.154
26–30 | AP | 0.906 | 0.838 | 0.931 | 0.659 | 0.327 | 0.75 | 0.945 | 0.028 | 0.994 | 0.673 | 0.325 |  0.759
AUC | 0.996 | 0.989 | 0.999 | 0.5 | 0.5 | 0.5 | 0.97 | 0.402 | 1 | 0.5 | 0.5 |  0.5
Time | 43.557 | 4.037 | 1160.43 | 3.935 | 2.002 | 6.1 | 87.738 | 28.326 | 233.279 | 29.611 | 15.747 |  45.045
BPI 12 (5) | 3–4 | AP | 0.941 | 0.853 | 0.953 | 0.877 | 0.564 | 0.948 | BPI 17 (5) | 0.946 | 0.52 | 0.992 | 0.945 | 0.343 |  0.991
AUC | 0.999 | 0.992 | 1 | 0.973 | 0.817 | 1 | 0.977 | 0.74 | 1 | 0.981 | 0.711 |  1
Time | 4.547 | 2.229 | 7.939 | 2.487 | 1.376 | 3.863 | 7.944 | 3.803 | 16.159 | 4.318 | 2.232 |  6.791
5–6 | AP | 0.922 | 0.849 | 0.948 | 0.799 | 0.46 | 0.933 | 0.972 | 0.616 | 0.991 | 0.913 | 0.402 | 0.991
AUC | 0.999 | 0.997 | 1 | 0.962 | 0.773 | 0.999 | 0.992 | 0.802 | 1 | 0.975 | 0.803 |  1
Time | 11.44 | 4.454 | 28.208 | 5.078 | 2.475 | 8.016 | 65.707 | 23.683 | 165.441 | 28.821 | 14.266 |  44.884
7–8 | AP | 0.873 | 0.665 | 0.936 | 0.795 | 0.507 | 0.899 | 0.962 | 0.818 | 0.981 | 0.813 | 0.362 |  0.973
AUC | 0.996 | 0.976 | 0.999 | 0.974 | 0.781 | 0.999 | 0.998 | 0.97 | 1 | 0.947 | 0.775 |  1
Time | 12.307 | 4.385 | 31.236 | 5.137 | 2.508 | 7.873 | 62.294 | 22.007 | 157.082 | 24.881 | 12.399 |  37.663
9–10 | AP | 0.8 | 0.533 | 0.909 | 0.661 | 0.293 | 0.846 | 0.927 | 0.692 | 0.98 | 0.7 | 0.251 |  0.964
AUC | 0.994 | 0.954 | 0.999 | 0.918 | 0.72 | 0.998 | 0.992 | 0.851 | 1 | 0.905 | 0.763 |  1
Time | 8.76 | 3.064 | 22.387 | 3.518 | 1.778 | 5.575 | 57.955 | 19.484 | 146.974 | 22.138 | 12.16 |  33.098
11–12 | AP | 0.73 | 0.511 | 0.798 | 0.61 | 0.239 | 0.768 | 0.891 | 0.071 | 0.975 | 0.637 | 0.167 |  0.966
AUC | 0.988 | 0.848 | 0.997 | 0.907 | 0.715 | 0.996 | 0.984 | 0.661 | 1 | 0.881 | 0.763 |  1
Time | 5.608 | 1.927 | 14.176 | 2.152 | 1.117 | 3.371 | 39.968 | 13.066 | 102.656 | 14.626 | 7.361 |  22.494
BPI 12 (10) | 2 | AP | 0.922 | 0.887 | 0.957 | 0.858 | 0.377 | 0.947 | BPI 17 (10) | 0.979 | 0.968 | 0.989 | 0.956 | 0.769 |  0.988
AUC | 0.998 | 0.989 | 0.999 | 0.965 | 0.728 | 0.999 | 0.999 | 0.998 | 1 | 0.988 | 0.908 |  1
Time | 2.423 | 1.449 | 3.78 | 1.748 | 1.122 | 2.916 | 4.299 | 2.469 | 7.444 | 2.957 | 1.754 |  4.751
3 | AP | 0.827 | 0.779 | 0.854 | 0.787 | 0.342 | 0.858 | 0.921 | 0.884 | 0.938 | 0.93 | 0.913 | 0.939
AUC | 0.997 | 0.994 | 0.998 | 0.979 | 0.705 | 0.999 | 0.998 | 0.977 | 0.999 | 0.999 | 0.995 |  0.999
Time | 5.585 | 2.679 | 12.431 | 3.404 | 1.815 | 5.906 | 32.042 | 14.555 | 73.366 | 19.018 | 9.618 |  34.132
4 | AP | 0.821 | 0.711 | 0.868 | 0.727 | 0.261 | 0.851 | 0.9 | 0.854 | 0.92 | 0.859 | 0.4 |  0.914
AUC | 0.997 | 0.99 | 0.999 | 0.96 | 0.698 | 0.998 | 0.999 | 0.997 | 0.999 | 0.982 | 0.738 |  0.999
Time | 6.08 | 2.567 | 14.182 | 3.253 | 1.761 | 5.269 | 29.948 | 12.292 | 71.597 | 15.735 | 8.166 |  27.556
5 | AP | 0.758 | 0.652 | 0.819 | 0.681 | 0.166 | 0.783 | 0.9 | 0.754 | 0.924 | 0.833 | 0.326 |  0.913
AUC | 0.994 | 0.987 | 0.997 | 0.974 | 0.665 | 0.997 | 0.997 | 0.954 | 0.999 | 0.982 | 0.734 |  0.999
Time | 4.352 | 1.733 | 10.35 | 2.206 | 1.064 | 3.546 | 27.997 | 10.725 | 67.197 | 13.343 | 6.21 |  22.833
6 | AP | 0.688 | 0.597 | 0.772 | 0.625 | 0.189 | 0.779 | 0.897 | 0.836 | 0.922 | 0.8 | 0.147 |  0.908
AUC | 0.991 | 0.978 | 0.995 | 0.951 | 0.653 | 0.997 | 0.998 | 0.996 | 0.999 | 0.981 | 0.73 |  0.999
Time | 2.898 | 1.131 | 7.01 | 1.356 | 0.692 | 2.144 | 20.053 | 7.425 | 50.366 | 8.973 | 4.414 |  14.846

Table 7: An overview of the performance of both neural network approaches for
a fixed window size (in brackets after event log name) trained over different
subsets of the 2012 and 2017 BPI Challenge data containing different numbers
of windows. The highest average and maximum AP is indicated in grey, the other
metrics in bold.

####  5.2.3 Neural network parameterisation

An overview of the impact of the network parameterisation on average precision
is not included but can be found for both network approaches at
https://github.com/JohannesDeSmedt/processes-as-movies. Overall, the number of
CONVLSTM layers used, the size of the filters, and kernel size have little
impact on the average precision for a fixed number of windows. Hence, the
intuition of Section 4.2 cannot necessarily be applied, potentially due to the
high overlap and slow changes in tensors between time steps even when few time
steps are used for longer traces. The kernel size does result in slightly
different results for BPI 17 with 10 windows per trace. For fixed window
sizes, the results also remain relatively stable, with a single drop in AP
typically for higher filter sizes in combination with higher kernel sizes,
however, the occasional spikes do not follow a specific trend. For more
windows per trace, regardless of the window size, the results become more
varied, possibly due to the fewer examples available. The results for window
size 10 contain the least variance.

For encoder-decoder LSTMs, the impact of the feature dimensionality used does
seem to have a strong effect. The higher the dimensionality, the lower the
average precision for 5 and 10 windows per trace for a fixed number of
windows. This could potentially indicate that the networks overfit quicker.
For the fixed window length, the results are relatively unstable with again
lower AP for higher dimensions. This is especially apparent for window size 2.
Given that this approach is more similar to other LSTM-based approaches, it
might indicate that the CONVLSTMs might even excel for single next-in-sequence
activity prediction.

Overall, we can notice that the results of the convolutional LSTMs are much
more stable over the full parameter space, which is in line with the standard
deviations from Tables 6–7.

###  5.3 Constraints and interpretation

Given that PAM resorts to predicting various types of constraints at once in
the output tensor, it is possible to retrieve the evaluation metrics per
constraint type as well. In Table 8, the results for the fixed number of
windows approach are shown, and in Tables 9 and 10 the results for the fixed
window length approach are shown for the maximum results from Section 5.2.2
for the BPI 17 log (the BPI 12 log produced similar results, which can be
found for encoder-decoder LSTMs here and CONVLSTMs here). Both the number of
constraints predicted to be present, and their average precision is included
as AUC is generally high at the same levels as in Tables 8-7.

Firstly, it is apparent that the average precision is strongly dependent on
the number of constraints present. Hence, infrequently occurring constraints
such as alternate precedence/response (which are only considered by iBCM in
case more than 1 occurrence of the consequent activities is present) have very
low precision. Absence, on the other hand, often dominates the overall
proportion of constraints present. Its presence is high as it checks for all
non-existing occurrences, which can be plenty in the case of shorter windows
where fewer activities of the full activity set appear. The average precision
of absence is typically very high at levels close to 99% for both a fixed
number of windows and fixed window size (except for longer traces with fixed
window length 2, although this might again be due to fewer observations
available). This seems to drive the average precision of the whole tensor
prediction up.

For a fixed number of windows, precedence, response, not succession, and co-
existence have a noticeably higher proportion as well. The average precision
is typically reasonably high (70-85%) for the BPI 17 log, while lower for BPI
12 (45-65%). There seems to be more of such constraints present in the BPI 17
log as well, with higher proportions for these constraints, meaning there are
more examples to learn from. While not present/predicted as often, the unary
constraints exactly (2)/existence/init/last all achieve very high average
precision scores (80-90%), although this is less the case for exactly (2) for
the BPI 17 log, and for a higher number of windows for BPI 12. In general,
this constraint occurs less often than the other unary constraints, so again,
the number of training samples is scarcer. There are no noticeable differences
between both neural network topologies.

For a fixed window size, the dominance of absence is even stronger, with other
constraints making up very low proportions of the other constraints. This is
mostly due to the shorter windows, in which fewer binary constraints can
manifest but more of the rest of the activity set is not present. Some
constraints can also not be present (e.g. existence (3)). For shorter window
lengths (2), all constraints have high average precision, which is likely
caused by the higher number of steps that are fed to the LSTM compared to
other settings. For the very long traces (21–25, 26–30 windows), the average
precision slacks off for encoder-decoder LSTMs and only absence is predicted
with reasonable precision, possibly due to fewer traces available for training
or too few constraints present in general. For longer window lengths (5/10),
constraints with a higher proportion perform better, especially for the BPI 17
log, although again this trails off for longer traces. For a window size of 5,
binary constraints perform around 50% for BPI 12, and 80-90% for BPI 17. For a
window size of 10, the binary constraints achieve 80% AP and up for BPI 12,
and 95% for BPI 17. Exactly and Init/Last perform at +85% for window sizes 5
and 10 for BPI 12, and 99% scores for BPI 17.

Again, there seems to be little difference in performance per constraint
between the best-performing models for either convolutional and encoder-
decoder LSTMs, however, for a window size of 2 in longer traces there are more
positive predictions by the convolutional LSTM over all constraints, with high
precision rates. The same is noticeable for a window length of 5 and longer
traces to some extent. Convolutional LSTMs seem to be better capable of making
these fine-granular predictions.

| Convolutional LSTMs |  Encoder-decoder LSTMs
---|---|---
constraint | count | ap | count | ap | count | ap | count | ap | count | ap | count |  ap
BPI 12 | 2 | 5 | 10 | 2 | 5 |  10
Prec. | 23,544 | 0.612 | 11,199 | 0.641 | 9,021 | 0.505 | 24,034 | 0.620 | 10,962 | 0.622 | 9,066 | 0.513
Alt. prec | 228 | 0.062 | 12 | 0.019 | 6 | 0.009 | 259 | 0.064 | 21 | 0.051 | 12 |  0.013
Chain prec. | 6,164 | 0.593 | 4,240 | 0.453 | 3,366 | 0.329 | 6,221 | 0.604 | 4,169 | 0.449 | 3,315 | 0.313
Resp. | 23,544 | 0.595 | 11,199 | 0.625 | 9,021 | 0.516 | 24,034 | 0.596 | 10,962 | 0.608 | 9,066 | 0.508
Alt. resp. | 315 | 0.082 | 77 | 0.182 | 83 | 0.118 | 364 | 0.067 | 85 | 0.122 | 74 | 0.165
Chain resp. | 6,413 | 0.629 | 4,214 | 0.490 | 3,391 | 0.349 | 6,466 | 0.639 | 4,160 | 0.497 | 3,349 | 0.344
Abs. | 52,820 | 0.995 | 39,768 | 0.997 | 28,403 | 0.993 | 52,708 | 0.995 | 39,830 | 0.997 | 28,439 | 0.993
Exactly | 7,163 | 0.839 | 4,811 | 0.826 | 3,761 | 0.726 | 7,259 | 0.847 | 4,758 | 0.813 | 3,762 | 0.701
Exactly (2) | 535 | 0.834 | 304 | 0.575 | 290 | 0.332 | 598 | 0.780 | 275 | 0.479 | 272 | 0.281
Existence (3) | 2,314 | 0.940 | 1,485 | 0.845 | 1,074 | 0.720 | 2,267 | 0.928 | 1,505 | 0.852 | 1,055 | 0.709
Init | 2,618 | 0.982 | 1,932 | 0.918 | 1,397 | 0.821 | 2,618 | 0.978 | 1,932 | 0.900 | 1,397 | 0.795
Last | 2,618 | 0.906 | 1,932 | 0.905 | 1,397 | 0.828 | 2,618 | 0.911 | 1,932 | 0.912 | 1,397 | 0.770
Not suc. | 17,347 | 0.552 | 7,463 | 0.548 | 6,197 | 0.424 | 17,611 | 0.558 | 7,271 | 0.531 | 6,325 | 0.450
Co-exist. | 23,544 | 0.661 | 11,199 | 0.753 | 9,021 | 0.661 | 24,034 | 0.670 | 10,962 | 0.734 | 9,066 | 0.641
BPI 17 | 2 | 5 | 10 | 2 | 5 |  10
Prec. | 146,134 | 0.842 | 55,815 | 0.818 | 48,131 | 0.740 | 144,965 | 0.839 | 56,501 | 0.825 | 47,601 | 0.720
Alt. prec | 4,730 | 0.677 | 42 | 0.141 | 10 | 0.004 | 4,329 | 0.611 | 69 | 0.073 | 18 | 0.002
Chain prec. | 29,875 | 0.869 | 18,119 | 0.830 | 16,960 | 0.748 | 29,643 | 0.865 | 18,217 | 0.830 | 16,893 | 0.729
Resp. | 146,134 | 0.805 | 55,815 | 0.799 | 48,131 | 0.726 | 144,965 | 0.806 | 56,501 | 0.806 | 47,601 | 0.714
Alt. resp. | 4,517 | 0.634 | 54 | 0.153 | 17 | 0.006 | 4,111 | 0.621 | 85 | 0.106 | 33 | 0.019
Chain resp. | 30,924 | 0.866 | 18,963 | 0.849 | 17,863 | 0.789 | 30,834 | 0.865 | 19,151 | 0.853 | 17,765 | 0.774
Abs. | 118,927 | 0.991 | 135,331 | 0.996 | 137,132 | 0.995 | 119,066 | 0.991 | 135,118 | 0.997 | 137,266 | 0.995
Exactly | 29,741 | 0.890 | 19,747 | 0.893 | 19,191 | 0.853 | 29,772 | 0.890 | 19,885 | 0.892 | 19,112 | 0.833
Exactly (2) | 3,920 | 0.572 | 1,621 | 0.599 | 1,989 | 0.612 | 3,723 | 0.571 | 1,629 | 0.560 | 1,935 | 0.439
Existence (3) | 11,264 | 0.971 | 7,153 | 0.917 | 5,540 | 0.814 | 11,291 | 0.971 | 7,220 | 0.920 | 5,539 | 0.777
Init | 6,302 | 0.946 | 6,302 | 0.941 | 6,302 | 0.906 | 6,302 | 0.913 | 6,302 | 0.941 | 6,302 | 0.815
Last | 6,302 | 0.825 | 6,302 | 0.915 | 6,302 | 0.856 | 6,302 | 0.827 | 6,302 | 0.913 | 6,302 | 0.825
Not suc. | 106,337 | 0.784 | 38,018 | 0.769 | 32,973 | 0.700 | 105,728 | 0.783 | 38,550 | 0.785 | 32,545 | 0.684
Co-exist. | 146,134 | 0.859 | 55,815 | 0.836 | 48,131 | 0.768 | 144,965 | 0.859 | 56,501 | 0.841 | 47,601 | 0.752

Table 8: An overview of the number of constraints present, and their average
precision for the highest scoring result from Table 6 for a fixed window size.
The grey scale indicates overall proportion per column/number of windows/data
set combination overall all constraints.

|  Encoder-decoder LSTMs - BPI 17
---|---
|  Window length 2
| 6–10 | 11–15 | 16–20 | 21–25 |  26–30 [t]
constraint | count | ap | count | ap | count | ap | count | ap | count |  ap [b]
Prec. | 397 | 0.994 | 1077 | 0.992 | 684 | 0.962 | 515 | 0.001 | 269 | 0.001
Alt. prec | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 397 | 0.995 | 1077 | 0.993 | 684 | 0.972 | 515 | 0.001 | 269 | 0.001
Resp. | 397 | 0.994 | 1077 | 0.989 | 684 | 0.961 | 515 | 0.001 | 269 | 0.001
Alt. resp. | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain resp. | 397 | 0.994 | 1077 | 0.992 | 684 | 0.972 | 515 | 0.001 | 269 | 0.001
Abs. | 8723 | 1.000 | 48939 | 1.000 | 34236 | 1.000 | 25357 | 0.998 | 15156 | 0.945
Exactly | 853 | 0.999 | 3160 | 0.998 | 2132 | 0.992 | 1578 | 0.571 | 869 | 0.002
Exactly (2) | 0 | 0.000 | 1 | 0.040 | 7 | 0.403 | 15 | 0.000 | 17 | 0.000
Existence (3) | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Init | 456 | 0.996 | 2084 | 0.994 | 1455 | 0.985 | 1078 | 0.291 | 617 | 0.001
Last | 456 | 1.000 | 2084 | 0.997 | 1455 | 0.990 | 1078 | 0.629 | 617 | 0.001
Not suc. | 397 | 0.994 | 1077 | 0.989 | 684 | 0.972 | 515 | 0.001 | 269 | 0.001
Co-exist. | 397 | 0.995 | 1077 | 0.993 | 684 | 0.973 | 515 | 0.001 | 269 | 0.001
|  Window length 5
| 3–4 | 5–6 | 7–8 | 9–10 |  11–12
Prec. | 1367 | 0.947 | 4235 | 0.908 | 3218 | 0.797 | 2481 | 0.745 | 1358 | 0.678
Alt. prec | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 882 | 0.938 | 2684 | 0.917 | 1960 | 0.810 | 1485 | 0.759 | 814 | 0.721
Resp. | 1367 | 0.986 | 4235 | 0.938 | 3218 | 0.862 | 2481 | 0.830 | 1358 | 0.791
Alt. resp. | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain resp. | 889 | 0.957 | 2692 | 0.939 | 1977 | 0.864 | 1500 | 0.841 | 821 | 0.832
Abs. | 8231 | 1.000 | 47142 | 1.000 | 32832 | 0.999 | 24302 | 0.999 | 14558 | 0.999
Exactly | 1184 | 0.988 | 4054 | 0.967 | 2962 | 0.909 | 2272 | 0.880 | 1262 | 0.863
Exactly (2) | 52 | 0.933 | 708 | 0.954 | 420 | 0.794 | 271 | 0.560 | 165 | 0.398
Existence (3) | 109 | 0.823 | 196 | 0.756 | 161 | 0.549 | 105 | 0.303 | 57 | 0.153
Init | 456 | 0.910 | 2084 | 0.973 | 1455 | 0.891 | 1078 | 0.799 | 617 | 0.841
Last | 456 | 0.998 | 2084 | 0.999 | 1455 | 0.987 | 1078 | 0.970 | 617 | 0.936
Not suc. | 1036 | 0.957 | 2770 | 0.925 | 2222 | 0.822 | 1844 | 0.789 | 1019 | 0.717
Co-exist. | 1367 | 0.992 | 4235 | 0.939 | 3218 | 0.867 | 2481 | 0.834 | 1358 | 0.798
|  Window length 10
| 2 | 3 | 4 | 5 |  6
Prec. | 3539 | 0.972 | 10283 | 0.808 | 6718 | 0.715 | 4405 | 0.710 | 2721 | 0.670
Alt. prec | 0 | 0.000 | 3 | 0.011 | 1 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 1409 | 0.958 | 4230 | 0.844 | 2857 | 0.745 | 2032 | 0.730 | 1202 | 0.671
Resp. | 3539 | 0.972 | 10283 | 0.817 | 6718 | 0.714 | 4405 | 0.738 | 2721 | 0.694
Alt. resp. | 0 | 0.000 | 8 | 0.046 | 1 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain resp. | 1545 | 0.956 | 4508 | 0.861 | 2970 | 0.787 | 2083 | 0.800 | 1213 | 0.747
Abs. | 7572 | 0.999 | 45171 | 0.998 | 31615 | 0.997 | 23596 | 0.998 | 14062 | 0.997
Exactly | 1558 | 0.990 | 5278 | 0.906 | 3581 | 0.869 | 2602 | 0.871 | 1534 | 0.843
Exactly (2) | 10 | 0.037 | 693 | 0.819 | 356 | 0.461 | 247 | 0.378 | 145 | 0.261
Existence (3) | 436 | 0.988 | 958 | 0.882 | 823 | 0.723 | 505 | 0.598 | 301 | 0.523
Init | 456 | 0.988 | 2084 | 0.970 | 1455 | 0.865 | 1078 | 0.802 | 617 | 0.718
Last | 456 | 0.992 | 2084 | 0.972 | 1455 | 0.893 | 1078 | 0.904 | 617 | 0.857
Not suc. | 2212 | 0.958 | 6703 | 0.783 | 4393 | 0.694 | 2900 | 0.707 | 1872 | 0.675
Co-exist. | 3539 | 0.977 | 10283 | 0.821 | 6718 | 0.760 | 4405 | 0.770 | 2721 | 0.744

Table 9: An overview of the number of constraints present, and their average
precision for the highest scoring result from Table 7 for a fixed window size
and encoder-decoder LSTMs.

|  Convolutional LSTMs - BPI 17
---|---
|  Window length 2
| 6–10 | 11–15 | 16–20 | 21–25 |  26–30 [t]
constraint | count | ap | count | ap | count | ap | count | ap | count |  ap [b]
Prec. | 402 | 0.989 | 1096 | 0.996 | 659 | 0.989 | 496 | 0.989 | 281 | 0.736
Alt. prec | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 402 | 0.989 | 1096 | 0.996 | 659 | 0.989 | 496 | 0.989 | 281 | 0.733
Resp. | 402 | 0.989 | 1096 | 0.996 | 659 | 0.989 | 496 | 0.989 | 281 | 0.736
Alt. resp. | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain resp. | 402 | 0.989 | 1096 | 0.996 | 659 | 0.989 | 496 | 0.989 | 281 | 0.731
Abs. | 8718 | 0.999 | 48920 | 1.000 | 34261 | 1.000 | 25376 | 1.000 | 15144 | 1.000
Exactly | 858 | 0.998 | 3179 | 0.999 | 2097 | 0.996 | 1546 | 0.996 | 876 | 0.968
Exactly (2) | 0 | 0.000 | 1 | 0.007 | 17 | 0.845 | 28 | 0.786 | 22 | 0.364
Existence (3) | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Init | 456 | 0.989 | 2084 | 0.998 | 1455 | 0.993 | 1078 | 0.994 | 617 | 0.913
Last | 456 | 1.000 | 2084 | 0.999 | 1455 | 0.996 | 1078 | 1.000 | 617 | 0.968
Not suc. | 402 | 0.992 | 1096 | 0.994 | 659 | 0.990 | 496 | 0.982 | 281 | 0.730
Co-exist. | 402 | 0.988 | 1096 | 0.995 | 659 | 0.992 | 496 | 0.983 | 281 | 0.738
|  Window length 5
| 3–4 | 5–6 | 7–8 | 9–10 |  11–12
Prec. | 1311 | 0.938 | 4180 | 0.897 | 3131 | 0.806 | 2466 | 0.783 | 1340 | 0.822
Alt. prec | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 852 | 0.946 | 2648 | 0.909 | 1937 | 0.818 | 1521 | 0.811 | 806 | 0.830
Resp. | 1311 | 0.983 | 4180 | 0.933 | 3131 | 0.872 | 2466 | 0.877 | 1340 | 0.883
Alt. resp. | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain resp. | 859 | 0.966 | 2672 | 0.934 | 1948 | 0.882 | 1535 | 0.875 | 809 | 0.891
Abs. | 8259 | 1.000 | 47170 | 1.000 | 32863 | 0.999 | 24277 | 0.999 | 14574 | 0.999
Exactly | 1145 | 0.981 | 4071 | 0.965 | 2930 | 0.927 | 2291 | 0.920 | 1258 | 0.932
Exactly (2) | 48 | 0.875 | 668 | 0.945 | 422 | 0.818 | 279 | 0.702 | 155 | 0.640
Existence (3) | 124 | 0.861 | 191 | 0.743 | 160 | 0.520 | 103 | 0.294 | 55 | 0.280
Init | 456 | 0.908 | 2084 | 0.973 | 1455 | 0.909 | 1078 | 0.865 | 617 | 0.859
Last | 456 | 0.998 | 2084 | 0.996 | 1455 | 0.995 | 1078 | 0.991 | 617 | 0.981
Not suc. | 972 | 0.956 | 2794 | 0.917 | 2144 | 0.826 | 1818 | 0.817 | 1027 | 0.799
Co-exist. | 1311 | 0.987 | 4180 | 0.933 | 3131 | 0.882 | 2466 | 0.869 | 1340 | 0.882
|  Window length 10
| 2 | 3 | 4 | 5 |  6
Prec. | 3491 | 0.967 | 10500 | 0.797 | 6588 | 0.730 | 4638 | 0.723 | 2710 | 0.710
Alt. prec | 0 | 0.000 | 2 | 0.029 | 1 | 0.000 | 0 | 0.000 | 0 | 0.000
Chain prec. | 1403 | 0.961 | 4303 | 0.835 | 2855 | 0.753 | 2096 | 0.741 | 1201 | 0.738
Resp. | 3491 | 0.967 | 10500 | 0.807 | 6588 | 0.726 | 4638 | 0.751 | 2710 | 0.729
Alt. resp. | 0 | 0.000 | 6 | 0.145 | 1 | 0.002 | 0 | 0.000 | 0 | 0.000
Chain resp. | 1528 | 0.957 | 4568 | 0.860 | 2942 | 0.790 | 2123 | 0.805 | 1214 | 0.777
Abs. | 7587 | 0.999 | 45103 | 0.998 | 31649 | 0.997 | 23508 | 0.997 | 14061 | 0.997
Exactly | 1542 | 0.984 | 5349 | 0.903 | 3543 | 0.873 | 2649 | 0.882 | 1534 | 0.871
Exactly (2) | 11 | 0.024 | 674 | 0.820 | 335 | 0.536 | 258 | 0.451 | 140 | 0.397
Existence (3) | 436 | 0.987 | 974 | 0.860 | 848 | 0.748 | 535 | 0.632 | 307 | 0.647
Init | 456 | 0.986 | 2084 | 0.967 | 1455 | 0.888 | 1078 | 0.823 | 617 | 0.788
Last | 456 | 0.991 | 2084 | 0.967 | 1455 | 0.889 | 1078 | 0.907 | 617 | 0.867
Not suc. | 2161 | 0.954 | 6867 | 0.771 | 4265 | 0.698 | 3086 | 0.711 | 1861 | 0.691
Co-exist. | 3491 | 0.969 | 10500 | 0.813 | 6588 | 0.767 | 4638 | 0.783 | 2710 | 0.763

Table 10: An overview of the number of constraints present, and their average
precision for the highest scoring result from Table 7 for a fixed window size
and CONVLSTMs.

###  5.4 Discussion

Distilling all results, it is apparent that PAM with convolutional LSTMs are
capable of achieving high predictive accuracy and precision over the high-
dimensional output consisting of activity pair-constraint information.
Especially for a log with longer traces (BPI 17), the approach achieves
results over 90% precision. Very granular input of short fixed window sizes
(length 2 or 5), are predicted very well, while on the other side of the
spectrum, precision of 70-80% can still be achieved even when dividing a trace
in 2 (fixed number of windows at 2) using only the first half of the trace to
predict the presence of constraints in the second half. Overall, the
performance of convolutional neural networks is better and more stable, but
come at a runtime cost. However, given that the results do not vary much
according to the parameters, even fast-running models can achieve good
results.

Given that the performance for short windows is strong, PAM can be used for
next-in-sequence prediction as well. Existence or absence constraints can tell
what activities will be executed next. Essentially, as discussed before, using
only one of these constraints would create a $|A|\times|A|\times 1$ tensor
which would result in a performance similar to a non-convolutional recurrent
neural network. Having the unary constraints in the network can make the other
constraints perform better as well, as they can correlate with the behaviour
of the presence/absence of particular activities.

We have also covered the impact of particular constraints. Unary constraints
perform well in most cases, and binary constraints’ performance is strongly
linked to their presence. Looking back to our motivation in Section 2, it
appears that it is easier to predict the occurrence of unary relationships
with high precision, making it possible to, e.g., verify whether an
application is going to be rejected (depending on whether absence is predicted
to hold for A_Declined), rather than whether a particular activity precedes or
always follows another (e.g. the chain response constraint between A_Declined
and W_Completeren aanvraag). Nevertheless, high precision is still achieved at
+80% and +90% levels for BPI 17. Hence, questions such as: ‘are all final
A_Declined preceded by W_Completeren aanvraag’ can be predicted well using PAM
similar to the objectives predicted in DBLP:journals/corr/TeinemaaDRM17 .
However, PAM predicts over 9,000 constraints/objectives simultaneously (e.g.
for 26 activities over 14 binary constraints for BPI 17), although they are
not as tailored towards a particular question and overlap to some extent (e.g.
exactly(a) overlaps with exactly(a,2) and precedence(a,b)). PAM allows to
combine various output constraints to create even stronger objective sets or
even declarative process models, but does not include any data variables
within the LTL constraints. Finally, PAM is not as flexible regarding the
input, where only a final window is predicted and the window size or number of
windows is determined up front. This might require multiple runs to finetune
results and depends on the outcome of the analysis. E.g., in some cases a
fixed window length might be preferred over predicting the last window, as it
might not be possible to determine how many more windows will be present.
Still, PAM can be run regardless of this knowledge and provide a prediction by
dividing the current execution trace in a fixed number of windows, or windows
of a fixed length.

##  6 Conclusion and Future Work

This paper introduced PAM, a new technique that encompasses a feature
generation model for processes such that processes can be treated as movies
fit for training high-dimensional recurrent neural networks. This allows
modelling the dynamic development of process models, offering a mix between
next-in-sequence and objective-based predictive process modelling. It was
shown that declarative process constraints can be used to make a multi-
dimensional representation of activity pairs to capture a process model at
various points in time, which is used towards predicting the constraint set
constituting the process model that will be present in subsequent windows of
the process execution. The neural network architecture has been proven to be
adequate to perform this prediction, with high accuracy and precision, for
various real-life event logs and different approaches to extract windows from
traces. Hence, this makes PAM effective to provide a forecast and early
warning of constraint violations and satisfaction, allowing the support of
monitoring process behaviour and even predictive conformance checking.

There are many future directions for this research. Firstly, a wider
experimental evaluation with an even larger hyperparameter search, focusing on
deeper or different network architectures such as PredRNN
DBLP:conf/icml/WangGLWY18  can be investigated. Next, a more in-depth analysis
of various constraint types would be worthwhile. Some constraint types might
be more interesting, which results in a trade-off between depth and
performance. If constraints are seldomly present, they might not provide
sufficient learning material for the network, while they increase the
dimensionality of the input. Hence, it might be sufficient to use PAM with
only a subset of constraints to obtain similar insights and results. The
window-based approach can be further improved significantly as well.
Currently, the setup of splitting up traces in windows of equal length or in
an equal number of windows could be improved in a variety of ways to account
for longer and shorter traces, and to find anchor points reflecting particular
milestones within a process which might be better suitable for creating models
that are self-contained. Finally, it would be interesting to see whether the
technique can run in an on-line fashion, by pre-training a network and
updating it according to newly-generated traces.

## References

  * (1) J. Evermann, J. Rehse, and P. Fettke, “Predicting process behaviour using deep learning,” _Decision Support Systems_ , vol. 100, pp. 129–140, 2017.
  * (2) N. Tax, I. Verenich, M. L. Rosa, and M. Dumas, “Predictive business process monitoring with LSTM neural networks,” in _CAiSE_ , ser. Lecture Notes in Computer Science, vol. 10253. Springer, 2017, pp. 477–492.
  * (3) F. M. Maggi, C. D. Francescomarino, M. Dumas, and C. Ghidini, “Predictive monitoring of business processes,” in _CAiSE_ , ser. Lecture Notes in Computer Science, vol. 8484. Springer, 2014, pp. 457–472.
  * (4) I. Teinemaa, M. Dumas, M. L. Rosa, and F. M. Maggi, “Outcome-oriented predictive process monitoring: review and benchmark,” _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , vol. 13, no. 2, p. 17, 2019.
  * (5) M. Pesic and W. M. van der Aalst, “A declarative approach for flexible business processes management,” in _Business Process Management Workshops_. Springer, 2006, pp. 169–180.
  * (6) X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo, “Convolutional LSTM network: A machine learning approach for precipitation nowcasting,” in _NIPS_ , 2015, pp. 802–810.
  * (7) A. E. Márquez-Chamorro, M. Resinas, and A. Ruiz-Cortés, “Predictive monitoring of business processes: A survey,” _IEEE Trans. Services Computing_ , vol. 11, no. 6, pp. 962–977, 2018.
  * (8) W. M. P. van der Aalst, M. H. Schonenberg, and M. Song, “Time prediction based on process mining,” _Inf. Syst._ , vol. 36, no. 2, pp. 450–475, 2011.
  * (9) G. T. Lakshmanan, D. Shamsi, Y. N. Doganata, M. Unuvar, and R. Khalaf, “A markov prediction model for data-driven semi-structured business processes,” _Knowl. Inf. Syst._ , vol. 42, no. 1, pp. 97–126, 2015.
  * (10) D. Breuker, M. Matzner, P. Delfmann, and J. Becker, “Comprehensible predictive models for business processes,” _MIS Quarterly_ , vol. 40, no. 4, pp. 1009–1034, 2016.
  * (11) N. Navarin, B. Vincenzi, M. Polato, and A. Sperduti, “LSTM networks for data-aware remaining time prediction of business process instances,” in _IEEE SSCI_. IEEE, 2017, pp. 1–7.
  * (12) N. Mehdiyev, J. Evermann, and P. Fettke, “A multi-stage deep learning approach for business process event prediction,” in _CBI (1)_. IEEE Computer Society, 2017, pp. 119–128.
  * (13) L. Lin, L. Wen, and J. Wang, “Mm-pred: A deep predictive model for multi-attribute event sequence,” in _Proceedings of the 2019 SIAM International Conference on Data Mining_. SIAM, 2019, pp. 118–126.
  * (14) V. Pasquadibisceglie, A. Appice, G. Castellano, and D. Malerba, “Using convolutional neural networks for predictive process analytics,” in _ICPM_. IEEE, 2019, pp. 129–136.
  * (15) P. De Koninck, S. vanden Broucke, and J. De Weerdt, “act2vec, trace2vec, log2vec, and model2vec: Representation learning for business processes,” in _BPM_ , ser. Lecture Notes in Computer Science, vol. 11080. Springer, 2018, pp. 305–321.
  * (16) C. D. Francescomarino, M. Dumas, F. M. Maggi, and I. Teinemaa, “Clustering-based predictive process monitoring,” _IEEE Trans. Services Computing_ , vol. 12, no. 6, pp. 896–909, 2019.
  * (17) C. D. Francescomarino, C. Ghidini, F. M. Maggi, G. Petrucci, and A. Yeshchenko, “An eye into the future: Leveraging a-priori knowledge in predictive business process monitoring,” in _BPM_ , ser. Lecture Notes in Computer Science, vol. 10445. Springer, 2017, pp. 252–268.
  * (18) A. E. M. Chamorro, M. Resinas, A. R. Cortés, and M. Toro, “Run-time prediction of business process indicators using evolutionary decision rules,” _Expert Syst. Appl._ , vol. 87, pp. 1–14, 2017.
  * (19) F. M. Maggi, A. Burattin, M. Cimitile, and A. Sperduti, “Online process discovery to detect concept drifts in ltl-based declarative process models,” in _OTM Conferences_ , ser. Lecture Notes in Computer Science, vol. 8185\. Springer, 2013, pp. 94–111.
  * (20) A. Burattin, M. Cimitile, and F. M. Maggi, “Lights, camera, action! Business process movies for online process discovery,” in _BPM Workshops_ , ser. LNBIP, vol. 202. Springer, 2014, pp. 408–419.
  * (21) M. Camargo, M. Dumas, and O. G. Rojas, “Learning accurate LSTM models of business processes,” in _BPM_ , ser. Lecture Notes in Computer Science, vol. 11675. Springer, 2019, pp. 286–302.
  * (22) M. Pesic, H. Schonenberg, and W. M. van der Aalst, “Declare: Full support for loosely-structured processes,” in _Enterprise Distributed Object Computing Conference, 2007. EDOC 2007. 11th IEEE International_. IEEE, 2007, pp. 287–287.
  * (23) A. Polyvyanyy, M. Weidlich, R. Conforti, M. L. Rosa, and A. H. M. ter Hofstede, “The 4c spectrum of fundamental behavioral relations for concurrent systems,” in _Petri Nets_ , ser. Lecture Notes in Computer Science, vol. 8489\. Springer, 2014, pp. 210–232.
  * (24) M. Weidlich, A. Polyvyanyy, N. Desai, J. Mendling, and M. Weske, “Process compliance analysis based on behavioural profiles,” _Inf. Syst._ , vol. 36, no. 7, pp. 1009–1025, 2011.
  * (25) A. Weijters, W. M. van der Aalst, and A. A. De Medeiros, “Process mining with the heuristics miner-algorithm,” _TUe, Tech. Rep. WP_ , vol. 166, 2006\.
  * (26) T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in neural information processing systems_ , 2013, pp. 3111–3119.
  * (27) P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _Proceedings of the 25th international conference on Machine learning_. ACM, 2008, pp. 1096–1103.
  * (28) V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * (29) S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * (30) K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in _EMNLP_. ACL, 2014, pp. 1724–1734.
  * (31) J. Chen, J. Zhang, X. Xu, C. Fu, D. Zhang, Q. Zhang, and Q. Xuan, “E-lstm-d: A deep learning framework for dynamic network link prediction,” _arXiv preprint arXiv:1902.08329_ , 2019.
  * (32) A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Commun. ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * (33) C. Finn, I. J. Goodfellow, and S. Levine, “Unsupervised learning for physical interaction through video prediction,” in _NIPS_ , 2016, pp. 64–72.
  * (34) M. Liang and X. Hu, “Recurrent convolutional neural network for object recognition,” in _CVPR_. IEEE Computer Society, 2015, pp. 3367–3375.
  * (35) T. Murata, “Petri nets: Properties, analysis and applications,” _Proceedings of the IEEE_ , vol. 77, no. 4, pp. 541–580, 1989.
  * (36) W. M. van der Aalst, T. Weijters, and L. Maruster, “Workflow mining: Discovering process models from event logs,” _Knowledge and Data Engineering, IEEE Transactions on_ , vol. 16, no. 9, pp. 1128–1142, 2004.
  * (37) S. J. J. Leemans, D. Fahland, and W. M. P. van der Aalst, “Discovering Block-Structured Process Models from Event Logs - A Constructive Approach,” in _Petri Nets_ , ser. Lecture Notes in Computer Science, vol. 7927. Springer, 2013, pp. 311–329.
  * (38) W. M. van der Aalst, A. Adriansyah, and B. Van Dongen, “Causal nets: a modeling language tailored towards process discovery,” in _CONCUR 2011–Concurrency Theory_. Springer, 2011, pp. 28–42.
  * (39) OMG, “Business Process Model and Notation (BPMN) 2.0,” 2011.
  * (40) T. T. Hildebrandt and R. R. Mukkamala, “Declarative event-based workflow as distributed dynamic condition response graphs,” _arXiv preprint arXiv:1110.4161_ , 2011.
  * (41) J. De Smedt, G. Deeva, and J. De Weerdt, “Mining behavioral sequence constraints for classification,” _IEEE Transactions on Knowledge and Data Engineering_ , 2019.
  * (42) J. Prescher, C. D. Ciccio, and J. Mendling, “From declarative processes to imperative models,” in _Proceedings of the 4th International Symposium on Data-driven Process Discovery and Analysis (SIMPDA 2014), Milan, Italy, November 19-21, 2014._ , 2014, pp. 162–173.
  * (43) J. De Smedt, S. K. L. M. vanden Broucke, J. De Weerdt, and J. Vanthienen, “A full R/I-net construct lexicon for declare constraints,” KU Leuven, Tech. Rep., 2015.
  * (44) M. Pesic, “Constraint-based workflow management systems: shifting control to users,” Ph.D. dissertation, Technische Universiteit Eindhoven, 2008.
  * (45) M. Westergaard, C. Stahl, and H. A. Reijers, “Unconstrainedminer: Efficient discovery of generalized declarative process models,” _BPM-13-28, BPMcenter_ , 2013.
  * (46) A. Leontjeva, R. Conforti, C. D. Francescomarino, M. Dumas, and F. M. Maggi, “Complex symbolic sequence encodings for predictive monitoring of business processes,” in _BPM_ , ser. Lecture Notes in Computer Science, vol. 9253\. Springer, 2015, pp. 297–313.
  * (47) H. Nguyen, M. Dumas, A. H. M. ter Hofstede, M. L. Rosa, and F. M. Maggi, “Business process performance mining with staged process flows,” in _CAiSE_ , ser. Lecture Notes in Computer Science, vol. 9694. Springer, 2016, pp. 167–185.
  * (48) F. M. Maggi, A. J. Mooij, and W. M. van der Aalst, “User-guided discovery of declarative process models,” in _Computational Intelligence and Data Mining (CIDM), 2011 IEEE Symposium on_. IEEE, 2011, pp. 192–199.
  * (49) C. Di Ciccio and M. Mecella, “A two-step fast algorithm for the automated discovery of declarative workflows,” in _Computational Intelligence and Data Mining (CIDM), 2013 IEEE Symposium on_. IEEE, 2013, pp. 135–142.
  * (50) C. D. Ciccio, F. M. Maggi, M. Montali, and J. Mendling, “Ensuring model consistency in declarative process discovery,” in _BPM_ , ser. Lecture Notes in Computer Science, vol. 9253. Springer, 2015, pp. 144–159.
  * (51) S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _ICML_ , ser. JMLR Workshop and Conference Proceedings, vol. 37. JMLR.org, 2015, pp. 448–456.
  * (52) C. D. Francescomarino, M. Dumas, M. Federici, C. Ghidini, F. M. Maggi, W. Rizzi, and L. Simonetto, “Genetic algorithms for hyperparameter optimization in predictive business process monitoring,” _Inf. Syst._ , vol. 74, no. Part, pp. 67–83, 2018.
  * (53) S. Ruder, “An overview of gradient descent optimization algorithms,” _arXiv preprint arXiv:1609.04747_ , 2016.
  * (54) M. D. Zeiler, “ADADELTA: an adaptive learning rate method,” _CoRR_ , vol. abs/1212.5701, 2012.
  * (55) W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” _arXiv preprint arXiv:1409.2329_ , 2014.
  * (56) Y. Wang, Z. Gao, M. Long, J. Wang, and P. S. Yu, “Predrnn++: Towards A resolution of the deep-in-time dilemma in spatiotemporal predictive learning,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 5110–5119.