#  Nirdizati: an Advanced Predictive Process Monitoring Toolkit

Williams Rizzi1,2, Chiara Di Francescomarino1,
Chiara Ghidini1, Fabrizio Maria Maggi2
1 Fondazione Bruno Kessler (FBK), Trento, Italy
2 Free University of Bozen-Bolzano, Bolzano, Italy
{wrizzi, dfmchiara<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Predictive Process Monitoring is a field of Process Mining that aims at
predicting how an ongoing execution of a business process will develop in the
future using past process executions recorded in event logs. The recent stream
of publications in this field shows the need for tools able to support
researchers and users in analyzing, comparing and selecting the techniques
that are the most suitable for them. Nirdizati is a dedicated tool for
supporting users in building, comparing, analyzing, and explaining predictive
models that can then be used to perform predictions on the future of an
ongoing case. By providing a rich set of different state-of-the-art
approaches, Nirdizati offers BPM researchers and practitioners a useful and
flexible instrument for investigating and comparing Predictive Process
Monitoring techniques. In this paper, we present the current version of
Nirdizati, together with its architecture which has been developed to improve
its modularity and scalability. The features of Nirdizati enrich its
capability to support researchers and practitioners within the entire pipeline
for constructing reliable Predictive Process Monitoring models.

##  1 Introduction

Predictive Process Monitoring (PPM) [2] is a branch of Process Mining that
aims at providing predictions on the future of an ongoing process execution by
leveraging past historical execution traces. An increasing number of PPM
approaches leverage machine and deep learning techniques in order to learn
from past historical execution traces the outcome of an ongoing process
execution, the time remaining till the end of an ongoing execution, or the
next activities that will be performed.

The importance of PPM has stimulated researchers and practitioners to create
dozens of Machine/Deep Learning approaches to build PPM models [3]. With such
a diverse range of Machine learning techniques an issue is how to sensibly
choose the most appropriate one(s), especially considering that Machine
Learning approaches performance can vary widely when the dataset supplied to
build the PPM predictive model are different.

This paper proposes an architecture for a PPM toolkit - Nirdizati\- which
offers researchers and practitioners with a rich set of features and
techniques to build, compare, and analyze predictive process monitoring
models. Given the growing importance of eXplainable AI (XAI), Nirdizati is
also equipped with a first set of features for explaining predictions.
Nirdizati has been implemented as an open-source toolkit. While stemming from
the research community, Nirdizati can be of interest to both practitioners and
research scientists. Once compared and selected, the chosen predictive models
can be used by practitioners to compute predictions on the future values of
performance indicators related to a currently running process execution.
Researchers instead can exploit the modular nature of the tool for extending
it with new solutions, and leverage the standardized Nirdizati pipeline to
comparatively evaluate new solutions against existing techniques for better
fitting models to data.

The structure of this paper is as follows. Section 2 provides a background on
the data used in PPM, the techniques used to build and evaluate a predictive
model, and the techniques used to explain the data and the predictive model to
the user. Section 3 provides a description of the typical pipeline used for
building predictive process monitoring models, and the capabilities a generic
tool for PPM should satisfy. Sections 4 and 5 describe the general Nirdizati
architectures and components to provide the capabilities described in Section
3, respectively. Sections 6 and 7 provide an overview of the current prototype
implementation and of some of its front-end features. The paper ends with an
assessment of the tool (Section 8), related work (Section 9) and concluding
remarks (Section 10).

##  2 Background

In this section we provide an overview of the main concepts in Predictive
Process Monitoring, and Machine Learning techniques and concepts used in the
construction of the Nirdizati tool.

###  2.1 Event Log

An _event log_ is composed of executions (also known as traces or cases) of
business processes. A _trace_ consists of a sequence of _events_ , each
referring to the execution of an activity (a.k.a. an event class). Events that
belong to a trace are _ordered_ and constitute a single “run” of the process.
For example, in trace $\sigma_{i}=\left\langle event_{1},event_{2},\ldots
event_{n}\right\rangle$, the first activity to be executed is the activity
associated to $event_{1}$. Events are also characterized by attributes (_event
attributes_), among which the _timestamp_ , indicating the time in which the
event has occurred, and possible _data payloads_ such as the resource(s)
involved in the execution of an activity or other data recorded with the
event. Some attributes, as for instance the personal birth date of a customer
in a loan request process, do not change throughout the different events in
the trace, and are called _trace attributes_.

The canonical representation format of the event logs is the XES (eXtensible
Event Stream) standard [4]: an XML-based format that maintains the general
standard structure of an event log.

###  2.2 Predictive Process Monitoring

Predictive Process Monitoring (PPM) [2] is a branch of Process Mining that
aims at predicting at runtime and as early as possible the future development
of ongoing cases of a process given their uncompleted traces. In the last few
years a wide literature about Predictive Process Monitoring techniques has
become available - see [5] for a survey - mostly based on Machine Learning
techniques.

Figure 1: The Predictive Process Monitoring pipeline.

Figure 1 shows a typical example of Predictive Process Monitoring pipeline.
These approaches usually require that trace prefixes are extracted from the
historical execution traces (_Prefix extraction phase_). This is due to the
fact that at runtime predictions are made on incomplete traces, so that
correlations between incomplete traces and what we want to predict (_target
variables_ or _labels_) have to be learned in the training phase. After
prefixes have been extracted, prefix traces and labels (i.e., the information
that has to be predicted) are encoded in the form of feature vectors
(_encoding phase_). Encoded traces are then passed to the (supervised
learning) techniques in charge of learning from the encoded data one (or more)
predictive model(s) (_Learning phase_). At runtime, the incomplete execution
traces i.e., the traces whose future is unknown, should also be encoded as
feature vectors and used to query the predictive model(s) so as to get the
prediction (_Predicting phase_).

Concerning “what we want to predict”, or better the label we aim at
predicting, we can follow the typical classification presented in [5] and
cluster predictions into three main categories:

  * •

_outcome-based predictions_ , that is, predictions related to predefined
categorical or boolean outcome values;

  * •

_numeric value predictions_ , that is, predictions related to measures of
interest taking numeric or continuous values; and

  * •

_next event predictions_ , that is, predictions related to sequences of future
activities and related data payloads.

Encodings are needed to transform labelled prefix traces of the form
$\sigma_{i}=\left\langle event_{1},event_{2},\ldots
event_{i},label\right\rangle$ into a format that is understandable by machine
learning techniques. This would allow the technique to train and hence learn,
from encoded data a predictive model. Due to the complex nature of event logs,
which are characterized by a temporal dimension, the presence of data
payloads, the possible inter-relation vs independence between execution
traces, and so on, several encodings have been proposed in the literature.
Typical encodings available in the literature [6] are: the _Boolean encoding_
and the _Simple-index encoding_ , which only take int account the control flow
(that is, the sequence of events) of a trace, and the _Complex index-based
encoding_ , which instead also consider data payloads in the encoding. Further
encodings, such as the inter-feature encoding presented in [7], make ti
possible to machine leaning algorithms to take into account the possible
interdependencies between temporally overlapping execution traces as this may
affect the result of a prediction (think, for instance, to “runs” competing
for the same resource).

When it comes to the learning phase, multiple approaches from the machine
learning literature have been used in the literature of Predictive Process
Monitoring. Examples range from traditional Machine Learning techniques such
as Decision Tree, Random Forest, or Support Vector Machine to deep learning
techniques such as LSTM Neural Networks [8, 2, 9, 6, 10]. One of the reasons
for such a variety of techniques is that the studies carried out have not
identified a Machine Learning algorithm strictly better than all the others.
Depending on the event log data, the encoding, and the prediction problem,
different techniques are known to perform better than others.

###  2.3 Hyperparameters optimization

The machine learning techniques to better fit the supplied data typically have
a variable number of hyperparameters. The tuning of the hyperparameters highly
influences the quality of the built predictive model. Indeed, although the
available techniques are expensive both in terms of computational power and
time, the gains in terms of prediction quality justify their use.

The hyperparameter optimization techniques receive as input the
hyperparameters space of correct input and a quality metric to maximize. The
technique will then proceed to maximizing the quality metric by producing
multiple different configurations of hyperparameters and then evaluating the
quality metric for each of them. The amount of hyperparameter configurations
produced by the hyperparameter optimization technique depends on how the
technique explores the hyperparameter optimization space. Various algorithms
exist to tackle this challenge [11], including approaches specific for the
Predictive Process Monitoring field [12].

###  2.4 Explainable Predictions

Explainable AI (XAI) is a growing body of research focused on providing
explanations on the outputs of, usually gray or black box, Machine Learning
algorithms. In the literature, there are two main sets of techniques used to
develop explainable systems, a.k.a. explainers: _post-hoc_ and _ante-hoc_
techniques. Post-hoc techniques, such as the Local Interpretable Model-
Agnostic Explanations (LIME) [13] or the SHapley Additive exPlanations (SHAP)
[14], allow models to be trained as usual, with explainability only being
incorporated at testing time. Ante-hoc techniques, such as Bayesian deep
learning (BDL) [15], entail integrating explainability into a model from the
beginning. Explainable techniques have recently started to be applied to the
Predictive Process Monitoring field, mainly with the usage of model agnostic
generic techniques (such LIME and SHAP mentioned above) with the aim of
discovering the rationale for a prediction. While the application of XAI
techniques to PPM is still at its infancy, and specific XAI techniques devoted
to event log data or Predictive Process Monitoring prediction problems are
still lacking, the field is developing quickly. The interested reader is
referred to [16] for a recent survey.

##  3 Building a Predictive Process Monitoring model

The task of building a Predictive Process Monitoring model can be seen as an
instance of a data science pipeline, tailored to the specific type of event
logs data. While several data science pipelines exist, they are all usually
composed of a set of tasks that start with the raw data, transform them into
event logs, and then move trough the steps of Data Engineering, Model
Construction (and Evaluation), and Model Deployment (see Figure 2).

Figure 2: Building a Predictive Process Monitoring model.

Nirdizati aims at supporting users in the _construction and the comparison of
different predictive models_ so as to:  (i) select the most appropriate
model(s), in case of end users who aim at deploying a predictive model, or
(ii) support experimental comparisons of different techniques in scientific
studies, in case of research scientists.  As such, it mainly focuses on
providing capabilities for Model Construction and its (comparative)
Evaluation, as highlighted in the blue boxes in Figure 2, and neglects steps
such as the extraction of raw data, the construction of an event log, and the
deployment of the predictive model, which are often tackled with ad hoc
approaches or different specific tools (see e.g. the _onprom_ tool for the
construction of an event log, described in [17]). Also, the Data Engineering
component is mainly tailored to support the splitting and labeling of data
into training and test sets, thus ignoring the important - but often similarly
ad hoc - step of Data Cleaning and Improving. Given the growing importance of
Explainable Artificial Intelligence (XAI), explanation is likely to become one
of the standard tasks of future ML-based pipelines and is therefore also
included in Nirdizati and discussed here. In the following we provide an
overview of the capabilities (that is the functionalities for the user) that a
tool for constructing and comparing different Predictive Process Monitoring
models should have, ranging from the creation of a good labeled training and
test set, up to the construction of Predictive Process Monitoring models, and
their (comparative) evaluation.

###  3.1 Data Engineering

In the context of Predictive Process Monitoring, data pre-processing is used
to determine the most effective representation of the data for the predictive
problem at hand. In particular, data pre-processing is used in Predictive
Process Monitoring to understand the distribution of attributes, events, and
timestamps in the data space, and whether the quality of data and their
interesting aspects make the specific prediction problem at hand feasible,
unfeasible, or only partly feasible.

The understanding of the quality of data and the feasibility of some/all of
the canonical prediction problems is usually tackled with data exploration,
which is concerned with the determination of the data types, the visualisation
of such data, and the analysis of the data space densities. In the context of
Predictive Process Monitoring data exploration must enable the user with the
exploration of the selected event log, which in turn consists in the
exploration of execution traces in the XES format, e.g., their start and end
time, the distribution of events, as well as of event or trace attributes.
Data exploration can trigger a refinement of the data space, which is usually
performed in a data cleaning phase, which can concern the removal of
unnecessary, duplicate, or inconsistent values, the addition of missing
values, or the modification of the sparsity of data, just to make simple
examples. Several open source or commercial tools exist to perform complex
event log exploration, analysis, and engineering tasks, usually as a
prerequisite of the task of process discovery (see e.g.,
ProM111https://www.promtools.org, Disco222https://fluxicon.com,
Celonis333https://www.celonis.com/, just to name a few). Thus, supporting
advanced event log engineering functionalities is not deemed crucial for a
Predictive Process Monitoring tool. Obviously an advanced tool for PPM _may
enable_ the user to perform some event log engineering as a way to support the
user within a single comprehensive tool.

###  3.2 Building the predictive model(s)

Once the event logs are in a suitable shape, they must be split into training
(the data used to build the predictive models), validation (the data used to
automatically optimize hyper-parameters, if used) and test (the data used to
evaluate the predictive models) sets. From here on, we will use the term
_training set_ to denote the data used for building a model with optimised
hyperparameters, that is for denoting both the training and validation set. In
addition to the choice of the size of training and test sets, the temporal
nature of Predictive Process Monitoring data implies that different strategies
can be chosen in order to split them. For instance one may opt for a random
split of traces for training and testing disregarding the time they were
executed; one could take the older444That is, the ones starting (resp.,
finishing) at the beginning of the period covered by the event log for
training and the newer555That is, the ones starting (resp., finishing) towards
the end of the period covered by the event log for testing so as to implement
a “not learn from the future” strategy; or one may decide further preferred
“orderings”. As a consequence, _an advanced tool for PPM should enable the
user to split an event log into training and test sets by choosing an
appropriate split strategy_.

Another important step in preparing the training and test set is the choice of
“when to predict” in the ongoing trace, that is the length(s) of the sequences
(prefixes) to be used to train the predictive model. Similarly to the previous
step, several strategies exist in Predictive Process Monitoring to select
prefixes in the construction of a predictive model, which need to be supported
by an advanced Predictive Process Monitoring tool. For example, one can choose
to build a model for predicting exactly at a prefix of length $n$ (that is,
after $n$ events in a sequence), for predicting at any prefix of length up to
$n$, or to build $n$ different predictive models, one for each prefix of
length $i\leq n$. A different strategy would be to choose to predict after
(the first occurrence of) a given activity “A” regardless of when it happens
in the trace. Once the prefix is chosen, an advanced Predictive Process
Monitoring tool must enable the user to decide how to deal with traces that
are shorter than the selected prefix length. Again, different strategies are
possible: for example, sequences shorter than $n$ can be discarded, or they
can be padded with “0” values by adopting a standard strategy when dealing
with input sequences of variable length. As a consequence, _an advanced tool
for PPM should enable the user to select the prefixes used to build the
predictive model and the appropriate prefix selection strategy_.

A similarly important step in building a training and test set is the support
for the labeling of execution traces. While providing support for any kind of
automatic labeling is rather challenging, an advanced tool for PPM should at
least provide support for typical types of labels. Examples are: attributes
used for outcome prediction, next activity(ies), categorical values referring
to the duration of the trace (e.g. fast or slow w.r.t. a given value or some
average trace duration), and so on. Thus, _an advanced tool for PPM should
provide extensive support for automatic labeling of training and test data._

Execution traces are not in a format that can be directly used by a Machine
Learning algorithm. As shown in Section 2.2 different encodings have been
proposed in the literature [6] so as to exploit different characteristics of
the event logs. Therefore, a key capability of an advanced Predictive Process
Monitoring tool is the availability of different encoding methods for the
dataset at hand. Furthermore, the Predictive Process Monitoring literature has
shown that Predictive Process Monitoring approaches can predict an e.g.
outcome of a running case as dependent (resp. independent) on the interplay of
all cases that are being executed concurrently [7]. Together with the choice
of the encoding method, an advanced Predictive Process Monitoring tool must
enable the user to specify whether traces are considered in an intercase vs.
intracase fashion. Thus, _an advanced tool for PPM should enable the user to
select among different encodings_ and _whether to perform predictions in an
intercase or intracase fashion_.

When it comes to the selection of the Machine Learning (ML) algorithm to be
used for training a model, it is well known that “ _no silver bullet exists_
”. Indeed, while it is true that some findings exist [18] one of the
challenges of the Predictive Process Monitoring field is exactly to understand
which specific algorithm is better suited for specific data or predictive
problems. To tackle the issue of building the most appropriate model for the
given data and prediction problem, an advanced Predictive Process Monitoring
tool should provide support for a variety of techniques together with their
parameter tuning. Thus, _an advanced tool for PPM should enable the user to
select among a wide variety of techniques and their configurations_.

Moreover, and most importantly, an advanced tool for PPM should enable the
user to compare configurations emerging from all the different choices
illustrated above. That is, _an advanced tool for PPM should enable the
setting up of different configurations covering all the steps above (from
splitting, to the selection of the ML algorithms and their parameters) and the
comparative construction of all their corresponding predictive models_.

The ability to store a number of alternatives - from the encodings to the ML
techniques - and the need to run the (often expensive) computation of several
predictive models at once imposes requirements on the software architecture
that will be discussed in Section 4. For what concerns the tool support
towards model construction, the building of several models triggers the need
to support some kind of _hyperparameter optimization_. Indeed the optimization
of the hyperparameters of a ML algorithm is known to be a difficult and time
expensive task, and end users may find unfeasible to autonomously optimize the
hyperparameters of multiple ML algorithms. Thus _an advanced tool for PPM
should enable the computation of an optimal configuration for each of the
supported ML algorithms_.

($\mathsf{C2}$) | support the split event log into training and test sets with appropriate split strategy
---|---
($\mathsf{C3}$) | support the selection of prefixes and the appropriate prefix selection strategy
($\mathsf{C4}$) | support for automatic labeling of event logs
($\mathsf{C5}$) | support for the selection of the appropriate encoding of event logs
($\mathsf{C6}$) | support intercase and intracase predictions
($\mathsf{C7}$) | support the choice among a variety of configurable ML techniques
($\mathsf{C8}$) | support the construction of different models
($\mathsf{C9}$) | support automatic hyperparameter optimization
($\mathsf{C10}$) | support different evaluation metrics, including the accuracy ones
($\mathsf{C11}$) | support models comparison in terms of metrics and characteristics of the configuration
($\mathsf{C1}$) | support event log exploration and engineering
($\mathsf{C12}$) | support explanations of predictions

Table 1: The capabilities of a Predictive Process Monitoring tool.

###  3.3 Evaluating the predictive model(s)

The evaluation step is of tantamount importance, when aiming at the
construction of a predictive model that is appropriate for the dataset at
hand. A tool must here support the user in two dimensions: the possibility to
rely on different metrics for each model, and the capability to comparatively
evaluate all the different configurations defined to build different models.

Focusing on the metrics to evaluate each single model, it is well known in
Machine Learning that different evaluation metrics are used for different
kinds of problems. Thus, _an advanced tool for PPM should enable the choice
between a wide set of evaluation metrics_. In particular, it should enable the
choice of appropriate accuracy metrics, as these metrics provide an estimate
of how well the predictive model understood the supplied data and is capable
to exploit new data to compute a value satisfying the prediction problem. In
addition, it should be able to accommodate further metrics useful to
scientists in their experimental evaluations of Predictive Process Monitoring
techniques. An example of such metrics are time metrics. These metrics measure
the amount of time spent by the tool to perform a set of actions, such as
e.g., the time spent in pre-processing the data, or the time spent in building
the predictive model.

Focusing on the comparative evaluation, the comparison between different
predictive models needs to be supported in terms of (at least) two different
instruments. The first instrument is composed of visual (e.g, tabular)
representations enabling the comparison of the different metrics illustrated
above. In that, a user may investigate which configurations have better, say,
F1 or AUC scores. This is a basic evaluation capability which could help for
instance an end user directly selecting the best performing model. The second
instrument should enable the user to drill down and understand how the
goodness of the model (e.g. the AUC) is related to a certain configuration
step, for example the prefix length, the encoding, or the chosen ML technique.
This latter more sophisticated method may be useful to  (i) tailor future
configurations for the construction of further models, in case of not fully
satisfactory ones, and  (ii) carry out scientific investigations, where it is
often required to understand why performances may vary.  Thus, _an advanced
tool for PPM should enable the comparison of different predictive models both
in terms of (performance) metrics and in terms of characteristics of the
configuration used for model construction_.

###  3.4 Explaining the predictive model(s) behaviors

Given the growing importance of Explainable Artificial Intelligence (XAI),
explanation is likely to become one of the standard tasks of future ML based
pipelines. Therefore, even if this task is not yet part of a standard
Predictive Process Monitoring pipeline, a good Predictive Process Monitoring
tool should include the possibility of easily incorporating XAI techniques,
both to explain a particular prediction or model and to be added as one of the
criteria upon which models can be compared. In addition to computing the
explanation, a Predictive Process Monitoring tool should be apt to tailor and
display the explanation to the target user. Given the recent nature of the XAI
field, and the different types of ML techniques, data, and prediction
problems, no general studies exist that indicate which XAI techniques are the
best ones for what. Therefore _an advanced tool for PPM should enable the
usage of different XAI algorithms and of different visualization methods._

Table 1 summarizes all the capabilities of a Predictive Process Monitoring
tool illustrated in this section. As discussed in the section, capabilities
($\mathsf{C2}$)–($\mathsf{C11}$) are considered necessary functionalities of
an advanced tool for PPM, while capabilities ($\mathsf{C1}$) and
($\mathsf{C12}$) are considered useful extra features part of which - as in
the case of ($\mathsf{C12}$) - may become necessary in the future. In the next
section, we describe the architecture of Nirdizati which is tailored towards
supporting the above-mentioned capabilities.

Figure 3: The capabilities of Nirdizati are provided by the concertation of
multiple architectural elements. The direction of the arrows represents the
flow of information. The user accesses the capability interacting with the
Front-end element. In turn, the Front-end element sends requests to the Server
element via the APIs. The Server, the Database and the Workers provide the
functionalities: the simple requests are handled by the Server or the Server
and the Database, whereas the Job requests are handled by the Worker.

##  4 The Nirdizati Architecture

The support of complex capabilities as those mentioned in Section 3 requires
both the ability to offer friendly user interfaces and the ability to ensure
fast processing and a good usage of resources, e.g., to train the models, so
as to exploit the available computational capabilities in an efficient manner.

At the high level, Nirdizati is composed of (i) a Front-end application, which
allows users to select the prediction methods and to assess the goodness-of-
fit of the built models, by means of a full / partial implementation of
capabilities ($\mathsf{C1}$) – ($\mathsf{C12}$), and (ii) of a Back-end
application responsible for the actual training and test (see Figure 3). Our
aim to offer Nirdizati as a service to the research community has triggered
the decision to built it as a web application. We postpone the description of
the different code modules, and their relations with capabilities
($\mathsf{C1}$)–($\mathsf{C12}$), to Section 5. Instead we focus here on the
architectural design that we adopted in order to support modularity and
scalability of the processing (see Figure 4).

Figure 4: Nirdizati follows a client-server architecture. The server exposes
APIs to communicate with the client. When the server receives a new job, it
puts it in a Queue. When a new job is available in the Queue, a worker takes
care of the job. Workers, whose number can vary according to the needs, share
access to the Queue, to the Cache, and to the Database.

In detail, Nirdizati 666Source code available at https://github.com/nirdizati-
research/ supports the multiprocessing capabilities of nowadays processors,
using a Master/Replica setting. In this setting, the Master - the _server_ in
Figure 4 \- creates and orchestrates the jobs and performs other small tasks,
thus providing functionalities such as the Job Validation and Enqueueing, the
Job Result retrieval, the Event Log Storage and Analysis and the Explanation
Generation. The Replica - the _workers_ in Figure 4 \- instead, take care of
performing (most of) the jobs (Job Dequeueing, Execution and Result storage).
They do it by exploiting _code modules_ that implement the required
functionalities in a modular setting (see Section 5).

When using such a setting for comparing different algorithms on the same task,
most of the processing time is taken for the pre-processing phases that are
repeated over and over for each different configuration. To overcome this time
and resource consumption, Nirdizati has been equipped with a _cache system_
that avoids pre-processing the data multiple times when evaluating multiple
approaches on a shared task and dataset. When using caching in the Nirdizati
architecture, however, two or more slaves performing the same pre-processing
step concurrently can generate duplicate cache files. A _database system_ able
to track in a granular way the status of each object of the tool, i.e., the
set of elements used in a given task, has hence been introduced. For instance,
the object representing the task of splitting a log into a training and a test
set is composed of a foreign key to the object representing the log, the
training and the test set sizes, the type of ordering used for the split, the
name of the split, and a unique key. Storing each object of the tool in a
table of the database allows, on the one hand, an easy retrieval of the
required information and, on the other hand, the capability to keep in memory
only the database keys instead of the objects themselves, thus resulting in a
lighter footprint in the memory. From the database it is possible to retrieve
the current jobs and their status, the available cached information, the
previously trained models and the corresponding accuracy metrics. By relying
on a stateful architecture and capitalizing the work carried out, Nirdizati is
able to exploit multiprocessing.

| Capabilities
---|---
Element | [$\mathsf{C1}$, $\mathsf{C7}$, $\mathsf{C11}$] | [$\mathsf{C3}$, $\mathsf{C12}$] | [$\mathsf{C2}$, $\mathsf{C4}$–$\mathsf{C6}$, $\mathsf{C8}$–$\mathsf{C10}$]
Front-end | x | x | x
API | x | x | x
Server | x | x | x
Database | x | x | x
Code | x | x | x
Queue |  |  | x
Cache |  | x | x
Worker |  |  | x
Table 2: Architectural Elements and Capabilities.

Table 2 describes how the different architectural elements of Figure 4 are
involved in the provision of Capabilities ($\mathsf{C1}$)–($\mathsf{C12}$).
For the sake of readability we have grouped capabilities that involve the same
architectural elements. The presence of a “x” on a given cell means that the
Architectural Element of that row is used to provide the group of Capabilities
of that column. For instance, the provision of $\mathsf{C1}$, $\mathsf{C7}$,
and $\mathsf{C11}$ involves the Front-end, the API, the Server, the Database,
and some Code Module.

Not surprisingly, the Front-end, the API, the Server, the Database, and Code
Modules are involved in the provision of all capabilities. The first two
groups of capabilities do not require any intensive computation777Actually
this may change for the provision of explanations ($\mathsf{C12}$), which are
currently handled by the Server only because of the still investigative nature
of this functionality. and are therefore handled directly by the Server, with
the only difference of ($\mathsf{C3}$) and ($\mathsf{C12}$) using the Cache
for optimization purposes. The third group, instead, is the one involved in
the setting up, computation, and comparison of the different predictive
models. Being time and resource expensive, the provision of these capabilities
takes advantage of the Nirdizati Master/Replica architecture based on the
usage of Workers and Caching.

##  5 Code Modules

To ensure modularity, capabilities ($\mathsf{C1}$)-($\mathsf{C12}$) are
supported in Nirdizati by means of nine specific code modules: Event Log,
Split, Label, Encode, Model Construction, Cache, Hyperparameter Optimization,
Model Evaluation, and Explanation, as summarized in Table 3. Each module
provides simple functionalities that compose the internals of the elements
discussed in Section 4 and allows for the provision of the complex
capabilities discussed in Section 3. In the following we briefly describe the
different modules by highlighting their main characteristics and thus
providing an illustration of how they support the required capabilities.

| Capabilities
---|---
Modules | $\mathsf{C1}$ | $\mathsf{C2}$ | $\mathsf{C3}$ | $\mathsf{C4}$ | $\mathsf{C5}$ | $\mathsf{C6}$ | $\mathsf{C7}$ | $\mathsf{C8}$ | $\mathsf{C9}$ | $\mathsf{C10}$ | $\mathsf{C11}$ | $\mathsf{C12}$
Event Log | x | x |  |  |  |  |  |  |  |  |  | x
Split |  | x | x | x | x | x |  |  |  |  | x |
Label |  |  |  | x |  |  |  |  |  | x | x |
Encode |  |  |  |  | x | x |  |  |  |  | x |
Model Construction |  |  |  |  |  |  | x | x |  | x | x | x
Hyper. Opt. |  |  |  |  |  |  |  |  | x |  |  |
Model Evaluation |  |  |  |  |  |  |  |  |  | x | x |
Explanation |  |  |  |  |  |  |  |  |  |  |  | x
Cache | x | x | x | x | x | x |  | x |  | x |  | x

Table 3: Code Modules and Capabilities.

#### Event Log

The Event Log module receives as input the data uploaded by the user, in the
form of an event log in the XES Format, and stores them in the Back-end ready
to be used. In order to help the user to establish which predictive model is
most likely to achieve better performance [19], a description of the event log
in terms of level of contention of resources in the data, parallelism in the
execution, and sparsity of the data is computed and stored at upload time.

#### Split

The Split module receives as input an event log and deals with the production
of the training and the test sets. These sets are subsets of the initial event
log (training and test logs), and their definition influences the quality of
the trained predictive models. To produce the training and test sets, the
Split module needs to support the sorting, filtering, and splitting of the
event log. The sorting of the event log allows for taking into consideration
different perspectives of the data, such as the seasonality or the variants
representation. The filtering of the event log allows for eliciting and
correcting underrepresented variants, wrongly represented traces, malformed
traces, biased traces, and in general information that we are not interested
in representing. The split of the event log allows for obtaining the training
and test sets. The relative sizes of the sets highly influence the final
performance of the predictive model; the split of the event log, indeed, might
create truncated traces, overlaps between the sets, or skewed sets, that can
invalidate the predictive model evaluation, as discussed in [20]. In addition
to the above, the Split module also supports prefix selection.

#### Label

The Label module receives a training or test event log and returns the same
log annotated with the desired label, that is, each trace in the event log is
annotated with the label of interest. Typical labels in PPM refer to
categorical outcomes, numeric values, and next activities. The Label module of
Nirdizati enables the automatic labeling of a log with all three types of
predictions. In detail, for categorical outcomes, the module allows for (i)
multiclass labels starting from categorical attributes and from the next
activity, as well as (ii) binary labels starting from numerical attributes and
trace durations by leveraging a threshold value. For numeric outcomes, the
module allows for numeric labels by starting from numeric attributes and trace
duration.

#### Encode

The Encode module receives a labeled training/test log, and returns its
encoded version as a Dataframe888https://www.databricks.com/glossary/what-are-
dataframes. To produce the Dataframe from a log, three different steps need to
be tackled:  (i) encoding information extraction,  (ii) feature encoding, and
(iii) data encoding.

The _encoding information extraction_ step receives as input a training/test
event log and returns some useful information for the encoding. In this step
the attribute(s) of the event log related to the control-flow (i.e., the
attribute holding the activity value), to the data flow (i.e., trace and event
attributes), as well as to the resource-flow (i.e., the attribute related to
the resource) are extracted. This mapping is useful in order to understand
what kind of information to take in (more) account in the actual encoding.

The _feature encoding_ step receives as input a training/test log and the
encoding information and returns the set of features that will be used to
represent each trace in the training/test sets. This step takes also care of
encoding in a way to support intercase or intracase predictions.

The _data encoding_ step receives as input the training/test log and the
feature set and returns the training/test Dataframe and the encoder object.
The Dataframe is built to be easy to understand for a machine and allows for a
smoother training. The operations performed to build a machine-understandable
object include steps like the one-hot encoding of categorical features, de-
noise and normalization for the numeric object categorization of numeric
features.

#### Model Construction

The Model Construction module receives the predictive model specification and
returns an instantiation of the predictive model with the specification of the
supported hyperparameters space. The Model Construction module is concerned
with the correct initialization of the predictive model. There are three
classes of predictive model building algorithms supported by Nirdizati namely,
(i) classification, (ii) regression, and (iii) clustering. These three classes
allow us tackling the three classic types of predictions: (i) categorical
outcomes, (ii) numerical values, and (iii) next-activities, respectively. In
turn, each class of predictive model building algorithms includes different
specific algorithms that can be further increased, due to the modular nature
of the tool. Section 7.1 provides a description of the Front-end features of
this module.

#### Hyperparameter Optimization

The Hyperparameter Optimization module receives in input the training
Dataframe, an instantiated predictive model, and the predictive model
hyperparameter space, and returns the optimized predictive model. Machine
Learning techniques are known to use model parameters and hyperparameters.
While the values of hyperparameters can influence the performance of the
predictive models in a relevant manner, their optimal values highly depend on
the specific dataset under examination, thus making their setting rather
burdensome. To support and automatize this onerous but important task, several
hyperparameter optimization techniques have been developed in the literature
[21] and included in the Hyperparameter Optimization module.

#### Model Evaluation

The Model Evalauation module receives in input the test Dataframe and the
(hyper)optimized predictive model and returns the evaluation of the model
using the selected metrics. There are two classes of metrics that one can use
in Nirdizati: (i) time and (ii) accuracy metrics. Time metrics are concerned
with understanding how fast the predictive model can train, update and return
predictions. Accuracy metrics, instead, are concerned with understanding how
well the predictions are in comparison with the “perfect predictions”
contained in the test set. Being one of the distinctive features of Nirdizati,
the Model Evaluation module provides a wide range of information for the
comparative evaluation of different predictive models based on their
configurations, results and prefixes. In addition to comparative tables, it
also supports the display of bubble chart visualizations. Section 7.2 provides
a description of the Front-end features of this module.

#### Explanation

The Explanation module receives in input a Dataframe and the trained
predictive model and returns one of the three explanation classes discussed in
[22]. The three classes of explanation refer to three different levels of
abstraction an explanation in Nirdizati can refer to: (i) event, (ii) trace,
and (iii) event log. At the _event_ level, the impact of each feature (and
corresponding value) on the prediction returned by the predictive model at the
current prefix length is shown to the user. At the _trace_ level, how the
impact (in terms of SHAP values) of each feature (and corresponding value) on
the prediction evolves at different prefix lengths is returned to the user.
Finally, at _log_ level, the average value of the prediction over all the
traces of the event log for each value of a given feature is shown to the user
(together with the number of traces characterized by that value for the
feature). Section 7.3 provides an example of one of the features supported by
these module.

#### Cache

The Cache module receives in input an object, such as an event log, a
Dataframe, or a trained predictive model and stores it. The cache module
determines an efficient storage position of the resource, proceeds to store
the object and it retrieves it when needed. In other words, the Cache module
is concerned with the efficient administration of the loaded and created
objects. The objects that are more suitable to be cached are the ones that
take a long time to be computed and occupy a small amount of space on the
disk. We hence cache the loaded log, the labeled dataframe, and the trained
predictive model. The loaded log is an event log that is loaded starting from
the XES format into an in-memory representation, that is then used for the log
encoding. The labeled dataframe is the log that is already transformed in the
Dataframe format. The trained predictive model is the predictive model that is
ready to be used to make predictions, meaning that it has been trained,
optimized and evaluated on a specified test set.

##  6 Implementation

Nirdizati is implemented as a Web application that follows the architecture
described in Section 6, so as to be a well structured, open, flexible and
documented platform for Predictive Process Monitoring supporting the
capabilities described in Section 3.

We implemented the modules described in Section 5 in the back-end of our
toolkit. Moreover, we provide a front-end web application that exploits the
APIs made available by the toolkit to allow the user to interact with
Nirdizati through a state-of-the-art User Interface (UI).

The _Front-end features_ can be accessed via a Web browser that exposes the
capabilities described in Section 3 through a graphical interface. The Web
page is supported by React.js 999https://reactjs.org. The stateful component
design of React.js allows us to load the interface and the state of the UI at
two different times, allowing the user to browse the static content while
waiting for the load of the dynamic content. After the dynamic content load,
the UI renders the information requested by the user without any lag due to
the state being stored locally, thus avoiding further requests to the Back-
end. To support the continuous improvement of Nirdizati we integrated Google
analytics anonymized tracking of the user flow through the pages of the
interface in order to identify the most common flows which can act as
candidates for future improvements.

The _Back-end modules_ described in Section 5 are implemented in Python
101010https://python.org. We exploited the
Django111111https://djangoproject.com framework to expose the APIs,
synchronize the database with the single code modules, and create the elements
of the architecture. The event logs are handled through the PM4py library
[23]. The logs are then encoded and get ready for the predictive model
training using the Pandas Dataframe
objects121212https://pandas.pydata.org/pandas-
docs/stable/reference/api/pandas.DataFrame.html. We exploit the Sklearn
131313https://scikit-learn.org init/fit/partial_fit/predict common signatures
to create, build, update, and evaluate the predictive models. In this way,
enriching the available techniques with new Machine/Deep Learning algorithms,
both included in Sklearn or taken from external libraries such as PyTorch or
TensorFlow through a wrapper, is a simple procedure. The queue handling is
done through the Redis 141414https://redis.io Library. The database manager is
Postgres 151515https://postgresql.org.

##  7 Using Nirdizati

In this section, we provide an overview of the tool interface. For the sake of
presentation we focus on the features that characterize more Nirdizati, that
is model construction, comparative evaluation and prediction explanation. The
reader interested in exploring all the Nirdizati features can do it either by
assessing the tool at http://research.nirdizati.org/ or by following the step-
by-step tutorial at https://bit.ly/tutorial_nirdizati.

###  7.1 Model Construction

The Nirdizati _Training page_ , accessible by clicking on the spanner icon of
the navigation menu (red box 1 in Figure 5), enables the user to configure the
construction of a predictive model. We illustrate the features provided in
this page with the example of a predictive model for a binary configuration
task with the help of Figure 5161616We illustrate here the main features of
the page leaving the rest to the tutorial at
https://bit.ly/tutorial_nirdizati.

Figure 5: The _Training page_ of Nirdizati.

The Nirdizati interface enables the user to select: the appropriate Prediction
method (red box A), the specific Classification method(s) (red box B); whether
to cluster the event log traces and, in this case, which Clustering method(s)
to use (red box C); the Encoding method(s) (red box D); whether to use padding
for the encoding (red box E); the Prefix length(red box G); and how to deal
with the prefixes up to the chosen prefix length (red box F). Finally, the
interface enables the user to select the label to be used (red box H) in terms
of Label type, Attribute name, and Threshold, the type of hyperparameter
optimization, here condensed for lack of space (red box I), and finally to
Submit the request (red box L).

###  7.2 Model Evaluation

Clicking on the histogram icon of the navigation menu (red box 2 in Figure 5),
the user can open the _Evaluation page_. In this page the user can (filter
and) select the predictive models to be evaluated and compared.

Figure 6: The features for the comparative evaluation of Nirdizati.

For reasons of space we skip the model selection part of the page and focus on
the comparative evaluation instruments, by assuming that the user is
interested in comparing the models obtained by using 2 techniques (Random
Forest and XGBoost), 2 encodings (simple and complex index), and 2 prefix
lengths (5 and 10 events), thus resulting in $2^{3}$ predictive models. The
names of the models in the table _Classification results_ , column _Task
Identity_ (see red box A) in Figure 6 provide a hint of the configuration
parameters selected for building the models.

The results of the evaluation of the different models are reported in three
different sections: the _Result_ , _Prefix length_ and _Bubble Chart_
sections. The _Result_ section (red box A) reports the values scored by the
different predictive models on the test set, using a wide range of metrics,
appropriate for the specific prediction task. This section allows the user to
compare the metrics, to dynamically re-rank the models depending on the
results in specific columns, and also to export them in a tabular format. The
_Prefix length_ section (red box B) displays the variation of performance
depending on the prefix length. In particular the left-hand side diagram
displays the trends of a specific metric (in Figure 6 the Elapsed time) for
each predictive model across the chosen prefix lengths, while the right-hand
side displays the performance of selected models (in Figure 6 all), for a
specific prefix (in our case the four models having prefix length equal to
10). The larger the polygon, the better the model. Finally, the Bubble Chart
section (red box C) displays four different bubble chart plots allowing the
user to compare the models (identified by their Model ID) using different
perspectives. In Figure 6 we can explore the comparison by focusing on the
predictive model construction and encoding techniques used, respectively. The
plots report on the two axes the AUC and F1 metrics, so that the closer to the
top right corner, the better the model for both metrics.

###  7.3 Explanation

Clicking on the _verify_ icon of the navigation menu (red box 3 in Figure 5),
the user can open the Explanation page from where she can select the
predictive model used and the trace prediction to be explained (step omitted
for space limitation). As illustrated in Section 5, Nirdizati supports
explanations at the level of event, trace, and event log. Figure 7 provides an
example of the trace level explanation. The plot shows the correlation between
3 features (Age=20, Weight=50 and Rehabilitation Prescription=false) related
to an incomplete trace of a patient who has carried out a treatment for a
broken bone and the corresponding prediction (the patient will recover soon
from the fracture) at different prefix lengths. As we can see from the plot,
at the beginning of the trace (prefix length $\leq 2$) the age of the patient
positively correlates with the patient fast recovery, while the absence of a
Rehabilitation Prescription has a negative impact on this outcome. We can also
note that the correlation of these two features with the prediction outcome
vary in a considerable manner along the incomplete trace, while the trend of
Weight=50 remains rather stable – it does not seem to have a strong impact on
the prediction along the whole prefix length. A detailed illustration of all
the plots provided in Nirdizati can be found in the aforementioned tutorial or
in [22]. All the plots support the interaction with the user with hovering (to
focus on an element) and clicking (to filter out unnecessary elements)
behavior.

Figure 7: The trace level plot of the the _Explanation page_.

##  8 Assessment

The implementation of Nirdizati is assessed through the use of reactive design
patterns [24] and load tests [25, 26]. The reactive design patterns aim at
assessing the interaction among the architectural elements, while the load
tests are designed to emulate the work of multiple users accessing the
prototype in a concurrent manner.

#### Assessing the interaction among architectural elements

The interaction among the architectural elements in Nirdizati is not trivial,
as illustrated in Figure 3.

We streamline the interaction by employing the reactive design patterns [24]
that concern:  (i) fault tolerance/recovery,  (ii) control flow  (iii)
replication,  (iv) resource management, and  (v) state management.

Fault Tolerance is ensured by delegating the predictive models’ training and
testing to the Workers, which are completely separated from the rest of the
toolkit. They are resilient towards code exceptions in the training/test of
the predictive model and are instructed to save the experienced exceptions in
the database. The logs can be monitored through a Grafana
171717https://grafana.com extension and, in case of unhandled exceptions, the
Fault Recovery is granted by the re-spawn feature of the Docker
181818https://docker.com daemon. The re-spawn of the Worker element and its
connection to the rest of the toolkit is allowed by the Queue element, which
can dynamically track the existence of any number of workers.

Concerning the Control Flow, the Queue element in combination with the Worker,
the Server, and the Database elements allows us to implement the Pull pattern
to give the Worker the time to produce the predictive model. The Server
element is not tied to wait for the Worker, which, once the training is
finished, publishes its results in the Database element.

The capability to handle heavy workloads in terms of the amount of Jobs to
carry out is ensured through the Master/Replica Replication pattern. The
Master role is covered by the Server element. On the one side, it creates new
jobs for the Workers and allows the workers to be aware of the existence of
such jobs by saving the Job identifiers (IDs) on the Queue element; on the
other side, it acts as an interface to the Front-end element exposing RESTful
191919https://restfulapi.net APIs. The Replica role is covered by the
instances of the Worker element. The Worker element instances have two states,
idle and running: when the Worker is in the idle state it is connected to the
Queue element and waits for a new job ID to be published; when a new Job ID is
available one Worker pops it from the Queue element, changes the Worker state
into running and processes the Job. The details regarding the Jobs are stored
in the Database element and are accessible through the Job ID inside the Back-
end or through the Server element APIs. Such APIs can be accessed by the
Front-end element which is a Web application allowing the user to access the
Back-end in a facilitated manner. However, a more experienced user is not
bound to use it. The experienced user can batch operations via simple scripts
that can interact through the RESTful APIs with the Back-end without involving
the Front-end.

Concerning Resource Management, the event log and cached objects concurrent
access is allowed by the Cache element. The Cache element is the sole owner of
every resource in the toolkit, and the Cache element then implements the
Resource Loan pattern allowing those who need the resource to access it but
not to modify it. Also, the Resource Pool pattern is implemented: every time
an element in the toolkit needs a resource it invokes the Cache element to
access it. Indeed, there is no other way to access a resource.

Finally, State Management is ensured by the Dissemination of Information
pattern which consists of storing multiple representations of the same object,
which then allows for accessing such an object with a lower burden on the
originating domain object (e.g., the original event log), thus enabling a
faster and more flexible use of it. This pattern is applied on the event log
objects: event logs are cached both in the pre-loaded and pre-processed form.

#### Assessing the workload

Nirdizati supports the users’ requests by means of three different workflows:
the upload workflow, the job computation request workflow, and the results
browsing workflow. The _upload workflow_ involves the Server element, the
Database element, and the main memory of the host machine. The _job
computation request workflow_ involves the Server element and the Database
element. The _results browsing workflow_ involves the Server element, the
Database element, and the Worker element.

We did assess these three workflows by achieving a stable CPU usage above
$80\%$ and an error per second $\leq 5\%$, as suggested in [25] for stress and
load testing. The stress test was performed using
Vegeta202020https://github.com/tsenart/vegeta, while Nirdizati was hosted on a
machine equipped with 4 CPUs and 16GB of RAM. Table 4 reports the results in
terms of requests per second (rps) sustained by each workflow both at peak
load and at sustained load. The tests show that the most computationally
demanding workflow is the result browsing workflow. By assuming an average of
10-15 rps by user, we can conclude that Nirdizati can support the concurrent
usage of about 40 users.

Workflow | Peak Load | Sustained Load
---|---|---
Upload | 753 | 62
Job computation | 767 | 64
Result browsing | 750 | 60
Table 4: Assessing the workloads.

##  9 Related Work

The most popular tools that implement PPM techniques are several ProM plug-
ins, Apromore, IBM Process Mining Suite, and a Camunda plug-in.

ProM [27] is a framework collecting a variety of independent plug-ins. Many of
them are devoted to compute a specific PPM prediction problem, that is,
providing  (i) numeric predictions [28, 29];  (ii) outcome-based predictions
[2, 30]; and  (iii) next activity predictions [31, 8].  Despite the
aforementioned wide amount of plug-ins tackling single PPM challenges, the
only plug-in in ProM implementing multiple complex capabilities, such as the
ones described in this work, is the one in [32]. This plugin, however, only
tackles the outcome and the numeric prediction problem, with no support for a
scalable and modular architecture such as the one at the basis of Nirdizati
and no support for explanations.

Apromore [33] is an advanced process mining tool that allows for the storage,
analysis, and re-use of large sets of process models. The tool is web-based
and therefore allows for the easy integration of new plug-ins in a service-
oriented manner. This tool aims both at allowing practitioners to deal with
the challenges of process stakeholders, and at enabling researchers to develop
and benchmark their own techniques with a strong emphasis on the separation of
concerns. The only plug-in performing PPM related challenges in Apromore is
the one in [34]. The plug-in performs outcome-based prediction, time-based
prediction, next activity prediction, and the computation of performance
metrics w.r.t. the event log. However, the amount of performance metrics
available in this tool is limited; no help/indication on the pre-processing of
the event log is given to the users, and there is no support for any
explanation. Differently from Nirdizati, Apromore also supports the model
deployment phase.

The IBM Process Mining Suite [35] 212121https://ibm.com/cloud/cloud-pak-for-
business-automation/process-mining is a commercial software suite allowing for
the discovery, conformance checking, monitoring, and simulation of processes.
The suite is multilevel and provides business analysts with a digital twin of
an organization (DTO). While this tool supports explanation and hyper-
parameter tuning, differently from Nirdizati, it mainly focus on a single
predictive technique (CatBoost), thus lacking the capabilities related to
model evaluation and comparison.

Finally, Camunda 222222https://camunda.com/ is an open-source workflow and
decision automation platform. The work in [36] presents a plugin that allows
for training, optimizing, and using PPM models. While the proposed plug-in is
able of tackling all three types of predictions of PPM, and automatically
choosing the hyperparameters of the predictive model, it supports a limited
amount of learning algorithms and a limited amount of evaluation metrics;
moreover, no help/indication on the pre-processing of the event log is
provided to the users and no support is given to explain the built predictive
models.

##  10 Conclusion

This paper presented the latest advancements of Nirdizati. Specifically, its
Back-end has been redesigned to provide a reliable and efficient framework
that allows for an easy comparison of different state-of-the-art Predictive
Process Monitoring approaches and the inclusion of a wide set of PPM
techniques, by leveraging its modular architecture.

The recent stream of publications in the Predictive Process Monitoring field
[5] shows the need for tools able to support researchers and users in
analyzing, comparing and selecting the techniques that are the most suitable
for them. This need is also reflected in the growth of Predictive Process
Monitoring plug-ins in well known process mining tools such as ProM [27] and
Apromore [34]. Nirdizati is a completely dedicated tool for running a very
rich set of Predictive Process Monitoring techniques and its latest
advancements make it even more robust, scalable and usable.

We assess the current Technology Readiness Level of Nirdizati to be 5. This
release offers indeed a well-defined structure of the software and code
documentation;232323https://nirdizati-research.readthedocs.io/ moreover, it is
equipped with a very large test suite, and a _Continuous Integration_
deployment pipeline. In addition to the workload tests presented in Section 8,
the tool has been extensively used and its features exercised on both
simulated and real data 242424The usage of Nirdizati is monitored employing
Google Analytics and the two years time span between the first prototype and
the writing of this work recorded 14.637 sessions filtering bounce traffic and
spending longer than a minute out of 23.871 sessions in total. We believe that
all these reasons make Nirdizati a mature and useful instrument for the BPM
community.

## References

  * [1]
  * [2] Maggi, F.M., Di Francescomarino, C., Dumas, M., Ghidini, C.: Predictive monitoring of business processes. In: Advanced Information Systems Engineering - 26th International Conference, CAiSE 2014. Proceedings. LNCS, vol. 8484, pp. 457–472. Springer, Heidelberg (2014)
  * [3] Di Francescomarino, C., Ghidini, C.: Predictive process monitoring. In: Process Mining Handbook. LNBIP, vol. 448, pp. 320–346. Springer, Heidelberg (2022)
  * [4] Verbeek, H.M.W., Buijs, J.C.A.M., van Dongen, B.F., van der Aalst, W.M.P.: XES, XESame, and ProM 6. In: Information Systems Evolution - CAiSE Forum 2010, Selected Extended Papers. LNBIP, vol. 72, pp. 60–75. Springer, Heidelberg (2010)
  * [5] Di Francescomarino, C., Ghidini, C., Maggi, F.M., Milani, F.: Predictive process monitoring methods: Which one suits me best? In: Business Process Management - 16th International Conference, BPM 2018, Proceedings. LNCS, vol. 11080, pp. 462–479. Springer, Heidelberg (2018)
  * [6] Leontjeva, A., Conforti, R., Di Francescomarino, C., Dumas, M., Maggi, F.M.: Complex symbolic sequence encodings for predictive monitoring of business processes. In: Business Process Management - 13th International Conference, BPM 2015, Proceedings. LNCS, vol. 9253, pp. 297–313. Springer, Heidelberg (2015)
  * [7] Senderovich, A., Di Francescomarino, C., Ghidini, C., Jorbina, K., Maggi, F.M.: Intra and inter-case features in predictive process monitoring: A tale of two dimensions. In: Business Process Management - 15th International Conference, BPM 2017, Proceedings. LNCS, vol. 10445, pp. 306–323. Springer, Heidelberg (2017)
  * [8] de Leoni, M., van der Aalst, W.M.P., Dees, M.: A general process mining framework for correlating, predicting and clustering dynamic behavior based on event logs. Inf. Syst. 56, 235–257 (2016)
  * [9] Di Francescomarino, C., Dumas, M., Maggi, F.M., Teinemaa, I.: Clustering-based predictive process monitoring. IEEE Trans. Serv. Comput. 12(6), 896–909 (2019)
  * [10] Tax, N., Verenich, I., Rosa, M.L., Dumas, M.: Predictive business process monitoring with LSTM neural networks. In: Advanced Information Systems Engineering - 29th International Conference, CAiSE 2017, Proceedings. LNCS, vol. 10253, pp. 477–492. Springer, Heidelberg (2017)
  * [11] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, Proceedings., pp. 2546–2554 (2011)
  * [12] Di Francescomarino, C., Dumas, M., Federici, M., Ghidini, C., Maggi, F.M., Rizzi, W., Simonetto, L.: Genetic algorithms for hyperparameter optimization in predictive business process monitoring. Inf. Syst. 74(Part), 67–83 (2018)
  * [13] Ribeiro, M.T., Singh, S., Guestrin, C.: “why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining., pp. 1135–1144. ACM, New York City (2016)
  * [14] Lundberg, S.M., Lee, S.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 4765–4774 (2017)
  * [15] Denker, J.S., Schwartz, D.B., Wittner, B.S., Solla, S.A., Howard, R.E., Jackel, L.D., Hopfield, J.J.: Large automatic learning, rule extraction, and generalization. Complex Systems 1(5) (1987)
  * [16] Stierle, M., Brunk, J., Weinzierl, S., Zilker, S., Matzner, M., Becker, J.: Bringing light into the darkness - A systematic literature review on explainable predictive business process monitoring techniques. In: 28th European Conference on Information Systems - Liberty, Equality, and Fraternity in a Digitizing World, ECIS 2020 (2021)
  * [17] Calvanese, D., Kalayci, T.E., Montali, M., Tinella, S.: Ontology-based data access for extracting event logs from legacy data: The onprom tool and methodology. In: Business Information Systems - 20th International Conference, BIS 2017, Proceedings. LNBIP, vol. 288, pp. 220–236. Springer, Heidelberg (2017)
  * [18] Teinemaa, I., Dumas, M., Rosa, M.L., Maggi, F.M.: Outcome-oriented predictive process monitoring: Review and benchmark. ACM Trans. Knowl. Discov. Data 13(2) (2019)
  * [19] Drodt, C., Weinzierl, S., Matzner, M., Delfmann, P.: The recomminder: A decision support tool for predictive business process monitoring. In: Proceedings of the Best Dissertation Award, Doctoral Consortium, and Demonstration & Resources Track at BPM 2021. CEUR Workshop Proceedings, vol. 2973, pp. 131–135. CEUR-WS.org, RWTH Aachen (2021)
  * [20] Weytjens, H., Weerdt, J.D.: Creating unbiased public benchmark datasets with data leakage prevention for predictive process monitoring. In: Business Process Management Workshops - BPM 2021 International Workshops, Revised Selected Papers. LNBIP, vol. 436, pp. 18–29. Springer, Heidelberg (2021)
  * [21] Bergstra, J.S., Bengio, Y.: Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012)
  * [22] Rizzi, W., Comuzzi, M., Di Francescomarino, C., Ghidini, C., Lee, S., Maggi, F.M., Nolte, A.: Explainable Predictive Process Monitoring: A User Evaluation. arXiv e-prints (2022)
  * [23] Berti, A., van Zelst, S.J., van der Aalst, W.M.P.: Process mining for python (pm4py): Bridging the gap between process- and data science. CoRR abs/1905.06169 (2019) 1905.06169
  * [24] Bonér, J., Farley, D., Kuhn, R., Thompson, M.: The reactive manifesto. Dosegljivo: http://www.reactivemanifesto.org/.[Dostopano: 21. 08. 2017] (2014)
  * [25] Shafique, M., Labiche, Y.: A systematic review of state-based test tools. Int. J. Softw. Tools Technol. Transf. 17(1), 59–76 (2015)
  * [26] Schieferdecker, I., Din, G., Apostolidis, D.: Distributed functional and load tests for web services. Int. J. Softw. Tools Technol. Transf. 7(4), 351–360 (2005)
  * [27] van Dongen, B.F., de Medeiros, A.K.A., Verbeek, H.M.W., Weijters, A.J.M.M., van der Aalst, W.M.P.: The ProM framework: A new era in process mining tool support. In: ICATPN 2005, pp. 444–454 (2005)
  * [28] Polato, M., Sperduti, A., Burattin, A., de Leoni, M.: Data-aware remaining time prediction of business process instances. In: 2014 International Joint Conference on Neural Networks, IJCNN 2014, pp. 816–823 (2014)
  * [29] Folino, F., Guarascio, M., Pontieri, L.: Discovering context-aware models for predicting business process performances. In: On the Move to Meaningful Internet Systems: OTM 2012, Confederated International Conferences: CoopIS, DOA-SVI, and ODBASE 2012, Proceedings, Part I, pp. 287–304 (2012)
  * [30] Castellanos, M., Salazar, N., Casati, F., Dayal, U., Shan, M.: Predictive business operations management. IJCSE 2(5/6), 292–301 (2006)
  * [31] Polato, M., Sperduti, A., Burattin, A., de Leoni, M.: Time and activity sequence prediction of business process instances. Computing 100(9), 1005–1031 (2018)
  * [32] Federici, M., Rizzi, W., Di Francescomarino, C., Dumas, M., Ghidini, C., Maggi, F.M., Teinemaa, I.: A ProM Operational Support Provider for Predictive Monitoring of Business Processes. In: Proceedings of the BPM Demo Session 2015. CEUR Workshop Proceedings, vol. 1418, pp. 1–5. CEUR-WS.org, RWTH Aachen (2015)
  * [33] La Rosa, M., Reijers, H.A., van der Aalst, W.M.P., Dijkman, R.M., Mendling, J., Dumas, M., García-Bañuelos, L.: APROMORE: an advanced process model repository. Expert Syst. Appl. 38(6), 7029–7040 (2011)
  * [34] Verenich, I., Mõskovski, S., Raboczi, S., Dumas, M., La Rosa, M., Maggi, F.M.: Predictive process monitoring in Apromore. In: Information Systems in the Big Data Era - CAiSE Forum 2018, Proceedings. LNBIP, vol. 317, pp. 244–253. Springer, Heidelberg (2018)
  * [35] Galanti, R., de Leoni, M., Marazzi, A., Bottazzi, G., Delsante, M., Folli, A.: Integration of an explainable predictive process monitoring system into ibm process mining suite (extended abstract). In: ICPM 2021 Doctoral Consortium and Demo Track 2021, Proceedngs. CEUR Workshop Proceedings, vol. 3098, pp. 53–54. CEUR-WS.org, RWTH Aachen (2021)
  * [36] Bartmann, N., Hill, S., Corea, C., Drodt, C., Delfmann, P.: Applied predictive process monitoring and hyper parameter optimization in camunda. In: Intelligent Information Systems - CAiSE Forum 2021, Proceedings. LNBIP, vol. 424, pp. 129–136. Springer, Heidelberg (2021)