# Explainability in Process Outcome Prediction: Guidelines to Obtain
Interpretable and Faithful Models

Alexander Stevens, Johannes De Smedt
Research Centre for Information Systems Engineering (LIRIS)
KU Leuven
Leuven
{alexander.stevens<EMAIL_ADDRESS>

###### Abstract

Predicting the outcome of various sorts of processes has seen a strong uptake
in recent years due to the advances in machine and deep learning. Although a
recent shift has been made in the field of process outcome prediction to use
models from the explainable artificial intelligence field, the evaluation
still occurs mainly through predictive performance-based metrics, thus not
accounting for the explainability, actionability, and implications of the
results of the models. This paper addresses explainability through the
properties _interpretability_ and _faithfulness_ in the field of process
outcome prediction. We introduce metrics to analyse these properties along the
main dimensions of (business) process data: the event, case, and control flow
attributes. This allows comparing inherently created explanations with post-
hoc explanations techniques, for which we benchmark seven classifiers on
thirteen real-life events logs covering a range of transparent and non-
transparent machine learning and deep learning models complemented with post-
hoc (model-agnostic or model-dependent) explainability techniques. Next, this
paper contributes a set of guidelines named X-MOP which allows selecting the
appropriate model by providing insight into how the varying preprocessing,
model complexity, and explainability techniques typical in process outcome
prediction influence the explainability of the model.

_K_ eywords Explainable Artificial Intelligence $\cdot$ Process Outcome
Prediction, Interpretability $\cdot$ Faithfulness $\cdot$ Deep Learning
$\cdot$ Machine Learning

##  1 Introduction

Both in operations research (OR) and business process management (BPM),
prevalent topics include the modelling of processes in order to identify
possible problems such as bottlenecks caused by a mismanagement or lack of
resources [1] with the goal to find root causes in the process flow [2]. Over
the past two decades, the BPM domain has seen a strong uptake of data-driven
process analysis, coined under the term process mining, which uses process
data generated by executed processes for cases within an information system
[3]. This follows a similar trend in OR, where research shifted because of the
access to large databases on (operational) transactions and a lack of back
testing [4]. The focus of this study lies with predictive process monitoring
[5], the umbrella term geared towards process mining for predictive
activities. It allows identifying process-related trends regarding particular
outcomes (e.g., will customers be awarded credit?), impeding bottlenecks
(e.g., how long will it take to process my credit application?), and whether
particular activities will occur in the future (e.g., will a credit check be
necessary for this application?). When the concrete objective is to predict
the outcome of an incoming, incomplete case, the field of study is referred to
as Process Outcome Prediction (POP). The process data used in this research
field is also referred to as _event logs_ , as the occurrence of a single
activity in a process (case) is referred to as ‘event’. Moreover, an event log
consists of traces, each a sequence of events produced in the context of one
case. In the situation of a loan application process, each event records the
occurrence of an activity (with the activity label being the _control flow
attribute_) up until the loan request is either accepted or rejected. Each
activity can have complementary _case attributes_ and _event attributes_ such
as the requested loan amount (case) and the current officer (event)
respectively. These case attributes remain unchanged (i.e. _static_) within
each case and hence in the whole trace, while the event attributes have
varying values (i.e. _dynamic_) for every event. This means that an event
possibly has three different attribute types: event, case, and control flow
attributes. In recent years, a wide array of traditional machine learning
models have been used for process outcome prediction [5]. An often anticipated
trend is the introduction of computationally complex models in order to
improve the predictive performance [6]. The use of such complex _black box_
models comes at the cost of obfuscating their inner workings and therefore
being unable to provide insights into why a certain business process
prediction was made. The field of eXplainable Artificial Intelligence (XAI)
focuses on gaining insights into _how_ and _why_ certain predictions were
made, while trying to maintain the predictive performance of these highly
performant models [7]. However, many XAI proponents state that trying to
explain these black box models comes with a loss of _faithfulness_ (or
fidelity) due to the fact that the relationship between input and output is
only an approximation [8, 9, 10]. Nonetheless, the use of increasingly complex
models has been widely adopted in high stake decision-making processes
throughout society [11].

This paper identifies and addresses two main issues that arise at the
intersection of POP and XAI. First, the existing POP research is constrained
to the predictive performance while neglecting the interpretation,
actionability, and implications of the results. Many of the existing
explainability metrics are either not model-agnostic (i.e. dependent on model
parameters), not able to compare transparent versus black box models, or not
adapted to a process-based analysis (i.e., do not take into account the
different dimensions of process data). In addition, recent literature pointed
out that the post-hoc explainability techniques should uncover, apart from an
interpretable explanation, the _true reasons_ for model predictions. Hence, a
more comprehensive study about the faithfulness and interpretability of XAI
techniques in POP is missing, which impedes the selection of the most
appropriate predictive algorithms and explainability models to achieve an
interpretable outcome. We address this issue by introducing model-agnostic
explainability metrics based on the properties interpretability and
faithfulness, which are able to assess and compare transparent models versus
black box models that require post-hoc model-agnostic (or model-dependent)
explainability methods. These metrics are not yet available in the field of
XAI, and are analysed along the event, case, and control flow perspective,
making them suitable for evaluation purposes in the field of POP. We perform a
benchmarking study with five machine learning (ML) and two deep learning (DL)
algorithms with different explainability approaches. In this sense, this paper
is complementary with the study of [5] and [6], as it compares the
experimental set-up with both deep learning and newly-introduced interpretable
models, i.e., the Logit Leaf Model (LLM) [12] and the Generalized Linear Rule
Models (GLRM) [13]. We also provide generic, model-agnostic implementations
for the XAI metrics which are also compatible with the latter models. Hence,
this study adds the XAI aspect and reconsiders what models excel in terms of
predictive accuracy, interpretability, and faithfulness. Second, the main
focus of this field is to verify its conformance with respect to the business
requirements and goals [14, 15], but there remains a need for a set of
guidelines towards obtaining faithful and interpretable explanations in the
context of business process monitoring. We provide a framework to obtain
accurate and eXplainable Methods for process Outcome Prediction (X-MOP). This
framework of guidelines is based on the evaluation of a wide benchmark of
preprocessing, traditional machine learning, deep learning and explainability
approaches.

This work extends the initial work of [16], focused on comparing inherently
created explanations with post-hoc explanations in the context of POP. This
paper is a considerable extension with new event logs, new algorithms and new
XAI metrics. Next, we further adapt the original metrics along the control,
case, and event perspective, which is typical for a process-based analysis.
Finally, this papers has bundled the insights obtained from the wide benchmark
into a framework of guidelines to obtain accurate and explainable models for
POP. The rest of the paper is organized as follows. First, a review of the
literature regarding explainability in predictive process monitoring is given
in Section 2, together with the motivation for this line of research. This is
followed by preliminaries defined in Section 3. Next, a definition for
explainability is given, together with the introduced metrics in Section 4.
The benchmark study and implementation details can be found in Section 5,
after we clarify the research questions in Subsection 5.1. In Section 6, the
insights obtained from the research questions are incorporated into the
framework of guidelines named X-MOP. Finally, the results and obtained
insights are concluded in Section 7.

##  2 Related Work and Motivation

Predictive process monitoring is concerned with providing insights about the
business processes of modern organizations. Most predictive efforts are
primarily driven by using classical machine learning (ML) approaches such as
logistic regression (LR) [5], ensemble methods such as XGBoost (XGB) and
Random Forest (RF) [17, 18], with recent works showing interest in applying
deep learning models [19, 20]. In [19], they propose to use attention-based
bidirectional Long Short-Term Memory (LSTM) neural networks for process
outcome prediction. Next, [20] compare LSTMs with Convolutional Neural Network
(CNN), stating that the latter models work better. The study of [5] provides
an extensive benchmark that comprises different event logs and classic
approaches used for process outcome prediction, with the study of [6]
describing which event log properties facilitate the use of deep learning (DL)
methods.

Nonetheless, the lack of transparency of these sophisticated models prohibits
the ability to understand the rationale of the decision-making process. Over
the last two years, the topic of explainability has gained traction in
predictive process monitoring. The different works are often divided into two
different trends based on how they deal with the explainability-performance
trade-off [7]. The first trend generates _post-hoc_ explanations for black box
models. Several papers have already suggested explainability techniques on top
of machine learning models, such as Local Interpretable Model-Agnostic
Explanations (LIME) SHapley or Additive exPlanations (SHAP) values [21, 22],
with similar developments in a deep learning context. As an example, [23]
visualizes the influence of certain attributes in the different steps of the
process in a LSTM model with the use of SHAP values. Next, [24] focuses on
creating local post-hoc explanations with the use of a surrogate decision
tree. In [25], Bidirectional LSTMs are used, where the hidden states of the
time steps of both RNNs are concatenated. After this, a context vector is
learnt that takes the different time steps into account. [26] visualizes the
impact of the activities on the predictions with the use of gated graph neural
networks. The second trend introduces interpretable models instead of trying
to _break open_ these black box models, stating that there are alternative
models that yield better explainability-performance trade-offs. In [27], a set
of fuzzy rules are learnt from neural networks. Here, the relationship between
inputs and output is determined by a set of IF-THEN rules. Nonetheless, this
approach requires domain knowledge in order to bin the attributes into
different interpretable terms. Next, a Bayesian network is used in [28] for
next event and suffix prediction. Even though the causal relationships are
inferred from historical data, it relies on domain knowledge assumptions.
Finally, [16] draws two advanced logistic regression models from the XAI
literature and adapts them to POP. These models, the Logit Leaf Model (LLM)
and the Generalized Logistic Rule Model (GLRM), are originally introduced by
[12] and [13] respectively.

In the field of XAI, a wide range of works have already evaluated these
different models with predictive performance-based metrics or with metrics
that assess the quality of the explanation methods. As an example, [29]
created a metric for explainability based on _human-friendly properties_ ,
while [30] introduced a metric based on three different properties that define
_model complexity_. Nonetheless, these metrics do not take into account the
different attributes that are typical of a process-based analysis. The
benchmark study of [21] compares different explainability models in the field
of predictive process monitoring, but the properties evaluated, _stability_
and _duration of execution_ , are not directly related to explainability
and/or take into account the dimensions of process data. Furthermore, none of
the above metrics evaluate the faithfulness of the explainability model.
However, work from related fields show that there are substantial problems
with the faithfulness of post-hoc explainability methods. First, the work of
[31] states that the learned attention weights are uncorrelated with the
gradient-based attribute importance, even though they mimic the predictions
rather accurately. In addition, they state that a Leave-One-Out (LOO)
attribute importance ranking correlates better with the gradient-based
attribute importance ranking. Similarly, the findings of [32] show that the
importance ranking of the attributes made by attention scores is not faithful
to the model decisions (i.e., what the model perceived as important). Second,
[33] demonstrated that there is a non-monotonic relationship between the SHAP
values and the predictive performance. Third, [34] introduced metrics to the
field of POP that assess the relevance of the attribute value range and the
relevance of the decision boundary by calculating the change in predictions by
permuting the attribute values inside and outside the attribute value range,
respectively. The results show that both LIME and SHAP report low-to-moderate
scores for both the faithfulness metrics. A comprehensive study about the
(un)faithfulness of explainability methods in POP is hence missing, due to the
fact that the accuracy, by which these post-hoc explanations reflect the
behaviour of the predictive model, is often inadequate [8, 16].

To conclude, there is a need for model-agnostic explainability metrics that
are adapted to POP and work for both transparent and non-transparent models.
Furthermore, the faithfulness of (post-hoc) explainability methods used for
POP needs to be evaluated with the use of metrics that are based on the
properties of _explainability_ , i.e. _interpretability_ and _faithfulness_.
These domain-specific metrics can guide practitioners to select the best model
for the task at hand [35] and can reduce the scope of research for human-based
studies by reducing the financial and time costs of such experiments. There is
a notable void in the POP literature in this respect, which this paper will
address through both introducing the notions of interpretability,
faithfulness, and a POP-specific set of guidelines.

##  3 Preliminaries

This section first describes the different nomenclatures inherited from the
XAI field, followed by the preliminary steps needed for predictive process
monitoring.

###  3.1 The different XAI nomenclatures

The _task model_ is defined by several studies as the predictive model that
generates the predictions [9, 8]. The _explainability model_ is the model that
generates the explanations for the predictions made by the task model. Recent
literature describes _transparency_ as the opposite of _blackbox-ness_ [7]. In
the case of a _transparent_ task model (also referred to as an _interpretable_
model), the model is able to generate its own explanations, where a _black
box_ model requires the need of an additional explainability model. This means
that a transparent model is technically also its own explainability model,
while the explainability model of a black box model can be e.g. a surrogate
model, attention layer, or SHAP values. Furthermore, _interpretability_ ,
originally described as _comprehensibility_ in [7], is the ability to provide
an explanation that consists solely out of single chunks of information,
preferably in a human understandable fashion. It is often quantified by the
related concept of model complexity [7, 30]. Note that an  _interpretable
model_ is different from the  _interpretability_. E.g., an interpretable model
(e.g., a logistic regression model) that creates its own explanations based on
more than 500 attributes has a low value for interpretability. In the XAI
literature, the explainability-accuracy trade-off compares the model
interpretability with the model accuracy, assuming that it is required to
strike a balance between either simple models (e.g., linear regression) or
models using complex inference structures (e.g., neural networks). By
contrast, this paper investigates whether the _interpretability of an
explanation_ is also in trade-off with the _predictive accuracy_. In this
sense, a non-interpretable deep neural network can have higher value for the
interpretability of explanations compared to a logistic regression model. This
is similar to the insights obtained in [9]. Next, even though often used
interchangeably [7], interpretability and _explainability_ differ
significantly due to the fact that an interpretable explanation is not always
faithful. To emphasize, a simple explanation generated for a rain forecast
prediction could be: ‘if the grass is green, it will rain’, which is easy to
interpret, but unfaithful. The necessity to distinguish between faithfulness
and interpretability has already been pointed out by prior research [9, 10].
The _faithfulness_ of an explainability model can be considered as the
accuracy by which the explainability model accurately mimics the behaviour of
the task model (and _not_ the predictions of the task model), as similar
predictions do not ensure that the behaviour of the task model is correctly
mimicked [8].

###  3.2 PPM setup

PPM relies on the use of historic process data recorded in an (event) log,
which is a set of traces representing the enactment of a process for a
particular case (e.g. a loan application) within an information system [3].
The occurrence of a single activity in a process (case) is referred to as
‘event’. Such an event may record three different attribute types typically
used for a process-based analysis: the _control flow attribute_ (the
activity), the _event attributes_ (i.e.  _dynamic_ attributes, which may
change from one event to another), and the _case attributes_ (i.e. _static_
attributes, which do not change throughout the lifetime of a case).

An event is a tuple $e=(c,a,t,(s_{1}\dots s_{m_{s}}),(d_{1}\dots d_{m_{d}}))$,
with $e\in\xi$ (the event universe), $c$ the case ID, $a\in A$ the activity,
and $t\in\mathbb{R}$ the timestamp. This event records both case attributes
$S=((s_{1}\dots s_{m_{s}}))$ and event attributes $D=(d_{1}\dots d_{m_{d}})$,
with $m_{s}$ and $m_{d}$ the number of case and event attributes,
respectively. A trace is a sequence of events $\sigma_{c}=[e_{1}\dots e_{n}]$
generated by executing activities in a process, sorted based on the timestamps
of the events, such that $\forall i\in{[1\dots|\sigma_{c}|]},e_{i}\in\xi$ and
$\forall i_{1},i_{2}\in{[1\dots|\sigma_{c}|]}$ $e_{i_{1}}.c=e_{i_{2}}.c$,
i.e., all the events in a trace belong to the same case. Consequently, an
event $e_{i}$ in a trace $\sigma_{j}$ of the event log $L$ is denoted as
$e_{i,j}=(c_{j},a_{i,j},t_{i,j},S_{j},D_{i,j})$. The outcome $y$ of a trace in
the case of POP is usually a binary attribute [36] and depends on the needs
and objectives of the process owner [5].

In order to learn, preferably incrementally over time as traces are sequences
over time, from the development of traces, (trace) prefixes are often
extracted from the completed cases. To this end, a prefix log $\mathbb{L}$ is
derived, which is the extracted event log $L$ that contains all the prefixes
of each case in the original event log. Next, trace cutting, i.e. limiting the
prefix up to a certain number of events, is typically performed for
computational reasons. Additionally, trace cutting is also performed when the
class label of the case is dependent on the occurrence of an event, otherwise
the label of the class becomes known and irreversible [5]. More information
about the labelling of cases is given in Section 5.2. Next, the use of an
encoding mechanism enables the user to work with a varying amount of
attributes, since each trace can have a different length. An often used
encoding mechanism is the aggregation encoding technique [5]. First, the
categorical static attributes are one-hot encoded, which means that each
static attribute that is categorical, results in a number of transformed
attributes based on the set of unique attribute values. These values can be
found with the use of $\theta$, also referred to as the uniqueness operator.
The numeric static attributes remain unchanged. Second, the timestamp is
transformed into three different dynamic numeric attributes:
_timesincelastevent_ , _timesincecasestart_ and _timesincemidnight_. Next, all
the dynamic (i.e. event) numeric attributes are replaced by their summary
statistics min, max, mean, sum, and std. The last transformation step relates
to the dynamic categorical attributes and the control flow attribute, where
the frequency of occurrence of an attribute value in a prefix is the value for
the new attribute. By contrast, the use of the above encoding mechanism in
step-based models such as recurrent neural networks becomes superfluous given
their sequential setup. To exploit this efficiently, a low-dimensional
representation of discrete attributes in the form of embeddings is an often
performed encoding technique [37]. This mapping transforms a categorical
attribute to a vector of continuous numbers, similarly to how one-hot encoding
works, although the latter ignores the similarity between the obtained
vectors. Finally, the tuple $(x_{j,1},x_{j,2},\dots,x_{j,p})$ consists of the
resulting attribute values (of all the attributes) of prefix trace
$\sigma_{j},j\in[1\dots l]$ with $l$ the number of prefix traces after prefix
extraction. Note that the case ID attribute is removed from all the traces.
This means that the total number of resulting attributes is p, and
$p=p_{a}+p_{s}+p_{d}$, with $p_{a}$, $p_{s}$ and $p_{d}$ the number of control
flow, case, and event attributes after the data transformation steps
respectively. This additionally means that the tuple $(x_{j,1}\dots
x_{j,p_{a}})$ denotes the attribute values of prefix trace $\sigma_{j}$ for
the attribute type _control flow_ , followed by $(x_{j,p_{a}+1}\dots
x_{j,p_{a}+p_{s}})$ and $(x_{j,p_{a}+p_{s}+1}\dots x_{j,p_{a}+p_{s}+p_{d}})$
for the attribute type _case_ and _event_ respectively.

Another data transformation step in the context of PPM is referred to as trace
bucketing, where traces are divided into different buckets while creating
separate models for each of them. This technique is commonly used to support
the discovery of heterogeneous segments in the data [38, 5]. Algorithms such
as K-Nearest Neighbours [39] or K-Means clustering measure the (dis)similarity
between traces depending on the parameter K. The prefix bucketing technique
[17] creates different buckets for the prefixes of different lengths, while
the state-based bucketing technique [40] creates a different model for each
different decision point within the process model. Although these bucketing
techniques can effectively diminish the runtime performance [5], they do not
necessarily result in an intuitive or interpretable outcome. E.g., clustering
techniques can base their grouping on a high number of dimensions that are not
interpretable. Nonetheless, the use of clusters is deemed a necessary action
when there is a notion of trace similarity [41]. The final transformed event
log is split into a train and test event log, where the former is used to
create a task model to predict the dependent variable based on independent
attributes. Moreover, the prediction for a prefix trace $\sigma_{j}$ is
denoted as: $\hat{y}_{i}=F(x_{j,1},x_{j,2},\dots,x_{j,p})$.

The final step is to interpret the predictions made by the task model $F$.
There already exist many model-agnostic methods that try to explain the
behaviour of the task model. Moreover, _attribution-based_ methods determine
the contribution that each attribute made to the final prediction and use this
as an explanation. Techniques such as accumulated local effect plots,
surrogate models or permutation feature importance are often used in this
context. For example, in [42], they calculate the importance of an attribute
by calculating the change in the prediction error of the model $F$ before and
after permuting the values of an attribute. An attribute is deemed _important_
when permuting the values of this attribute increases the prediction error (in
mean squared error (MSE)), and is deemed _unimportant_ when the MSE remains
unchanged after permutation (as the model did not use this attribute for the
prediction). The use of this technique is intuitive, as it clearly shows how
much _influence_ each attribute has.

Algorithm 1 Attribute Importance

$X\leftarrow(x_{1,1},x_{1,2},\dots,x_{1,p},x_{2,1},\dots
x_{l,p}),z\leftarrow[\ ],w_{PI}\leftarrow[\ ]$

for $i\in[1\dots p]$ do $\triangleright$ loop over all the attributes

$X^{copy}\leftarrow Copy(X)$ $\triangleright$ make a copy of the original data

$values\\_list\leftarrow[\ ]$ $\triangleright$ empty list for each attribute

$z\leftarrow\theta(x_{1,i},\dots,x_{l,i})$ $\triangleright$ the unique
attribute values of attribute $i$

for $j\in[1\dots l]$ do $\triangleright$ loop over all the (prefix) traces

$value\leftarrow x_{i,j}$ $\triangleright$ the current attribute value of
instance $i$

$\displaystyle z^{*}\leftarrow\\{x^{*}\in z:x^{*}\notin value\\}$
$\triangleright$ remove the current value from the list

$values\\_list[j]\leftarrow Random(z^{*})$ $\triangleright$ take a random
value and add to the list

$X^{copy}_{i}\leftarrow values\\_list$ $\triangleright$ replace attribute $i$
values with permuted values

$\hat{y}\leftarrow predict(X)$ $\triangleright$ original predictions

$y^{*}\leftarrow predict(X^{copy})$ $\triangleright$ predictions after
permuting attribute $x_{i}$

$MSE_{y,\hat{y}}\leftarrow\sqrt{\frac{(y-\hat{y})^{2}}{n}}$ $\triangleright$
MSE between y and $\hat{y}$

$MSE_{y,y^{*}}\leftarrow\sqrt{\frac{(y-y^{*})^{2}}{n}}$ $\triangleright$ MSE
between y and $y^{*}$

$effect\leftarrow MSE_{y,y^{*}}-MSE_{y,\hat{y}}$ $\triangleright$ attribute
importance calculated by change in MSE

$w_{PI}[i]\leftarrow effect$ $\triangleright$ save the effects of all the
attributes return $w_{PI}$

The use of this permutation-based attribute importance technique is described
in Algorithm 1. Note that some remarks about the design of this algorithm
should be made. First, in the case of ML models, permuting the values for
original attributes, that are later transformed through frequency aggregation
and/or summary statistics, does not make sense, as, e.g., the mean value of
attribute values before and after permutation remains the same. Therefore, the
attribute importance is calculated for the attributes after transformation
$(x_{1},\dots,x_{p})$. On the other hand, in the case of DL models the
attribute importance is calculated by permuting the original attributes. The
attribute weights defined by this permutation attribute importance technique
$PI$ are defined as: $w_{PI}=w_{PI,1},...,w_{PI,p}$.

Likewise, transparent models are able to produce their own attribute
importances. As an example, the coefficients in a logistic regression model
indicate the importances of the different attributes on the dependent
variable. On the other hand, in case of a black box model, the attribute
importances can also be measured with the use of a post-hoc explainability
model, such as SHAP or LIME, which approximates the attribute weights of the
black box model. The attribute importance calculated by the explainability
model $E$ (can be both the transparent model itself, or a post-hoc
explainability model) is denoted as: ($w_{E,1}\dots w_{E,p}$).

##  4 Explainability in Process Outcome Prediction

Despite the recent growth of literature in the field of XAI, it still remains
unclear what identifies as a suitable explanation and how to evaluate them.
One of the reasons is that the quality of an explanation is dependent on many
things, such as the use case, the stakeholder, and the explanation method
itself [10]. Although _explainability_ is a subjective matter, many papers
have already stated that there exists properties of explainability (which can
be objectively quantified) that a _good_ explanation should satisfy. These
papers often make the distinction between interpretability and faithfulness,
and identify different metrics to quantify these properties [9, 10].
Nonetheless, most of the available metrics are not model-agnostic (e.g.
dependent on model parameters), or do not work for both transparent and non-
transparent models. Furthermore, it is necessary to divide the attributes into
the different attribute types, allowing the metrics to take into account the
process-based perspectives. In this paper, the separation of explainability
into interpretability and faithfulness is used to obtain model-agnostic
metrics suitable for a process-based analysis. These metrics consequently
allow defining and assessing explainability in the context of POP, by being
able to quantitatively compare transparent versus non-transparent models
extended with explainability methods. An overview of the different properties
and metrics introduced by this paper is given in Figure 1. For more
information about the different XAI definitions, properties and/or metrics, we
refer to [7], [9] and [10].

Figure 1: Explainability defined through its properties Interpretability and
Faithfulness.

###  4.1 Parsimony

Parsimony ($C$) is a property of interpretability that represents the
complexity of a model, and is an often used metric for linear regression
models. This can be seen as the number of attributes with non-zero weights
defined by the explainability model $E$. These are non-zero coefficient
weights in a linear regression model, or for a black box model, the non-zero
weights provided by the post-hoc explainability model.

First, we reconsider the definition of parsimony as a property for the
_interpretability of an explanation_ rather than a property for _model
interpretability_. Next, we adapt this metric to take into account the
different perspectives of a process-based analysis by determining the
parsimony for each attribute type separately. The parsimony of the total model
$C_{F}$ is equal to the sum of the values for parsimony of the different
attribute types. Moreover, a parsimonious (i.e. simple) model corresponds to a
low value $C_{F}=C_{control}+C_{event}$ \+ $C_{case}$.

Assume an attribute $x_{i}$ with $w_{E,j}$ the weight for that attribute
$i\in{\\{1\dots p\\}}$ defined by the explainability model $E$. Then, the
parsimony for the control flow attribute type is calculated as follows:

$C_{control}=\sum_{i=1}^{p_{a}}C(x_{i})\text{\quad with
\quad}C(x_{i})=\begin{dcases*}1,&if $w_{E,i}>0$,\\\
0,&otherwise.\end{dcases*}$ (1)

The parsimony for the other attribute types is calculated analogously.

###  4.2 Functional Complexity

Functional complexity ($FC$) is a measure for model complexity in line with
other FC calculations such as the one described by [30] and is made suitable
for POP data below as[43] states that small random perturbations to data such
as test images can drastically change the generated explanations, without the
predicted label being altered. For this, we adapt the metric further by
investigating how many altered predictions there would be when permuting the
attribute values of an attribute type, and consequently measures how strongly
the explanations depend on that attribute type. This permutation-based metric
is consequently used to retrieve the dependency of the attribute types (i.e.,
event/case/control flow) on the explanations. In Algorithm 2, the pseudocode
for the proposed FC algorithm is given and how the different permutations are
made in order to obtain the FC values for the attribute type $control$. Again,
the FC for the other attribute types is calculated analogously.

Algorithm 2 Functional Complexity of _Control Flow_

$X_{test}\leftarrow(x_{1,1},x_{1,2},\dots,x_{1,p},x_{2,1},\dots
x_{l,p}),FC_{control}\leftarrow 0$

$X^{test,copy}\leftarrow Copy(X^{test})$ $\triangleright$ make a copy of the
original test data

for $i\in[1\dots p_{a}]$ do $\triangleright$ loop over the control flow
attributes

$values\\_list\leftarrow[\ ]$ $\triangleright$ empty list for each attribute

$z\leftarrow\theta(x_{1,i},\dots,x_{l,i})$ $\triangleright$ the unique
attribute values of attribute $i$

for $j\in[1\dots l]$ do $\triangleright$ loop over all the (prefix) traces

$value=x_{i,j}$ $\triangleright$ the current attribute value of instance $i$

$\displaystyle z^{*}\leftarrow\\{x\in z:x\notin value\\}$ $\triangleright$
remove the current value from the list

$values\\_list[j]\leftarrow Random(z^{*})$ $\triangleright$ take a random
value and add to the list

$X^{test,copy}_{i}\leftarrow values\\_list$ $\triangleright$ replace attribute
$i$ values with permuted values

$\hat{y}_{i}=F(X_{test})$ $\triangleright$ original predictions

$\hat{y}^{*}=F(X^{test,copy})$ $\triangleright$ predictions after permuting
all control flow attributes

$FC_{control}\leftarrow\frac{distance(\hat{y},y^{*})}{n}$ $\triangleright$ FC
for attribute type control

return $FC_{control}$

The algorithm starts by looping over the attributes of the _control flow_
attribute type. For each attribute $i$, a random new value is assigned for
each prefix trace $j$. The difference with the attribute importance (Algorithm
1) is that all the attributes are permuted simultaneously. Next, the
predictions before and after these permutations are calculated. Finally, the
Hamming Distance, which calculates the number of bit positions in which the
two bits (i.e. predictions for a prefix trace) are different, is used as the
measure between the different prediction vectors. A low value for $FC_{t}$
means that the predictions are created seemingly independently of this
attribute type, and should therefore not be regarded as an important attribute
type when interpreting the explanations.

###  4.3 Importance Ranking Correlation (IRC)

($IRC$) is a measure for the faithfulness of an explainability method and is
quantified with the use of the non-parametric Spearman’s correlation
coefficient. The original metric [44] is model-dependent on a neural network
model. Therefore, this paper has adapted the metric to make it model-agnostic
by evaluating how faithful the attribute importance ranking of the
explainability model is to the ranking made by the permutation attribute
importance (PI). Furthermore, this metric does not make a distinction between
the different attribute types, as the focus is on the relative ranking of the
attributes in general. For a transparent model such as LR, we quantify the
$IRC$ between the ranking of the attributes measured with PI and the ranking
made by the explainability model (the LR coefficients). For a post-hoc
explainability model, we quantify the IRC between the ranking of the
attributes measured with PI and the ranking made by the explainability model
(e.g. SHAP values). The $IRC$ is defined as:

$IRC=\rho(w_{PI},w_{E})$ (2)

with $w_{PI}=w_{PI,1},\dots,w_{PI,p}$ the attribute weights measured with PI
and $w_{E}=w_{E,1},\dots,w_{E,p}$ the weights of the explainability model.
This correlation coefficient takes a value between [-1,1] and describes the
association of rank. A perfectly faithful model has a correlation coefficient
of +1, where a loss in faithfulness corresponds with a value closer to 0.
Consequently, a negative value corresponds to a negative rank association
between the two attribute importance weights.

###  4.4 Level Of Disagreement (LOD@10)

($LOD@10$) is a metric of faithfulness defined by [45], which computes the
percentage of similar predictions between attribute importance measured with
PI and explainability model. In predictive process monitoring, it is useful to
know the importance of the attribute types, as the obtained insights are
relevant in order to improve the early prediction problem [5]. In this paper,
the LOD@10 therefore investigates whether the PI and the explainability model
focus on the same attribute type. The $LOD@10$, different from IRC, neglects
the ranking of the attributes but only looks at the relative frequency of the
attribute types in the top ten most important attributes (based on their
weights). For this, the metric is quantified with the Euclidean Distance
between the relative frequency of top ten attributes of the PI and the
explainability model, with $w_{PI}^{10}=argmax_{w_{PI}^{\prime}\subset
w_{PI},|w_{PI}^{\prime}|=10}\sum_{w\in w_{PI}^{\prime}}w$ and
$w_{E}^{10}=argmax_{w_{E}^{\prime}\subset w_{E},|w_{E}^{\prime}|=10}\sum_{w\in
w_{E}^{\prime}}w$ indicating the top ten highest weights measured with PI and
by the explainability model respectively. Next, the control flow attributes
that are in the top ten attributes determined by PI and the explainability
model are defined as follows: $w_{PI,control}^{10}={|\\{w_{PI}^{10}\mid
w_{PI}=w_{PI,1},\dots,w_{PI,p_{a}}}\\}|$ and
$w_{E,control}^{10}={|\\{w_{E}^{10}\mid w_{E}=w_{E,1},\dots,w_{E,p_{a}}}\\}|$.
The calculation is similarly for the other attribute types. As an example, the
$LOD@10$ is 1.41 when the relative ranking made by the PI is
$(w_{PI,control}^{10},w_{PI,case}^{10},w_{PI,event}^{10})=(1,2,7)$ and the
relative ranking of the explainability model $E$
is$(w_{E,control}^{10},w_{E,casel}^{10},w_{E,event}^{10})=(2,2,6)$.

$\begin{split}LOD@10&=\sqrt{\left(w_{PI,control}^{10}-w_{E,control}^{10}\right)^{2}+\left(w_{PI,case}^{10}-w_{E,case}^{10}\right)^{2}+\left(w_{PI,event}^{10}-w_{E,event}^{10}\right)^{2}}\\\
\end{split}$ (3) $\displaystyle\text{with }\
\begin{dcases*}w_{PI,control}^{10}+w_{PI,case}^{10}+w_{PI,event}^{10}=10\\\
w_{E,control}^{10}\ \ +w_{E,case}^{10}\ +w_{E,event}^{10}\ =10\\\
\end{dcases*}$ (4)

This metric is introduced to take into account that the number of attributes
used in the task model has a negative influence on the IRC value. Furthermore,
a high value for this LOD@10 metric indicates that the explainability model
focus on rather different attribute types, which impairs the faithfulness of
the explainability model.

##  5 Experimental Setup

In this section, a detailed build-up to the research questions related to the
interpretability and faithfulness of POP models is provided, which will be
used in a benchmark setting. Next, the different event logs and their
specifications are described, followed by a description of the benchmark
models and explainability techniques often used for POP purposes. Finally, the
hyperoptimization settings and implementation details of the different setups
are given.

###  5.1 Research Questions

This paper aims at investigating the influence of the most important steps in
an POP context on the predictive performance and XAI metrics from Section 4,
in order to establish a set of guidelines to obtain accurate and explainable
POP solutions. For this, an experimental pipeline similar to the benchmark
studies of [5] and [6], extended with an XAI dimension, is used. To this
purpose, the following research questions are investigated:

  1. RQ1.

How do the different POP methods compare in terms of interpretability versus
predictive performance?

  2. RQ2.

What do the different XAI metrics tell about the inherent architecture of the
methods?

  3. RQ3.

How should the faithfulness of explanations be evaluated when compared with
interpretability and predictive performance?

The first research question (RQ1) investigates the trade-off between
interpretability (measured with _parsimony_) and predictive performance. This
additionally allows assessing whether transparent models typically
underperform compared to the black box models in the case of sequential and
high-dimensional data [6].

The second research question (RQ2) investigates how the different attribute
types (i.e. _event_ , _case_ and _control flow_) relate to the different XAI
metrics and the different model architecture of the POP methods. We assess
whether deep learning models focus more on the control flow perspective (i.e.
higher $C_{control}$) compared to the traditional cross-sectional statistical
and machine learning models, as their sequential architecture is more tailored
towards modelling time-dependent and sequential data tasks without such
aggregation. Next, we compare the interpretable LLM model, introduced to POP
in [16], as an alternative to the bucketing technique (see Section 3) in
conjunction with the LR model based on the performance and four explainability
metrics. Finally, the parsimony values of the different attribute types are
compared with the values for functional complexity. Although both metrics are
a measure for model complexity, the parsimony evaluates the importance of the
attribute types as perceived by the task model (calculated on the training
data), while the functional complexity investigates the importance of the
attribute types on the predictions (calculated on the test data).

The third research question (RQ3) investigates how we should interpret the
different faithfulness metrics, and how they relate to the interpretability
metrics and the predictive performance. First, we assess whether the
predictive outperforming of certain models comes at the expense of the
interpretability and/or faithfulness. Next, we investigate whether the
explanations generated by post-hoc explainability techniques (e.g. SHAP
values) are less faithful compared to the explanation method that contribute
to the predictions of the black box model (e.g. attention values). Moreover,
we implicitly assume that the faithfulness of the latter should be higher
compared to the SHAP values.

###  5.2 Event logs

Table 1: The Different Specifications of the Event Logs.

Event Log | Traces | Events | Med. | Max. | Prefix | Var. | $\frac{Act}{Trace}$ | Stat. cat. | Dyn. cat. | $\frac{Var.}{Trace}$ | $\frac{Events}{Trace}$ | $\frac{Events}{Act}$ | $\frac{Dyn.}{Stat.}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---
BPIC2011(1) | 1140 | 67480 | 25 | 1814 | 36 | 815 | 193 | 961 | 290 | 0.71 | 59 | 3 | 0.3
BPIC2011(2) | 1140 | 149730 | 54.5 | 1814 | 40 | 977 | 251 | 994 | 370 | 0.86 | 131 | 5 | 0.37
BPIC2011(3) | 1121 | 70546 | 21 | 1368 | 31 | 793 | 190 | 886 | 283 | 0.71 | 63 | 3 | 0.32
BPIC2011(4) | 1140 | 93065 | 44 | 1432 | 40 | 977 | 231 | 993 | 338 | 0.86 | 82 | 3 | 0.34
BPIC2015(1) | 696 | 28775 | 42 | 101 | 40 | 677 | 380 | 19 | 433 | 0.97 | 41 | 1 | 22.79
BPIC2015(2) | 753 | 41202 | 55 | 132 | 40 | 752 | 396 | 7 | 429 | 1.00 | 55 | 1 | 61.29
BPIC2015(3) | 1328 | 57488 | 42 | 124 | 40 | 1280 | 380 | 18 | 428 | 0.96 | 43 | 1 | 23.78
BPIC2015(4) | 577 | 24234 | 42 | 82 | 40 | 576 | 319 | 9 | 347 | 1.00 | 42 | 1 | 38.56
BPIC2015(5) | 1051 | 54562 | 50 | 134 | 40 | 1048 | 376 | 8 | 420 | 1.00 | 52 | 1 | 52.5
SEPSIS(1) | 782 | 13120 | 14 | 185 | 29 | 684 | 14 | 195 | 38 | 0.87 | 16 | 1 | 0.54
SEPSIS(2) | 782 | 10924 | 13 | 60 | 13 | 656 | 15 | 200 | 40 | 0.84 | 14 | 1 | 0.19
SEPSIS(4) | 782 | 12463 | 13 | 185 | 22 | 709 | 15 | 200 | 40 | 0.91 | 16 | 2 | 0.2
Production | 220 | 2489 | 9 | 78 | 23 | 203 | 26 | 37 | 79 | 0.92 | 11 | 3 | 2.14

This study is based on four different real-life event logs that can be found
at the website of 4TU Centre for Research Data111https://data.4tu.nl/, and are
often used in the field of POP [5, 6, 26, 24]. These event logs are split with
Linear Temporal Logic (LTL) rules as defined in [5] to obtain objectives for
the process. Moreover, the event log is split based on the labelling functions
defined by the four LTL rules, therefore creating four different binary
prediction tasks. The event log specifications are defined in Table 1. The
first two columns indicate the number of traces and events in the event log.
The next columns describe the median length, original maximum length and the
length after prefix cutting (see 3) respectively. The column $Var$ indicates
the total number of different activities, with the columns $Stat.cat.$ and
$Dyn.cat$ indicating the number of static (i.e. case) and dynamic (i.e. event)
categorical attributes, respectively. The remaining columns are intuitive. The
first log, BPIC2011, describes the medical history of patients from the
Gynaecology department of a Dutch Academic hospital. The applied procedures
and treatments of the different cases represent the activities in this event
log, with the label being either true or false if the LTL rule is violated or
not, respectively. The four different LTL rules to generate the event logs:
bpic2011(1), bpic2011(2), bpic2011(3) and bpic2011(4), are described in [5].
Next, similar trace prefixing and cutting preprocessing steps are performed as
in [5]. The second log, BPIC2015, assembles events pertaining to the building
permit application process from five Dutch municipalities. A single LTL rule
is applied on the event log and split for each of the five municipalities. The
LTL rule defines that a certain activity _send confirmation receipt_ must
always be followed by _retrieve missing data_ (and not the other way around),
where the latter activity has to always be present in the trace if the former
was. No trace cutting was performed on this event log. Next, the sepsis cases
event log contains the discharge information of patients with symptoms of
sepsis in a Dutch hospital, starting from the admission in the emergency room
until the discharge of the patient. Here, the labelling is performed based on
the discharge of the patient instead of LTL rules [5]. Last, the Production
log, contains information about the activities in a manufacturing process,
together with the workers and/or machines of the production of the items
itself. The labelling function is based on whether the number of work orders
rejected is larger than zero or not.

###  5.3 Benchmark models and explainability models

The wide variety of classifiers and explainability models are chosen based on
their frequent presence in other studies, such as [5, 6, 23, 25]. The first
model is the logistic regression model, an often used interpretable predictive
technique to model the probability of a discrete variable. The two advanced
logistic regression models, LLM [12] and GLRM [13], are interpretable models
that were introduced to the field of POP in [16]. The former clusters the data
with a decision tree and builds linear models in the leave nodes. The latter
creates binary rules with a generalized logistic rule model. Here, the
probability of being classified as ‘deviant’ is calculated similarly to a
regular logistic regression model [16].

The next two models are ensemble machine learning models, i.e. XGBoost (XGB)
and Random Forest (RF), which are not interpretable models, as the inherent
complexity is what bestows their predictive abilities. In the XGB model, weak
learners are iteratively improved to a final strong learner by incorporating
the loss function of the previous weak learner(s). On the other hand, the RF
trains a number of decision trees on various subsets of the data. Different to
XGB, the voting scheme is based on the majority votes of predictions. By
contrast to the transparent models, ensemble methods require an explainability
model. For this, the explanations for the ensemble methods are created with
SHAP values [46] created with Tree SHAP, which are calculations for each
instance-attribute combination based on coalitional game theory. Here, a
prediction is explained by assuming that each attribute value is a player in a
game, where the prediction is the payout. This model-agnostic technique tells
how to distribute the payout among the attributes, as the SHAP values are the
average marginal contribution of an attribute value across all possible
coalitions.

Next, two different neural network models are used. The first model is the
LSTM, often used for POP [23, 47, 6] with the long-term relations and
dependencies encoded in the cell state vectors, designed to solve the
vanishing gradient problem. The advantage of LSTM over classical machine
learning models lies in the ability to model time-dependent and sequential
data tasks, where the categorical values are encoded in embeddings. The second
model is a Convolutional Neural Network (CNN) which is a deep-forward
artificial neural network with information flow starting from the input
layers, through the hidden layers, until the output layer (therefore only in
one direction). To the best of our knowledge, this was only implemented by
[20] for POP, but is incorporated in this study due to the frequent use in
related fields such as next activity prediction [48]. Similar to the ensemble
methods, the internal representation of a deep neural network (LSTM and CNN),
does not allow for inherent explanations of predictions. The use of attention
layers is a model-specific post-hoc explainability technique in the strict
sense of the meaning, as attention is contributing to the prediction but
typically calculated afterwards to obtain attribute importance scores [49].
These attention layers calculate non-negative weights (multiplied by their
corresponding representations) for each input that together sums to one, and
finally sums the resulting vectors into a single fixed-length representation
[32].

###  5.4 Preprocessing steps

First, an equivalent train-test split as in [5] is performed. To this end, a
temporal train-test split is performed that ensures that the period of
training data does not overlap with the period of the test data, while the
events of the cases in the train data that did overlap with the test data are
cut. Next, the traces are cut with maximum length as defined in [5].

Then, the aggregation encoding is used to encode the data for the machine
learning algorithms. Note that the aggregation encoding is unique to processes
and is, together with the three perspectives, the primordial reason to
investigate a dedicated POP XAI-based approach. On the other hand, the deep
learning models are built to work with sequential models and therefore do not
need the use of such a sequence encoding mechanism. Moreover, the deep
learning models use an embedding mechanism for categorical attributes, which
can be seen as one-hot encoding along with dimensionality reduction. The
bidirectional LSTM neural network architecture with attention layer for
interpretation purposes stems from [25]. Compared to the original set-up, we
have optimized the code to work for multiple dynamic attributes, and multiple
numerical attributes (by adding them as input layers to the LSTM model).
Finally, the predictive function of [25] is transformed into a binary outcome-
oriented prediction by stripping off the final layer and inserting a sigmoid
output layer instead. In order to compare, ceteris paribus, the overall
performance of the LSTM with the CNN, we ensure that both models have a
similar set-up. Therefore, the CNN model starts from the same architecture as
the LSTM, and the Bidirectional LSTMs are replaced with 1D convolutional
layers. Different to the LSTM model, the attention is calculated directly
after the input layers and embeddings (similar to [50]) and the kernel size is
set to be the same as the length of the sequences and the filter as the length
of the concatenated input (to ensure that we can extract attention values)
[50]. Additionally, an extra dense layer with Rectified Linear Unit (ReLU)
activation was added before the final dense layer, in order to ensure that the
output is correctly linked back to the inputs.

The event logs are filtered out for which the average obtained AUC over all
the classifiers was lower than 50 (i.e. _sepsis cases(2)_ , as no analysis
should be performed where random luck has a better ability compared to the
classifiers in order to distinguish between the classes). For the XAI
evaluation, we additionally filtered out the event logs that had an average
AUC below 75 (sepsis cases(3) and production). This was to overcome the issue
that explainability techniques can only perform well when the original task
model is performant enough.

###  5.5 hyper optimization details

Finally, a 4-fold cross validation was performed with the use of hyperopt for
the machine learning models. The hyper optimization for the deep learning
models was done by a 10-fold cross validation, as the neural networks need
additional evaluation to overcome the plethora of local minima. For the LR,
XGB and RF model, similar hyperparameter settings are taken from [5]. For the
LLM model, we additionally incorporate the maximum depth of the decision tree
and the minimum samples per leaf to ensure that the tree splitting does not
allow overfitting. According to [19], the optimal dropout rate for the
bidirectional LSTM with attention was around 0.2. Therefore, we have set the
maximal dropout rate to 0.3 (as a margin of error). More detailed information
about design implementations and parameters are provided on
GitHub222https://github.com/AlexanderPaulStevens/Evaluation-Metrics-and-
Guidelines-for-Process-Outcome-Prediction to enhance the reproducibility
results.

##  6 Experimental Evaluation

First, the research questions are answered in-depth. Next, a case-based
evaluation in the context of POP is performed with the use of the event log
BPIC2015(1). Finally, this section ends with the framework of guidelines for
Explainable AI in POP in order to guide the practitioner to the correct model
selection.

###  6.1 Benchmark results

Table 2: Predictive Performance (AUC) per Event Log and Classifier

Event Log | LR | LLM | GLRM | XGB | RF | LSTM | CNN
---|---|---|---|---|---|---|---
BPIC2011(1) | 95.88 | $\underline{\textbf{97.34}}$ | 91.98 | 95.59 | 93.09 | 75.5 | 79.69
BPIC2011(2) | 96.56 | 98.18 | 97.4 | 98.21 | $\underline{\textbf{98.49}}$ | 86.55 | 80.67
BPIC2011(3) | 98.71 | 98.55 | 98.01 | $\underline{\textbf{98.9}}$ | 98.87 | 78.78 | 85.01
BPIC2011(4) | 88.21 | $\underline{\textbf{89.75}}$ | 80.63 | 86.46 | 89.04 | 87.22 | 85.83
BPIC2015(1) | 93.42 | $\underline{\textbf{93.67}}$ | 89.21 | 87.96 | 92.16 | 90.29 | 90.66
BPIC2015(2) | 94.74 | 94.99 | 87.18 | $\underline{\textbf{95.65}}$ | 93.89 | 94.34 | 92.55
BPIC2015(3) | $\underline{\textbf{96.13}}$ | 95.92 | 93.75 | 93.71 | 96 | 94.56 | 93.72
BPIC2015(4) | $\underline{\textbf{94.79}}$ | 94.28 | 91.33 | 92.39 | 93.93 | 90.7 | 85.99
BPIC2015(5) | 93.54 | 93.55 | 90.49 | 93.78 | $\underline{\textbf{94.86}}$ | 91.99 | 93.5
SEPSIS(1) | 54.03 | 45.67 | 47.18 | 39.38 | 32.14 | $\underline{\textbf{56.27}}$ | 50.45
SEPSIS(2) | $\underline{\textbf{92.34}}$ | 89.47 | 73.04 | 87.45 | 83.04 | 84.69 | 82.13
SEPSIS(4) | $\underline{\textbf{74.39}}$ | 63.05 | 64.72 | 71.63 | 73.89 | 66.29 | 64.84
Production | 59 | 61.33 | 58.98 | $\underline{\textbf{75.91}}$ | 71.35 | 60.55 | 69.46

The first research question (RQ1) investigates how the different POP models
compare in terms of interpretability versus predictive performance. Table 2
provides us with an overview of the predictive performance results for the
classifiers and event logs. The bottom-up ranking of the average AUC of the
classifiers is as follows: GLRM, CNN, LSTM, XGB, RF, LLM, LR. Next, the
bottom-up ranking based on the model interpretability (see [7] for more
information) is: LR, LLM, GLRM, RF, XGB, CNN and LSTM. This means that there
is no trade-off between model interpretability and predictive performance, as
simple models such as LR perform better than complex black-box structures such
as LSTMs. Furthermore, the ML models (86.08 AUC) perform better than the DL
models (81.23 AUC) on average. A possible explanation why the DL models
underperform compared to the ML models is due to the trace cutting of the
prefixes (see Section 3). Without trace cutting (i.e.retaining very long
prefixes), the DL models could potentially obtain equal (or higher) AUC values
than the ML models, as the former are better capable of handling long-term
dependencies. Nonetheless, as the field of POP is also labelled as early
sequence prediction [5], the use of trace cutting is justified (as we want the
prediction as soon as possible). Another possible explanation is that the DL
models are disadvantaged, due to the lack of aggregation in their encoding
(which seems to be an encoding mechanism that works well, see [5])

In [6] it is stated that DL models generally outperform classical ML
approaches when it comes to process outcome prediction, especially for event
logs that have many different activities per trace (i.e. a high value for
$\frac{Var.}{Trace}$) and a high ratio of event versus case attributes (i.e. a
high value for $\frac{Dyn.}{Stat.}$). When comparing Table 1 and Table 2, we
see that the DL models obtain accurate results for the BPIC2015 event logs
(which have high values for $\frac{Var.}{Trace}$ and $\frac{Dyn.}{Stat.}$),
while they perform poorly (compared to the ML models) for the BPIC2011 logs.
Although [6] used different event logs, it was found that the results apply
for logs outside their study due to the purposive sampling of logs and
techniques. The only difference is therefore in the sequence encoding
mechanism (one-hot encoding instead of aggregation encoding), meaning that the
ML model perform better when the dynamic behaviour of dynamic attributes is
made static through techniques like frequency aggregation and summary
statistics. Furthermore, in line with the results from [20] and [48], where it
was shown that CNN models perform better than LSTM models, we see that CNNs
perform better than LSTMs in 5 of 13 event logs. Next, the two transparent
models LR and LLM obtain the highest AUC (on average). This means that
transparent models are able to attain the performance level of the black box
models in the case of sequential and high-dimensional data. Interestingly, the
LR (the best overall AUC performance in 4 out of 13 event logs) is ranked
higher compared to the LLM model, with the latter model performing better than
the LR model in 7 out of 13 event logs. This is due to two reasons. First, the
LLM model performs poorly for the sepsis logs. When the sepsis logs are not
taking into account (and only the other 10 event logs), then the LLM is ranked
higher than the LR. Second, the LLM model is implemented as such that at least
one split is _enforced_ by the decision tree. This enforced split causes that
at least two LR models are created (both on a subset of the data), which might
explain the negative effect on the performance of the LLM compared to LR. From
the first subplot of Figure 2, it is clear to see that the CNN, GLRM and LSTM
model have significantly lower (better) values for parsimony compared to the
other models, with the XGB model as the next in line. Coincidentally, these
are the four models with the lowest predictive performance on average. This
means that, when only taking into account parsimony as the metric for
_interpretability_ , the interpretability-predictive performance trade-off
holds in the context of POP. In contrast, although the RF model has higher
values for $C$ for all the attribute types, it is ranked lower than LR based
on predictive performance (AUC).

Figure 2: XAI metrics per classifier (averaged over the event logs). The top
subplot describes the parsimony of each attribute type (_control_ ,_case_ and
_event_) individually. The second subplot shows the fraction of the parsimony
per attribute type and the total attributes of that attribute type. The third
subplot shows the Functional Complexity per attribute type. The fourth and
fifth subplot describe the Importance Ranking Correlation ($IRC$) and the
Level of Disagreement ($LOD@10$) respectively.

The second research question (RQ2) investigates what the different XAI metrics
tell about the inherent architecture of the models. To start, we can see in
the second subplot of Figure 2 that only the CNN and LSTM models use 100% of
the event and control attributes. Two interesting remarks can be made for this
observation. First, the DL models extract more information from the
sequential, control perspective compared to the traditional cross-sectional
statistical and machine learning models, as their sequential architecture is
more tailored towards modelling time-dependent and sequential data tasks
without such aggregation. Second, both the LSTM and CNN also use all the event
attributes, which means that these models have greater preference for
attributes characterized by their dynamic nature. In contrast, the
permutation-based $FC_{event}$ and $FC_{control}$ of the DL models is
extremely low. This means that, although the event and control attributes are
deemed important by the model (high $\frac{C_{event}}{total_{event}}$ and
$\frac{C_{control}}{total_{control}}$), the $FC$ value state otherwise. This
casts considerable doubt on the interpretation of the generated explanations,
as permuting all the attribute values of the control and event attributes that
the model uses has no influence on the prediction made (i.e. the FC remains
unchanged). Additionally, interesting insights can be made about whether the
relative ranking of parsimony values are similar (or not) to the ranking of
the functional complexity values. First, the random forest model seems to use
a lot of case attributes (high $C_{case}$), but permuting the values of these
attributes does not have a high influence on the predictions (relatively low
$FC_{case}$). This shows that the relative ranking of the $FC$ values is not
always similar to the relative ranking of the parsimony $C_{F}$ values. This
would be a considerable problem in cases where the parsimony indicates that a
particular attribute type is relatively important (i.e. $C_{i}=0$), but
permuting these values has a very high influence on the predictions (i.e.
$FC_{i}>>>0$, meaning that they are still somewhat important). On the other
hand, we see in Figure 2 that the FC values of the different attribute types
of the GLRM model are rather high, while the parsimony values are rather low.
This is desirable, as this means that the model uses few, but influential
attributes.

The third research question (RQ3) investigates how we should evaluate the
faithfulness of the explainability models when compared with the
interpretability metrics and predictive performance. It is clear to see that
the models with the highest average AUC value have the highest value for
$LOD@10$, i.e. the LR, LLM, and RF model. We additionally see a relatively low
value for $IRC$ for these models. Additionally, we can also see that the LR
(the model with the highest ranking based on AUC), is the least faithful based
on the metric $IRC$ (apart from the CNN model) and the metric $LOD@10$,
whereas the model with the lowest ranking (GLRM) is the most faithful based on
these metrics. In addition, we can also see that RF models, which are higher
ranked than LSTM model based on AUC, seems to have lower faithfulness values
compared to LSTMs. This means that there seems to be a faithfulness-predictive
performance trade-off as well. Next, in Section 3, it was made clear that the
explanation ‘if the grass is green, it will rain’ is easy to interpret but
unfaithful to the underlying task model behaviour. One could argue that the
exploded attribute space of the ML model is detrimental for faithfulness of
the model, but the results indicate otherwise. Interestingly, the value for
the Importance Ranking Correlation $IRC$ for the XGB(RF) is better in 9(10)
out of 13 event logs compared to LSTM, and 12(3) out of 13 compared to the
CNN. For the LOD@10, the LSTM only has 1(2) event log where the value is lower
when compared with RF(XGB). On the other hand, the CNN model has a lower LOD
value in 10(4) event logs in comparison with RF(XGB). In addition, based on
the $IRC$, the LSTM seems to have more faithful explanations than the CNN in
11 of the 13 event logs, only in 9 out of 13 event logs when taking into
account the $LOD@10$. The IRC of the CNN model is very compromised (even
reports negative values). Although the inherent architecture of LSTMs are the
most suited for the sequential structure of process data, the results show
that the faithfulness of post-hoc explanations (e.g. SHAP values) is _less_
compromised in comparison with explanations that contribute to the predictions
of the black box model but are calculated afterwards (i.e. attention values),
which is rather counterintuitive as it is intuitive that the faithfulness of
the latter should be higher compared to the SHAP values. This can be due to
the fact that attention values focus on explaining the representation of data
inside a network rather than explaining the processing of data [51].

To summarize the obtained insights, the introduction of model-agnostic XAI
metrics to the field of POP allow obtaining insights into how different POP
methods differ based on the properties interpretability and faithfulness.
First, we establish that there is a trade-off between predictive performance
and interpretability (based on the metric $C$), while there is no trade-off
when considering the traditional predictive performance versus model
interpretability trade-off. Next, there is a trade-off between predictive
performance and faithfulness (measured with $IRC$ and $LOD$), meaning that
more faithful models are often at the expense of less accurate models.
Finally, the $FC$ (which is a metric to indicate how much predictions change
when the attributes of an attribute type are permuted) of control and event
attributes of the DL models are almost zero, which means that we can
practically change all the values of all the attributes of a certain attribute
type, and the predictions will always remain unchanged (which is not something
we desire).

###  6.2 Event log Analysis: BPIC2015(1)

Table 3: Overview of predictive performance (AUC) and XAI metrics for the
event log BPIC2015(1).

BPIC2015(3) | AUC | event | ctrl | case | event | $C_{control}$ | $C_{case}$ | $C_{event}$ | $\dfrac{C_{control}}{total_{control}}$ | $\dfrac{C_{case}}{total_{case}}$ | $\dfrac{C_{event}}{total_{event}}$ | $\dfrac{C_{control}}{total_{control}}$ | $FC_{control}$ | $FC_{case}$ | $FC_{event}$ | $IRC$ | $LOD_{@10}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$XGB$ | 87.96 | 88 | 266 | 35 | 88 | 50 | 21 | 64 | 18.8 | 60 | 72.73 | 18.8 | 91.78 | 5.7 | 18.27 | 0.45 |  1.41
$GLRM$ | 89.21 | 88 | 266 | 35 | 88 | 4 | 2 | 0 | 1.5 | 5.71 | 0 | 1.5 | 94.81 | 26.68 | 0 | 0.91 |  1.41
$LSTM$ | 90.29 | 11 | 1 | 17 | 11 | 1 | 1 | 11 | 100 | 5.88 | 100 | 100 | 0.02 | 0.01 |  0.03 | 0.4 | 1.41
$CNN$ | 90.66 | 11 | 1 | 17 | 11 | 1 | 1 | 11 | 100 | 5.88 | 100 | 100 | 0.03 | 0.01 |  0.03 | -0.07 | 3.61
$RF$ | 92.16 | 88 | 266 | 35 | 88 | 222 | 34 | 86 | 83.46 | 97.14 | 97.73 | 83.46 | 90.32 | 2.13 | 6.34 | 0.34 |  4.24
$LR$ | 93.42 | 88 | 266 | 35 | 88 | 12 | 3 | 0 | 4.51 | 8.57 | 0 | 4.51 | 95.84 | 1.46 | 0 | 0.47 |  2.83
$LLM$ | 93.67 | 88 | 266 | 35 | 88 | 149.5 | 24.5 | 49.5 | 56.2 | 70 | 55.68 | 56.2 | 98.75 | 4.37 | 0.8 | 0.37 |  3.74

A more in-depth analysis is given for the event log BPIC2015(1) as an
exemplary POP exercise. As mentioned in Section 5.2, this event log describes
the building permit application process of the first municipality. Here, the
label is dependent on whether a certain activity _send confirmation receipt_
is always be followed by _retrieve missing data_ or not. The label _regular_
is given when this rule is followed, and _deviant_ otherwise. In table 3, the
predictive performance and XAI metrics results for the event log BPIC2015(1)
is given.

First, it is clear to see that the GLRM, LSTM, CNN and LR are the most
parsimonious models (lowest $C_{F}$, with
$C_{F}=C_{control}+C_{case}+C_{event}$), with a maximum of 15 attributes used
in the resulting model. A little remark has to be made for the DL models, as
both the LSTM and CNN do not require the aggregation sequence encoding in
order to work with sequential data (and their dynamic behaviour). This is
visible in Table 3, where the _control_ , _case_ and _event_ columns indicate
that the ML models (266 control, 35 case and 99 event attributes) have a much
bigger attribute space compared to the DL models (1 control, 17 case, 11 event
attributes). Another interesting insight is that, although the DL models use
100% of their control and event attributes, the $FC_{control}$ and
$FC_{event}$ are almost zero. This means that, although the model uses all the
control and event attributes, they do not seem to have any influence on the
predictions made (as the $FC$ investigates the number of prediction changes
after permutating an attribute type). Third, the LTL rule used for this event
log is based on the presence of control flow attributes. Intuitively, we
expect a high $FC_{control}$ (when we assume that the model has learnt that
the label is dependent on the presence of control flow attributes, then this
is something we desire), which is clearly visible in the ML models. As an
example, the $FC_{control}$ value of the GLRM model for BPIC2015(1) is very
high (more than 94% of the predictions change after permuting the control flow
attributes), despite only using 4 control attributes. The $FC_{control}$
values for the DL models, on the other hand, are almost zero, meaning that
permuting the attribute values of the control flow attributes does not cause
the predictions to change (as the $FC$ investigates the number of prediction
changes after permutating an attribute type). Fourth, related to the previous
insight, the $C_{control}$ of the LR model is zero, meaning that the LR model
did not use any control flow attribute. Therefore, it is clear that the LR did
not learn that the label of the event log is dependent on the control flow
attributes (although having correct predictions in almost 94% of the cases?).
Fifth, it is also possible to compare the interpretability between different
classifiers. Moreover, we see an $FC_{control}$ of 91.78% for the XGB model,
with a value $C_{control}$ of 50. For the RF model, we observe an
$FC_{control}$ of 90.32%, with a value $C_{control}$ of 222 control columns.
This means that the XGB model uses fewer attributes compared to RF, but the
attributes used are more important for the predictions made (as 91% percent of
the predictions change if these attributes are permuted).

From a predictive performance perspective, the LLM, LR or RF models are
preferred. Nonetheless, the $IRC$ and $LOD@10$ values show that the
faithfulness of the explainability model of the RF model is compromised the
most. These values indicate that the explainability model is not able to
correctly mimic the model behaviour of the task model (which is estimated
through an attribute permutation importance). This can be due to the fact that
the RF model uses all of its attributes based on the values for
$\frac{C_{event}}{event}$,$\frac{C_{case}}{case}$ and
$\frac{C_{control}}{control}$ (indicated with a red colour). Next, although
the LLM model obtains the best predictive performance, it is less
interpretable than LR (higher $C_{event}$, $C_{case}$ and $C_{control}$).
Nonetheless, the LR did not seem to learn that the label is dependent on the
control flow attributes. Finally, the GLRM uses only 4 control attributes
($C_{control}$), and two case ($C_{case}$) attributes, which means that the
generated rules are roughly based on the presence (or absence) of certain
control attributes in combination with case attributes. The model is still
able to obtain a high AUC (89.21) value. This means that the use of the GLRM
model is advised, as the model has the best values for both IRC and LOD@10 and
is highly interpretable while remaining performant in terms of AUC. This
exemplary event log illustrates that the trade-off between predictive
performance, interpretability and/or faithfulness is a difficult one. By being
able to resort to the full framework of metrics, it becomes possible to make
an informed decision on how to approach the POP problem.

Figure 3: X-MOP: the guidelines for explainable AI purposes in POP

###  6.3 XMOP: Guidelines for XAI in POP

The insights obtained with the use of the research questions and event log
analysis are summarized in an XAI-based guidelines map for POP in Figure 3.
This figure describes the guidelines for Explainable AI purposes in POP. The
white boxes are the questions to guide partitioners and researchers in
obtaining POP results according to their preferences in terms of predictive
accuracy and explainability. The blue circles are the recommendations for the
POP predictive models and explainability models, together with indications
whether the advised methods tend to score well (green) or poor (red) for the
explainability metrics. Note that these recommendations do not apply to all
event logs, but are meant to guide authors to models that comply the best with
their requirements (based on the explainability metrics).

The guideline starts with the start event, whereafter a first question arises:
_Is explainability a lot more important than predictive performance?_. If so,
the guidelines directly advise using the GLRM model, as this model tends to be
parsimonious and faithful, while obtaining an average high AUC. Next, the
question, _Is parsimony very important?_ arises, which advises the use of deep
learning models if this is the case. Deep learning models are designed to work
with sequential data and do not need the aggregation sequence encoding as
mentioned in Section 3. Then, only when the response to the question: _is the
metric IRC unimportant?_ is yes, the guidelines advise using CNN over LSTM.
If, on the other hand, the response to the question _Is faithfulness
important?_ is positive, the guidelines advise using the XGB model. It is
necessary to mention that the faithfulness values for the GLRM are better (on
average), but the predictive performance is better (on average) for the XGB
model. Therefore, we take this trade-off into account. The next question, _Is
parsimony unimportant?_ is meant to distinguish between RF and the LR (and
advanced LLM). When parsimony is still essential, it is suggested to first use
LR and/or LLM, depending on whether the data is heterogeneous or not. The RF
model is recommended when you prefer models that use all the attributes. Next,
the XGB model is advised when a low value for LOD is required. The LLM model
should be preferred over the LR model when the data is heterogeneous.

Note that in the case of BPIC2015(1) (see Section 6), the LR and LLM were both
more parsimonious, more faithful and obtained a higher AUC than RF. This means
that, although the guidelines advise the use of GLRM (or DL models) to obtain
interpretable models (based on parsimony), the LR was also very interpretable.
This is due to the fact that we are working with aggregated values, so these
guidelines serve rather as a guideline (and therefore should not be taken
strictly).

##  7 Conclusion

This paper introduced a framework of metrics to evaluate the explainability of
predictive models used for POP purposes. Compared to typical performance-based
metrics and standard XAI metrics, they take into account the event, case, and
control perspective and describe the _interpretability_ and the
_faithfulness_. We provide an extensive benchmark study of seven models and
thirteen event logs, which illustrate how these metrics can capture different
interpretations relevant to process data. Finally, we provide the reader with
a consistent overview of the insights obtained by this study in the field of
POP through a framework of guidelines contrasting traditional machine
learning, deep learning and explainability approaches to guide the
practitioner to the best model selection. Not only can we conclude that the
transparent GLRM model exhibits very interpretable explanations for high
dimensional and sequential data (for only a small loss of performance), but
also that machine learning models tend to perform better than the deep
learning models based on predictive performance. Next, we also show that the
interpretability-predictive performance trade-off holds in the field of POP
when considering the introduced interpretability metric  _parsimony_. To
conclude, we show that each model has its advantages and disadvantages, where
naively opting for the model with the highest performance can have a strong
detrimental effect on both the interpretability and on the faithfulness of the
explanations. As a result, identifying faithful explanations, while remaining
interpretable, still imposes a challenge for black box models.

One of the limitations of this research is the fact that the metrics do not
take the loss of information of the aggregation encoding into account.
Therefore, it is assumed that the summary statistics of a certain variable
contain no loss in information compared to the original attribute values.
Future work will focus on the concept of Responsible AI, by introducing causal
inference to assess the causal insights obtainable from predictive models, and
focus on the creation of fairness-aware decision models.

## Acknowledgments

This was was supported in part by……

## References

  * [1] Milind Dawande, Zhichao Feng, and Ganesh Janakiraman.  On the structure of bottlenecks in processes.  Manag. Sci., 67(6):3853–3870, 2021.
  * [2] Martin J Land, Matthias Thürer, Mark Stevenson, Lawrence D Fredendall, and Kirstin Scholten.  Inventory diagnosis for flow improvement—a design science approach.  Journal of Operations Management, 67(5):560–587, 2021.
  * [3] Wil M. P. van der Aalst.  Process Mining - Data Science in Action, Second Edition.  Springer, 2016.
  * [4] Stephen C. Graves.  Reflections on the evolution of operations management.  Manag. Sci., 67(9):5379–5388, 2021.
  * [5] Irene Teinemaa, Marlon Dumas, Marcello La Rosa, and Fabrizio Maria Maggi.  Outcome-oriented predictive process monitoring: Review and benchmark.  ACM Trans. Knowl. Discov. Data, 13(2):17:1–17:57, 2019.
  * [6] Wolfgang Kratsch, Jonas Manderscheid, Maximilian Röglinger, and Johannes Seyfried.  Machine learning in business process monitoring: a comparison of deep learning and classical approaches used for outcome prediction.  Business & Information Systems Engineering, pages 1–16, 2020.
  * [7] Alejandro Barredo Arrieta, Natalia Díaz Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI.  Inf. Fusion, 58:82–115, 2020.
  * [8] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215, 2019.
  * [9] Aniek F. Markus, Jan A. Kors, and Peter R. Rijnbeek.  The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies.  J. Biomed. Informatics, 113:103655, 2021.
  * [10] Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger.  Evaluating the quality of machine learning explanations: A survey on methods and metrics.  Electronics, 10(5):593, 2021.
  * [11] Cynthia Rudin and Berk Ustun.  Optimized scoring systems: Toward trust in machine learning for healthcare and criminal justice.  Interfaces, 48(5):449–466, 2018.
  * [12] Arno De Caigny, Kristof Coussement, and Koen W De Bock.  A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees.  European Journal of Operational Research, 269(2):760–772, 2018\.
  * [13] Dennis Wei, Sanjeeb Dash, Tian Gao, and Oktay Gunluk.  Generalized linear rule models.  In International Conference on Machine Learning, pages 6687–6696. PMLR, 2019.
  * [14] Ingrid Nunes and Dietmar Jannach.  A systematic review and taxonomy of explanations in decision support and recommender systems.  User Modeling and User-Adapted Interaction, 27(3):393–444, 2017\.
  * [15] Marlon Dumas, Marcello La Rosa, Jan Mendling, and Hajo A. Reijers.  Fundamentals of Business Process Management, Second Edition.  Springer, 2018.
  * [16] Alexander Stevens, Johannes De Smedt, and Jari Peeperkorn.  Quantifying explainability in outcome-oriented predictive process monitoring.  In Jorge Munoz-Gama and Xixi Lu, editors, Process Mining Workshops - ICPM, volume 433 of Lecture Notes in Business Information Processing, pages 194–206. Springer, 2021.
  * [17] Anna Leontjeva, Raffaele Conforti, Chiara Di Francescomarino, Marlon Dumas, and Fabrizio Maria Maggi.  Complex symbolic sequence encodings for predictive monitoring of business processes.  In International Conference on Business Process Management, pages 297–313. Springer, 2016.
  * [18] Arik Senderovich, Chiara Di Francescomarino, Chiara Ghidini, Kerwin Jorbina, and Fabrizio Maria Maggi.  Intra and inter-case features in predictive process monitoring: A tale of two dimensions.  In International Conference on Business Process Management, pages 306–323. Springer, 2017.
  * [19] Jiaojiao Wang, Dongjin Yu, Chengfei Liu, and Xiaoxiao Sun.  Outcome-oriented predictive process monitoring with attention-based bidirectional lstm neural networks.  In 2019 IEEE International Conference on Web Services (ICWS), pages 360–367. IEEE, 2019.
  * [20] Hans Weytjens and Jochen De Weerdt.  Process outcome prediction: Cnn vs. lstm (with attention).  In International Conference on Business Process Management, pages 321–333. Springer, 2020.
  * [21] Ghada El-khawaga, Mervat Abu-Elkheir, and Manfred Reichert.  Xai in the context of predictive process monitoring: An empirical analysis framework.  Algorithms, 15(6):199, 2022.
  * [22] Renuka Sindhgatta, Chun Ouyang, and Catarina Moreira.  Exploring interpretability for predictive process analytics.  In ICSOC, volume 12571 of Lecture Notes in Computer Science, pages 439–447. Springer, 2020.
  * [23] Riccardo Galanti, Bernat Coma-Puig, Massimiliano de Leoni, Josep Carmona, and Nicolò Navarin.  Explainable predictive process monitoring.  In ICPM, pages 1–8. IEEE, 2020.
  * [24] Nijat Mehdiyev and Peter Fettke.  Explainable artificial intelligence for process mining: A general overview and application of a novel local explanation approach for predictive process monitoring.  Interpretable Artificial Intelligence: A Perspective of Granular Computing, pages 1–28, 2021.
  * [25] Renuka Sindhgatta, Catarina Moreira, Chun Ouyang, and Alistair Barros.  Exploring interpretable predictive models for business processes.  In BPM, volume 12168 of Lecture Notes in Computer Science, pages 257–272. Springer, 2020.
  * [26] Maximilian Harl, Sven Weinzierl, Mathias Stierle, and Martin Matzner.  Explainable predictive business process monitoring using gated graph neural networks.  Journal of Decision Systems, pages 1–16, 2020.
  * [27] Vincenzo Pasquadibisceglie, Giovanna Castellano, Annalisa Appice, and Donato Malerba.  Fox: a neuro-fuzzy model for process outcome prediction and explanation.  In 2021 3rd International Conference on Process Mining (ICPM), pages 112–119. IEEE, 2021.
  * [28] Stephen Pauwels and Toon Calders.  Bayesian network based predictions of business processes.  In International Conference on Business Process Management, pages 159–175. Springer, 2020.
  * [29] Sheikh Rabiul Islam, William Eberle, and Sheikh K. Ghafoor.  Towards quantification of explainability in explainable artificial intelligence methods.  In Roman Barták and Eric Bell, editors, Proceedings of the Thirty-Third International Florida Artificial Intelligence Research Society Conference, May 17-20, 2020, pages 75–81. AAAI Press, 2020.
  * [30] Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl.  Quantifying model complexity via functional decomposition for better post-hoc interpretability.  In PKDD/ECML Workshops (1), volume 1167 of Communications in Computer and Information Science, pages 193–204. Springer, 2019.
  * [31] Sarthak Jain and Byron C Wallace.  Attention is not explanation.  arXiv preprint arXiv:1902.10186, 2019.
  * [32] Sofia Serrano and Noah A. Smith.  Is attention interpretable?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [33] Sisi Ma and Roshan Tourani.  Predictive and causal implications of using shapley value for model interpretation.  In Proceedings of the 2020 KDD Workshop on Causal Discovery, pages 23–38. PMLR, 2020.
  * [34] Mythreyi Velmurugan, Chun Ouyang, Catarina Moreira, and Renuka Sindhgatta.  Evaluating fidelity of explainable methods for predictive process analytics.  In International Conference on Advanced Information Systems Engineering, pages 64–72. Springer, 2021.
  * [35] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  CoRR, abs/2103.11251, 2021.
  * [36] Jongchan Kim, Marco Comuzzi, Marlon Dumas, Fabrizio Maria Maggi, and Irene Teinemaa.  Encoding resource experience for predictive process monitoring.  Decision Support Systems, 153:113669, 2022.
  * [37] Joerg Evermann, Jana-Rebecca Rehse, and Peter Fettke.  Predicting process behaviour using deep learning.  Decision Support Systems, 100:129–140, 2017.
  * [38] Chiara Di Francescomarino, Marlon Dumas, Fabrizio Maria Maggi, and Irene Teinemaa.  Clustering-based predictive process monitoring.  IEEE transactions on services computing, 12(6):896–909, 2016.
  * [39] Fabrizio Maria Maggi, Chiara Di Francescomarino, Marlon Dumas, and Chiara Ghidini.  Predictive monitoring of business processes.  In CAiSE, volume 8484 of Lecture Notes in Computer Science, pages 457–472. Springer, 2014.
  * [40] Geetika T. Lakshmanan, Songyun Duan, Paul T. Keyser, Francisco Curbera, and Rania Khalaf.  Predictive analytics for semi-structured case oriented business processes.  In Business Process Management Workshops, volume 66 of Lecture Notes in Business Information Processing, pages 640–651. Springer, 2010\.
  * [41] Massimiliano De Leoni, Wil MP van der Aalst, and Marcus Dees.  A general process mining framework for correlating, predicting and clustering dynamic behavior based on event logs.  Information Systems, 56:235–257, 2016.
  * [42] Christoph Molnar.  Interpretable machine learning.  2020\.
  * [43] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681–3688, 2019.
  * [44] An-phi Nguyen and María Rodríguez Martínez.  On quantitative aspects of model interpretability.  CoRR, abs/2007.07584, 2020.
  * [45] Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec.  Interpretable & explorable approximations of black box models.  CoRR, abs/1707.01154, 2017.
  * [46] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30, 2017.
  * [47] Niek Tax, Ilya Verenich, Marcello La Rosa, and Marlon Dumas.  Predictive business process monitoring with LSTM neural networks.  In CAiSE, volume 10253 of Lecture Notes in Computer Science, pages 477–492. Springer, 2017.
  * [48] Vincenzo Pasquadibisceglie, Annalisa Appice, Giovanna Castellano, and Donato Malerba.  Using convolutional neural networks for predictive process analytics.  In 2019 international conference on process mining (ICPM), pages 129–136. IEEE, 2019.
  * [49] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [50] Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou.  Abcnn: Attention-based convolutional neural network for modeling sentence pairs.  Transactions of the Association for Computational Linguistics, 4:259–272, 2016.
  * [51] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining explanations: An overview of interpretability of machine learning.  In DSAA, pages 80–89. IEEE, 2018.