*

11institutetext: Research Centre for Information Systems Engineering (LIRIS),
Faculty of Economics and Business,
KU Leuven, Leuven, Belgium
11email<EMAIL_ADDRESS>
# Timing Process Interventions with Causal Inference and Reinforcement
Learning

Hans Weytjens 11 0000-0003-4985-0367 Wouter Verbeke 2233 0000-0002-8438-0535
Jochen De Weerdt 33 0000-0001-6151-0504

###### Abstract

The shift from the understanding and prediction of processes to their
optimization offers great benefits to businesses and other organizations.
Precisely timed process interventions are the cornerstones of effective
optimization. Prescriptive process monitoring (PresPM) is the sub-field of
process mining that concentrates on process optimization. The emerging PresPM
literature identifies state-of-the-art methods, causal inference (CI) and
reinforcement learning (RL), without presenting a quantitative comparison.
Most experiments are carried out using historical data, causing problems with
the accuracy of the methods’ evaluations and preempting online RL. Our
contribution consists of experiments on timed process interventions with
synthetic data that renders genuine online RL and the comparison to CI
possible, and allows for an accurate evaluation of the results. Our
experiments reveal that RL’s policies outperform those from CI and are more
robust at the same time. Indeed, the RL policies approach perfect policies.
Unlike CI, the unaltered online RL approach can be applied to other, more
generic PresPM problems such as next best activity recommendations.
Nonetheless, CI has its merits in settings where online learning is not an
option.

###### Keywords:

Prescriptive Process Monitoring Process Optimization Timed Interventions
Causal Inference Reinforcement Learning.

##  1 Introduction

Moving from predicting the outcome of a running process to optimizing it with
respect to a goal implies making decisions about actions that will change its
course. In its most basic form, the optimization of a process assumes a
correctly timed intervention (or sometimes non-intervention) in it. Examples
range from the escalation of a customer complaint process to higher management
echelons, the maintenance of a machine, a customer call to speed up an
administrative process or to maximize turnover, an additional test to conduct
to reduce a patient’s length of stay at hospitals, etc. Prescriptive Process
Monitoring (PresPM) is a young subfield of Process Mining (PM) studying
business process optimization methods. Optimization in the PresPM context
concerns decisions an agent has to take to optimize the outcome of a running
case given certain goals (metrics). It does not concern enhancing the
underlying process itself, as practiced in PM. Our PresPM (see below)
literature review reveals that two methods, reinforcement learning (RL)
(Sutton and Barto, 2018) and causal inference (CI) (Pearl, 2000; Imbens and
Rubin, 2015), emerge as pathways. However, a quantitative comparison is
currently missing. Most of the research on PresPM works with offline
historical data creating two limitations: Online RL is not possible and
experimental results prove difficult to quantify accurately for lack of
counterfactuals.

This research gap defines our contribution. In our experiments, we introduce
online RL to business processes and benchmark it against CI. Our use of
synthetic data, rather than historical event logs as in earlier PresPM
research, is not only instrumental in permitting both online RL and CI, but
also enables deeper insights, a correct evaluation of the experiments’
results, and the calculation of perfect policies as an absolute benchmark.

Solutions to timed process interventions can be seen as a gateway to solving
the more generic problem of recommending the next best activities in a
process. In timed interventions, the agent has one chance to make an
intervention sometime during the process, whereas, in next best activity
problems, the agent has to choose between all possible activities at every
step in the process. Both problem types are structurally the same, the RL
algorithms for the former can be transferred to the latter problem without
modification. The relative simplicity of timed process interventions in terms
of combinatorial possibilities (state space) results in fewer data and
computational requirements. It will permit easier insights into the
characteristics of the used models and find faster real-world adoption.
Furthermore, a vast number of relevant applications for timed process
interventions exist. For these reasons, our experiments focus on timed process
interventions, rather than next best activity recommendations.

This paper is structured as follows. Section 2 introduces concepts pertaining
to PresPM and refers to related work around them. We then move on to the
experimental Section 3 comparing RL to CI using two synthetic datasets. The
insights gained from the literature review and experiments lead to a deeper
discussion of the two PresPM methods in Section 4. We conclude this paper and
suggest avenues for future work in Section 5.

##  2 Background and related work

###  2.1 Preliminaries

The goal of PresPM (Bozorgi et al., 2021; Kubrak et al., 2021; Shoush and
Dumas, 2022; Weinzierl et al., 2020) is to recommend actions for ongoing cases
in order to optimize their outcomes as measured by a certain metric. Before
the appearance of ML, most prescriptive problems relating to processes
concerned industrial processes and were approached by operations research
methods, requiring mathematical models (Winston, 2022) of the problems. Later,
ML opened new, model-free opportunities, leading to a substantial body of
research about its application in predictive maintenance and process control
(regulating a system to keep certain parameters within a defined range). For
overviews, the interested reader is referred to Carvalho et al. (2019) and
Dalzochio et al. (2020). Within PresPM, a much more recent discipline, CI and
RL emerged as two promising methods for process outcome optimization and will
be the subject of this research.

The business processes studied in PresPM exhibit a substantial degree of
variation, in practice often explained by the humans in the loop. We believe
that the high degree of variation in these processes makes them generic, and
hence, the results of this work also apply to more constrained or structured
processes such as industrial processes.

In this work, we restrict our focus to the optimization of a single process in
isolation. This is an important assumption given that in practice, many
processes affect and even interact with each other. Action recommendations may
impact each other e.g., in case of limited resources. To the best of our
knowledge, the very large majority of PresPM research assumes process
independence. We also assume that decision points in processes are known after
either consulting experts and/or applying PM techniques to identifying
decision rules (Scheibel and Rinderle-Ma, 2022) and causal relationships
(e.g., Qafari and van der Aalst (2020), Leemans and Tax (2022), Koorn et al.
(2020), Polyvyanyy et al. (2019).

###  2.2 Causal inference

By default, CI (Pearl, 2000; Imbens and Rubin, 2015) works with offline,
logged data. The field can be subdivided into two components. The first
concerns the detection of causal relationships: “Which treatment(s) have an
effect on the process’ outcomes?”. The second CI component involves estimating
the effect of treatments. We concentrate on the individual treatment effect
(ITE) (Shalit et al., 2017), which is the difference between predicted
outcomes of (possible) treatment(s) and non-treatment for a given sample. For
example, when our model predicts that calling (treatment) customer $x$ will
increase revenue by $200$€, while $x$ is expected to reduce sales by $100$€ if
not called (non-treatment), then the ITEx is $300$€. Note that the ITE is an
expectation, not a hard-coded causality. Usually, a threshold (e.g., $50$€ in
our example) is determined to arrive at a policy for selecting (non-)
treatments. The main challenge is the absence of counterfactuals in the
dataset. A counterfactual is the unobserved outcome of a case assuming another
treatment than the one factually applied (not to be confused with negative or
forbidden events or traces as in Ponce de León et al. (2018)). In the absence
of randomized controlled trials (RCTs), realized by a policy of random
interventions, selection bias will occur as the data-gathering policy leads to
different distributions of treatment and non-treatment samples in the
datasets. Combating selection bias is an important aspect of CI (e.g., Shalit
et al. (2017)). Real-world CI applications include marketing (e.g., churn
reduction: Devriendt et al. (2021), discounting, …), education (e.g., Olaya et
al. (2020b)), recommender systems, etc. Most of these applications, however,
are cross-sectional rather than longitudinal: There are no timing issues, let
alone sequential treatments as seen in processes.

To the best of our knowledge, CI plays no role in the optimization of
industrial processes. In line with the overview provided by Kubrak et al.
(2021), we found no papers in the sparse PresPM literature published before
2020 claiming to use CI for process outcome optimization. de Leoni et al.
(2020) and Weinzierl et al. (2020) did apply a form of indirect CI with that
aim, albeit without carrying the CI label. The indirect approach consists of
first predicting the most likely (or distribution of) suffix(es) for every
possible treatment given a certain prefix. In the second step, another model
predicts outcomes for all these suffixes, which will then be used to choose a
treatment. In the direct CI approach, the process outcomes for all possible
treatments for a given prefix are directly predicted. Direct CI
implementations can be found in Bozorgi et al. (2023) (without timing
considerations) and Bozorgi et al. (2021) and Shoush and Dumas (2022)
(including timing). With the exception of Bozorgi et al. (2023), none of the
PresPM papers addresses selection bias. Bica et al. (2020), in contrast, use a
sequence-to-sequence recurrent NN that automatically builds a treatment-
invariant representation of the prefixes to combat the selection bias in a
medical treatment problem.

The lack of counterfactuals in the test set stemming from the use of offline
data hinders the accurate evaluation of CI methods’ results: For a given
prefix, the action recommended by the CI model may be absent from the cases in
the test set. Researchers cope with this problem by relying on a predictive
model to estimate outcomes, a distance-minimizing algorithm to find the
nearest case in the training or dataset, or a generative model that produces
augmented data (Neal et al., 2020).

###  2.3 Reinforcement learning

RL (Sutton and Barto, 2018) is an important class of ML algorithms learning
policies that guide an agent’s behavior or sequence of actions in an
environment in order to maximize an expected cumulative reward. Early
successes in computer games drew much attention to RL, which has since then
expanded not only into industrial processes but also into many other fields
such as robotics (e.g., autonomous driving: Sallab et al. (2017)), healthcare
(Gottesman et al., 2019), engineering, finance, etc. RL comes in many flavors.
We will discuss and use the widespread Q-learning variant. In processes, the
most important reward is often the process outcome that becomes known at the
conclusion (last event) of the case. Regardless, intermediate rewards could be
easily included in RL should they occur. The cost of actions can be viewed as
a negative reward. At its core, RL assumes an online environment that the
agent can interact with. RL does not need an environment ($\rightarrow$
process) model. Instead, real (or simulated) episodes ($\rightarrow$ cases)
are just executed and their rewards ($\rightarrow$ outcomes) are observed. For
every encountered state ($\rightarrow$ prefix), a state-action value (Q) is
learned for every possible action. Q represents the state-action value for the
next state ($\rightarrow$ prefix) plus the reward minus the cost of that
action to get to that next state (a transition). For any given prefix, the
state-action values can be interpreted similarly as the effects of the
possible treatments learned by CI. The difference between the state-action
value for a treatment and the one for the non-treatment corresponds to the ITE
at that state (prefix). The state-action value of the last prefix of a
(completed) process is its final outcome. Given the size of the state space
($\rightarrow$ number of possible prefixes) in most processes, these state-
action values cannot be stored in tabular form (Q-table). Instead, they are
approximated by an NN. This is called deep reinforcement learning (Mnih et
al., 2015). At every state ($\rightarrow$ prefix), the policy will be to
choose the action with the highest relative state-action value. Learning is
achieved by playing out many processes and iteratively updating the Q-table NN
after each (batch of) observed rewards ($\rightarrow$ outcomes). In order to
explore all areas of the state space and to prevent prematurely settling into
a sub-optimal policy, a certain degree of exploration is introduced: The agent
will sometimes overrule the policy and choose another action, especially at
the beginning of the learning process. RL has found many applications in
process outcome optimization, e.g., in robotics (Ibarz et al., 2021) and
industrial process control (Paraschos et al., 2020; Spielberg et al., 2020),
but few researchers (Branchi et al., 2022; Bugaenko, 2021) apply RL to PresPM
process optimization.

In practice, the real-life form of data gathering is often too slow and too
expensive. It can even be dangerous at the early stages of learning when the
NN is insufficiently trained and significant exploration happens. An entire
spectrum of alternative data-gathering methods at different proximities to
reality exists. In academia, synthetic data are instrumental in investigating
and comparing methods. For instance, Paraschos et al. (2020) work with
synthetic data for industrial process control. Simulation models can be rooted
in the laws of physics or even social sciences. Spielberg et al. (2020) use
simulation in industrial processes, Tan et al. (2018) use simulation to train
robots and investigate pathways to close the reality gap, the mismatch between
the reality and the simulation. In Lisowska et al. (2021), a Fogg behavior
model is used for healthy habit formation for cancer patients. Digital twins
(Park et al., 2021) are extended simulations benefiting from imputed real-time
data. Burggraef et al. (2022) deploy a miniature factory in which their agent
can act. In practice and academia alike, there is great interest in offline RL
i.e. to work with data from existing datasets (supervised data). This can be
achieved by mining models from the data. PM discovery techniques, for example,
yield grid graphs of business processes as representations of the agent’s
environment in Branchi et al. (2022) and Bugaenko (2021). Research on offline
RL (Levine et al., 2020) also suggests using predictive models trained on the
dataset to guide the agent through its environment and estimate outcomes,
somehow similar to indirect CI. Alternatively, nearest-neighbors algorithms
can force the agent to remain in the vicinity of the data-gathering policy.
The RL agent can even be forced to remain within the boundaries of the
dataset, which of course makes it harder to improve upon the original data-
gathering policy. With the exception of purely synthetic data, these data-
gathering strategies can all be used to derive a policy with which to
initialize an online learning agent. This dual strategy greatly speeds up
training, avoids expenses, and minimizes mistakes.

###  2.4 Problem complexity

From the analysis of the related work, we identify two main drivers for
problem complexity: action width and action depth. These drivers facilitate
understanding problems at hand and will be helpful in positioning our
experiments (Section 3) and the subsequent discussion of CI and RL (Section
4). Action width relates to the number of different actions (set size)
available to an agent: Actions can be binary (interventions e.g., “apply” or
“don’t apply”), continuous (e.g., a regulating valve) or multi-class (e.g.,
several options such as “visit”, “call” or “email” customer or “do nothing”).
In this paper, we define multi-class actions as treatments. Interventions are
thus a binary subclass of treatments. Asymptotically, the set of multi-class
actions becomes the set of all possible activities, even including attributes,
as in the next best activity prediction. The action depth is a measure of the
longitudinal dimension and depends on how many consecutive actions can be
taken within a process, and when. An action’s timing can be predetermined
(fixed or irrelevant) or an action can happen once at any time during the
process’ lifetime (one-off, as in timed process interventions). Sequences of
(repeated) actions are another, more complicated, setting. Finally, actions
can be continuous such as steering an autonomous vehicle to keep it in its
lane.

###  2.5 Research gap

There exists no quantitative comparative analysis of CI and RL process outcome
optimizations (nor any other) problems. This is the main research gap we
address in this paper. As explained in subsection 2.2, the use of historical
data for the test sets hampers the evaluation of CI methods for lack of
counterfactuals. A similar issue appears in the RL literature that exhibits a
prevalence of non-real-life work. Here, simulations or models based on reality
are used to train and test online models without considering the performance
on the original problems, thus ignoring the reality gap (exception: Spielberg
et al. (2020)). We also address this issue by making use of entirely
artificial synthetic data in our experiments. This form of data allows us to
accurately evaluate CI, to test online RL and eliminate the reality gap, and
to share the same test set between both methods. Additionally, none of the
aforementioned papers compared their results to perfect policy results needed
to gain an intuition for the absolute performance of their methods. This can
be explained by the majority of the discussed papers treating rather complex
problems for which computing such a perfect policy is intractable, hence the
need for techniques such as CI and RL. We opt for timed interventions, which
have narrow action widths (binary) and shallow action depths (at most once per
process) so that we can easily compute results for a perfect policy. The next
Section 3 describes our contribution: making an accurately evaluated CI-RL-
perfect-solution comparison based on synthetic datasets.

##  3 Experimental comparison of CI and RL

In the following three subsections, we describe our data generation,
experimental setup, and results.

###  3.1 Data generation

We work with synthetic processes generating the environment and data for our
experiments in order to compute counterfactuals that are not available in
real-world data. Knowing the counterfactuals allows for accurate evaluations
of the experiments. Moreover, given a sufficiently small state space, a
perfect policy can be derived and used to judge the absolute performance of
methods. The same synthetic generative model can create both the offline
dataset for CI and the online environment required for online RL. We first
describe the two processes and then motivate our choice.

####  3.1.1 Two synthetic processes

The process models as Petri nets and key features of our two synthetic
processes are shown in Figure 1 and Table 1 respectively. Process_1 is a
sequence of three activities, either “$A$” or “$B$” with an according integer
attribute. At one of the three events, a (free) intervention can be made. The
outcome of the process is the sum of the attributes, where the attribute of
the event where the intervention took place is multiplied by $2$ if activity
“$A$” occurred at least once in the process, otherwise by $-2$. Process_2
consists of five events and includes both an AND and an XOR construct. Every
Process_2 case carries an integer case attribute known from the start. Event
attributes are integers as well, and an intervention can be made once in a
process at any event. When an intervention is made (at a cost of $5$), the
attribute corresponding to the first of “$D1$” or “$D2$” to occur thereafter
(if any) will be multiplied by $2$ in case the process passes through its
“$B$” branch, by $-4$ otherwise. The final outcome is the sum of the
attributes times the case attribute.

(a) Process model representing Process_1

(b) Process model representing Process_2

Figure 1: Synthetic generative processes: Petri net visualization Table 1:
Synthetic generative processes: key features

|  Process_1 |  Process_2
---|---|---
Structure |  sequence of OR |  AND with OR in lower branch
Activities |  every event activity from $[A,B]$ with prob $[.25,.75]$ |  as per graph. “$B$” and “$C$” with prob $[.2,.8]$
Attributes ($Att$) |  uniformly random from $[0,1,2,5]$ |  uniformly random from $[1,2,3]$ for “$D1$” and from $[1,2,3,4]$ for “$D2$”, else $0$
Case variables ($CV$) |  - |  uniformly random integer from $[1-10]$
State space size |  512 |  720
Interventions |  at all steps or not at all |  at all steps or not at all
Intervention cost |  0 |  5
Intervention effect (intervention at event $n$) |  Multiply $Att_{n}$ by 2 in case an “$A$” occurred in the process, otherwise by -2. |  $Att$ corresponding to next of “$D1$” or “$D2$” multiplied by $2$ (in case of “$B$”-process) or $-4$ (in case of “$C$”-process)
Outcome |  $\sum Att$ |  $CV*\sum Att$

####  3.1.2 Motivation

These processes and interventions were designed to be simple for clear
insights, yet representative of real-world processes by incorporating their
main challenges. Since PresPM concerns actionable decisions, we can reduce
sub-processes that do not contain any decision points (and are not a branch of
a parallel structure with another branch containing such a decision point)
into one event (e.g., the subprocess “$R-V$” collapses into event “$A$” in
Fig. 2), thus significantly shrinking the process model. In our experiments,
the interventions only change event attributes but in reality, they may alter
the control-flow as well. That would not change the CI and RL algorithms.
Moreover, when all control-flow variations starting from a given decision
point (after event “$W$” in Fig. 2) merge together in one location/activity
later in the process model (“$Z$”) without containing any further intermediate
decision points, then they can be reduced to one event (“$B$”) as well. The
value of this event’s attribute will vary according to which decision was made
and which control-flow variant was followed earlier.

Figure 2: Example of how a process can be reduced to a simpler process for
PresPM purposes. “$A$” summarizes the subprocess without decision points
“$R-V$”, whereas “$B$” summarizes the subprocess ‘$W-Z$” which includes a
decision point “$W$”.

Both processes have a strong stochastic component to reflect the uncertainty
accompanying real-life processes. The values of the three activities and
attributes in Process_1 are sampled from probability distributions, whereas
activities in Process_2 are governed by the given structure, with the
attributes and case variables sampled from probability distributions as well.
A real-life decision-maker is not only confronted by stochasticity, but the
information available to make decisions may also differ between cases. Our
synthetic processes also incorporate this aspect: as long as no “$A$” appears
in Process_1 or Process_2 hasn’t passed through its “$B$” or “$C$” branch, it
cannot be known for sure whether intervening will be beneficial or
detrimental.

As in many real-world processes, the outcomes of both processes will only be
known at their conclusion. Including intermediary rewards or penalties,
however, would not significantly alter the CI or RL algorithms.

Our experiments investigate binary actions (interventions). This
simplification allows for clearer insights without loss of generalization. As
direct CI is generally not suited for sequences of actions, we further
simplified by opting for one-off actions (timed interventions) to permit a CI-
RL comparison; the RL method, however, can be extended to sequential or
continuous actions without modification. In combination with the use of
synthetic data, the small state space resulting from a narrow action width and
shallow action depth also renders calculating perfect policies practical.

###  3.2 Experimental setup

Even though increasingly performant next event prediction algorithms exist
(e.g., Pasquadibisceglie et al. (2022)), the indirect CI approach inevitably
compounds the errors of two successive prediction models. The direct CI
approach circumvents the next event/suffix prediction stage. The simplicity of
working with one model favors the direct approach and we used it in our
experiments. We use one NN to predict process outcomes for CI; the
intervention (Boolean) is part of its inputs, and the batch size is 1,024. RL
is achieved with a standard Q-learning architecture with a 1,024-transition
samples memory for stabilization (experience replay: Mnih et al. (2015)).
Transition samples are retained in the memory according to the first-in-first-
out (FIFO) principle. A penalty of 100 is applied for intervening more than
once. An NN predicts Q for both possible actions (“intervention” and “non-
intervention”) at every encountered state (prefix). For a balanced comparison,
the same NN architecture is used for both CI and RL. Our NNs have two LSTM and
two dense layers as displayed in Fig 3.

Figure 3: The NN architectures (regression model) are almost identical for CI
and RL. The first layer includes an additional “intervention” feature for CI.
The last layer outputs a scalar (outcome) for CI and a 2-dimensional vector (Q
for “intervention´´ and “non-intervention”) for RL. Table 2: Experimental
settings.

Setting | Value
---|---
Input features | “activity”, “attribute”,
| and “case variable” (for Process_2)
Categorical features representation | one-hot-encoding
Sequence length (padding for shorter prefixes) | 3 and 5 (Process_1 and Process_2)
Loss function | MAE
Optimizer | ADAM
Metric | Uplift
Nr. runs per experiment | 5
Batch size | 1,024
Memory size (RL) | 1,024
Nr. epochs | early stopping
Patience | 5
Test set size | 1,000 samples
Validation set size (% of training set) | 20% (threshold calculation in CI)

For the CI learning phase, an RCT dataset of 10,000 samples is generated. This
largely exceeds both processes’ state space size and should, therefore, offset
CI’s offline handicap. For RL, data are generated on the fly. The test set
consists of 1,000 samples for which all counterfactuals are computed (feasible
thanks to the relatively simple processes and the binary one-off action
design). The data generated by the synthetic processes are preprocessed as
follows: The activity levels are one-hot encoded. The outcomes, attributes,
and case variables (Process_2) are standardized. For CI, the intervention
decision ($1$ or $0$) is concatenated with the other event features. For every
case sample, we build a sequence (sequence length = total process length) for
every prefix, using padding to complete the sequence for ongoing process
instances. We thus arrive at a two-dimensional data structure that is fed into
the models’ input layer. For Process_2, the case variable enters the models
separately after the LSTM layers.

Every experiment is carried out five times and learning stopped using an
early-stopping algorithm for both methods. A policy based on CI requires
identifying the threshold, which we identify as the value that maximizes the
ITE score on a 20% validation set. Uplift (Devriendt et al., 2021; Olaya et
al., 2020a) is the metric to evaluate the results. It is the difference
between the process outcomes of implementing the policy and not intervening at
all, cumulated over the complete test set. The experimental settings are
summarized in Table 2.

###  3.3 Results

We summarized our experimental results in Table 3. RL clearly outperforms CI
for both processes: The mean scores significantly higher. The standard
deviations of the RL scores are much lower, making RL by far the more robust
method. Most or all of this outperformance can be attributed to RL’s innate
superior ability to find the optimal policy (see Section 4). The fact that
online RL permits exploring all parts of the state space plays virtually no
role here, as the CI training sets in our experiments contain the complete
state space as well. Were this not the case, the observed CI-RL divergence
would certainly widen.

Table 3: Experimental results comparing CI, online RL, a perfect and random
policy for both processes. Online RL reaches the highest uplift but requires
much more computational effort than CI.

| Uplift | Computational effort
---|---|---
| Mean | StDev. | Unit | Mean | StDev.
| CI | 1,526 | 36.8 | epochs | 188 | 27.6
Process_1 | RL | 1,616 | 5.0 | transitions | 6,000 | 2,549
| perfect | 1,651 |  |  |  |
| RCT | -515 |  |  |  |
| CI | 1,682 | 203.1 | epochs | 212 | 51.7
Process_2 | RL | 1,806 | 34.7 | transitions | 26,800 | 10,628
| perfect | 1,845 |  |  |  |
| RCT | -50,336 |  |  |  |

The precise knowledge of the (stochastic) synthetic generative processes
enables computing perfect policies. This is done by drawing the complete state
space in tree form and then calculating the best policy (intervene/don’t
intervene) from the leaves (512 or 720 for our processes) back to the prefixes
of length one, always assuming no intervention happened before. Table 3 shows
that RL comes to within 3% of the perfect policy results for both processes
(some stochasticity is normal). The CI policy constitutes a substantial
improvement over the RCT data-gathering policy that originally created the
dataset as well, albeit to a lesser extent than RL.

Having set both RL’s memory and CI’s batch size to 1,024, one optimization
step of the NN involves the same number of samples for both methods. Since
every RL transition (except for the first 1,023 ones) was followed by an NN
optimization step, we can directly compare the number of RL transitions to the
number of CI epochs. Table 3 shows that RL’s computational requirements are
two orders of magnitude higher than those for CI.

##  4 Discussion

In this section, we discuss the suitability of CI and RL for PresPM and show
why RL outperformed CI in our experiments. We also address the issues of RL’s
online requirement, reward specification, and inefficiency.

####  4.0.1 Causal inference

Learning counterfactuals and treatment effects is at the core of CI. The
sequential aspect of processes, however, poses a problem: The decision to not
treat at a certain time in a running process does not preclude treatments
later on in the process. For any given prefix in our experiments, direct CI
relied on a predictive model to estimate the process outcomes for both
intervention and non-intervention. This is problematic in the latter case: The
predictive model cannot discern the optimal path from that prefix, and will
instead consider the outcomes for all encountered treatments under the data-
gathering policy that produced the relevant samples in the training set, as
illustrated in the simplified example in Figure 4a.

(a) CI: policy requires threshold (TH), e.g., TH=0 $\rightarrow$ intervene at
1st event; TH=2 $\rightarrow$ intervene at 2nd event.

(b) RL: optimal policy follows maximum Q values: do not intervene at 1st
event, intervene at 2nd event.

Figure 4: Simple process to compare CI to RL. Both agree on the policy at the
second event. At the first event, CI correctly estimates the outcome for
intervention ($Y_{I}$), whereas the prediction model for the outcome for non-
intervention ($Y_{NI}$), will observe two different outcomes ($5$ and $0$) and
summarize (here: average) those to $2.5$. This value depends on the loss
function and the samples’ distribution (percentages in the graph) in the
training set, which itself depends on the data-gathering policy. In contrast,
RL selects the maximum of the two Q values in the second event.

Direct CI, therefore, only operates safely on problems without any action
depth (“fixed” or “irrelevant”), and will become increasingly suboptimal when
moving to real processes with action depths “once” or “multiple”. The action
width for CI realistically comprises “binary” and “multi-class” treatments. CI
cannot handle permanently-running processes. Thresholds are sub-optimal
compromises and products of optimization algorithms themselves. Dependencies
between processes, e.g., when resources (space, manpower) are limited or
processes interact with each other, cannot be incorporated in the CI
framework. Because of these deficits, optimal policies are theoretically out
of CI’s reach, as confirmed in our experiments. Nevertheless, CI policy
results are still better than those that the data-gathering policy yields.

Similar to all other predictive models used for prescriptive or decision-
making purposes, feedback loops (Sculley et al., 2015) risk deteriorating
results: Implementing the CI policy will progressively shift the real-life
data distribution away from the original training data, decaying the models’
predictive accuracy. Frequent updates of the CI models would help but at the
same time introduce new bias in the data (new data-gathering policy). However,
with a sufficient degree of randomness in the decisions taken (as in RCTs and
similar to exploration in RL), this iterative, in the limit online CI,
approach would neutralize the feedback loops.

####  4.0.2 Reinforcement learning

RL has many theoretical advantages over CI. It does not require a prediction
model and can rely on observed outcomes. RL is entirely generic:
Theoretically, it can deal with any action width or depth as well as with
continuous processes. Next best activity prediction, which represents the
ultimate action width and depth, requires no change to the RL algorithms we
used for timed process interventions. RL models are very flexible:
Constraints, rewards, and penalties can be added at liberty to avoid
detrimental or unacceptable actions, pursue secondary goals, etc. With online
RL, agents can freely interact with their environment, and dependencies
between processes can be taken into account if the processes are treated
concurrently by one model. Exploration in online RL theoretically visits the
complete state-space (all possible prefixes). Given sufficient exploration,
online RL policies will automatically adapt to a changing environment (concept
drift). Proven theorems even show that online Q-learning algorithms converge
given enough time. Both online and offline RL, however, are known to be
inefficient, requiring many transitions to converge to the optimal policy, as
demonstrated by our experiments.

The max operator over the Q-values (see Figure 4b) explains RL outperformance
versus direct CI with equal data access. For every prefix, the learned Q
values represent the expected outcomes for intervention and non-intervention
respectively, assuming a (calculated) perfect policy after that, whereas the
ITEs in CI represent the difference between the expected outcomes, each of
which depends on the sample distribution from the data-gathering policy and
the loss function. Note, however, that with an online CI approach (with real-
time updating after every finished process observed) and allowing exploration,
this data-gathering policy would converge to the optimal policy as well, thus
practically obliterating the differences between CI and RL.

####  4.0.3 Real-world implementation

Despite its power and versatility, RL suffers from some important drawbacks.
Yet, many of these are not entirely unique to RL but apply to CI and PresPM in
general as well. The first such drawback is the risk of committing errors
during real-time implementation. This implementation risk, however, can be
reduced to that of the data-gathering policy (the de facto policy in place
upon which the CI dataset is based) by inserting constraints into the RL
algorithm that can easily deal with those. Rules mined earlier with a process
discovery algorithm can frame the agent’s actions as prescribed by Dumas et
al. (2022). Even human intuition can be inserted by allowing the human agent
to overrule the RL algorithm’s proposed action. In other words, implementing
RL should not be riskier either than the original, existing policy or than
implementing CI. The latter two policies occasionally make or propose costly
mistakes too. If necessary, a two-stage offline-online approach can further
reduce the risk: Offline RL based on simulations or predictive models can
serve as an initialization to an online RL that then continues to learn acting
in the real world, thereby closing the reality gap.

A similar argument can be made for the related challenge of reward
specification. The desired outcome for a process to be optimized will not
always be one-dimensional: The primary goal may be to reduce throughput time,
however, without compromising employees’ well-being and product quality.
Moreover, such goals may shift over time or may need adjustment in the face of
concept drift. Again, this challenge is not unique to RL, and exists
regardless of the solution method, if any. When possible, these goals will be
consolidated into one metric for use by both CI and RL. If not, RL can be
extended to include constraints on undesired actions and/or rewards/penalties
that promote secondary goals. As before, the human agent can also overrule the
RL’s algorithms suggestions.

RL is inefficient: It is data-hungry and slow to converge. Our experiments
were based on relatively short and simple processes. Longer and more
complicated processes (great action width/depth) will have an exponentially
larger state space, suggesting that RL will no longer be a viable option where
CI could still be. Yet, in deep RL, the Q-table is replaced by an NN, which to
some extent obsoletes the need to visit the complete state space as unseen
state-action (prefix-action) pairs can be interpolated. Working examples of
this are video games with very large, and autonomous driving with near-
infinite state spaces. The more similar regions the state space contains, the
better this will work. Additionally, limiting the number of actions to the
most relevant ones with causal discovery techniques (first CI component in
Pearl (2000)) may be a worthwhile investment before starting with RL (and CI
as well). PM has an arsenal of causal discovery techniques that can be used to
this end (Scheibel and Rinderle-Ma, 2022; Qafari and van der Aalst, 2020;
Leemans and Tax, 2022).

Online RL implies working with event streams rather than event logs. Streaming
is an active field of PM research. Burattin (2022) identifies different types
of incomplete cases in the observation window. This is not an issue for an
online RL as it always starts from a case’s beginning and updates itself after
observing rewards for every new event it encounters (transition) until the
case is complete.

The process independence assumption underlying both methods warrants caution
when generalizing the results from our experiments. The larger the
dependencies between processes and the larger the share of processes being
optimized, the higher the risk of mutual process interference jeopardizing the
expected results.

##  5 Conclusions and future work

We conducted experiments on timed process interventions with synthetic data
that render genuine online RL and the comparison to CI possible and allow for
an accurate evaluation of the results. We showed how the theoretical problems
burdening CI can be overcome by online RL, contingent upon the strong
assumption of real-time implementation of the learned policies in the real
world. In our experiments, online RL produced better and more robust policies
than CI. In fact, RL nearly reached the theoretically optimal solution, which
can be inferred because of the use of synthetic data. The used RL methods can
also be applied without any modification to similar problems with greater
action width and depth (next best activity prediction in the limit). When
computational effort and/or the real-time implementation requirement preclude
online RL, CI may be a viable alternative in scenarios where the dataset
covers a large and evenly distributed share of the state space and action
depth is limited.

With this work, we contributed to the nascent field of PresPM. We chose a
simplified setting to gain some important insights. Reaching PresPM maturity
will depend on exploring other, perhaps more sophisticated approaches, in ever
more realistic settings. Further extensions of this work are, therefore,
plentiful. First, an initial investigation of the merits of loss attenuation
(Weytjens and De Weerdt, 2022), uncertainty (Weytjens and De Weerdt, 2022),
and future individual intervention effects (Shoush and Dumas, 2022) revealed
promising insights but should be corroborated. Future work could also shed
light on the conditions under which RL remains efficient enough on realistic
problems with sequences of multiple possible actions (greater action width and
depth). Further complications could include the introduction of outcome noise,
uncertain inputs, and concept drift. Since the rewards of processes often only
happen (or become known) at their conclusion, MC learning (as in Branchi et
al. (2022)) could be a faster alternative to the classical Q-learning we used.
The FIFO principle for the online RL transition samples memory could be
replaced by more sophisticated sampling techniques such as described in Fani
Sani et al. (2023) for PredPM, or in Liu and Zou (2018) for experience replay
in RL. Leveraging uncertainty estimates could be another option to improve
sampling. RL does adapt to concept drift, but only very slowly. As a
consequence, RL is not suited to deal with disruptions (e.g., caused by a
pandemic). Digital twins for processes or organizations have been proposed as
a solution (van der Aalst et al., 2021) and are an avenue for future research.
Instead of including the complete state space in the data for CI, as we did,
it could be investigated to what extent CI would fall further behind online RL
when the dataset only covers part of the state space (and contains selection
bias caused by the data-gathering policy). For applications where online RL is
not an option, more research on offline RL is recommended. Lifting the
assumption of process independence would move the problem setting even closer
to reality and would pose additional challenges: Process independence is a
requirement to satisfy the stable unit treatment value assumption (SUTVA)
(Imbens and Rubin, 2015) in CI. The combinatorial explosion caused by
interdependent processes is challenging for RL as well and possibly demands
additional heuristics (e.g., Huang et al. (2011)). In the domain of CI,
adaptations to the standard algorithms could lead to more capabilities in
terms of action depth (possibly with a discounting mechanism as used in RL).
Indirect CI’s theoretical ability to handle sequences of actions could be
weighed against the accuracy loss due to the compounding of two predictive
models. Combating selection bias in processes (as in Bica et al. (2020) for an
environment without exogenous actors) beckons more research as well. Causal
Reinforcement Learning (Zeng et al., 2023) enriches RL with the first
component of CI (causal relationship detection) by means of causal graphs. It
requires either a priori causal graphs (which are rarely available in PresPM,
or deriving them from the observational data under a set of assumptions (e.g.
Potoniec et al. (2022) and Bozorgi et al. (2020) for business processes). In
our discussion about action width and depth (Subsection 2.4), we did not
elaborate on how the decision points and the set of possible actions available
to the agents at those points are elaborated. Next to human expertise, both PM
and other methods should be reviewed from a PresPM perspective.

## References

  * Bica et al. (2020) Bica, I., Alaa, A. M., Jordon, J., and van der Schaar, M.  Estimating counterfactual treatment outcomes over time through adversarially balanced representations.  _CoRR_ , abs/2002.04083, 2020.  URL https://arxiv.org/abs/2002.04083.
  * Bozorgi et al. (2020) Bozorgi, Z. D., Teinemaa, I., Dumas, M., La Rosa, M., and Polyvyanyy, A.  Process mining meets causal machine learning: Discovering causal rules from event logs.  In _2020 2nd International Conference on Process Mining (ICPM)_ , pages 129–136, 2020.  doi: 10.1109/ICPM49681.2020.00028.
  * Bozorgi et al. (2021) Bozorgi, Z. D., Teinemaa, I., Dumas, M., Rosa, M. L., and Polyvyanyy, A.  Prescriptive process monitoring for cost-aware cycle time reduction.  In _2021 3rd International Conference on Process Mining (ICPM)_ , pages 96–103, 2021.  doi: 10.1109/ICPM53251.2021.9576853.
  * Bozorgi et al. (2023) Bozorgi, Z. D., Irene Teinemaa, I., Dumas, M., La Rosa, M., and Polyvyanyy, A.  Prescriptive process monitoring based on causal effect estimation.  _Information Systems_ , 2023.  doi: https://doi.org/10.1016/j.is.2023.102198.  URL https://www.sciencedirect.com/science/article/abs/pii/S0306437923000340.
  * Branchi et al. (2022) Branchi, S., Di Francescomarino, C., Ghidini, C., Massimo, D., Ricci, F., and Ronzan, M.  Learning to act: A reinforcement approach to learn to best activities.  In Di Ciccio, C., Dijkman, R., del Río Ortega, A., and Rinderle-Ma, S., editors, _Business Process Management Forum_ , pages 137–154, Cham, 2022. Springer International Publishing.  ISBN 978-3-031-16171-1.
  * Bugaenko (2021) Bugaenko, A. A.  Application of reinforcement learning to optimize business processes in the bank.  _Turkish Journal of Computer and Mathematics_ , 12, 2021.
  * Burattin (2022) Burattin, A.  Streaming process mining.  In van der Aalst, W. M. and Carmona, J., editors, _Process Mining Handbook_ , pages 349–372. Springer International Publishing, Cham, 2022\.  ISBN 978-3-031-08848-3.  doi: 10.1007/978-3-031-08848-3-1.  URL https://doi.org/10.1007/978-3-031-08848-3-1.
  * Burggraef et al. (2022) Burggraef, P., Steinberg, F., Heinbach, B., and Bamberg, M.  Reinforcement learning for process time optimization in an assembly process utilizing an Industry 4.0 demonstration cell.  _Procedia CIRP_ , 107:1095–1100, 2022.  doi: https://doi.org/10.1016/j.procir.2022.05.114.
  * Carvalho et al. (2019) Carvalho, T. P., Soares, F. A., Vita, R., da P. Francisco, R., Basto, J. P., and Alcaliá, S. G.  A systematic literature review of machine learning methods applied to predictive maintenance.  _Computers & Industrial Engineering_, 137:106024, 2019\.  ISSN 0360-8352.  doi: https://doi.org/10.1016/j.cie.2019.106024.
  * Dalzochio et al. (2020) Dalzochio, J., Kunst, R., Pignaton, E., Binotto, A., Sanyal, S., Favilla, J., and Barbosa, J.  Machine learning and reasoning for predictive maintenance in Industry 4.0: Current status and challenges.  _Computers in Industry_ , 123:103298, 2020.  ISSN 0166-3615.  doi: https://doi.org/10.1016/j.compind.2020.103298.
  * de Leoni et al. (2020) de Leoni, M. d., Dees, M., and Reulink, L.  Design and evaluation of a process-aware recommender system based on prescriptive analytics.  In _2020 2nd International Conference on Process Mining (ICPM)_ , pages 9–16, 2020.  doi: 10.1109/ICPM49681.2020.00013.
  * Devriendt et al. (2021) Devriendt, F., Berrevoets, J., and Verbeke, W.  Why you should stop predicting customer churn and start using uplift models.  _Information Sciences_ , 548:497–515, 2021.  ISSN 0020-0255.  doi: https://doi.org/10.1016/j.ins.2019.12.075.
  * Dumas et al. (2022) Dumas, M., Fournier, F., Limonad, L., Marrella, A., Montali, M., Rehse, J.-R., Accorsi, R., Calvanese, D., De Giacomo, G., Fahland, D., et al.  AI-augmented business process management systems: A research manifesto.  _ACM Transactions on Management Information Systems_ , 2022.
  * Fani Sani et al. (2023) Fani Sani, M., Vazifehdoostirani, M., Park, G., Pegoraro, M., van Zelst, S. J., and van der Aalst, W. M.  Performance-preserving event log sampling for predictive monitoring.  _Journal of Intelligent Information Systems_ , 2023.  ISSN 1573-7675.  doi: 10.1007/s10844-022-00775-9.  URL https://doi.org/10.1007/s10844-022-00775-9.
  * Gottesman et al. (2019) Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., and Celi, L. A.  Guidelines for reinforcement learning in healthcare.  _Nature Medicine_ , 25(1):16–18, jan 2019.  doi: 10.1038/s41591-018-0310-5.  URL https://doi.org/10.1038/s41591-018-0310-5.
  * Huang et al. (2011) Huang, Z., van der Aalst, W. M., Lu, X., and Duan, H.  Reinforcement learning based resource allocation in business process management.  _Data & Knowledge Engineering_, 70(1):127–145, 2011.  ISSN 0169-023X.  doi: https://doi.org/10.1016/j.datak.2010.09.002.
  * Ibarz et al. (2021) Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., and Levine, S.  How to train your robot with deep reinforcement learning: Lessons we have learned.  _The International Journal of Robotics Research_ , 40(4-5):698–721, 2021.  doi: 10.1177/0278364920987859.  URL https://doi.org/10.1177/0278364920987859.
  * Imbens and Rubin (2015) Imbens, G. W. and Rubin, D. B.  _Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction_.  Cambridge University Press, 2015.  doi: 10.1017/CBO9781139025751.
  * Koorn et al. (2020) Koorn, J. J., Lu, X., Leopold, H., and Reijers, H. A.  Looking for meaning: Discovering action-response-effect patterns in business processes.  In Fahland, D., Ghidini, C., Becker, J., and Dumas, M., editors, _Business Process Management_ , pages 167–183, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-58666-9.
  * Kubrak et al. (2021) Kubrak, K., Milani, F., Nolte, A., and Dumas, M.  Prescriptive process monitoring: Quo vadis?  _CoRR_ , abs/2112.01769, 2021.  URL https://arxiv.org/abs/2112.01769.
  * Leemans and Tax (2022) Leemans, S. J. J. and Tax, N.  Causal reasoning over control-flow decisions in process models.  In Franch, X., Poels, G., Gailly, F., and Snoeck, M., editors, _Adv. Information Systems Engineering_ , pages 183–200, Cham, 2022. Springer International Publishing.  ISBN 978-3-031-07472-1.
  * Levine et al. (2020) Levine, S., Kumar, A., Tucker, G., and Fu, J.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _ArXiv_ , abs/2005.01643, 2020.
  * Lisowska et al. (2021) Lisowska, A., Wilk, S., and Peleg, M.  From personalized timely notification to healthy habit formation: A feasibility study of reinforcement learning approaches on synthetic data.  In _SMARTERCARE@AI*IA_ , pages 7–18, 2021.
  * Liu and Zou (2018) Liu, R. and Zou, J.  The effects of memory replay in reinforcement learning.  In _2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_ , pages 478–485, 2018.  doi: 10.1109/ALLERTON.2018.8636075.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, A. A., David Rusu, Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518:529–533, 2015.  doi: 10.1038/nature14236.  URL https://doi.org/10.1038/nature14236.
  * Neal et al. (2020) Neal, B., Huang, C.-W., and Raghupathi, S.  Realcause: Realistic causal inference benchmarking, 2020.  URL https://arxiv.org/abs/2011.15007.
  * Olaya et al. (2020a) Olaya, D., Coussement, K., and Verbeke, W.  A survey and benchmarking study of multitreatment uplift modeling.  _Data Mining and Knowledge Discovery_ , 34:273–308, 2020a.  ISSN 1573-756X.  doi: 10.1007/s10618-019-00670-y.  URL https://doi.org/10.1007/s10618-019-00670-y.
  * Olaya et al. (2020b) Olaya, D., Vásquez, J., Maldonado, S., Miranda, J., and Verbeke, W.  Uplift modeling for preventing student dropout in higher education.  _Decision support systems_ , 134:113320, 2020b.
  * Paraschos et al. (2020) Paraschos, P. D., Koulinas, G. K., and Koulouriotis, D. E.  Reinforcement learning for combined production-maintenance and quality control of a manufacturing system with deterioration failures.  _Journal of Manufacturing Systems_ , 56:470–483, 2020.  ISSN 0278-6125.  doi: https://doi.org/10.1016/j.jmsy.2020.07.004.
  * Park et al. (2021) Park, K. T., Son, Y. H., Ko, S., and Noh, S. D.  Digital twin and reinforcement learning-based resilient production control for micro smart factory.  _Applied Sciences_ , 11, 03 2021.  doi: 10.3390/app11072977.
  * Pasquadibisceglie et al. (2022) Pasquadibisceglie, V., Appice, A., Castellano, G., and Malerba, D.  A multi-view deep learning approach for predictive business processes monitoring.  In _2022 IEEE World Congress on Services (SERVICES)_ , pages 26–26, 2022.  doi: 10.1109/SERVICES55459.2022.00039.
  * Pearl (2000) Pearl, J.  _Causality: Models, Reasoning, and Inference_.  Cambridge University Press, Cambridge, 2000.  ISBN 0-521-77362-8.
  * Polyvyanyy et al. (2019) Polyvyanyy, A., Pika, A., Wynn, M. T., and ter Hofstede, A. H.  A systematic approach for discovering causal dependencies between observations and incidents in the health and safety domain.  _Safety Science_ , 118:345–354, 2019.  ISSN 0925-7535.  doi: https://doi.org/10.1016/j.ssci.2019.04.045.  URL https://www.sciencedirect.com/science/article/pii/S0925753518316230.
  * Ponce de León et al. (2018) Ponce de León, H., Nardelli, L., Carmona, J., and vanden Broucke, S. K.  Incorporating negative information to process discovery of complex systems.  _Information Sciences_ , 422:480–496, 2018.  ISSN 0020-0255.  doi: https://doi.org/10.1016/j.ins.2017.09.027.  URL https://www.sciencedirect.com/science/article/pii/S0020025516315444.
  * Potoniec et al. (2022) Potoniec, J., Sroka, D., and Pawlak, T. P.  Continuous discovery of causal nets for non-stationary business processes using the online miner.  _European Journal of Operational Research_ , 303(3):1304–1320, 2022.  ISSN 0377-2217.  doi: https://doi.org/10.1016/j.ejor.2022.03.046.  URL https://www.sciencedirect.com/science/article/pii/S0377221722002685.
  * Qafari and van der Aalst (2020) Qafari, M. S. and van der Aalst, W. M.  Root cause analysis in process mining using structural equation models.  In Del Río Ortega, A., Leopold, H., and Santoro, F. M., editors, _Business Process Management Workshops_ , pages 155–167, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-66498-5.
  * Sallab et al. (2017) Sallab, A. E., Abdou, M., Perot, E., and Yogamani, S.  Deep reinforcement learning framework for autonomous driving.  _Electronic Imaging_ , 29(19):70–76, jan 2017\.  doi: 10.2352/issn.2470-1173.2017.19.avm-023.  URL https://doi.org/10.2352\%2Fissn.2470-1173.2017.19.avm-023.
  * Scheibel and Rinderle-Ma (2022) Scheibel, B. and Rinderle-Ma, S.  Decision mining with time series data based on automatic feature generation.  In Franch, X., Poels, G., Gailly, F., and Snoeck, M., editors, _Advanced Information Systems Engineering_ , pages 3–18, Cham, 2022. Springer International Publishing.  ISBN 978-3-031-07472-1.
  * Sculley et al. (2015) Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D.  Hidden technical debt in machine learning systems.  In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://proceedings.neurips.cc/paper\\_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf.
  * Shalit et al. (2017) Shalit, U., Johansson, F. D., and Sontag, D.  Estimating individual treatment effect: Generalization bounds and algorithms.  In Precup, D. and Teh, Y. W., editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3076–3085. PMLR, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/shalit17a.html.
  * Shoush and Dumas (2022) Shoush, M. and Dumas, M.  When to intervene? prescriptive process monitoring under uncertainty and resource constraints.  In Di Ciccio, C., Dijkman, R., del Río Ortega, A., and Rinderle-Ma, S., editors, _Business Process Management Forum_ , pages 207–223, Cham, 2022. Springer International Publishing.  ISBN 978-3-031-16171-1.
  * Spielberg et al. (2020) Spielberg, S. P., Tulsyan, A., Lawrence, N. P., Loewen, P. D., and Gopaluni, R. B.  Deep reinforcement learning for process control: A primer for beginners.  _ArXiv_ , abs/2004.05490, 2020.
  * Sutton and Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement Learning: An Introduction_.  MIT Press, Cambridge, 2018.  ISBN 9780262039246.
  * Tan et al. (2018) Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., and Vanhoucke, V.  Sim-to-real: Learning agile locomotion for quadruped robots.  _arXiv preprint arXiv:1804.10332_ , 2018.
  * van der Aalst et al. (2021) van der Aalst, W. M., Hinz, O., and Weinhardt, C.  Resilient digital twins.  _Business & Information Systems Engineering_, 63:615–619, 2021.  ISSN 1867-0202.  doi: 10.1007/s12599-021-00721-z.  URL https://doi.org/10.1007/s12599-021-00721-z.
  * Weinzierl et al. (2020) Weinzierl, S., Dunzer, S., Zilker, S., and Matzner, M.  Prescriptive business process monitoring for recommending next best actions.  In Fahland, D., Ghidini, C., Becker, J., and Dumas, M., editors, _Business Process Management Forum_ , pages 193–209, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-58638-6.
  * Weytjens and De Weerdt (2022) Weytjens, H. and De Weerdt, J.  Learning uncertainty with artificial neural networks for predictive process monitoring.  _Applied Soft Computing_ , 125:109134, 2022.  ISSN 1568-4946.  doi: https://doi.org/10.1016/j.asoc.2022.109134.
  * Winston (2022) Winston, W. L.  _Operations research: applications and algorithms_.  Cengage Learning, 2022.
  * Zeng et al. (2023) Zeng, Y., Cai, R., Sun, F., Huang, L., and Hao, Z.  A survey on causal reinforcement learning, 2023.  URL https://arxiv.org/abs/2302.05209.