# Adaptive Contrast Test for Dose-Response Studies and Modeling

Masahiro Kojima  Kyowa Kirin Co., Ltd  The Graduate University for Advanced
Studies

###### Abstract

We propose a powerful adaptive contrast test with ordinal constraint contrast
coefficients determined by observed responses. The adaptive contrast test can
perform using easily calculated contrast coefficients and existing statistical
software. We provide the sample SAS program codes of analysis and calculation
of power for the adaptive contrast test. After the adaptive contrast test
shows the statistically significant dose-response, we consider to select the
best dose-response model from multiple dose-response models. Based on the best
model, we identify a recommended dose. We demonstrate the adaptive contrast
test for sample data. In addition, we show the calculation of coefficient,
test statistic, and recommended dose for the actual study. We perform the
simulation study with eleven scenarios to evaluate the performance of the
adaptive contrast test.
We confirmed the statistically significant dose-response for the sample data
and the actual study. In the simulation study, we confirmed that the adaptive
contrast test has higher power in most scenarios compared to the conventional
method. In addition, we confirmed that the type 1 error rate of the adaptive
contrast test was maintained at a significance level when there was no
difference between the treatment groups.
We conclude that the adaptive contrast test can be applied unproblematically
to the dose-response study.

##  1 Introduction

A primary objective of a dose-response trial is to verify a statistically
significant dose-response relationship. After confirming the dose-response, a
recommended dose is selected based on efficacy, safety, pharmacokinetic,
efficiency of a manufacturing, and so on. In general, various analyses are
proposed to confirm the dose-response relationship. For example, there are to
identify a recommended dose according to the dose-response from the viewpoint
of safety [1, 2, 3, 4, 5, 6]. The analyses method are also proposed to
identify a recommended dose from the perspective of safety and efficacy [7, 8,
9]. In this paper, we consider an analysis method to verify the statistically
significant dose-response to verify a proof-of-concept (PoC) in terms of
efficacy. Various analyses to confirm dose-response have been proposed [10,
11, 12]. In particular, multiple comparison procedures with modeling
techniques (MCP-Mod) have been used in various clinical trials [13, 14, 15,
16, 17, 18, 19, 20, 21]. In an MCP part of the MCP-Mod, contrast coefficients
are given based on multiple dose-response models, and statistically
significant dose-responses are confirmed from contrast tests adjusted for
multiplicity. After the dose-response is confirmed, in a Mod part, a dose-
response model is selected by using Akaike information criterion (AIC) or
Tmax. A recommended dose is selected based on the clinical meaningful
difference from the control group. However, we feel a hassle to calculate the
contrast coefficient based on the dose-response model and the multivariate
t-distribution to adjust for multiplicity. In particular, we can not analyzed
the existing statistical analysis software (SAS) procedure. SAS is basically
used in a new drug applications. In the R software, there is a MCP-Mod
procedure. However, the U.S. Food and Drug Administration (FDA), for example,
requires that the MCP-Mod package follow the guidelines of the General
Principles of Software Validation. We consider it not easy to confirm to the
FDA that the MCP-Mod package is in compliance with the guidelines and that
there is no problem using the MCP-Mod package.

In this paper, we propose the simple adaptive contrast test with ordinal
constraint contrast coefficients determined by observed response. The adaptive
contrast test can perform using easily calculated contrast coefficients and
existing statistical software. We provide the sample SAS program codes of
analysis and calculation of power for the adaptive contrast test. After the
adaptive contrast test shows the statistically significant dose-response, we
consider to select the best dose-response model from multiple dose-response
models. Based on the best model, we identify a recommended dose. We
demonstrate the adaptive contrast test for sample data given by Bretz et
al.[10]. In addition, we show the calculation of coefficient, test statistic,
and recommended dose for the actual study by Akizawa et al. [22]. We perform
the simulation study to evaluate the performance of the adaptive contrast test
compared to MCP-Mod.

This paper is organized as follows. Chapter 2 introduces the adaptive contrast
test and model selection. In addition, the analyses of sample data and actual
study are demonstrated. We have shown the configuration of simulation. Chapter
3 describes the results of simulation. Chapter 4 is discussion. Chapter 5
shows a sample program codes for power calculation and analysis in SAS.

##  2 Methods

We consider the randomized, placebo-controlled, multicenter, parallel-group,
dose-finding study. The number of arms including the placebo group is $k$. The
number of patients treated is $n_{i}$ $(i=1,2,\ldots,k)$. The subscript ”1” of
$n_{1}$ refers to the placebo group. The observed responses are
${\overline{\text{\boldmath$Y$}}}=({\overline{Y}}_{1},\ldots,{\overline{Y}}_{k})^{T}$
such as the sample means or means adjusted by an analysis of covariance or
mixed-effects model for repeated measures. We assume that a larger
${\overline{Y}}$ indicates a trend toward improvement. However, even if the
improvement trend is reversed, an analysis can conduct without any problem. In
Section 2.3, we show an example of improvement as ${\overline{Y}}$ is lower.
The standard deviations are $S_{1},\ldots,S_{k}$. The statistical hypothesis
testing for verifying proof of concept (PoC) is conducted by a contrast test.
The test statistic is
$T=\frac{\sum_{i=1}^{k}c_{i}{\overline{Y}}_{i}}{\sqrt{\left(\sum_{i=1}^{k}\frac{c_{i}^{2}}{n_{i}}\right)S^{2}}}$,
where $\sum_{i=1}^{k}c_{i}=0$ and
$S^{2}=\frac{1}{\sum_{i=1}^{k}n_{i}-k}\sum_{i=1}^{k}\left(n_{i}-1\right)S_{i}^{2}$.
When $T$ exceeds the upper 2.5% point of the distribution followed by the
statistic, a statistically significant dose-response is shown, and the PoC for
an investigational drug is accepted.

###  2.1 Adaptive contrast test

We propose a novel adaptive contrast test. First of all, we give an ordinal
constraint of each element of the contrast coefficients $c$. For example, we
assume that $c$ increases quasi-monotonically in a dose-dependent, the ordinal
constraint of $c$ is $c_{1}\leq c_{2}\leq c_{3}\leq c_{4}\leq\cdots\leq
c_{k}$. The constraint should be defined before the start of study. Under the
constraint, the each of $c$ is calculated based on the observed responses
${\overline{\text{\boldmath$Y$}}}$. $c_{1}$ is given by
$\frac{1}{k}\left((k-1){\overline{Y}}_{1}-\sum^{k}_{i=2}\underset{j\in{1,\ldots,i}}{\operatorname{max}}({\overline{Y}}_{j})\right)$
and
$c_{i}=\underset{j\in{1,\ldots,i}}{\operatorname{max}}({\overline{Y}}_{j})-\underset{j\in{1,\ldots,i-1}}{\operatorname{max}}({\overline{Y}}_{j})+c_{i-1}$,
$i=2,3,\ldots,k$. The reason for taking the maximum value is to satisfy the
ordinal constraint $c_{i}\leq c_{j}$ at a dose $j$ that is larger than dose
$i$. We show examples of observed response ${\overline{\text{\boldmath$Y$}}}$
and contrast coefficient $c$ for four arms in Figure 1. The constraint of $c$
is $c_{1}\leq c_{2}\leq c_{3}$. The $c_{4}$ has no constraint because we are
interested in the $c$ adapting flexibly to an umbrella shape. For the case 1
to case 5, the $c$ shows a similar trend in the observed response. For the
case 6, the observed response of dose $3$ is lower than that of dose 2. Hence,
the contrast coefficient of dose $2$ is the same with the coefficient of dose
$3$. The formulas for each $c$ in the example are
$c_{1}=\frac{1}{4}\left(3{\overline{Y}}_{1}-\underset{j\in{1,2}}{\operatorname{max}}({\overline{Y}}_{j})-\underset{j\in{1,2,3}}{\operatorname{max}}({\overline{Y}}_{j})-{\overline{Y}}_{4}\right)$,
$c_{2}=\underset{j\in{1,2}}{\operatorname{max}}({\overline{Y}}_{j})-{\overline{Y}}_{1}+c_{1}$,
$c_{3}=\underset{j\in{1,2,3}}{\operatorname{max}}({\overline{Y}}_{j})-\underset{j\in{1,2}}{\operatorname{max}}({\overline{Y}}_{j})+c_{2}$,
$c_{4}={\overline{Y}}_{4}-\underset{j\in{1,2,3}}{\operatorname{max}}({\overline{Y}}_{j})+c_{3}$.
As an example of the specific calculation of $c$ using actual response values,
in the case 1, for ${\overline{\text{\boldmath$Y$}}}=(0.2,0.4,0.6,0.8)$, each
of $c$ is $c_{1}=\frac{1}{4}(3\times 0.2-0.4-0.6-0.8)=-0.3$,
$c_{2}=0.4-0.2+(-0.3)=-0.1$, $c_{3}=0.6-0.4+(-0.1)=0.1$, and
$c_{4}=0.8-0.6+0.1=0.3$. In the case 6, under
${\overline{\text{\boldmath$Y$}}}=(0.2,0.4,{\bf 0.2},0.6)$, each of $c$ is
$c_{1}=\frac{1}{4}(3\times 0.2-0.4-{\bf 0.4}-0.6)=-0.2$,
$c_{2}=0.4-0.2+(-0.2)=0$, $c_{3}={\bf 0.4}-0.4+0=0$, and $c_{4}=0.6-{\bf
0.4}+0=0.2$. The test statistic $T$ is calculated using the calculated $c$.
Because the test statistic using $c$ with ordinal constraints does not follow
the t-distribution, we use the permutation method to calculate the p-value.
The permutation method is design-based analysis method which is suitable for
randomized design in dose-response studies. In other words, the randomized
design is not a random sampling design. If all response values are the same or
all the investigational drug groups are lower than the placebo group, the test
statistic is set to zero.

Figure 1: Examples of actual responses ${\overline{\text{\boldmath$Y$}}}$ and
coefficients $c$

####  2.1.1 Modeling

When a statistically significant dose-response is verified, we are interested
in identifying the recommended dose from a dose-response model fitting the
observed response. We consider to use the AIC to select a dose-response model.
Candidate models include Linear, Log-Linear, Emax, Exponential, Quadratic, and
Logistic models. If the best model is selected, the recommended dose should be
selected using minimal effective dose (MED). The MED is a clinically
meaningful difference from placebo. If there is a clinically meaningful change
in response from baseline in the medical guideline, the recommended dose can
select from the doses that are changed meaningful rather than looking at the
difference from placebo.

###  2.2 Analysis of sample dataset

We demonstrate the adaptive contrast test by using the sample dataset given by
Bretz et al.[10]. The sample dataset consists of data from 20 patients per
group in the placebo and four drug groups (dosages: 0.05, 0.20, 0.60, and 1)
in a randomized trial. The responses of each group follow a normal
distribution. The sample means are
${\overline{\text{\boldmath$Y$}}}=(0.345,0.457,0.810,0.934,0.949)^{T}$, the
standard deviations are $S_{1}=0.517$, $S_{2}=0.490$, $S_{3}=0.740$,
$S_{4}=0.765$, and $S_{5}=0.947$. The elements of $c$ are
$c_{1}=\frac{1}{5}(4\times 0.345-0.457-0.810-0.934-0.949)=-0.354$,
$c_{2}=0.457-0.345+c_{1}=-0.242$, $c_{3}=0.810-0.457+c_{2}=0.111$,
$c_{4}=0.934-0.810+c_{3}=0.235$, and $c_{5}=0.949-0.934+c_{4}=0.250$. The test
statistic is $T=3.330$, the one-sided p-value of the permutation method is
$p=0.0003$. We can have confirmed the statistically significant dose-response.

We consider the model selection. We choose the best model from Emax, Linear
log, Linear, Exponential, Quadratic, and Logistic models in terms of
prediction for each dose response by using the AIC. Because the AIC of Emax
model is the smallest, Emax is selected as the dose-response model. We have
summarized the transition for each model in Figure 2.

Figure 2: Dose-response model and observe responses shown as dots

###  2.3 Actual study (Phase 2b study of evocalcet)

We re-analyze the phase 2b study of evocalcet for hemodialysis patients with
secondary hyperparathyroidism using the summary data. The objective of the
study is to confirm the PoC of efficacy for the randomized, double-blind,
placebo-controlled, multicenter, parallel-group, dose-finding design. The
patients were assigned randomly to a placebo, 0.5, 1, 2 mg/day of evocalcet
group for 3 weeks treatment period. The primary endpoint is the percent change
from baseline in intact parathyroid hormone (PTH) at the end of treatment. The
primary analysis is contrast test with seven contrast patterns for a dose-
response. The PoC was shown by a statistically significant decrease in the
percent change in iPTH. The secondary analysis calculated the sample mean and
standard deviation of percent change from baseline in the intact PTH of each
group at end of treatment. The results (Mean$\pm$SD) of percent change from
baseline in the intact PTH were 5.44$\pm$25.85% in placebo, -8.40$\pm$25.43%
in 0.5 mg, -10.56$\pm$22.86% in 1 mg, and -20.16$\pm$34.23% in 2 mg. Because a
lower value for the percent change indicates an clinical improvement, a
constraint on the contrast coefficient is given as $c_{1}\geq c_{2}\geq
c_{3}\geq c_{4}$. By calculating coefficients $c$ based on the formula
replacing the maximum function with a minimum function, $c_{1}=13.86$,
$c_{2}=0.02$, $c_{3}=-2.14$, and $c_{4}=-7.56$. Pooled variance is
$S^{2}=773.17$.
$T=\frac{13.86*5.44-0.02*8.40+2.14*10.56+11.74*20.16}{\sqrt{S^{2}*(13.86^{2}/28+0.02^{2}/30+(-2.14)^{2}/30+(-11.74)^{2}/28)}}=3.54$.
Because we can not have access to the individual data for this study, we show
the upper points for t-distribution. The upper $2.5\%$ point of the
t-distribution is $1.98$, The upper $0.25\%$ point of the t-distribution is
$2.86$. The upper $0.05\%$ point of the t-distribution is $3.38$. We have
confirmed that the result is statistically significant even when the
significance level is sufficiently small. We confirmed the power based on the
sample mean and standard deviation in this study, and the power was 92.04%.
Based on these results, we assume that the permutation method shows
statistically significant. Although this study shows the 90.0% power via
multiple contrasts test, the power for the adaptive contrast test following
the setting of sample size in this study was 91.7%.

We consider the model selection. Emax model is $E_{0}+E_{max}d/({\theta}+d)$.
$E_{0}$ is initial value $5.44$ at placebo. Emax is the minimum value $-20.16$
at 2 mg, and ${\theta}$ is parameter. The estimator ${\hat{\theta}}$ is
$0.40$. The AIC is $22.4$. Linear log-dose model is $E_{0}+{\theta}\log(d+1)$.
The estimator ${\hat{\theta}}$ is $-24.21$. The AIC is $21.3$. Linear model is
$E_{0}+{\theta}d$. The estimator ${\hat{\theta}}$ is -14.12. The AIC is 25.9.
Exponential model is $E_{0}+{\theta}_{1}\exp(d/{\theta}_{2})$. ${\theta}_{1}$
and ${\theta}_{2}$ is parameter. The estimators ${\hat{\theta}}_{1}$ and
${\hat{\theta}}_{2}$ are $1.48$ and $-6.87$, respectively. The AIC is 35.1.
Quadratic model is $E_{0}+{\theta}_{1}d+{\theta}_{2}d^{2}$. The estimator
${\hat{\theta}}_{1}$ is -24.19 and ${\hat{\theta}}_{2}$ is $5.79$. The AIC is
$22.9$. Logistic model is
$E_{0}+E_{max}/(1+\exp(({\theta}_{1}-d)/{\theta}_{2}))$. The estimator
${\hat{\theta}}_{1}$ is 0.66 and ${\hat{\theta}}_{2}$ is $0.36$. The AIC is
$25.8$. The minimum AIC is shown for the Linear log-dose model, we select the
Linear log-dose model as the best dose-response model. We show the all models
in Figure 3. We show examples of recommended dose selection. If a 10% decrease
in the rate of change in iPTH has clinical implications, we can select the
dose 1.0 or more. If a difference of 10% or more from placebo is a clinical
meaningful, we can select the dose 0.5 or more.

Figure 3: Figure 3. Model

###  2.4 Simulation study

We evaluate the statistical power of adaptive contrast test compared to the
MCP-Mod via simulation study. We assume the randomized dose-response study
with five arms and one-sided significance level 2.5%. The dosages are
${\text{\boldmath$d$}}=(d_{1},d_{2},d_{3},d_{4},d_{5})^{T}=(0,0.05,0.2,0.6,1.0)^{T}$.
The number of simulations was set to 10,000. The constraint of contrast
coefficient is $c_{1}\leq c_{2}\leq c_{3}\leq c_{4}$. The coefficient $c_{5}$
for the highest dose has no constraint because we also consider the umbrella
shape. The number of permutations for permutation method is 50,000. The MCP
part of MCP-Mod evaluates the models shown in the Table 1 referred by [10].
The true mean value of each dose is shown in Table 2. The scenario 1 refers to
the constant mean values to confirm the significance level maintaining at
2.5%. For the scenario 2 to the scenario 7, the true mean values are generated
by the dose-response models in the Table 1. For the scenario 8 to the scenario
11, we assume the results with the dose-response relationship that is not
based on a dose-response model. The standard deviation is 1.5.

Table 1: Dose-response model Model name | Equation
---|---
Constant | $0.2$
Linear | $0.2+0.6d_{i}$
Linear in log-dose | $0.2+0.6\log(5d_{i}+1)/\log(6)$
Emax | $0.2+0.7d_{i}/(0.2+d_{i})$
Exponential | $0.183+0.017\exp(2d_{i}\log(6))$
Quadratic | $0.2+2.049d_{i}-1.749d_{i}^{2}$
Logistic | $0.193+0.607/(1+\exp(10\log(3)(0.4-d_{i})))$
Table 2: True mean value | true mean values
---|---
Scenario 1 (Constant) | $(0.2,0.2,0.2,0.2,0.2)$
Scenario 2 (Linear) | $(0.2,0.23,0.32,0.56,0.8)$
Scenario 3 (Linear in log-dose) | $(0.2,0.275,0.432,0.664,0.8)$
Scenario 4 (Emax) | $(0.2,0.34,0.55,0.725,0.783)$
Scenario 5 (Exponential) | $(0.2,0.201,0.206,0.226,0.264)$
Scenario 6 (Quadratic) | $(0.2,0.298,0.54,0.8,0.5)$
Scenario 7 (Logistic) | $(0.271,0.289,0.362,0.631,0.767)$
Scenario 8 | $(0.2,0.4,0.6,0.6,0.8)$
Scenario 9 | $(0.2,0.4,0.6,0.6,0.6)$
Scenario 10 | $(0.2,0.6,0.6,0.6,0.6)$
Scenario 11 | $(0.2,0.6,0.6,0.8,0.8)$

##  3 Results

The results of the simulation study are shown in Table 4 and Figure 4. The
scenario 1 confirmed that the significance level of 2.5% was maintained for
all adaptive contrast test and MCP-Mod. The power increased with increasing
sample size for the adaptive contrast test and MCP-Mod except in Scenarios 1
and 5. For the adaptive contrast test (N=100), the power was higher than the
MCP-Mod in scenarios 3, 4, 6, 8, 9, 10, and 11, while MCP had higher power in
scenarios 2, 5, and 7. MCP had lower power in scenarios 9, 10, and 11, which
were not generated from the model, compared to scenarios 2 to 7, which were
generated from the model. Supplementally, we show the results of quasi-
monotonically increasing for the ordinal constraint of all contrast
coefficients (not apply Umbrella shape) in Supplementary Analysis 6.

Table 3: Power of each scenario in simulation study | Adaptive contrast test | MCP-Mod
---|---|---
| $N=50$ | $N=75$ | $N=100$ | $N=50$ | $N=75$ | $N=100$
Scenario 1 | 2.40 | 2.45 | 2.52 | 2.39 | 2.46 | 2.66
Scenario 2 | 51.82 | 75.71 | 85.63 | 56.81 | 75.65 | 86.86
Scenario 3 | 52.26 | 79.32 | 88.01 | 55.27 | 75.04 | 86.81
Scenario 4 | 49.92 | 76.84 | 87.61 | 49.97 | 69.79 | 82.69
Scenario 5 | 2.20 | 1.15 | 1.06 | 3.45 | 3.80 | 3.85
Scenario 6 | 50.01 | 70.85 | 76.57 | 26.37 | 39.42 | 51.97
Scenario 7 | 40.57 | 62.43 | 67.57 | 43.81 | 61.97 | 75.41
Scenario 8 | 36.59 | 68.59 | 78.57 | 40.12 | 57.98 | 71.42
Scenario 9 | 21.80 | 42.50 | 51.79 | 18.27 | 27.92 | 38.06
Scenario 10 | 23.53 | 42.27 | 49.80 | 10.61 | 15.30 | 20.84
Scenario 11 | 50.17 | 75.31 | 85.46 | 37.82 | 54.42 | 69.23
Figure 4: Power of each scenario in simulation study

##  4 Discussion

We proposed the adaptive contrast test and model selection. The contrast
coefficients are given the ordinal constraint before the study starts and
adaptively determined in a data-dependent. We have confirmed that the adaptive
contrast test has higher power because of the contrast coefficients determined
adaptively. On the practical side, the contrast coefficients are easy to
calculate. The statistical test is performed by permutation method, which can
be easily computed using, for example, the multtest procedure in SAS, permute
in STATA, or perm package in R. In addition, we provide the sample SAS program
codes of analysis and calculation of power for the adaptive contrast test in
Chapter 5. We proposed a procedure to choose a dose-response model from
candidate models and select a recommended dose after a statistically
significant dose-response has been confirmed. We also provide the sample
program codes using the existing SAS procedure for model selection in Chapter
5.

We demonstrated the adaptive contrast test for the sample study given by Bretz
et al.[10]. We have confirmed the statistically significant dose-response. In
addition, we selected the dose-response model via the AIC. In addition, We re-
analyzed the actual phase 2b study by Akizawa et al.[22]. We showed the
determined contrast coefficients and the contrast test statistic. The test
statistic implied that the results were statistically significant, and we
selected the best dose-response model. We presented recommended doses with
clinically meaningful efficacy based on the dose-response models. The power of
the adaptive contrast test showed higher than the permutation test for
multiple contrasts used in the actual phase 2b study.

We performed the simulation study to evaluate the power of adaptive contrast
test compared to the MCP-Mod. In many scenarios, the adaptive contrast test
has higher power than the MCP-Mod. We consider that the power was high by
identifying the optimal contrast coefficients in a data-dependent. We
confirmed that the one-sided type 1 error rate of the adaptive contrast test
was maintained at 2.5% when there was no difference between the treatment
groups. Hence, there was no problem with the performance. The power of MCP-Mod
was relatively high when the true mean of each group was based on the dose-
response model. However, the power of MCP-Mod decreased when the true mean was
generated not-dose-response model. In reality, because the true mean values do
not transition based on the dose-response model the MCP-Mod may not be able to
maintain the expected power. For the ordinal constraint of the contrast
coefficient, when the quasi-monotonic increase assumption was made for all
coefficients without assuming an umbrella type, the power increased for the
all scenarios except for the scenario generated from an umbrella type. We
recommend the assumption of quasi-monotonic increase for all coefficients when
no umbrella type is assumed in the efficacy data.

The adaptive contrast test is a powerful test that can perform using easily
calculated contrast coefficients and existing statistical software. We
confirmed that the adaptive contrast test is the higher power than not only
the permutation test with multiple contrast patterns but also the MCP-Mod.
When we plan to use the permutation test with multiple contrast patterns and
MCP-Mod, we need to explain the procedure of those methods, assumption of
dose-response models and adjustment of multiplicity to clinicians and
decision-makers. We proposed the analysis method that can avoid the
multiplicity and be easy-to-understand of analysis procedure. The adaptive
contrast test can be easy to execute by the simple analysis program. We
provide the sample SAS codes, SAS is basically used in new drug applications.
Therefore, we hope that the adaptive contrast test will be used in many dose-
response studies.

##  5 Software

Listing 1: Sample program code of analysis of biom dataset

⬇

proc import out=dat

datafile=”\biom.xlsx”

/* Add path. biom.xlsx converted from data(biom) of R MCPMoD package*/

dbms=Excel replace;

getnames=no;

run;

/*Variable name tn is arm name (numeric) and res is response (numeric)*/

data dat;

length t $200.;

set dat;

if tn=0 then t=”1_0”;

else if tn=0.05 then t=”2_0.05”;

else if tn=0.2 then t=”3_0.2”;

else if tn=0.6 then t=”4_0.6”;

else if tn=1 then t=”5_1”;

run;

proc univariate data=dat noprint;

where tn=0;

var res;

output out=out1 mean=mean1 std=std1;

run;

proc univariate data=dat noprint;

where tn=0.05;

var res;

output out=out2 mean=mean2 std=std2;

run;

proc univariate data=dat noprint;

where tn=0.2;

var res;

output out=out3 mean=mean3 std=std3;

run;

proc univariate data=dat noprint;

where tn=0.6;

var res;

output out=out4 mean=mean4 std=std4;

run;

proc univariate data=dat noprint;

where tn=1;

var res;

output out=out5 mean=mean5 std=std5;

run;

data out;

merge out1-out5;

run;

%macro _do;

data out;

set out;

mean1=round(mean1,1E-5);

mean2=round(mean2,1E-5);

mean3=round(mean3,1E-5);

mean4=round(mean4,1E-5);

mean5=round(mean5,1E-5);

max2=max(of mean1-mean2);

max3=max(of mean1-mean3);

max4=max(of mean1-mean4);

max5=max(of mean1-mean5);

c1=round(-(max2+max3+max4+max5-4*mean1)/5,1E-5);

c2=round((max2-mean1)+c1,1E-5);

c3=round((max3-max2)+c2,1E-5);

c4=round((max4-max3)+c3,1E-5);

c5=round((max5-max4)+c4,1E-5);

call symput(”cc1”, c1);

call symput(”cc2”, c2);

call symput(”cc3”, c3);

call symput(”cc4”, c4);

call symput(”cc5”, c5);

if c1=0 and c2=0 and c3=0 and c4=0 and c5=0 then do; %let _FL=Y; end;

else do;

%let _FL=N;

end;

run;

%if &_FL.=Y %then %do;

data pValues_1;

set pValues_1;

Permutation=1;

run;

%end;

%else %do;

ods output pValues = pValues_1;

proc multtest data=dat permutation nsample=10000 seed=2021;

class t;

test mean (res / ddfm=pooled upper);

contrast ’Adaptive Contrast’ &cc1. &cc2. &cc3. &cc4. &cc5.;

run;

ods listing;

%end;

data pValues_1;

set pValues_1;

c1=&cc1.;

c2=&cc2.;

c3=&cc3.;

c4=&cc4.;

c5=&cc5.;

run;

%mend;

%_do;

/*Dataset pValues_1 shows p-value.*/

/*AIC is derived below codes*/

data dr;

input d res E0 Emax;

datalines;

0 0.34491 0.34491 0.94871

0.05 0.45675 0.34491 0.94871

0.2 0.81032 0.34491 0.94871

0.6 0.93444 0.34491 0.94871

1 0.94871 0.34491 0.94871

;

run;

title ”Emax model” ;

ods output FitStatistics=AIC_Emax ;

proc nlmixed data = dr;

parms ED50 = 1 SD=1;

mu = E0 +Emax*d / (ED50+d);

model res ~ normal(mu, SD**2);

run;

ods output close;

title ”Linear log-dose model” ;

ods output FitStatistics=AIC_Lld ;

proc nlmixed data = dr;

parms de = 1 SD=1;

mu = E0 +de*log(d+1);

model res ~ normal(mu, SD**2);

run;

ods output close;

title ”Linear model” ;

ods output FitStatistics=AIC_L ;

proc nlmixed data = dr;

parms de = 1 SD=1;

mu = E0 +de*d;

model res ~ normal(mu, SD**2);

run;

ods output close;

title ”Exponential model” ;

ods output FitStatistics=AIC_Exp ;

proc nlmixed data = dr;

parms sl=1 de = 1 SD=1;

mu = E0 +sl*exp(d/de);

model res ~ normal(mu, SD**2);

run;

ods output close;

title ”Quadratic model” ;

ods output FitStatistics=AIC_Q ;

proc nlmixed data = dr;

parms be1 = 1 be2=1 SD=1;

mu = E0 +be1*d+be2*d**2;

model res ~ normal(mu, SD**2);

run;

ods output close;

title ”Logistic model” ;

ods output FitStatistics=AIC_Log ;

proc nlmixed data = dr;

parms ED50=1 de=1 SD=1;

mu = E0 +Emax/(1+exp((ED50-d)/de));

model res ~ normal(mu, SD**2);

run;

ods output close;

Listing 2: Sample program code for calculation of power for the adaptive
contrast test

⬇

data res1; set _NULL_; run;

%macro _func(_num,_m1,_m2,_m3,_m4,_m5,_sd);

data test;

CALL STREAMINIT(100);

do i=1 to 10000;

do j=1 to &_num.;

x1=rand(’NORMAL’,&_m1.,&_sd.);

x2=rand(’NORMAL’,&_m2.,&_sd.);

x3=rand(’NORMAL’,&_m3.,&_sd.);

x4=rand(’NORMAL’,&_m4.,&_sd.);

x5=rand(’NORMAL’,&_m5.,&_sd.);

output;

end;

end;

run;

proc univariate data=test noprint;

by i;

var x1;

output out=out1 mean=mean1 std=std1;

run;

proc univariate data=test noprint;

by i;

var x2;

output out=out2 mean=mean2 std=std2;

run;

proc univariate data=test noprint;

by i;

var x3;

output out=out3 mean=mean3 std=std3;

run;

proc univariate data=test noprint;

by i;

var x4;

output out=out4 mean=mean4 std=std4;

run;

proc univariate data=test noprint;

by i;

var x5;

output out=out5 mean=mean5 std=std5;

run;

data out;

merge out1-out5;

by i;

run;

data out;

set out;

mean1=round(mean1,1E-5);

mean2=round(mean2,1E-5);

mean3=round(mean3,1E-5);

mean4=round(mean4,1E-5);

mean5=round(mean5,1E-5);

max2=max(of mean1-mean2);

max3=max(of mean1-mean3);

max4=max(of mean1-mean4);

max5=max(of mean1-mean5);

c1=round(-(max2+max3+max4+mean5-4*mean1)/5,1E-5);/

*Non-Umblella, round(-(max2+max3+max4+max5-4*mean1)/5,1E-5);*/

c2=round((max2-mean1)+c1,1E-5);

c3=round((max3-max2)+c2,1E-5);

c4=round((max4-max3)+c3,1E-5);

c5=round((mean5-max4)+c4,1E-5);/*Non-Umblella, round((max5-max4)+c4,1E-5);*/

call symput(”cc1”, c1);

call symput(”cc2”, c2);

call symput(”cc3”, c3);

call symput(”cc4”, c4);

call symput(”cc5”, c5);

S=(std1**2+std2**2+std3**2+std4**2)*(&_num.-1)/(&_num.*4-4);

if c1=0 and c2=0 and c3=0 and c4=0 and c5=0 then do; %let _FL=Y; end;

else do;

%let _FL=N;

end;

run;

*proc freq data=out noprint;

* table FL/out=res;

*run;

data test1(keep=val trtpn i);

set test;

rename x1=val;

TRTPN=1;

run;

data test2(keep=val trtpn i);

set test;

rename x2=val;

TRTPN=2;

run;

data test3(keep=val trtpn i);

set test;

rename x3=val;

TRTPN=3;

run;

data test4(keep=val trtpn i);

set test;

rename x4=val;

TRTPN=4;

run;

data test5(keep=val trtpn i);

set test;

rename x5=val;

TRTPN=5;

run;

data testt;

set test1-test5;

run;

%if &_FL.=Y %then %do;

data pValues_1;

set pValues_1;

Permutation=1;

run;

%end;

%else %do;

ods output pValues = pValues_1;

proc sort data=testt;by i ;run;

proc multtest data=testt permutation nsample=50000 seed=2021;

by i;

class TRTPN;

test mean (val / ddfm=pooled upper);

contrast ’Adaptive Contrast’ &cc1. &cc2. &cc3. &cc4. &cc5.;

run;

ods listing;

%end;

data pValues_1;

set pValues_1;

if Permutation<0.025 then FL=”Y”;

else if Permutation>=0.025 then FL=”N”;

run;

proc freq data=pValues_1 noprint;

table FL/out=res2;

run;

data res2;

set res2;

SS=&_num.;

m1=&_m1.;

m2=&_m2.;

m3=&_m3.;

m4=&_m4.;

m5=&_m5.;

sd=&_sd.;

run;

data res1;

set res1 res2;

run;

%mend _func;

%macro _loop(_n);

%_func(&_n.,0.2,0.2,0.2,0.2,0.2,1.5);

%_func(&_n.,0.2,0.23,0.32,0.56,0.8,1.5);

%_func(&_n.,0.2,0.275,0.432,0.664,0.8,1.5);

%_func(&_n.,0.2,0.34,0.55,0.725,0.783,1.5);

%_func(&_n.,0.2,0.201,0.206,0.226,0.264,1.5);

%_func(&_n.,0.2,0.298,0.54,0.8,0.5,1.5);

%_func(&_n.,0.271,0.289,0.362,0.631,0.767,1.5);

%_func(&_n.,0.2,0.4,0.6,0.6,0.8,1.5);

%_func(&_n.,0.2,0.4,0.6,0.6,0.6,1.5);

%_func(&_n.,0.2,0.6,0.6,0.6,0.6,1.5);

%_func(&_n.,0.2,0.6,0.6,0.8,0.8,1.5);

%mend;

%_loop(50);

%_loop(75);

%_loop(100);

proc export data = res1

outfile = ”output.xlsx” /*Add path*/

dbms = xlsx replace;

run;

##  6 Supplementary Result

We show supplemental result with the ordinal constraint with assuming an
umbrella type (Adaptive contrast test 1) and without assuming an umbrella type
(Adaptive contrast test 2). The adaptive contrast test 1 assumes $c_{1}\leq
c_{2}\leq c_{3}\leq c_{4}$. The adaptive contrast test 2 assumes $c_{1}\leq
c_{2}\leq c_{3}\leq c_{4}\leq c_{5}$. The adaptive contrast test 2 was higher
power than the adaptive contrast test 1 except for the scenario 6 (Umbrella
shape).

Table 4: Power of each scenario in supplemental simulation study | Adaptive contrast test 1 | Adaptive contrast test 2
---|---|---
| $N=50$ | $N=75$ | $N=100$ | $N=50$ | $N=75$ | $N=100$
Scenario 1 | 2.40 | 2.45 | 2.52 | 2.47 | 2.45 | 2.26
Scenario 2 | 51.82 | 75.71 | 85.63 | 61.32 | 83.42 | 88.47
Scenario 3 | 52.26 | 79.32 | 88.01 | 62.53 | 83.31 | 88.06
Scenario 4 | 49.92 | 76.84 | 87.61 | 62.94 | 80.2 | 85.27
Scenario 5 | 2.20 | 1.15 | 1.06 | 3.06 | 4.59 | 3.36
Scenario 6 | 50.01 | 70.85 | 76.57 | 48.37 | 64.76 | 72.12
Scenario 7 | 40.57 | 62.43 | 67.57 | 49.86 | 72.44 | 77.18
Scenario 8 | 36.59 | 68.59 | 78.57 | 51.73 | 72.62 | 80.3
Scenario 9 | 21.80 | 42.50 | 51.79 | 38.07 | 50.43 | 54.88
Scenario 10 | 23.53 | 42.27 | 49.80 | 39.27 | 50.64 | 66.03
Scenario 11 | 50.17 | 75.31 | 85.46 | 63.28 | 79.6 | 86.49

Acknowledgments. The author would like to thank Associate Professor Hisashi
Noma for his encouragement and helpful suggestions.

## References

  * [1] Liu S., Yuan Y.. Bayesian optimal interval designs for phase I clinical trials. JRSS C. 2015;64(Part 3):507-523.
  * [2] Yan F., Mandrekar S.J., Yuan Y.. Keyboard: A Novel Bayesian Toxicity Probability Interval Design for Phase I Clinical Trials. Clin Cancer Res. 2017;23(15):3994-4003.
  * [3] Lee S., Ursino M., Cheung Y.K., Zohar S.. Dose-finding designs for cumulative toxicities using multiple constraints. Biostatistics. 2019;20:17-29.
  * [4] Lin R., Yuan Y.. Time-to-event model-assisted designs for dose-finding trials with delayed toxicity. Biostatistics. 2020;21(4):807-824.
  * [5] Kojima M.. Early completion of phase I cancer clinical trials with Bayesian optimal interval design. Statistics in Medicine. 2021;40:3215-3226.
  * [6] Kojima M.. Early Completion of Model-Assisted Designs for Dose-Finding Trials. JCO PO. in press;.
  * [7] Mozgunov P., Jaki T.. A benchmark for dose finding studies with continuous outcomes. Biostatistics. 2020;21(2):189-201.
  * [8] Mozgunov P., Paoletti X., Jaki T.. A benchmark for dose-finding studies with unknown ordering. Biostatistics. 2020;.
  * [9] Lin R., Zhou Y., Yan F., Li D., Yuan Y.. BOIN12: Bayesian Optimal Interval Phase I/II Trial Design for Utility-Based Dose Finding in Immunotherapy and Targeted Therapies. JCO PO. 2020;:1393-1402.
  * [10] Bretz F., Pinheiro J.C., Branson M.. Combining Multiple Comparisons and Modeling Techniques in Dose-Response Studies. Biometrics. 2005;61:738-748.
  * [11] Pinheiro J., Bornkamp B., Glimm E., Bretz F.. Model-based dose finding under model uncertainty using general parametric models. Statistics in Medicine. 2014;33:1646-1661.
  * [12] Ma S., McDermott M.P.. Generalized multiple contrast tests in dose-response studies. Statistics in Medicine. 2020;39:757-772.
  * [13] Bornkamp B., Bretz F., Pinheiro J.. Efficient statistical methodology for model-based design and analysis of Phase II dose finding studies under model uncertainty. : Novartis Pharmaceuticals; 2013.
  * [14] ClinicalTrials.gov . Targeted Dose Finding of Canakinumab (ACZ885) for Management of Acute Flare in Refractory or Contraindicated Gout Patients.  2012\.
  * [15] ClinicalTrials.gov . Cardiovascular Risk Reduction Study (Reduction in Recurrent Major CV Disease Events) (CANTOS).  2022\.
  * [16] ClinicalTrials.gov . Safety, Tolerability, Efficacy and Optimal Dose Finding Study of BAF312 in Patients With Relapsing-remitting Multiple Sclerosis.  2020\.
  * [17] ClinicalTrials.gov . Double-blind, Randomized Study Evaluating the Efficacy and Safety of Brivaracetam in Adults With Partial Onset Seizures.  2021\.
  * [18] ClinicalTrials.gov . A Randomized, Double-blind, Placebo Controlled Study to Assess Efficacy, Safety and Tolerability of LCQ908 in Subjects With Familial Chylomicronemia Syndrome.  2015\.
  * [19] ClinicalTrials.gov . A Pilot Study to Assess the Efficacy and Safety of LCQ908 Alone and in Combination With Fenofibrate or Lovaza® in Patients With Severe Hypertriglyceridemia.  2020\.
  * [20] ClinicalTrials.gov . Dose-finding Study of LIK066 Compared With Placebo or Sitagliptin to Evaluate Change in HbA1c in Patients With Diabetes.  2014\.
  * [21] ClinicalTrials.gov . Dose Finding Study for QAW039 in Asthma.  2020\.
  * [22] Akizawa T., Shimazaki R., Fukagawa M.. Phase 2b study of evocalcet (KHK7580), a novel calcimimetic, in Japanese patients with secondary hyperparathyroidism undergoing hemodialysis: A randomized, double-blind, placebo-controlled, dose-finding study. Pros One. 2018;.
  * [23] Menon S., Zink R.C.. Modern Approaches to Clinical Trials Using SAS: Classical, Adaptive, and Bayesian Methods.  SAS Institute; 2015.
  * [24] Lin R., Yin G., Shi H.. Bayesian adaptive model selection design for optimal biological dose finding in phase I/II clinical trials. Biostatistics. 2021;.
  * [25] Kennes L.N., Volkers G., Kralidis G.. Study design aspects and inter‐subject variability in longitudinal clinical phase II dose‐finding trials. Pharmaceutical Statistics. 2019;18:248-259.


# RACCER: Towards Reachable and Certain Counterfactual Explanations for
Reinforcement Learning

Jasmina Gajcin<EMAIL_ADDRESS>0000-0002-8731-1236 Trinity College
DublinCollege GreenDublinIreland and  Ivana Dusparic  0000-0003-0621-5400
<EMAIL_ADDRESS>Trinity College DublinCollege GreenDublinIreland

###### Abstract.

While reinforcement learning (RL) algorithms have been successfully applied to
numerous tasks, their reliance on neural networks makes their behavior
difficult to understand and trust. Counterfactual explanations are human-
friendly explanations that offer users actionable advice on how to alter the
model inputs to achieve the desired output from a black-box system. However,
current approaches to generating counterfactuals in RL ignore the stochastic
and sequential nature of RL tasks and can produce counterfactuals which are
difficult to obtain or do not deliver the desired outcome. In this work, we
propose RACCER, the first RL-specific approach to generating counterfactual
explanations for the behaviour of RL agents. We first propose and implement a
set of RL-specific counterfactual properties that ensure easily reachable
counterfactuals with highly-probable desired outcomes. We use a heuristic tree
search of agent’s execution trajectories to find the most suitable
counterfactuals based on the defined properties. We evaluate RACCER in two
tasks as well as conduct a user study to show that RL-specific counterfactuals
help users better understand agent’s behavior compared to the current state-
of-the-art approaches.

Reinforcement Learning, Explainability, Counterfactual Explanations

††ccs: Computing methodologies Reinforcement learning††ccs: Human-centered
computing User studies

##  1\. Introduction

Reinforcement learning (RL) algorithms have shown remarkable success in many
fields in the recent years and are being developed for high-risk areas such as
healthcare and autonomous driving (Arulkumaran et al., 2017). However, RL
algorithms often use neural networks to represent their policies, which makes
them difficult to understand and hinders their applicability to real-life
tasks. Explainable RL (XRL) is a growing research field that addresses the
need for improving the understanding of black-box RL models. XRL compiles
research in developing methods for explaining RL models both locally, focusing
on a decision in a specific state, and globally, explaining the behavior of a
model as a whole (Puiutta and Veith, 2020). For example, saliency maps are a
local explanation method that is used to identify parts of the image that most
contributed to a decision in a specific state (Huber et al., 2021; Rosynski et
al., 2020). On a global level, RL models have been distilled into
interpretable formats, such as decision trees (Coppens et al., 2019). However,
most methods in XRL generate explanations targeted at developers and expert
users. Such explanations often deal in low-level, domain-specific terms which
are not easy to comprehend for non-expert users. Non-expert users require more
abstract, high-level explanations, that help them better understand and
interact with the system.

Counterfactual explanations are local user-friendly explanations for
interpreting decisions of black-box algorithms. In machine learning,
counterfactuals are defined as an answer to the question: “Given that the
black-box model M outputs $A$ for input features $f_{1},...,f_{k}$, how can
the features change to elicit output B from M?” (Verma et al., 2021).
Counterfactual explanations can help users by giving them actionable advice on
how to change their input to obtain a desired output. For example, a user
rejected for a loan by an AI system, might not only be interested to know why
the decision was made, but also how they can improve their application, in
order to be approved in the future. Additionally, counterfactuals are inherent
to human reasoning, as we rely on them to assign blame and understand events
(Byrne, 2019).

In the recent years, numerous methods for generating counterfactual
explanations for supervised learning tasks have been proposed (Wachter et al.,
2017; Dandl et al., 2020; Poyiadzi et al., 2020; Looveren and Klaise, 2021;
Mothilal et al., 2020; Samoilescu et al., 2021; Laugel et al., 2017). The
majority of these methods rely on optimizing counterfactual properties such as
proximity, sparsity and data manifold closeness, which leads to the a
realistic and easily obtainable counterfactual instance. In RL, to the best of
our knowledge, there currently exists only one approach for generating
counterfactual explanations. Olson et al. (2019) use generative deep learning
to create counterfactual explanations and rely on the same feature-based
properties proximity, sparsity and data manifold closeness used in supervised
learning to guide the generating process. However, relying on these
traditional counterfactual properties in RL tasks can result in
counterfactuals that are similar in features to the original instance but
difficult to reach, due to the sequential nature of RL tasks or do not deliver
the desired output with certainty, due to stochasticity in the RL environment
(Gajcin and Dusparic, 2022). Previous work recognizes that counterfactual
explanations can suggest user potentially life-changing actions and as such
carry great responsibility (Gajcin and Dusparic, 2022). Offering users
counterfactuals which are not easy to reach or do not deliver on the promised
outcome can cost users substantial time, and cause them to lose trust in the
AI system.

In this work, we propose RACCER (Reachable And Certain Counterfactual
Explanations for Reinforcement Learning), to the best of our knowledge the
first approach for generating counterfactual explanations for RL tasks which
takes into account the sequential and stochastic nature of the RL framework.
Firstly, we propose three novel RL-specific counterfactual properties –
reachability, stochastic certainty and cost-efficiency, that should be
considered instead of the commonly used proximity, sparsity and data manifold
closeness properties when searching for easily obtainable counterfactuals.
These counterfactual properties rely on the stochastic and sequential nature
of RL tasks and ensure that counterfactuals are easy to reach and deliver the
desired outcome with high probability. RACCER searches for the most suitable
counterfactual by optimizing a loss function consisting of the three RL-
specific properties using heuristic tree search of agent’s execution tree. We
evaluate RACCER in two environments of varying complexity – Stochastic
GridWorld task and chess, and show that our approach produces counterfactuals
that can be reached faster and deliver the desired output more often compared
to the baseline methods relying on the traditional counterfactual properties.
Additionally, we conduct a user study in which we compare the effect of
counterfactual explanations on user understanding of RL agents and show that
RACCER generates counterfactuals that help humans better understand and
predict the behavior of RL agents.

Our contributions are as follows:

  1. (1)

We design three RL-specific counterfactual properties – reachability,
stochastic certainty and cost-efficiency, and provide metrics for their
estimation.

  2. (2)

We propose RACCER, the first algorithm for generating RL-specific
counterfactual explanations, which relies on the above counterfactual
properties.

  3. (3)

We conduct a user study and show that RACCER can produce counterfactuals which
help humans better understand agent’s behavior compared to the baseline
approaches.

The implementation of RACCER and evaluation details can be found at
https://github.com/anonymous902109/RACCER.

##  2\. Related Work

In this section, we offer a short overview of counterfactual explanations and
counterfactual properties, and summarize some of the most notable methods for
generating counterfactuals in supervised and RL.

###  2.1. Counterfactual Explanations

In the first work on counterfactual explanations for black-box models, Wachter
et al. (2017) define them as follows: “Score p was returned because variables
V had values (v1, v2 , . . .) associated with them. If V instead had values
(v1’, v2’, . . .), and all other variables had remained constant, score p’
would have been returned”.  Counterfactual explanations for an instance $x$
are offered in the form of a counterfactual instance $x^{\prime}$ that is
similar to $x$ but achieves the desired outcome. They offer users actionable
advice on how they can change their features to achieve a desired outcome, and
help users better interact with the system. Additionally, they are selective,
suggesting users to change only few features. Counterfactual explanations are
also inherent to human reasoning, as we use them to assign blame (Byrne,
2019). All of this makes counterfactuals user-friendly explanations (Molnar,
2019).

For counterfactual explanations to be useful to users, they need to produce
the desired output, and be easy to obtain, in order to minimize user effort.
To that end, multiple counterfactual properties have been proposed to evaluate
the quality of different counterfactuals (Verma et al., 2021). For example,
validity is used to measure whether counterfactual achieves the desired
output, proximity is a feature-based similarity measure which ensures
counterfactual features are similar to those in the original instance, and
sparsity measures the number of features changed. By optimizing these
counterfactual properties current state-of-the-art approaches search for
counterfactuals that can be easily reached from the original instance with
minimal user effort, are realistic and produce the desired outcome.
Counterfactuals can suggest life-altering actions to the user, and as such
carry a great responsibility. Offering users counterfactual explanations that
require large amounts of effort or do not deliver the desired outcome can
decrease user trust and hinder their interaction with the system.

###  2.2. Generating Counterfactual Explanations

In supervised learning, counterfactual explanations have been used to propose
changes of input features that elicit a desired prediction from a black-box
model. In the recent years, numerous works have proposed methods for
generating counterfactual explanations in supervised learning (Wachter et al.,
2017; Dandl et al., 2020; Looveren and Klaise, 2021; Laugel et al., 2017;
Mothilal et al., 2020; Poyiadzi et al., 2020; Samoilescu et al., 2021). The
majority of these methods follow the same approach. Firstly, a loss function
is defined by combining different counterfactual properties, such as validity,
proximity and sparsity. The loss function is then optimized over a training
data set in order to find the most suitable counterfactual. The methods differ
in their design of the loss function and the choice of the optimization
method. For example, in the first work on counterfactual explanations for
supervised learning, Wachter et al. (2017) use gradient descent to optimize a
loss function based on proximity and validity properties. Similarly, Mothilal
et al. (2020) propose DICE, which introduces a diversity property to the
approach of Wachter et al. (2017) to ensure users are offered a set of
diverse, high-quality explanations. Dandl et al. (2020) pose the problem of
counterfactual search as multi-objective optimization and use genetic
algorithm to optimize validity, proximity, sparsity and data manifold
closeness of counterfactual instances.

In RL, counterfactual explanations aim to explain a decision of a black-box RL
model in a specific state by proposing an alternative state in which the model
would choose the desired action. Olson et al. (2019) propose the only method
for generating counterfactuals in RL so far. The approach relies on generative
modelling to create counterfactuals which are realistic, similar in features
to the original instance and produce a desired output. The approach is not
model-agnostic and requires access to the internal parameters of the black-box
model that is being explained. While the approach proposed by Olson et al.
(2019) generates realistic counterfactuals that can help users better
understand agent’s decisions and even detect faulty behavior in Atari agents,
they focus on the same feature-based counterfactual properties such as
proximity and sparsity as supervised learning methods. However, in RL where
two states can be similar in features but distant in terms of execution,
feature-based metrics are not sufficient for measuring how obtainable a
counterfactual is. Relying only on feature-based similarity measures can
produce counterfactuals which are not easily (or at all) obtainable, and
decrease human trust in the system. In contrast, our work proposes the first
approach for generating RL-specific counterfactuals that take into account the
stochastic and sequential nature of RL tasks.

While the goal of counterfactual explanations is to deliver the desired
outcome, this is often uncertain due the environment in which the system
operates. For example, even if the loan applicant fulfills all conditions
stipulated in a counterfactual, the bank might change the conditions for
approving a loan. Delaney et al. (2021) recognized the need for estimating and
presenting the uncertainty associated with counterfactuals to the user in
supervised learning tasks. On the other hand, in this work we use estimate
uncertainty from an RL perspective, and use it not only as additional
information for the user, but as an important factor during search and choice
of the counterfactual explanation.

##  3\. RACCER

In this section, we describe RACCER, the approach for generating
counterfactual explanations for RL tasks. To generate a counterfactual
explanation $x^{\prime}$, we require oracle access to the black-box model $M$
being explained, the state $x$ being explained, and the desired outcome
$a^{\prime}$. Additionally, the approach needs access to the RL environment.
RACCER then generates a counterfactual state $x^{\prime}$ that can be easily
reached from $x$ and in which the black-box model $M$ chooses $a^{\prime}$
with a high probability. We propose a fully model-agnostic method, which does
not require information on model parameters and can be used for generating
counterfactual explanations of any RL model.

The are two main directions in which the search for counterfactual
explanations can be conducted. Namely, we can search either directly for a
counterfactual instance $x^{\prime}$ (Wachter et al., 2017; Dandl et al.,
2020; Mothilal et al., 2020; Poyiadzi et al., 2020) or for a sequence of
actions $A$ that can transform the original instance into a counterfactual
(Karimi et al., 2020b; Ustun et al., 2019). The second approach corresponds to
the field of actionable recourse which has often been investigated alongside
counterfactual explanations (Karimi et al., 2020a). Once the sequence of
actions is found, counterfactual can be obtained by performing the actions on
the original state. To estimate how far away in terms of execution two RL
states are, we need access to actions used to tranform one into the other. For
that reason, in this work we utilize the indirect approach to counterfactual
generation, where we search for a sequence of actions to transform the
original to the counterfactual instance. By following the sequence of actions
from the original instance $x$, a counterfactual $x^{\prime}$ can be obtained
and presented to the user. This way of conducting counterfactual search is
more informative for the user, as they can be presented with not just the
counterfactual instance, but also the sequence of actions they need to perform
to obtain their desired outcome.

To that end, we set out to find the optimal sequence of actions $A$ that can
transform $x$ into a counterfactual state $x^{\prime}$. In the remainder of
this section we first describe how we can compare and evaluate different
action sequences that lead to counterfactual states (Sections 3.1 and 3.2) and
describe our approach to searching for the optimal one (Section 3.3).

###  3.1. Counterfactual Properties for Reinforcement Learning

Counterfactual properties guide the counterfactual search and are used to
select the most suitable counterfactual explanation. In supervised learning,
they have been designed to ensure minimal user effort is needed to transform
the original instance to the counterfactual. These properties have so far been
mostly defined as feature-based, assuming that if two instances are similar in
features one can easily be reached from the other. However, due to the
sequential nature of RL tasks, two states can be similar in terms of features
but far away in terms of execution (Wang et al., 2016). For example, consider
a state in Atari game of Breakout, and another state obtained by removing the
ball from the first state. The two states differ in only a few pixels,
however, one can never be transformed into the other using available Breakout
actions as the ball cannot be removed from the game. Similarly, stochasticity
in the environment can affect the process of transforming the original
instance to the counterfactual. Only if the user is presented with a
counterfactual that considers these stochastic and sequential constraints can
they find the fastest and securest path to the desired outcome.

In this section we propose three RL-specific counterfactual properties that
take into account the sequential and stochastic nature of RL tasks. These
properties ensure that counterfactuals are easily obtainable from the original
instance, and produce desired output with high certainty. Unlike
counterfactual properties in supervised learning which are often defined with
respect to the counterfactual instance, we define these properties as a
functions of action sequence $A$ that transforms $x$ into counterfactual
$x^{\prime}$.

####  3.1.1. Reachability property

In RL two states can be similar in terms of state features, but far away in
terms of execution. This means that, despite appearing similar, a large number
of actions might be required to reach the counterfactual from the original
state. Conversely, a state can be very easily reachable by RL actions even if
it appears different based on its feature values. Additionally, state features
in counterfactual instances can be affected by stochastic processes outside of
agent’s control. Relying solely on feature-based similarity measures could
dismiss easily reachable counterfactuals where changes in features are beyond
agent’s control and do not affect action choice.

To account for sequential and stochastic nature of RL tasks, we propose
measuring reachability. For a state $x$ and a sequence of actions $A$, we
define reachability as:

(1) $R(x,A)=len(A)$

$R(x,A)$ measures the number of actions within the sequence that navigates to
the counterfactual instance. By minimizing this property we ensure that
counterfactual can be reached within a small number of steps.

####  3.1.2. Cost-efficiency property

Current work on counterfactual explanations assumes that each action that
changes the original instance carries the same cost. In RL, however, actions
often have costs associated with them. If a counterfactual can be obtained
through a less costly path, then it should be presented to the user, in order
to minimize user effort. For example, if either pawn or a queen sacrifice can
bring about piece capture for a chess player, they should be advised to
sacrifice the pawn, i.e., the piece of lower value and therefore with a lower
cost.

We propose cost-efficiency as a counterfactual property which prioritizes
instances reachable through least costly actions. For a state $x$ and a
sequence of actions $A$, cost-efficiency is defined as:

(2) $C(x,A)=rew(x,A)$

where $rew(x,A)$ is the cumulative reward obtained when all actions in $A$ are
applied to state $x$. In this way, user can choose a sequence of least costly
actions to transform the original instance into the counterfactual one.

####  3.1.3. Stochastic certainty property

One of the main qualities of counterfactual explanations is that they deliver
the desired outcome. Asking the user to put their time and effort into
changing the model inputs, only to obtain another unsatisfactory output can
have detrimental effects on user trust in the system. During the time that is
needed to convert the original instance into a counterfactual, conditions of
the task can change, rendering the counterfactual invalid. For example,
imagine a user unsuccessfully applying for a loan, and receiving a
counterfactual explanation, suggesting them to increase their income to be
approved. Conditions for approving a loan can change during the time it takes
the user to change their income (e.g., change jobs, get promoted), and
previously proposed counterfactual can lead to another denied loan request.

Similarly, in RL, the stochastic nature of the environment can make a
counterfactual instance invalid during the time it takes the user to obtain
it. To ensure that users are presented with counterfactuals that are likely to
produce desired output, we propose stochastic certainty. For instance $x$, a
sequence of actions $A$, black-box model $M$ and the desired action
$a^{\prime}$ stochastic certainty is defined as:

(3) $S(x,A,a^{\prime})=P[M(x^{\prime})=a^{\prime}\quad|\quad x^{\prime}=A(x)]$

where $A(x)$ is a state obtained by applying actions from $A$ to state $x$.
Intuitively, stochastic certainty measures the probability of the desired
outcome still being chosen by $M$ after the time it takes to navigate to the
counterfactual state. By maximizing stochastic certainty we promote sequences
of actions that more often lead to the desired outcome.

###  3.2. Loss Function

In order to optimize the counterfactual properties, we design a weighted loss
function encompassing RL-specific objectives. For a state $x$, sequence of
actions $A$, desired output $a^{\prime}$, loss function is defined as:

(4) $L(x,A,a^{\prime})=\alpha R(x,A)+\beta C(x,A)+\gamma(1-S(x,A,a^{\prime}))$

where $\alpha,\beta$ and $\gamma$ are parameters determining the importance of
different properties. By minimizing $L$ we can find a sequence of actions
which quickly and certainly leads to a counterfactual explanation. However,
$L(x,A,a^{\prime})$ does not verify that $a^{\prime}$ is predicted in the
obtained counterfactual. To that end, we ensure that a validity constraint is
satisfied:

(5) $V(x,x^{\prime},a^{\prime})=M(x^{\prime})==a^{\prime}$

where $x^{\prime}$ is obtained by performing actions from $A$ in $x$. Validity
is used to filter potential counterfactual instances as is described in more
detail in the next part of this section.

Figure 1. Heuristic tree search: in each iteration a node is selected by
navigating the tree from the root to a leaf by choosing actions according to
the UCT formula. The node is expanded by performing all possible actions and
appending all obtained states as children of the node. Finally, newly
generated nodes are evaluated and their values are propagated back to the root
to update the values of parent nodes. The white nodes represent states, while
black nodes are determination nodes, that serve to instantiate all possible
children states of a node in a stochastic environment.

###  3.3. Counterfactual Search

Our goal is to obtain a sequence of actions $A$ that minimizes the loss
function $L$ and satisfies the validity constraint. Unlike traditional
counterfactual search which directly searches for a counterfactual in a data
set, we are looking for an optimal sequence of actions that can transform the
original state into a counterfactual one. This means that we cannot directly
optimize $L$ over a data set of states to find a counterfactual as this would
give us no information about how difficult this counterfactual is to reach in
terms of RL actions. To this end, we propose a counterfactual search algorithm
that utilizes heuristic tree search to find a sequence of actions that
transform the original into counterfactual state that minimizes the loss
function $L$. The details of the algorithm are given in Algorithm 1 and shown
in Figure 1.

The proposed algorithm builds a tree to represent agent’s execution – each
node corresponds to a state, and each edge to one action. Each node $n$ is
also associated with a value $val(n)$ and each edge is assigned a value
$Q(n,a)$. These values are based on the loss function $L$ and are used to
determine which node should be expanded in the next iteration. Children of a
node are obtained by taking a specific action in that node. To account for the
stochasticity in the environment, we apply determinization to the expanding
process by adding hidden determinization nodes each time an action is
performed. The children of determinization nodes are sampled from the possible
states that result from performing a specific action. To calculate $val(n)$ we
compute the value of $L(x,A,a^{\prime})$, where $A$ is the sequence of actions
that navigates from root $x$ to node $n$ in the tree. $Q(n,a)$ is calculated
for each node $n$ and action $a$ as the average of values $val$ of the
children nodes obtained when performing $a$ in $n$. To estimate
$L(x,A,a^{\prime})$ we need to calculate the values of individual
counterfactual properties of reachability, cost-efficiency and stochastic
uncertainty for nodes in the tree. We calculate reachability of node $n$ as
the length of the path between the root and $n$. To calculate cost-efficiency
of $n$ we record and sumate the environment’s rewards along the path from the
root to $n$. Finally, to calculate stochastic certainty, we perform N
simulations by unrolling the sequence of actions $A$ from $x$ in the
environment, and record the number of times a desired outcome is obtained in
the resulting state. We then calculate stochastic certainty as:

(6) $S(x,A,a^{\prime})=\frac{N(M(x^{\prime})==a^{\prime})}{N}$

where $x^{\prime}$ is a state obtained after following $A$ in $x$. We
normalize the values for reachability and cost-efficiency so that they fall
within $[0,1]$ range, while stochastic uncertainty values naturally belong to
that range. We can then evaluate a node in tree by combining and weighting the
three counterfactual properties to obtain $L(x,A,a^{\prime})$ as shown in
Equation 4.

At the start of the search, a tree is constructed with only the root node
corresponding to the state $x$ that is being explained. At each step of the
algorithm, a node in the tree is chosen and tree is expanded with the node’s
children. All actions are expanded simultaneously in the node. The resulting
children nodes are then evaluated against $L$, and the results are propagated
back to the tree root to update the value of nodes and edges. To decide which
node is expanded in each iteration we navigate the tree from the root, at each
node $n$ taking the action decided by the Upper Confidence Bound applied for
Trees (UCT) formula (Kocsis and Szepesvári, 2006):

(7) $a^{*}=\arg\max_{a\in
A}\left\\{Q(a,n)+C\sqrt{\frac{\ln(N(n))}{N(s,a)}}\right\\}$

where $C$ is the exploration constant, $N(n)$ number of times $n$ was visited
and $N(n,a)$ number of times $a$ was chosen in $n$. UCT balances between
following the paths of high value and exploring underrepresented paths through
the exploration constant $C$. The process is repeated until a predetermined
maximum number of iterations $T$ is reached.

Once the tree is fully grown, all nodes are first filtered according to the
validity constraint to remain only with the states that deliver the desired
output. The remaining nodes are potential counterfactual explanations. Then
all nodes are evaluated against $L$. The state corresponding to the node in
the tree with minimum value for $L$ is presented to the user as the best
counterfactual.

Algorithm 1 Counterfactual heuristic tree search

1: Input: state $x$, desired outcome $a^{\prime}$, black-box model $M$,
environment $E$

2: Parameters:number of iterations $T$

3: Output: counterfactual state $x^{\prime}$

4: $t=\\{x\\}$ {Initializing search tree}

5: $i=0$

6: while i ¡ T do

7: n = select(t) {Select state $n$ to be expanded}

8: S = expand(n) {Expand $n$ by performing available actions and obtain a set
of new states $S$}

9: for all $s\in S$ do

10: $val(s)=L(x,A,a^{\prime})$ {Evaluate new states in $S$ according to $L$}

11: $t+={s}$

12: end for

13: backpropagate() {Propagate newly evaluated values back to the root}

14: $i+=1$

15: end while

16: $p=[]$

17: for all $s\in t$ do

18: if $valid(s)$ then

19: $p+=s$ {Filter valid counterfactuals}

20: end if

21: end for

22: $cf=\arg\min_{s\in p}L(x,s(A),a^{\prime})$ {Select best counterfactual as
the valid counterfactual which minimizes $L$}

##  4\. Experiments

In this section we outline the experiment setup for evaluating RACCER. We
describe the baseline approaches we evaluate RACCER against (Section 4.1) and
the evaluation tasks (Section 4.2).

###  4.1. Baseline Approaches

In this work, we proposed a model-agnostic approach for generating
counterfactual explanations for RL. In the current state-of-the-art there is
only one other method for generating counterfactuals for RL (Olson et al.,
2019), but it requires substantial information about the RL model parameters.
For that reason we cannot compare our work to Olson et al. (2019). Instead, we
implement two baseline models based on current state-of-the-art approaches in
supervised learning and RL. Both baseline approaches optimize feature-based
metrics that are used in the majority of current counterfactual approaches.
Specifically, all baselines optimize the following $6$ counterfactual
properties:

  1. (1)

Validity: we use simple binary metrics for determining whether the desired
outcome $a^{\prime}$ is obtained in the counterfactual state $x^{\prime}$:

(8) $d_{v}(x,x^{\prime})=M(x^{\prime})==a^{\prime}$

  2. (2)

Proximity: as we evaluate our approach in environments with discrete features,
we decide on measuring the feature-based proximity using the Euclidian
distance between the original and the counterfactual state in the encoding
space:

(9) $d_{p}(x,x^{\prime})=|enc(x)-enc(x^{\prime})|^{2}_{2}$

The encoder-decoder pair is trained on a dataset of rollout trajectories of
black-box policy $M$ that is being explained.

  3. (3)

Sparsity: to calculate sparsity we count the number of different features
between the original and counterfactual instance:

(10) $d_{s}(x,x^{\prime})=|x-x^{\prime}|_{1}$

  4. (4)

Data manifold closeness: to estimate how realistic the counterfactual instance
is we use the encoding loss, similar to methods in (Looveren and Klaise, 2021;
Dhurandhar et al., 2018):

(11) $d_{dmc}(x,x^{\prime})=|dec(enc(x^{\prime}))-x^{\prime}|^{2}_{2}$

  5. (5)

Actionability: actionability refers to maintaining the values of immutable
features. As different tasks have different immutable features, we define
actionability depending on the task. More detail is given in Section 4.2.

  6. (6)

Game fidelity: generating counterfactuals can often involve changing or
deleting features and comes with the risk that the obtained state no longer
complies with the game rules. We ensure that the generated counterfactual
abides by the rules of the game by implementing game fidelity constraint. Game
fidelity depends on the task, and is described in more detail for specific
environments in Section 4.2.

To search for the best counterfactual, we define a baseline loss function
$L_{BO}$ which relies on the proximity, sparsity and data manifold closeness
properties:

(12)
$L_{BO}(x,x^{\prime},a^{\prime})=\theta_{0}d_{p}(x,x^{\prime},a^{\prime})+\theta_{1}d_{s}(x,x^{\prime},a^{\prime})+\theta_{2}d_{dmc}(x,x^{\prime},a^{\prime})$

Parameters $\theta_{0}$, $\theta_{1}$ and $\theta_{2}$ determine the
importance of different objectives. For simplicity, we use
$\theta_{0}=\theta_{1}=\theta_{2}=-1$ for our experiments, resulting in a loss
function which favors all properties equally (Table 1). The remaining
properties validity, actionability and game fidelity are used as constraints
to filter the obtained instances for those that satisfy the game rules, do not
change immutable features and deliver the desired outcome.

To optimize baseline loss $L_{BO}$, we implement two baseline approaches:

  1. (1)

BO+GEN: this approach uses a genetic algorithm to find the best
counterfactuals based on the baseline loss function $L_{BO}$. Genetic
algorithm is a model-agnostic optimization approach that has previously been
used to search for counterfactuals in supervised learning (Dandl et al.,
2020). We use a basic $(\mu+\lambda)$ genetic algorithm with $L_{BO}$ as the
fitness function (Blank and Deb, 2020). The parameters of the algorithm are
provided in Table 1.

  2. (2)

BO+TS: we optimize the loss function $L_{BO}$ using heuristic tree search. The
optimization algorithm is the same heuristic tree search as used in RACCER and
described in Section 3.3, except BO+TS uses $L_{BO}$ to evaluate nodes and
expand the tree, and ultimately choose the best counterfactual. Parameters
used in the approach for different environments can be found at Table 1.

Table 1. Parameters used for generating counterfactual explanations for
$BO+GEN$, $BO+TS$ and $RACCER$ approaches in Stochastic GridWorld and chess
environments.

Parameter Task Stochastic GridWorld Chess Number of iterations ($T$) 300 1
Number of simulations ($N$) 100 20 Maximum number of actions ($k$) 5 1
Evaluation dataset size ($|D|$) 500 63 Generation sample size 1000 100 Genetic
iterations 30 10

Loss Parameter Value $\alpha$ -1 $\beta$ -1 $\gamma$ -1 $\Theta_{0}$ -1
$\Theta_{1}$ -1 $\Theta_{2}$ -1

###  4.2. Evaluation Tasks

We evaluate our approach in two environments – Stochastic GridWorld and Chess.

####  4.2.1. Stochastic GridWorld

Stochastic GridWorld is a simple $5\times 5$ gridworld, where agent is tasked
with shooting the dragon. To successfully shoot the dragon, agent has to be in
the same file or row as the dragon, and the space between them has to be
empty. In that situation agent can successfully perform the SHOOT action and
win the game. Environment also contains different types of trees, located in
the middle file of the grid, that can block agent’s shooting path to the
dragon. Agent can chop down the tree by performing a required number of CHOP
actions when located directly next to the tree. Different tree types require
different number of consecutive CHOP actions to disappear. At each step, agent
can move one step in any of the directions and perform SHOOT and CHOP actions.
Additionally, the middle file of the board is extremely fertile, and trees can
regrow along this file with different probabilities. Agent’s actions are
penalized with $-1$ reward, while successfully shooting the dragon brings
$+10$ reward. The episode ends when the dragon is shot or when the maximum
number of time steps is reached. The only immutable feature in the environment
is the dragon’s location, as it cannot move within one episode. We consider
all states that contain an agent, a dragon, a have trees only along the middle
file of the grid to correspond to the rules of the games.

In this environment two states can appear very similar but be far away in
terms of execution. For example, even if the only difference between two
states is one tree, depending on its type chopping it down might be a lengthy
process. Chopping down a tree in order to be able to shoot the dragon might be
less preferable than simply going around it, and suggesting this to the user
could save them time and effort. Similarly, due to the stochastic nature of
the task, during the time needed to obtain a counterfactual, new trees can
regrow and potentially block agent’s path to the dragon.

####  4.2.2. Chess

We evaluate our approach in Chess environment, with the aim of assisting users
with understanding simple tactics. Specifically, we focus on positions in
which user might prematurely attack, before the attack is fully formed. In
this situation, user might be interested to know why the attack is not the
best option, and a counterfactual explanation could give them actionable
advice on how to prepare and execute the attack. In the chess environment we
do not consider any features as immutable, due to the complexity of the game
and high number of possible situations that can arise from one state. To check
if counterfactual states correspond to valid game states we use
functionalities provided within the Stockfish package (Zhelyabuzhsky, 2022).

Due to the rules of the game, even two states that differ only in one piece
can be unreachable from one another, as pieces are difficult or impossible to
appear back on the board. Similarly, the game is highly stochastic due to
opponent’s moves, and planning an attack has to include an analysis of its
probable success depending on the opponent’s choices. Suggesting to the user a
counterfactual which is unobtainable in the game terms or one that is only
successful for a small number of opponent’s responses will not assist the user
to perform the attack.

##  5\. Evaluation

In this section we describe the evaluation process of RACCER in the Stochastic
GridWorld and Chess environments. Firstly, we evaluate the counterfactuals
against counterfactual properties of reachability, cost-efficiency and
stochasticity, as well as feature-based properties proximity, sparsity and
data manifold closeness. Additionally, we conduct a user study, to investigate
how different types of counterfactual explanations affect user understanding
of agent’s behavior.

For both tasks, we first obtain a black-box model $M$ which is being
explained. For Stochastic GridWorld we train a DQN (Mnih et al., 2013), while
for the chess tasks we use Stockfish engine. Additionally, we assume access to
the environment in both tasks. For each task we generate a data set of factual
states for which we generate counterfactual explanations. In chess environment
we manually created a dataset of $63$ game states in which a player can
perform a simple, multi-step tactical attack. The final action in the attack
is used as the desired outcome. In this way, the counterfactual explanation
can demonstrate to the user what preparatory steps need to be take for the
attack to be successful. In the Stochastic GridWorld we sample a dataset with
$100$ factual states by unrolling expert policy $M$ in the environment. For
each state, we explain each alternative action that agent did not choose in
that state, resulting in $500$ generated counterfactuals.

Table 2. Average values of counterfactual properties for counterfactual
explanations generated using $BO+GEN$, $BO+TS$ and $RACCER$ approaches in
Stochastic GridWorld and Chess tasks.

Task Stochastic Gridworld Chess Metric Approach BO + GEN BO + TS RACCER BO +
GEN BO + TS RACCER Generated counterfactuals ($\%$) $74.60\%$ $56.80\%$
$\mathbf{73.40\%}$ $98.41$ % $95.24\%$ $\mathbf{100\%}$ Proximity -0.23 -0.31
-0.45 -0.06 -0.25 -0.24 Sparsity -2.02 -2.09 -3.24 -5.54 -3.71 -3.75 Data
manifold closeness -0.37 -0.36 -0.57 -14.58 -13.05 -14.80 $L_{BO}$ -2.62 -2.76
-4.26 -20.19 -17.02 -18.80 Reachability $-0.58$ $-0.59$ $\mathbf{-0.41}$ -1 -1
-1 Cost-efficiency $\mathbf{-1.0}$ $\mathbf{-1.0}$ $\mathbf{-1.0}$ -1 -0.48
-0.45 Stochastic uncertainty -0.45 -0.33 $\mathbf{-0.21}$ -1 -0.68 -0.44 $L$
-2.03 -1.93 -1.62 -3 -2.14 -1.86

Figure 2. Sample question from the conducted user study

###  5.1. Evaluating Counterfactual Properties

We evaluate counterfactual explanations produced by baselines $BO+GEN$ and
$BO+TS$ and compare them with RACCER based on their reachability, cost-
efficiency and stochastic certainty, and feature-based properties proximity,
sparsity and data manifold closeness. For each factual state, we run the all
three approaches to select the best counterfactual, and evaluate
counterfactual properties for them. We limit the search for counterfactuals to
$k$ actions. Parameters used in $BO+GEN$, $BO+TS$, and $RACCER$ approaches are
given in Table 1.

Evaluating RL-specific counterfactual properties for $BO+TS$ and RACCER is
straightforward as both use tree search to navigate to the counterfactual and
properties can be calculated by analysing the sequence of actions leading from
the root to the counterfactual. Genetic search, however, generates a
counterfactual by combining different states and uses no notion of actions. To
measure reachability, cost-efficiency and stochastic certainty for a
counterfactual $x^{\prime}$ generated by $BO+GEN$, we build a tree of agent’s
execution of length $k$ rooted in $x$ and find $x^{\prime}$ in it. In that way
we can estimate properties which rely on actions even for explanations
generated through direct search for counterfactual states. If $x^{\prime}$
cannot be found in the tree, it is assigned the lowest possible value for each
property which is $-1$.

We present the average results for each counterfactual property and the loss
function $L$ value for all three approaches in the Stochastic GridWorld and
Chess environment in Table 2. We also record the values of baseline
counterfactual properties of proximity, sparsity and data manifold closeness,
as well the $L_{BO}$ value for each generated counterfactual (Table 2). We
record values of properties already multipled with their weighing factors
($\alpha,\beta,\gamma,\theta_{0},\theta_{1},\theta_{2}$) from Table 1. For
each approach we also record the percentage of states for which a
counterfactual was successfully found.

In the Stochastic GridWorld environment, $RACCER$ and $BO+GEN$ approaches
generate counterfactuals for over $70\%$ of factual states. $BO+TS$ algorithm,
however, provides counterfactuals for only $56.80\%$ of states. We assume that
this is a consequence of the algorithm’s reliance on feature-based
counterfactual properties when deciding which node to expand in the execution
tree. As $BO+TS$ uses proximity, sparsity and data manifold closeness metrics
to decide which node to expand in each iteration, it prefers nodes whose
features are similar to the root. For this reason, $BO+TS$ navigates the tree
by often choosing to follow the action $SHOOT$ that does not change features.
This behavior leads to a lack of diversity within the explored nodes, and
ultimately to fewer generated counterfactuals. While baseline algorithms
$BO+GEN$ and $BO+TS$ perform better than $RACCER$ in feature-based metrics
(proximity, sparsity, data manifold closeness and baseline loss function
$L_{BO}$), $RACCER$ produces counterfactuals that perform better in
reachability and stochastic uncertainty and report lower values for $L$. As
the normalized cost is $-1$ for any sequence of actions in the Stochastic
GridWorld, there is no difference in cost-efficiency property values between
the approaches.

In the chess task, all three approaches can successfully generate
counterfactuals for almost all provided factual states. While showing the best
results for proximity property, $BO+GEN$ reports the worst performance on
other baseline properties sparsity and data manifold closeness, as well as the
baseline objective $L_{BO}$. Additionally, $BO+GEN$ produces counterfactuals
that cannot be reached from the original instance in the allotted number of
steps, as it often removes, adds or replaces pieces contrary to the game
rules. This results in $BO_{GEN}$ reporting the worst performance according to
the RL-specific metrics and $L$. Baseline approaches $BO+TS$ produces
counterfactuals with lowest values for baseline properties proximity, sparsity
and data manifold closeness, as well as $L$. However, RACCER performs better
in RL-specific metrics reachability, cost-efficiency and stochastic
uncertainty, as well as RL-specific loss function L.

While baseline methods perform better on feature-based metrics, RACCER
produces counterfactuals which are easier to reach through less costly paths
and deliver the desired outcome more frequently.

###  5.2. User Study

In Section 5.1 we have shown that RACCER generates counterfactuals that are
easier to reach and more probable to deliver the desired outcome compared to
the baseline approaches. However, counterfactual explanations are ultimately
intended to assist humans in real-life tasks, and evaluating them in this
context is necessary to ensure their usefulness. To that end, we conduct a
user study in which we evaluate the effect of different types of
counterfactual explanations on user understanding of agent’s behavior.
Specifically, we compare the counterfactual explanations produced by the
baseline $BO+GEN$ and those produced by RACCER. We conduct the study in the
Stochastic GridWorld environment, as it has simple rules, and requires no
prior knowledge from users (which also ensures that results are not skewed by
different levels of prior knowledge, like for example, the case might be with
chess).

We sourced $50$ participants through the Prolific platform from English-
speaking countries (UK, Ireland, Canada, USA, Australia and New Zealand) and
split them into two groups. The first group received counterfactuals generated
by BO+GEN and the second counterfactuals produced by RACCER. After filtering
participants for those who have passed attention checks, we have remained with
46 participants, 23 in each group.

Figure 3. Users’ rating of different explanations properties on a 1 - 5 Likert
scale for different explanation types.

The study consisted of $10$ questions, and in each question participants were
shown a game state from the Stochastic GridWorld task (Figure 2). Participants
were offered multiple possible action sequences and asked to choose the one
they believe agent will take in the shown game state. The participants were
then shown a counterfactual explanation for that state, that explains in which
situation agent would have chosen action SHOOT. Finally, participants were
again presented with the original state, and asked to predict a sequence of
agent’s actions. They could remain with their original answer, or change it
based on the presented explanations. We focus on counterfactual explanation
for the action SHOOT, as performing this action is agent’s goal, and as such
it carries the most information about agent’s behavior. To generate questions
for the user study, we assume a black-box agent $M$ in the Stochastic
GridWorld as described in Section 5. For $M$, we generate counterfactual
explanations using algorithms $BO+GEN$ and RACCER approaches as described in
Sections 4.1 and 3 respectively. At the end of the study users also ranked the
explanations based on the explanation goodness metrics (Hoffman et al., 2018)
on a $1-5$ Likert scale ($1$ \- strong disagreement, $5$ \- strong agreement).
Specifically, users reported whether they found explanations to be useful,
satisfying, complete, detailed, actionable, trustworthy and reliable.
Additionally, we included a question about how confident users are about their
predictions, in order to estimate the effect of counterfactuals on user
confidence. A sample user study is available at:
https://forms.gle/4DLPhcMABwkLTahx5.

To evaluate the effect of counterfactuals on user understanding we measure
users’ accuracy in predicting the correct sequence of actions after seeing the
explanation. The correct sequence of actions is the one the agent $M$ would
take. There are two reasons for choosing prediction accuracy as the evaluation
metrics for this study. Firstly, successful prediction of agent’s behavior
indicates that user understands and can anticipate system’s behavior. From a
perspective of actionable advice, on the other hand, prediction accuracy tell
us how good the user is at choosing the best path to performing the SHOOT
action and winning the game after seeing a counterfactual explanations. In
other words, accurate prediction of agent’s behavior indicates that the
counterfactual has helped the user identify the best path to achieving the
goal, which is the ultimate purpose of these explanations.

Users presented with counterfactual explanations generated by $BO+GEN$ have
chosen a correct sequence of actions in $23.04\%$ of cases. In contrast, users
that saw RL-specific counterfactuals generated by $RACCER$ chose the correct
sequence in $56.52\%$ of situations. We performed a Wilcoxon Signed-rank test
with significance level $0.05$ and found significant difference in accuracy
prediction between participants who received counterfactuals generated by
$BO+GEN$ compared to $RACCER$ algorithm ($W=4.0,p=0.0015$).

We record the results of user’s ranking of explanation goodness metrics in
Figure 3 for counterfactuals generated by $BO+GEN$ and $RACCER$ algorithms. We
perform a Wilcoxon Signed-rank test with significance level $0.05$ to evaluate
the differences in user rankings of explanations goodness metrics. However, we
found no significant difference in ratings for any of the explored metrics.
This indicates users find explanations generated using $BO+GEN$ and $RACCER$
approaches equally satisfying. Even though users perceive baseline
explanations as satisfactory, they do not help them understand agent behavior,
indicating that traditional feature-based methods can generate misleading
counterfactuals.

##  6\. Conclusion and Future Work

In this work, we presented RACCER, the first RL-specific approach to
generating counterfactual explanations. We designed and implemented three
novel counterfactual properties that reflect the sequential and stochastic
nature of RL tasks, and provided a heuristic tree search approach to finding a
counterfactual that optimizes these properties. We evaluated our approach in a
Stochastic GridWorld and a more complex chess tasks, and showed that RACCER
generates counterfactuals that are easier to reach and provide the desired
outcomes more often compared to baseline approaches. We have also conducted a
user study, and shown that users presented with counterfactuals generated by
RACCER could correctly predict behavior of RL agents twice more frequently
compared to users presented with baseline explanations. This indicates that
RL-specific counterfactuals help users better understand and anticipate
agent’s behavior.

In this work we have limited our search to only the best counterfactual. In
the future work, we hope to expand our search to include a set of diverse
counterfactual explanations optimizing different counterfactual properties. In
this way, users would have a wider choice of potential actionable advice.
Similarly, we have also assumed in our work that all counterfactual properties
are of the same importance to the user. However, some users might be more
interested in a shorter but riskier path, while others might prefer safety
over speed. In the future work we hope to utilize a human-in-the-loop approach
to generate personalized counterfactual explanations that fit users
preferences.

## Acknowledgement

This publication has emanated from research conducted with the financial
support of a grant from Science Foundation Ireland under Grant number
18/CRT/6223. For the purpose of Open Access, the author has applied a CC BY
public copyright licence to any Author Accepted Manuscript version arising
from this submission.

## References

  * (1)
  * Arulkumaran et al. (2017) Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017\.  Deep reinforcement learning: A brief survey.  _IEEE Signal Processing Magazine_ 34, 6 (2017), 26–38.
  * Blank and Deb (2020) J. Blank and K. Deb. 2020.  pymoo: Multi-Objective Optimization in Python.  _IEEE Access_ 8 (2020), 89497–89509.
  * Byrne (2019) Ruth MJ Byrne. 2019\.  Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning.. In _IJCAI_. 6276–6282.
  * Coppens et al. (2019) Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. 2019.  Distilling deep reinforcement learning policies in soft decision trees. In _Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence_. 1–6.
  * Dandl et al. (2020) Susanne Dandl, Christoph Molnar, Martin Binder, and Bernd Bischl. 2020.  Multi-objective counterfactual explanations. In _International Conference on Parallel Problem Solving from Nature_. Springer, 448–469.
  * Delaney et al. (2021) Eoin Delaney, Derek Greene, and Mark T Keane. 2021\.  Uncertainty estimation and out-of-distribution detection for counterfactual explanations: Pitfalls and solutions.  _arXiv preprint arXiv:2107.09734_ (2021).
  * Dhurandhar et al. (2018) Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018.  Explanations based on the missing: Towards contrastive explanations with pertinent negatives.  _arXiv preprint arXiv:1802.07623_ (2018).
  * Gajcin and Dusparic (2022) Jasmina Gajcin and Ivana Dusparic. 2022.  Counterfactual Explanations for Reinforcement Learning.  _arXiv preprint arXiv:2210.11846_ (2022).
  * Hoffman et al. (2018) Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for explainable AI: Challenges and prospects.  _arXiv preprint arXiv:1812.04608_ (2018).
  * Huber et al. (2021) Tobias Huber, Katharina Weitz, Elisabeth André, and Ofra Amir. 2021.  Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps.  _Artificial Intelligence_ 301 (2021), 103571.
  * Karimi et al. (2020a) Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. 2020a.  A survey of algorithmic recourse: definitions, formulations, solutions, and prospects.  _arXiv preprint arXiv:2010.04050_ (2020).
  * Karimi et al. (2020b) Amir-Hossein Karimi, Julius Von Kügelgen, Bernhard Schölkopf, and Isabel Valera. 2020b.  Algorithmic recourse under imperfect causal knowledge: a probabilistic approach.  _Advances in Neural Information Processing Systems_ 33 (2020), 265–277.
  * Kocsis and Szepesvári (2006) Levente Kocsis and Csaba Szepesvári. 2006.  Bandit based monte-carlo planning. In _European conference on machine learning_. Springer, 282–293.
  * Laugel et al. (2017) Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017\.  Inverse classification for comparison-based interpretability in machine learning.  _arXiv preprint arXiv:1712.08443_ (2017).
  * Looveren and Klaise (2021) Arnaud Van Looveren and Janis Klaise. 2021.  Interpretable counterfactual explanations guided by prototypes. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 650–665.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * Molnar (2019) Christoph Molnar. 2019\.  _Interpretable Machine Learning_.
  * Mothilal et al. (2020) Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020\.  Explaining machine learning classifiers through diverse counterfactual explanations. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 607–617.
  * Olson et al. (2019) Matthew L Olson, Lawrence Neal, Fuxin Li, and Weng-Keen Wong. 2019\.  Counterfactual states for atari agents via generative deep learning.  _arXiv preprint arXiv:1909.12969_ (2019).
  * Poyiadzi et al. (2020) Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. 2020\.  FACE: feasible and actionable counterfactual explanations. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 344–350.
  * Puiutta and Veith (2020) Erika Puiutta and Eric Veith. 2020.  Explainable reinforcement learning: A survey. In _International cross-domain conference for machine learning and knowledge extraction_. Springer, 77–95.
  * Rosynski et al. (2020) Matthias Rosynski, Frank Kirchner, and Matias Valdenegro-Toro. 2020\.  Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?  _arXiv preprint arXiv:2012.01281_ (2020).
  * Samoilescu et al. (2021) Robert-Florian Samoilescu, Arnaud Van Looveren, and Janis Klaise. 2021\.  Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning.  _arXiv preprint arXiv:2106.02597_ (2021).
  * Ustun et al. (2019) Berk Ustun, Alexander Spangher, and Yang Liu. 2019\.  Actionable recourse in linear classification. In _Proceedings of the conference on fairness, accountability, and transparency_. 10–19.
  * Verma et al. (2021) Sahil Verma, John Dickerson, and Keegan Hines. 2021\.  Counterfactual Explanations for Machine Learning: Challenges Revisited.  _arXiv preprint arXiv:2106.07756_ (2021).
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017\.  Counterfactual explanations without opening the black box: Automated decisions and the GDPR.  _Harv. JL & Tech._ 31 (2017), 841.
  * Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016\.  Dueling network architectures for deep reinforcement learning. In _International conference on machine learning_. PMLR, 1995–2003.
  * Zhelyabuzhsky (2022) Ilya Zhelyabuzhsky. 2022\.  Stockfish.  https://github.com/zhelyabuzhsky/stockfish