# Light kaonic atoms: from ”corrected” to ”summed up” Deser formula

N.V. Shevchenko<EMAIL_ADDRESS>Nuclear Physics Institute, 25068 Řež,
Czech Republic

###### Abstract

Accuracy of ”corrected Deser” and ”summed up Deser” formulas was checked for
the $K^{-}p$ and $K^{-}d$ systems. It was found that the last one is much more
accurate and should be used for connection the $1s$ level shift to the
corresponding scattering length.

###### pacs:

13.75.Jz, 36.10.Gv

Interaction of antikaons with nucleons is the basic input for studying quasi-
bound states in exotic nuclei containing antikaons. Experimental data, which
can be used for construction of a model of $\bar{K}N$ interaction, are not too
rich. Some of the data are old and not very accurate, such as $K^{-}p$ cross-
sections. They can be reproduced by an antikaon-nucleon strong potential
directly, in a model independent way. Reproduction of others cannot be used
for parameter fitting directly, it needs some additional model-dependent
assumptions. It is the case of $K^{-}p$ correlation functions recently
measured by ALICE at CERN.

The most promising is kaonic hydrogen, which is an atom consisting of $K^{-}$
and proton. It’s $1s$ level is shifted in respect to the pure Coulomb value
due to strong interaction between the particles. In principle, the shift and
the width of kaonic hydrogen is the data, which can be calculated directly
using the strong plus Coulomb interactions. However, it is much easier to
calculate $K^{-}p$ scattering length using only strong $\bar{K}N$ interaction,
and then use some formula, which connects the scattering length to the $1s$
level shift. Such formula was suggested by Deser for pion-nucleon system [1].

Later the authors of [2] suggested a formula, which is more accurate for the
case when the meson is heavier, like an antikaon. This ”corrected Deser”
formula is widely used nowadays in a form

$\Delta E_{K^{-}p}^{cD}-i\frac{\Gamma_{K^{-}p}^{cD}}{2}=\\\
-2\alpha^{3}\mu^{2}a_{K^{-}p}[1-2\alpha\mu a_{K^{-}p}(\ln{\alpha}-1)]$ (1)

by experimentalists and by theorists, who are not able to calculate $1s$ level
of kaonic hydrogen directly. It was shown in [3, 4], that the accuracy of the
formula is about $10\%$ for the two-body $K^{-}p$ system and much worse for
the three-body $K^{-}d$ system.

However, the series, which leads to the corrected Deser formula in [2] can be
summed up. The result is a formula, which I will call ”summed up Deser”
formula

$\Delta
E_{K^{-}p}^{sD}-i\,\frac{\Gamma_{K^{-}p}^{sD}}{2}=-2\alpha^{3}\mu^{2}a_{K^{-}p}/[1+2\alpha\mu
a_{K^{-}p}(\ln{\alpha}-1)].$ (2)

To the best of my knowledge, first it was wrote down in [5]. The authors of
[5] introduce it in a footnote: ”In the case of kaonic atoms, higher-order
Coulomb corrections may turn out to be not completely negligible numerically
…This issue is, however, relatively easy to cure since the large contribution
comes from an iteration of a particular diagram to all orders. Replacing the
factor $1-2\mu\alpha(\ln{\alpha}-1)a_{p}$ by
$(1+2\mu\alpha(\ln{\alpha}-1)a_{p})^{-1}$ already captures the bulk of the
effect. We shall not further elaborate on this issue.”

Such summed up formula should be more accurate than the first two terms of the
series. The question is, how accurate is it, especially for the three-body
system.

Knowledge of the accuracy of the approximate Deser-type formulas is useful not
only for theorists, but for experimentalists as well. Quite a few experiments
plan or performing measurements of antikaon-nucleon or -nucleus systems. In
particular, SIDDHARTA-2 [6] or J-PARK E57 [7] experiments aim to measure $1s$
level shift in kaonic deuterium.

Authors of [8, 9] mentioned in passing ”summed up” Deser formula as
”substantially improving the agreement” or ”reproducing the level shifts
considerably better”. Accuracy of ”summed up” Deser formula was checked in
[10], where for the two-body $K^{-}p$ system it was found to be much higher
than for the ”corrected” Deser formula. The results for the three-body
$K^{-}d$ system, however, are not so clear. The problem is that $1s$ level
shift and width of kaonic deuterium were evaluated in [10] from variational
calculation with energy-independent $\bar{K}N$ potential, while $K^{-}d$
scattering length, necessary for the approximate formulas, was calculated
using the fixed-center approximation. Therefore, it is not absolutely clear,
what is the accuracy of the approximate formulas themselves, and what are the
errors introduced by the approximations.

In previous years we performed a series of calculations of diffferent states
and reactions in $\bar{K}NN$ and $\bar{K}\bar{K}N$ systems [11]. The
calculations were performed using dynamically exact Faddeev-type AGS
equations. As an input we used three different versions of the $\bar{K}N$
interaction, constructed by us. The potentials reproduce low-energy $K^{-}p$
experimental data including $1s$ level shift and width of kaonic hydrogen. In
contrast to most of the authors of $\bar{K}N$ interaction models, we
calculated energy of the $1s$ level and its width directly by solving
Lippmann-Schwinger equation with strong plus Coulomb potentials. The
scattering lengths, given by the strong antikaon-nucleon potential, were also
calculated, so that we can compare the directly calculated characteristics of
kaonic hydrogen with those given by approximate formulas.

In addition, $1s$ shift caused by strong interaction in kaonic deuterium was
evaluated using Faddeev-type three-body equations with directly included
strong plus Coulomb interactions in [12]. Since $K^{-}d$ scattering length was
calculated before as well, it is a unique possibility to check the approximate
formulas, connecting scattering length with characteristics of kaonic atoms
for two- and three-body systems, comparing them with the exact results.

Three our antikaon-nucleon potentials are: two phenomenological potentials
having one- $V_{\bar{K}N}^{\rm 1,SIDD}$ or two-pole $V_{\bar{K}N}^{\rm
2,SIDD}$ structure of $\Lambda(1405)$ resonance [13] , and a chiral potential
$V_{\bar{K}N}^{\rm Chiral}$ [14]. All three potentials reproduce the most
recent and most accurate result of the $1s$ level shift of kaonic hydrogen
measurement performed by SIDDHARTA collaboration [15].

The $K^{-}p$ scattering lengths of the three antikaon-nucleon potentials [13,
14], reproducing SIDDHARTA data are:

$\displaystyle a_{K^{-}p}^{\rm 1,SIDD}=-0.76+i\,0.89\,{\rm fm}$ (3)
$\displaystyle a_{K^{-}p}^{\rm 2,SIDD}=-0.74+i\,0.90\,{\rm fm}$ (4)
$\displaystyle a_{K^{-}p}^{\rm Chiral}=-0.77+i\,0.84\,{\rm fm.}$ (5)

I also used our previously constructed phenomenological potentials from [16],
which do not reproduce SIDDHARTA data, but can be useful for the checks of the
approximate formulas. The $K^{-}p$ scattering lengths given by the one-pole
$V_{\bar{K}N}^{\rm 1,KEK}$ and the two-pole $V_{\bar{K}N}^{\rm 1,KEK}$
potentials are:

$\displaystyle a_{K^{-}p}^{\rm 1,KEK}=-1.00+i\,0.68\,{\rm fm}$ (6)
$\displaystyle a_{K^{-}p}^{\rm 2,KEK}=-0.96+i\,0.80\,{\rm fm}.$ (7)

Table 1: $1s$ level shift $\Delta E_{1s}^{K^{-}p}$ (eV) and width $\Gamma_{1s}^{K^{-}p}$ (eV) of kaonic hydrogen calculated using the ”corrected Deser” and ”summed up Deser” formulas together with the exact results. | Corrected | Summed up | Exact
---|---|---|---
| Deser | Deser |
| $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$ | $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$ | $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$
$V_{\bar{K}N}^{\rm 1,SIDD}$ | -328 | 579 | -318 | 593 | -313 | 597
$V_{\bar{K}N}^{\rm 2,SIDD}$ | -322 | 589 | -312 | 603 | -308 | 602
$V_{\bar{K}N}^{\rm Chiral}$ | -326 | 544 | -318 | 559 | -313 | 561
$V_{\bar{K}N}^{\rm 1,KEK}$ | -383 | 404 | -381 | 429 | -377 | 434
$V_{\bar{K}N}^{\rm 2,KEK}$ | -381 | 482 | -376 | 509 | -373 | 514

The $1s$ level shifts and widths of kaonic hydrogen calculated using the
”corrected Deser” Eq.(1) and ”summed up Deser” Eq.(2) formulas are presented
in Table 1 together with the exact results. It is seen that the ”summed up
Deser” formula lead to much more accurate $\Delta E_{1s}^{K^{-}p}$ (eV) and
$\Gamma_{1s}^{K^{-}p}$ values corresponding to the $K^{-}p$ scattering lengths
Eqs.(3,4,5) and Eq.(6,7) than the ”corrected Deser” formula. The accuracy of
the ”summed up Deser” formula variates within $0.8-1.6\%$ for the shift and
$0.2-1.1\%$ for the width.

The three potentials reproducing SIDDHARTA experimental data: phenomenological
$V_{\bar{K}N}^{\rm 1/2,SIDD}$ [13] and chiral potential $V_{\bar{K}N}^{\rm
Chiral}$ [14], - were used in the three-body calculations of the $\bar{K}NN$
system with spin $1$. The $K^{-}d$ scattering lengths were evaluated by
solving the dynamically exact Faddeev-type AGS equations with strong
$\bar{K}N$ potentials only:

$\displaystyle a_{K^{-}d}^{\rm 1,SIDD}=-1.49+i1.24\,{\rm fm}$ (8)
$\displaystyle a_{K^{-}d}^{\rm 2,SIDD}=-1.51+i1.25\,{\rm fm}$ (9)
$\displaystyle a_{K^{-}d}^{\rm Chiral}=-1.59+i1.32\,{\rm fm.}$ (10)

Dynamically exact calculation of the kaonic deuterium characteristics were
performed using Faddeev-type equations with directly included Coulomb
interaction in [12]. The same three strong antikaon-nucleon potentials were
used there. The $1s$ level shifts and widths of the three-body kaonic atom
corresponding to the scattering lengths of $K^{-}d$ system Eq.(8,9,10) are
presented in Table 2. We used the same Eq.(1) and Eq.(2) for the ”corrected”
and ”summed up Deser” formulas as for the two-body $K^{-}p$ case with $\mu$
being the $K^{-}d$ reduced mass.

The results of one more method of kaonic deuterium characteristics
calculations are denoted in Table 2 as ”$K^{-}d$ optical potential”. To obtain
them we: calculated low-energy $K^{-}d$ scattering amplitudes using
dynamically exact three-body AGS equations, constructed an optical $K^{-}-d$
two-body potential, reproducing the three-body amplitudes, and evaluated the
$1s$ level shift and width using two-body Lippmann-Schwinger equations with
the optical $K^{-}-d$ and Coulomb potentials [13].

Table 2: $1s$ level shift $\Delta E_{1s}^{K^{-}d}$ (eV) and width $\Gamma_{1s}^{K^{-}d}$ (eV) of kaonic deuterium calculated using the ”corrected Deser”, ”summed up Deser” formulas and using $K^{-}d$ optical potentials together with the exact results. | Corrected | Summed up | Exact |  $K^{-}d$ optical
---|---|---|---|---
| Deser | Deser |  | potential
| $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$ | $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$ | $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$ | $\Delta E_{1s}^{K^{-}p}$ | $\Gamma_{1s}^{K^{-}p}$
$V_{\bar{K}N}^{\rm 1,SIDD}$ | -826 | 731 | -792 | 921 | -767 | 928 | -785 | 1018
$V_{\bar{K}N}^{\rm 2,SIDD}$ | -835 | 727 | -800 | 923 | -782 | 938 | -797 | 1025
$V_{\bar{K}N}^{\rm Chiral}$ | -876 | 724 | -836 | 951 | -835 | 1004 | -828 | 1055

It is seen from Table 2 that in this three-body case ”summed up Deser” result
is slightly worse in reproducing the exact results: $0.1-3.3\%$ for the shift
and $0.8-5.3\%$ for the width, - but is much better than the results of
”corrected Deser” formula. As for the ”$K^{-}d$ optical potential” method, it
gives comparable accuracy for the $1s$ shift ($0.8-2.4\%$), but is worse for
the width ($5.1-9.7\%$) than the ”summed up Deser”.

Keeping all the above in mind, I suggest to use ”summed up” Deser formula
Eq.(2) for evaluation $1s$ level shift and width of kaonic hydrogen and kaonic
deuterium from $K^{-}p$ and $K^{-}d$ scattering length correspondingly. Its
accuracy is better than $2\%$ for the two-body $K^{-}p$ system and $6\%$ for
the three-body system.

###### Acknowledgements.

I am thankful to M.Mai for drawing my attention to existence of the summed up
version of the formula. The work was supported by the Czech GACR grant
19-19640S.

## References

  * [1] S. Deser et al., Phys. Rev. 96, 774 (1954).
  * [2] U.-G. Meißner, U. Raha, A. Rusetsky, Eur. Phys. J. C 35, 349 (2004).
  * [3] J. Révai, N.V. Shevchenko, Phys. Rev. C 79, 035202 (2009).
  * [4] A. Cieplý, J. Smejkal, Eur. Phys. J. A 34, 237 (2007).
  * [5] V. Baru, E. Epelbaum, A. Rusetsky, Eur. Phys. J. A 42, 111 (2009).
  * [6] M. Miliucci et al. (SIDDHARTA-2 collaboration), Springer Proc. Phys. 238, 965 (2020).
  * [7] T. Hashimoto et al., JPS Conf. Proc. 26, 023013 (2019).
  * [8] M. Mai et al., Phys. Rev. D 91, 054016 (2015).
  * [9] J. Révai, Few-Body Syst. 59 (2018).
  * [10] T. Hoshino et al., Phys. Rev. C 96, 045204 (2017).
  * [11] N.V. Shevchenko, Few-Body Syst 58,1 (2017).
  * [12] J. Révai, Phys. Rev. C 94, 054001 (2016).
  * [13] N.V. Shevchenko, Nucl. Phys. A 890-891, 50 (2012).
  * [14] N. V. Shevchenko, J. Révai, Phys. Rev. C 90, 034003 (2014).
  * [15] Bazzi, M. et al. (SIDDHARTA Collaboration), Phys. Lett. B 704, 113 (2011).
  * [16] N.V. Shevchenko, Phys. Rev. C 85, 034001 (2012).


# CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual
Explanation Algorithms

Martin Pawelczyk
University of Tübingen
<EMAIL_ADDRESS>
&Sascha Bielawski
University of Tübingen
<EMAIL_ADDRESS>
&Johannes van den Heuvel
University of Tübingen
<EMAIL_ADDRESS>
&Tobias Richter
CarePay International
<EMAIL_ADDRESS>
&Gjergji Kasneci 22footnotemark: 2
University of Tübingen
<EMAIL_ADDRESS>
Corresponding authorEqual senior author contribution

###### Abstract

Counterfactual explanations provide means for prescriptive model explanations
by suggesting actionable feature changes (e.g., increase income) that allow
individuals to achieve favourable outcomes in the future (e.g., insurance
approval). Choosing an appropriate method is a crucial aspect for meaningful
counterfactual explanations. As documented in recent reviews, there exists a
quickly growing literature with available methods. Yet, in the absence of
widely available open–source implementations, the decision in favour of
certain models is primarily based on what is readily available. Going forward
– to guarantee meaningful comparisons across explanation methods – we present
CARLA (Counterfactual And Recourse LibrAry), a python library for benchmarking
counterfactual explanation methods across both different data sets and
different machine learning models. In summary, our work provides the following
contributions: (i) an extensive benchmark of 11 popular counterfactual
explanation methods, (ii) a benchmarking framework for research on future
counterfactual explanation methods, and (iii) a standardized set of integrated
evaluation measures and data sets for transparent and extensive comparisons of
these methods. We have open sourced CARLA and our experimental results on
Github, making them available as competitive baselines. We welcome
contributions from other research groups and practitioners.

##  1 Introduction

Machine learning (ML) methods have found their way into numerous everyday
applications and have become an indispensable asset in various sensitive
domains, like disease diagnostics [14], criminal justice [5], or credit risk
scoring [30]. While ML models bear the great potential to provide effective
support in human decision making processes, their predictions may have
considerable impact on personal lives, where the final decisions might be
disadvantageous for an end user. For example, the rejection of a loan or the
denial of parole might have negative effects on the future development of the
corresponding person’s life.

When ML systems involve humans in the loop, it is crucial to build a strong
foundation for long-term acceptance of these methods. To this end, it is
critical (1) to _explain_ the predictions of a model and (2) to _offer
constructive means for the improvement_ of those predictions to the advantage
of the end–user. Counterfactual explanations – popularized by the seminal work
of [63] – provide means for prescriptive model explanations by suggesting
actionable feature changes (e.g., increase income) that allow individuals to
achieve favourable outcomes in the future (e.g., insurance approval).

When counterfactual explainability is employed in systems that involve humans
in the loop, the community refers to it as _recourse_. Algorithmic recourse
subsumes precise recipes on how to obtain desirable outcomes after being
subjected to an automated decision, emphasizing feasibility constraints that
have to be taken into account. Those explanations are found by making the
smallest possible change to an input vector to influence the prediction of a
pretrained classifier in a positive way; for example, from ‘loan denial’ to
‘loan approval’, subject to the constraint that an individual’s sex may not
change. As documented in recent reviews, there exists a quickly growing
literature with available methods (see Figure 1 and [55, 25, 61]), reflecting
the insight that the understanding of complex machine learning models is an
elementary ingredient for a wide and safe technology adoption.

Figure 1: ArXiv submissions over time on explainability, counterfactual
explanations and algorithmic recourse.

In practice, the counterfactual explanation (CE) that an individual receives
crucially depends on the method that computes the recourse suggestions. Hence,
there is a substantial need for a standardized benchmarking platform, which
ensures that methods can be compared in a transparent and meaningful way.
Researchers need to be able to easily evaluate their proposed methods against
the overwhelming diversity of already available methods and practitioners need
to make sure that they are using the right recourse mechanism for the problem
at hand. Therefore, a standardized framework for comparison and quality
assurance is an essential and indispensable prerequisite.

In this work, we present CARLA (Counterfactual And Recourse LibrAry), a python
library with the following merits: First, CARLA provides competitive baselines
for researchers to benchmark new counterfactual explanation and recourse
methods for the standardized and transparent comparison of CE methods on
different integrated data sets. Second, CARLA is a common framework with more
than 10 counterfactual explanation methods in combination with the possibility
to easily integrate new methods into a commonly accessible and easily
distributable Python library. Moreover, the built-in integrated evaluation
measures allow users to plug-in their custom black-box predictive models into
the available counterfactual explanation methods and conduct extensive
evaluations in comparison with other recourse mechanisms across different data
sets. The same is true for researchers, who can use CARLA to extensively
benchmark available counterfactual methods on popular data sets across various
ML models. Third, CARLA supports popular optimization frameworks such as
Tensorflow [2] and PyTorch [44], and provides a generic abstraction layer to
support custom implementations. Users can can define problem–specific data set
characteristics like immutable features and explicitly specify hyperparameters
for the chosen counterfactual explanation method.

The remainder of this work is structured as follows: Section 2 presents
related work, Section 3 formally introduces the recourse problem, Section 4
presents the benchmarking process. In Section 5, we describe our main
findings, before concluding in Section 6. Appendices A \- E describe CARLA’s
software architecture and usage instructions, as well as additional
experimental results, used ML classifiers, data sets and hyperparameters
settings.

##  2 Related Work

Explainable machine learning is concerned with the problem of providing
explanations for complex ML models. Towards this goal, various streams of
research follow different explainability paradigms which can be categorized
into the following groups [18, 15].

###  2.1 Feature Highlighting Explanations

Local input attribution techniques seek to explain the behaviour of ML models
instance by instance. Those methods aim to understand how all inputs available
to the model are being used to arrive at a certain prediction. Some popular
approaches for model explanations aim at explainability by design [35, 3, 6,
64]. For white-box models – the internal model parameters are known –
gradient-based approaches, e.g. [28, 7] (for deep neural networks), and rule-
based or probabilistic approaches for tree ensembles, e.g. [20, 10] have been
proposed. In cases where the parameters of the complex models cannot be
accessed, model-agnostic approaches can prove useful. This group of approaches
seeks to explain a model’s behavior locally by applying surrogate models [51,
36, 52, 37], which are interpretable by design and are used to explain
individual predictions of black-box ML models.

###  2.2 Counterfactual Explanations

The main purpose of counterfactual explanations is to suggest constructive
interventions to the input of a complex model so that the output changes to
the advantage of an end user. By emphasizing both the feature importance and
the recommendation aspect, counterfactual explanation methods can be further
divided into three different groups: independence-based, dependence-based, and
causality-based approaches.

In the class of independence-based methods, where the input features of the
predictive model are assumed to be independent, some approaches use
combinatorial solvers or evolutionary algorithms to generate recourse in the
presence of feasibility constraints [58, 53, 50, 24, 29, 9]. Notable
exceptions from this line of work are proposed by [57, 33, 32, 19, 16], who
use decision trees, random search, support vector machines (SVM) and
information networks that are aligned with the recourse objective. Another
line of research deploys gradient-based optimization to find low-cost
counterfactual explanations in the presence of feasibility and diversity
constraints [11, 39, 40, 54, 60, 47]. The main problem with these approaches
is that they abstract from input correlations. That implies that the
intervention costs (i.e., the costs of changing the input to achieve the
proposed counterfactual state) are too optimistically estimated. In other
words, the estimated costs do not reflect the true costs that an individual
would need to incur in practical scenarios, where feature dependencies are
usually present: e.g., _income_ is dependent on _tenure_ , and if _income_
changes, _tenure_ also changes (see Figure 2 for a schematic comparison).

In the class of causality-based approaches, all methods make use of Pearl’s
causal modelling framework [48]. As such, they usually require knowledge of
the system of causal structural equations [21, 17, 26, 43] or the causal graph
[27]. The authors of [26] show that these models can generate minimum-cost
recourse, if the access to the true causal data generating process was
available. However, in practical scenarios, the guarantee for such minimum-
cost recommendations is vacuous, since, in complex settings, the causal model
is likely to be miss-specified [27]. Since these methods usually require the
_true_ causal graph – which is the limiting factor in practice – we have not
considered them at this point, but we plan to do that in the future.

Dependence-based methods bridge the gap between the strong independence
assumption and the strong causal assumption. This class of models builds
recourse suggestions on generative models [45, 12, 21, 38, 46]. The main idea
is to change the geometry of the intervention space to a lower dimensional
latent space, which encodes different factors of variation while capturing
input dependencies. To this end, these methods primarily use variational
autoencoders (VAE) [31, 42]. In particular, Mahajan et al. [38] demonstrate
how to encode various feasibility constraints into VAE-based models. Most
recently, [4] proposed CLUE, a generative recourse model that takes a
classifier’s uncertainty into account. Work that deviates from this line of
research was done by [49, 23]. The authors of [49] provide FACE, which uses a
shortest path algorithm on graphs to find counterfactual explanations. In
contrast, Kanamori et al. [23] use integer programming techniques to account
for input dependencies.

##  3 Preliminaries

In this Section, we review the algorithmic recourse problem and draw a
distinction between two observational (i.e., non–causal) methods.

$X_{1}$$X_{2}$${\color[rgb]{1,0.39,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.39,0.13}\pgfsys@color@cmyk@stroke{0}{0.61}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.61}{0.87}{0}X_{3}}$$X_{1}$${\color[rgb]{1,0.39,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.39,0.13}\pgfsys@color@cmyk@stroke{0}{0.61}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>(a) Indep. recourse intervention

$X_{1}$$X_{2}$${\color[rgb]{1,0.39,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.39,0.13}\pgfsys@color@cmyk@stroke{0}{0.61}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.61}{0.87}{0}X_{3}}$$X_{1}$${\color[rgb]{1,0.39,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.39,0.13}\pgfsys@color@cmyk@stroke{0}{0.61}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>(b) Dep. recourse intervention

Figure 2: Different views on recourse generation. In (2(a)) a change to
$X_{3}$ only impacts $f$ through $X_{3}$ , while in (2(b)) the same change
induces indirect effects on $f$, if $X_{3}$ is correlated with other inputs.

###  3.1 Counterfactual Explanations for Independent Inputs

Let $\mathcal{D}$ be the data set consisting of $N$ input data points,
$\mathcal{D}=\\{(x_{1},y_{1}),\dots,(x_{N},y_{N})\\}$. We denote by
$f:\mathbb{R}^{d}\xrightarrow{}[0,1]$ the fixed classifier for which recourse
is to be determined. We denote the set of outcomes by $y(x)\in\\{0,1\\}$,
where $y=1$ indicates the desirable outcome. Moreover,
$\hat{y}=\mathbb{I}[f(x)>\theta]$ is the predicted class, where
$\mathbb{I}[\cdot]$ denotes the indicator function and $\theta$ is a threshold
(e.g., 0.5). Our goal is to find a set of actionable changes in order to
improve the outcomes of instances $x$, which are assigned an undesirable
prediction under $f$. Moreover, one typically defines a distance measure in
inputs space
$c:\mathbb{R}^{d}\times\mathbb{R}^{d}\xrightarrow{}\mathbb{R}_{+}$. We discuss
typical choices for $c$ in Section 4.

Assuming inputs are pairwise statistically independent, the recourse problem
is defined as follows:

$\displaystyle\begin{split}\delta_{x}^{*}&=\operatorname*{arg\,min}_{\delta_{x}\in\mathcal{A}_{d}}c\big{(}x,\check{x}\big{)}\text{
s.t. }\check{x}=x+\delta_{x},f(x+\delta_{x})>\theta,\end{split}$
($\mathbf{I}$)

where $\mathcal{A}_{d}$ is the set of admissible changes made to the factual
input $x$. For example, $\mathcal{A}_{d}$ could specify that no changes to
sensitive attributes such as age or sex may be made. For example, using the
independent input assumption, existing approaches [58] use mixed-integer
linear programming to find counterfactual explanations. In the next paragraph,
we present a problem formulation that relaxes the strong independence
assumption by introducing generative models.

###  3.2 Recourse for Correlated Inputs

We assume the factual input $x\in\mathcal{X}=\mathbb{R}^{d}$ is generated by a
generative model $g$ such that:

$x=g(z),$

where $z\in\mathcal{Z}=\mathbb{R}^{k}$ are latent codes. We denote the
counterfactual explanation in an input space by $\check{x}=x+\delta_{x}$.
Thus, we have $\check{x}=x+\delta_{x}=g(z+\delta_{z})$. Assuming inputs are
dependent, we can rewrite the recourse problem in ($\mathbf{I}$) to faithfully
capture those dependencies using the generative model $g$:

$\displaystyle\begin{split}&\delta_{z}^{*}=\operatorname*{arg\,min}_{\delta_{z}\in\mathcal{A}_{k}}c\big{(}x,\check{x}\big{)}\text{
s.t. }\check{x}=g(z+\delta_{z}),f(\check{x})>\theta,\end{split}$
($\mathbf{D}$)

where $\mathcal{A}_{k}$ is the set of admissible changes in the
$k$-dimensional latent space. For example, $\mathcal{A}_{k}$ would ensure that
the counterfactual latent space lies within range of $z$. The problem in
($\mathbf{D}$) is an abstraction from how the problem is usually solved in
practice: most existing approaches first train a type of autoencoder model
(e.g., a VAE), and then use the model’s trained decoder as a deterministic
function $g$ to find counterfactual explanations [21, 45, 38, 12, 4]. Our
benchmarked explanation models roughly fit in one of these two categories.

(a) Adult Data

(b) Give Me Some Credit Data

Figure 3: Evaluating the distribution of costs of counterfactual explanations
on 2 different data sets (the results on COMPAS are relegated to Appendix B).
For all instances with a negative prediction
($\\{x\in\mathcal{D}:f(x)<\theta\\}$), we plot the distribution of $\ell_{0}$
and $\ell_{1}$ costs of algorithmic recourse as defined in (1) for a logistic
regression and an artificial neural network classifier. The white dots
indicate the medians (lower is better), and the black boxes indicate the
interquartile ranges. We distinguish between independence based and dependence
based methods. The results are discussed in Section 5

##  4 Benchmarking Process

In this Section, we provide a brief explanation model overview and introduce a
variety of explanation measures used to evaluate the quality of the generated
counterfactual explanations. In Table 1 we present a concise explanation model
overview.

Approach | Method | Model Type | Algorithm | Immutable | Categorical | Other
---|---|---|---|---|---|---
Independent (I) | AR | Linear | Integer Prog. | Yes | Binary | Direction of change
AR--LIME | Agnostic | Integer Prog. | Yes | Binary | Direction of change
CEM | Gradient based | Gradient based | No | No | None
DICE | Gradient Based | Gradient based | Yes | Binary | Generative model
GS | Agnostic | Random search | Yes | Binary | None
Wachter | Gradient based | Gradient based | No | Binary | None
Dependent (D) | CEM-VAE | Gradient based | Gradient based | No | No | Gen. Model regularizer
CLUE | Gradient based | Gradient based | No | No | Generative model
FACE--EPS | Agnostic | Graph search | Binary | Binary | CE is from data set
FACE--KNN | Agnositc | Graph search | Binary | Binary | CE is from data set
REVISE | Gradient based | Gradient based | Binary | Binary | Generative model

Table 1: Explanation method summary: we categorize different approaches based
on their underlying assumptions and list what kind of ML model they work with
(Model Type), the Method’s underlying algorithm (Algorithm), whether the
method can handle immutable features (Immutable), whether it can handle
categorical features (Categorical) and any other outstanding characteristics
(Other).

###  4.1 Counterfactual Explanation Methods

#### AR $\mathbf{(I)}$

Ustun et al. [59] provide a method to generate minimal cost actions
$\delta^{*}_{x}$ for linear classification models such as logistic regression
models. AR requires the linear model’s coefficients, and uses these
coefficients for its search for counterfactual explanations. To provide
reasonable actions it is possible to restrict $\delta_{x}^{*}$ to
_user–specified constraints_ (e.g., has_phd can only change from False to
True) or to set a subset of inputs as _immutable_ (e.g., age). The problem to
find these changes is a discrete optimization problem. Given a set of actions,
AR finds the action which minimizes a defined cost function, using integer
programming solvers like CPLEX or CBC.

#### AR--LIME $\mathbf{(I)}$

Most classification tasks do not have linearly separable classes and complex
non–linear models usually provide more accurate predictions. Non–linear models
are not per se interpretable and usually do not provide coefficients similar
to linear models. We use a reduction to apply AR to non–linear models by
computing a local linear approximation for the point of interest $x$, using
LIME [51]. For an arbitrary black–box model $f$, LIME estimates post–hoc local
explanations in form of a set of linear coefficients per instance. Using the
coefficients we apply AR.

#### CEM $\mathbf{(I)}$

Dhurandhar et al. [11] use an elastic–net regularization inspired objective to
find low-cost counterfactual instances. Different weights can be assigned to
$\ell_{1}$ and $\ell_{2}$ norms, respectively. There exists no immutable
feature handling. However, we provide support for their VAE type regularizer,
which should help ensure that counterfactual instances look more realistic.

#### CLUE $\mathbf{(D)}$

Antorán et al. [4] propose CLUE, a generative recourse model that takes a
classifier’s uncertainty into account. This model suggests feasible
counterfactual explanations that are likely to occur under the data
distribution. The authors use a variational autoencoder (VAE) to estimate the
generative model. Using the VAE’s decoder, CLUE uses an objective that guides
the search of CEs towards instances that have low uncertainty measured in
terms of the classifier’s entropy.

#### DICE $\mathbf{(I)}$

Mothilal et al. [41] suggest DICE, which is an explanation model that seeks to
generate minimum costs counterfactual explanations according to ($\mathbf{I}$)
subject to a diversity constraint which aims to promote a diverse set of
counterfactual explanations. Diversity is achieved by using the whole range of
suggested changes, while still keeping proximity to a given input. Regarding
the optimization problem, DICE uses gradient descent to find a solution that
trades-off proximity and diversity. Domain knowledge – in form of feature
ranges or immutability constraints – can be added.

#### FACE $\mathbf{(D)}$

The authors of [49] provide FACE, which uses a shortest path algorithm (for
graphs) to find counterfactual explanations from high–density regions. Those
explanations are actual data points from either the training or test set.
Immutability constraints are enforced by removing incorrect neighbors from the
graph. We implemented two variants of this model: the first variant uses an
epsilon–graph (FACE--EPS), whereas the second variant uses a knn–graph (FACE--
KNN).

#### Growing Spheres (GS) $\mathbf{(I)}$

Growing Spheres – suggested in [33] – is a random search algorithm, which
generates samples around the factual input point until a point with a
corresponding counterfactual class label was found. The random samples are
generated around $x$ using growing hyperspheres. For binary input dimensions,
the method makes use of Bernoulli sampling. Immutable features are readily
specified by excluding them from the search procedure.

#### REVISE $\mathbf{(D)}$

Joshi et al. [21] propose a generative recourse model. This model suggests
feasible counterfactual explanations that are likely to occur under the data
distribution. The authors use a variational autoencoder (VAE) to estimate the
generative model. Using the VAE’s decoder, REVISE uses the latent space to
search for CEs. No handling of immutable features exists.

#### Wachter et al. (Wachter) $\mathbf{(I)}$

The optimization approach suggested by Wachter et al. [62] generates
counterfactual explanations by minimizing an objective function using gradient
descent to find counterfactuals $\check{x}$ which are as close as possible to
$x$. Closeness is measured in $\ell_{1}$-norm.

| Artificial Neural Network | Logistic Regression
---|---|---
Data Set | Method | yNN | redund. | violation | success | $\overline{t}(s)$ | yNN | redund. | violation | success | $\overline{t}(s)$
Adult | AR(--LIME) | 0.62 | 0.00 | 0.14 | 0.28 | 1.59 | 0.72 | 0.67 | 0.13 | 0.52 | 10.49
CEM | 0.26 | 3.96 | 0.66 | 1.00 | 1.10 | 0.20 | 3.98 | 0.66 | 1.00 | 0.92
DICE | 0.71 | 0.53 | 0.17 | 1.00 | 0.13 | 0.58 | 0.51 | 0.23 | 1.00 | 0.13
GS | 0.30 | 3.77 | 0.09 | 1.00 | 0.01 | 0.30 | 3.94 | 0.10 | 1.00 | 0.01
Wachter | 0.23 | 4.45 | 0.83 | 0.50 | 15.72 | 0.16 | 1.67 | 0.94 | 1.00 | 0.03
GMC | AR(--LIME) | 0.89 | 0.00 | 0.29 | 0.07 | 0.55 | 1.00 | 2.33 | 0.14 | 0.39 | 3.42
CEM | 0.95 | 5.46 | 0.65 | 1.00 | 0.97 | 0.74 | 5.07 | 0.67 | 1.00 | 0.87
DICE | 0.90 | 0.58 | 0.27 | 1.00 | 0.28 | 0.88 | 0.61 | 0.27 | 1.00 | 0.29
GS | 0.40 | 6.64 | 0.17 | 1.00 | 0.01 | 0.49 | 5.29 | 0.17 | 1.00 | 0.01
Wachter | 0.58 | 6.56 | 0.71 | 1.00 | 0.02 | 0.59 | 6.12 | 0.83 | 1.00 | 0.01

(a) Independence based methods

| Artificial Neural Network | Logistic Regression
---|---|---
Data Set | Method | yNN | redund. | violation | success | $\overline{t}(s)$ | yNN | redund. | violation | success | $\overline{t}(s)$
Adult | CEM--VAE | 0.12 | 9.68 | 1.82 | 1.00 | 0.93 | 0.43 | 10.05 | 1.80 | 1.00 | 0.81
CLUE | 0.82 | 8.05 | 1.28 | 1.00 | 2.70 | 0.33 | 7.30 | 1.33 | 1.00 | 2.56
FACE--EPS | 0.65 | 5.19 | 1.45 | 0.99 | 4.36 | 0.64 | 5.11 | 1.44 | 0.94 | 4.35
FACE--KNN | 0.60 | 5.11 | 1.41 | 1.00 | 4.31 | 0.57 | 4.97 | 1.38 | 1.00 | 4.31
REVISE | 0.20 | 8.65 | 1.33 | 1.00 | 8.33 | 0.62 | 7.92 | 1.23 | 1.00 | 7.52
GMC | CEM--VAE | 1.00 | 8.40 | 0.66 | 1.00 | 0.87 | 1.00 | 8.54 | 0.36 | 1.00 | 0.88
CLUE | 1.00 | 9.39 | 0.90 | 0.93 | 1.91 | 1.00 | 9.56 | 0.96 | 1.00 | 1.76
FACE--EPS | 0.99 | 8.06 | 0.99 | 1.00 | 19.44 | 0.98 | 7.98 | 0.96 | 1.00 | 19.50
FACE--KNN | 0.98 | 9.00 | 0.98 | 1.00 | 15.87 | 0.98 | 7.88 | 0.95 | 1.00 | 16.09
REVISE | 1.00 | 9.50 | 0.97 | 1.00 | 4.56 | 1.00 | 9.59 | 0.96 | 1.00 | 3.76

(b) Dependence based methods

Table 2: Summary of a subset of results for independence and dependence based
methods. For all instances with a negative prediction
($\\{x\in\mathcal{D}:f(x)<\theta\\}$), we compute counterfactual explanations
for which we then measure yNN (higher is better), redundancy (lower is
better), violation (lower is better), success rate (higher is better) and time
(lower is better). We distinguish between a logistic regression and an
artificial neural network classifier. Detailed descriptions of these measures
can be found in Section 4. The results are discussed in Section 5.

###  4.2 Evaluation Measures for Counterfactual Explanation Methods

As algorithmic recourse is a multi–modal problem we introduce a variety of
measures to evaluate the methods’ performances. We use six baseline evaluation
measures. Besides distance measures it is important to consider measures that
emphasize the _quality_ of recourse.

#### Costs

When answering the question of generating the nearest counterfactual
explanation, it is essential to define the distance of the factual $x$ to the
nearest counterfactual $\check{x}$. The literature has formed a consensus to
use either the normalized $\ell_{0}$ or $\ell_{1}$ norm or any convex
combination thereof (see for example [50, 40, 46, 24, 58, 63]). The $\ell_{0}$
norm puts a restriction on the number of feature changes between factual and
counterfactual instance, while the $\ell_{1}$ norm restricts the average
change:

$\displaystyle c_{0}(\check{x},x)$ $\displaystyle=\frac{1}{d}\lVert
x-\check{x}\rVert_{0}=\frac{1}{d}\lVert\delta_{x}\rVert_{0},$ $\displaystyle
c_{1}(\check{x},x)$ $\displaystyle=\frac{1}{d}\lVert
x-\check{x}\rVert_{1}=\frac{1}{d}\lVert\delta_{x}\rVert_{1}.$ (1)

#### Constraint violation

This measure counts the number of times the CE method violates user-defined
constraints. Depending on the data set, we fixed a list of features which
should not be changed by the used method (e.g., _sex_ , _age_ or _race_).

#### yNN

We use a measure that evaluates how much data support CEs have from positively
classified instances. Ideally, CEs should be close to positively classified
individuals which is a desideratum formulated by Laugel et al. [34]. We define
the set of individuals who received an undesirable prediction under $f$ as
$H^{-}:=\\{x\in\mathcal{D}:f(x)<\theta\\}$. The counterfactual instances
(instances for which the label was successfully changed) corresponding to the
set $H^{-}$ are denoted by $\check{H}^{-}$. We use a measure that captures how
differently neighborhood points around a counterfactual instance $\check{x}$
are classified:

$\text{yNN}=1-\frac{1}{nk}\sum_{i\in\check{H}^{-}}^{n}\sum_{j\in\text{kNN}(\check{x}_{i})}\big{|}f_{b}(\check{x}_{i})-f_{b}(x_{j})\big{|},$
(2)

where kNN denotes the $k$-nearest neighbours of $x$, and
$f_{b}(x)=\mathbb{I}[f(x)>0.5]$ is the binarized classifier. Values of yNN
close to 1 imply that the neighbourhoods around the counterfactual
explanations consists of points with the same predicted label, indicating that
the neighborhoods around these points have already been reached by positively
classified instances. We use a value of $k:=5$, which ensures sufficient data
support from the positive class.

#### Redundancy

We evaluate how many of the proposed feature changes were not necessary. This
is a particularly important criterion for independence–based methods. We
measure this by _successively_ flipping one value of $\check{x}$ after another
back to $x$, and then we inspect whether the label flipped from $1$ back to
$0$: e.g., we check whether flipping the value for the second dimension would
change the counterfactual outcome $1$ back to the predicted factual outcome of
$0$:
$\mathbb{I}[f_{b}\big{(}[\check{x}_{1},x_{2},\check{x}_{3},\dots,\check{x}_{d}]\big{)}=0]$.
If the predicted outcome does not change, we increase the redundancy counter,
concluding that a sparser counterfactual explanation could have been found. We
iterate this process over all dimensions of the input vector.111We do not
consider all possible subsets of changes. A low number indicates few
redundancies across counterfactual instances.

#### Success Rate

Some generated counterfactual explanations do not alter the predicted label of
the instance as anticipated. To keep track how often the generated CE does
hold its promise, the success rate shows the fraction of respective models’
correctly determined counterfactuals.

#### Average Time

By measuring the average time a CE method needs to generate its result, we
evaluate the effectiveness and feasibility for real–time prediction settings.

##  5 Experimental Evaluation

Using CARLA we conduct extensive empirical evaluations to benchmark the
presented counterfactual explanations methods using three real-world data
sets. Our main findings are displayed in Figure 3, and Table 2. We split the
benchmarking evaluation by CE method category. In the following Sections, we
provide an overview over the used data sets (see Table 3) and the
classification models. Detailed information on hyperparameter search for the
CE methods is provided in Appendix E.

#### Data sets

The Adult data set [13] originates from the 1994 Census database, consisting
of 14 attributes and 48,842 instances. The classification consists of deciding
whether an individual has an income greater than 50,000 USD/year. Since
several CE methods cannot handle non-binary categorical data, we binarized
these features by partitioning them into the most frequent value, and its
counterpart (e.g., _US_ and _Non-US_ , _Husband_ and _Non-Husband_). The
features age, sex and race are set as immutable. The Give Me Some Credit (GMC)
data set [8] from a 2011 Kaggle Competition is a credit scoring data set,
consisting of 150,000 observations and 11 features. The classification task
consists of deciding whether an instance will experience financial distress
within the next two years (SeriousDlqin2yrs is 1) or not. We dropped missing
data, and set age as immutable.

Data Set | Task | Positive Class | Size $(N\;|\;d)$ | Features | Immutable Features
---|---|---|---|---|---
Adult | Predict Income | High Income (24%) | (45,222 | 20) | Work, Education, Income | Sex, Age, Race
COMPAS | Predict Recidivism | No Recid. (65%) | (10,000 | 8) | Crim. History, Jail & Prison Time | Sex, Race
GMC | Predict Financial Distress | No deficiency (93%) | (150,000 | 11) | Pay. History, Balance, Loans | Age

Table 3: Summarizing the used data sets, where $N$ and $d$ are the number of
samples and input dimension after processing of the data. Results on the
COMPAS data set are relegated to Appendix B.

#### Black-box models

We briefly describe how the black–box classifiers $f$ were trained. CARLA
supports different ML libraries (e.g., Pytorch, Tensorflow) to estimate these
classifiers as the implementations of the various explanation methods work
particular ML libraries only. The first model is a multi-layer perceptron,
consisting of three hidden layers with 18, 9 and 3 neurons, respectively. To
allow a more extensive comparison (AR only works on linear models) between CE
methods, we chose logistic regression models as the second classification
model for which we evaluate the CE methods. Detailed information on the
classifiers’ training for each data set is provided in Appendix C.

#### Benchmarking

For independence based methods, we find that no one single CE method
outperformed all its competitors. This is not too surprising since algorithmic
recourse is a multi–modal problem. Instead, we found that some methods
dominated certain measures across all data sets. AR, AR--LIME, DICE performed
strongest with respect to $\ell_{0}$ (see the top left panels in Figures 3(a)
and 3(b)). AR--LIME does so despite our use of the LIME reduction. Therefore,
it makes sense that AR, AR-LIME and DICE offer the lowest _redundancy_ scores
(Table 2(a)). CEM performed strongest with respect to the overall cost measure
$\ell_{1}$ across data sets. GS is the clear winner when it comes to the
measurement of time (Table 2(a)). Since the algorithm behind GS is based on a
rather rudimentary sampling strategy, we expect that savvier sampling
strategies should boost its cost performance significantly.

For dependence based methods, the results are mixed as well. While CLUE and
REVISE are the winner with respect to the cost of recourse ($\ell_{1}$), the
margins between these generative recourse models and the graph-based ones
(FACE) are small (Figure 3). The FACE-EPS method performs strongest with
respect to the $ynn$ measure (usually well above 0.60) (Table 2(b)) indicating
that the generated CEs have sufficient data support from positively classified
individuals relatively to the remaining dependence–based methods. As expected,
the ynn measures are on average higher for the dependence based methods. This
suggest that dependence based CEs are less often outliers. Notably, CLUE and
REVISE perform best with respect to $\ell_{1}$ (with REVISE being the clear
winner on 3 out of 4 cases), while they perform worst on $\ell_{0}$ – likely
due to the decoder’s imprecise reconstruction. In this respect, it is not
surprising that these methods have average redundancy values that are up to
twice as high as those by FACE. Finally, the generative model approaches (CEM-
VAE, CLUE, REVISE) performed best with respect to time since the autoencoder
training time amortizes with more samples.

##  6 Conclusion and Broader Impact of CARLA

The current implementations of recourse methods, mentioned in Section 4.1 are
based on the original implementation of the respective research groups.
Researchers mostly implement their experiments and models for specific ML
frameworks and data sets. For example, some explanation methods are restricted
to Tensorflow and are not applicable to Pytorch models. In the future, we will
extend CARLA to decouple each recourse method from the frameworks and data
contraints.

When trying to combine different CE methods into a common benchmarking
framework we encountered the following issues: First, a great number of
repositories only contain remarks about installation and script calls to
recreate the results from the corresponding research papers. Second, missing
information about interfaces for data sets or black–box models further
complicated the process of integrating different CE methods into the
benchmarking workflow. In order to add more CE methods and data sets to CARLA,
we are currently in contact with several authors in this exciting and rapidly
growing field. With a growing open-source community, CARLA can evolve to be
the main library for generating counterfactual explanations and benchmarks for
recourse methods. Therefore we are continuously expanding the catalog of
explanation methods and data sets, and welcome researchers to add their own
recourse methods to the library. To facilitate this process, we provide a
step-by-step user-guide to integrate new CE methods into CARLA, which we
present in Appendix A.

The rapidly growing number of available CE methods calls for standardized and
efficient ways to assure the quality of a new technique in comparison with
other approaches on different data sets. Quality assurance is a key aspect of
actionable recourse, since complex models and CE mechanisms can have a
considerable impact on personal lives. In this work, we presented CARLA, a
versatile benchmarking platform for the standardized and transparent
comparison of CE methods on different integrated data sets. In the
explainability field, CARLA bears the potential to help researchers and
practitioners alike to efficiently derive more realistic and use–case–driven
recourse strategies and assure their quality through extensive comparative
evaluations. We hope that this work contributes to further advances in
explainability research.

## References

  * [1]
  * Abadi et al. [2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: A system for large-scale machine learning. In _12th $\\{$USENIX$\\}$_.
  * Alvarez-Melis and Jaakkola [2018] David Alvarez-Melis and Tommi S Jaakkola. 2018.  Towards robust interpretability with self-explaining neural networks. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Antorán et al. [2021] Javier Antorán, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2021\.  Getting a CLUE: A Method for Explaining Uncertainty Estimates. In _International Conference on Learning Representations (ICLR)_.
  * Berk [2012] Richard Berk. 2012\.  _Criminal justice forecasts of risk: A machine learning approach_.  Springer Science & Business Media.
  * Broelemann and Kasneci [2019] Klaus Broelemann and Gjergji Kasneci. 2019.  A gradient-based split criterion for highly accurate and transparent model trees. In _Proceedings of the International Joint Conference on Artificial Intelligence IJCAI_.
  * Chattopadhay et al. [2018] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. 2018\.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE.
  * Competition [[n.d.]] Kaggle Competition. [n.d.].  "Give Me Some Credit. Improve on the state of the art in credit scoring by predicting the probability that somebody will experience financial distress in the next two years".
  * Dandl et al. [2020] Susanne Dandl, Christoph Molnar, Martin Binder, and Bernd Bischl. 2020.  Multi-objective counterfactual explanations. In _International Conference on Parallel Problem Solving from Nature_. Springer, 448–469.
  * Deng [2019] Houtao Deng. 2019\.  Interpreting tree ensembles with intrees.  _International Journal of Data Science and Analytics_ (2019).
  * Dhurandhar et al. [2018] Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018.  Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives. In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Downs et al. [2020] Michael Downs, Jonathan L. Chu, Yaniv Yacoby, Finale Doshi-Velez, and Weiwei Pan. 2020.  CRUDS: Counterfactual Recourse Using Disentangled Subspaces.  _ICML Workshop on Human Interpretability in Machine Learning (WHI 2020)_ (2020).
  * Dua and Graff [2017] Dheeru Dua and Casey Graff. 2017.  UCI Machine Learning Repository.
  * Fatima et al. [2017] Meherwar Fatima, Maruf Pasha, et al. 2017\.  Survey of machine learning algorithms for disease diagnostic.  _Journal of Intelligent Learning Systems and Applications_ (2017).
  * Gade et al. [2019] Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly. 2019\.  Explainable AI in industry. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_.
  * Ghazimatin et al. [2020] Azin Ghazimatin, Oana Balalau, Rishiraj Saha Roy, and Gerhard Weikum. 2020.  PRINCE: provider-side interpretability with counterfactual explanations in recommender systems. In _Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM)_.
  * Goyal et al. [2019] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. 2019\.  Explaining classifiers with causal concept effect (cace).  _arXiv preprint arXiv:1907.07165_ (2019).
  * Guidotti et al. [2018] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A survey of methods for explaining black box models.  (2018).
  * Gupta et al. [2019] Vivek Gupta, Pegah Nokhiz, Chitradeep Dutta Roy, and Suresh Venkatasubramanian. 2019\.  Equalizing recourse across groups.  _arXiv preprint arXiv:1909.03166_ (2019).
  * Hara and Hayashi [2018] Satoshi Hara and Kohei Hayashi. 2018.  Making tree ensembles interpretable: A bayesian model selection approach. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_. PMLR.
  * Joshi et al. [2019] Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019.  Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.  _arXiv preprint arXiv:1907.09615_ (2019).
  * Julia Angwin and Kirchner [2016] Surya Mattu Julia Angwin, Jeff Larson and Lauren Kirchner. 2016.  Machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks".
  * Kanamori et al. [2020] Kentaro Kanamori, Takuya Takagi, Ken Kobayashi, and Hiroki Arimura. 2020.  DACE: Distribution-Aware Counterfactual Explanation by Mixed-Integer Linear Optimization. In _Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI_.
  * Karimi et al. [2020a] Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. 2020a.  Model-Agnostic Counterfactual Explanations for Consequential Decisions. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_.
  * Karimi et al. [2020b] Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. 2020b.  A survey of algorithmic recourse: definitions, formulations, solutions, and prospects.  _arXiv preprint arXiv:2010.04050_ (2020).
  * Karimi et al. [2021] Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. 2021.  Algorithmic Recourse: from Counterfactual Explanations to Interventions. In _Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT)_.
  * Karimi et al. [2020c] Amir-Hossein Karimi, Julius von Kügelgen, Bernhard Schölkopf, and Isabel Valera. 2020c.  Algorithmic recourse under imperfect causal knowledge: a probabilistic approach. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Kasneci and Gottron [2016] Gjergji Kasneci and Thomas Gottron. 2016.  Licon: A linear weighting scheme for the contribution ofinput variables in deep artificial neural networks. In _CIKM_.
  * Kenny and Keane [2020] Eoin M. Kenny and Mark T. Keane. 2020.  On Generating Plausible Counterfactual and Semi-Factual Explanations for Deep Learning.  arXiv:2009.06399 [cs.LG]
  * Khandani et al. [2010] Amir E. Khandani, Adlar J. Kim, and Andrew W. Lo. 2010.  Consumer credit-risk models via machine-learning algorithms.  _Journal of Banking & Finance_ (2010).
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes. In _International Conference on Learning Representations_.
  * Lash et al. [2017] Michael T Lash, Qihang Lin, Nick Street, Jennifer G Robinson, and Jeffrey Ohlmann. 2017.  Generalized inverse classification. In _Proceedings of the 2017 SIAM International Conference on Data Mining_. SIAM.
  * Laugel et al. [2017] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017\.  Inverse Classification for Comparison-based Interpretability in Machine Learning.  _arXiv preprint arXiv:1712.08443_ (2017).
  * Laugel et al. [2019] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2019\.  The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations. In _Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI_.
  * Lou et al. [2012] Yin Lou, Rich Caruana, and Johannes Gehrke. 2012.  Intelligible models for classification and regression. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_.
  * Lundberg and Lee [2017] Scott Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Lundberg et al. [2020] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020\.  From local explanations to global understanding with explainable AI for trees.  _Nature machine intelligence_ (2020).
  * Mahajan et al. [2019] Divyat Mahajan, Chenhao Tan, and Amit Sharma. 2019.  Preserving causal constraints in counterfactual explanations for machine learning classifiers.  _arXiv preprint arXiv:1912.03277_ (2019).
  * Mittelstadt et al. [2019] Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019\.  Explaining explanations in AI. In _Proceedings of the conference on fairness, accountability, and transparency (FAT*)_.
  * Mothilal et al. [2020a] Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. 2020a.  Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. In _Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*)_.
  * Mothilal et al. [2020b] R. K. Mothilal, Amit Sharma, and Chenhao Tan. 2020b.  Explaining machine learning classifiers through diverse counterfactual explanations.
  * Nazabal et al. [2020] Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. 2020.  Handling incomplete heterogeneous data using vaes.  _Pattern Recognition_ (2020).
  * O’Shaughnessy et al. [2020] Matthew O’Shaughnessy, Gregory Canal, Marissa Connor, Mark Davenport, and Christopher Rozell. 2020.  Generative causal explanations of black-box classifiers. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _arXiv preprint arXiv:1912.01703_ (2019).
  * Pawelczyk et al. [2020a] Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. 2020a.  Learning Model-Agnostic Counterfactual Explanations for Tabular Data. In _Proceedings of The Web Conference 2020 (WWW)_. ACM.
  * Pawelczyk et al. [2020b] Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. 2020b.  On Counterfactual Explanations under Predictive Multiplicity. In _Conference on Uncertainty in Artificial Intelligence (UAI)_. PMLR, 809–818.
  * Pawelczyk et al. [2021] Martin Pawelczyk, Shalmali Joshi, Chirag Agarwal, Sohini Upadhyay, and Himabindu Lakkaraju. 2021\.  On the Connections between Counterfactual Explanations and Adversarial Examples.  arXiv:2106.09992 [cs.LG]
  * Pearl [2009] Judea Pearl. 2009\.  _Causality_.  Cambridge university press.
  * Poyiadzi et al. [2020] Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. 2020\.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES)_.
  * Rawal and Lakkaraju [2020] Kaivalya Rawal and Himabindu Lakkaraju. 2020.  Beyond Individualized Recourse: Interpretable and Interactive Summaries of Actionable Recourses. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  Why should i trust you?: Explaining the predictions of any classifier. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_. ACM.
  * Ribeiro et al. [2018] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018\.  Anchors: High-precision model-agnostic explanations. In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Russell [2019] Christopher Russell. 2019\.  Efficient Search for Diverse Coherent Explanations. In _Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*)_.
  * Schut et al. [2021] Lisa Schut, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021\.  Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 1756–1764.
  * Stepin et al. [2021] Ilia Stepin, Jose M Alonso, Alejandro Catala, and Martín Pereira-Fariña. 2021\.  A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence.  _IEEE Access_ (2021).
  * Tieleman and Hinton [2012] Tijmen Tieleman and Geoffrey Hinton. 2012.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ (2012).
  * Tolomei et al. [2017] Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017.  Interpretable predictions of tree-based ensembles via actionable feature tweaking. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_. ACM.
  * Ustun et al. [2019a] Berk Ustun, Alexander Spangher, and Yang Liu. 2019a.  Actionable recourse in linear classification. In _Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*)_.
  * Ustun et al. [2019b] Berk Ustun, Alexander Spangher, and Y. Liu. 2019b.  Actionable Recourse in Linear Classification. In _Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*)_.
  * Van Looveren and Klaise [2019] Arnaud Van Looveren and Janis Klaise. 2019.  Interpretable counterfactual explanations guided by prototypes.  _arXiv preprint arXiv:1907.02584_ (2019).
  * Verma et al. [2020] Sahil Verma, John Dickerson, and Keegan Hines. 2020\.  Counterfactual Explanations for Machine Learning: A Review.  _arXiv preprint arXiv:2010.10596_ (2020).
  * Wachter et al. [2017] Sandra Wachter, B. Mittelstadt, and Chris Russell. 2017\.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.  _Cybersecurity_ (2017).
  * Wachter et al. [2018] Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018\.  Counterfactual explanations without opening the black box: automated decisions and the GDPR.  _Harvard Journal of Law & Technology_ 31, 2 (2018).
  * Wang et al. [2019] Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019\.  Designing theory-driven user-centric explainable AI. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–15.

## Checklist

The checklist follows the references. Please read the checklist guidelines
carefully for information on how to answer these questions. For each question,
change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly
encouraged to include a justification to your answer, either by referencing
the appropriate section of your paper or providing a brief inline description.
For example:

  * •

Did you include the license to the code and datasets? [Yes] See Section
LABEL:gen_inst.

  * •

Did you include the license to the code and datasets? [No] The code and the
data are proprietary.

  * •

Did you include the license to the code and datasets? [N/A]

Please do not modify the questions and only use the provided macros for your
answers. Note that the Checklist section does not count towards the page
limit. In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] . As we state in the abstract, our
goal is to provide a Python framework for benchmarking counterfactual
explanation methods. Users can easily evaluate our results by accessing our
Github repository, where we host our Python framework and our benchmarking
results.

    2. (b)

Did you describe the limitations of your work? [Yes] . In Section 6, we
discuss the current limitations of our approach. The counterfactual
explanation methods are based on the original implementation of the respective
research groups. Researchers mostly implement their experiments and models for
specific ML frameworks and data sets. For example, some explanation methods
are restricted to Tensorflow and are not applicable to Pytorch models.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A] .
We discuss the broader impact of our benchmarking library in Section 6; we
mainly see positive impacts on the literature of algorithmic recourse.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes] . We have read the ethics review guidelines and attest
that our paper conforms to the guidelines.

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A] .
We did not provide theoretical results.

    2. (b)

Did you include complete proofs of all theoretical results? [N/A] . We did not
provide theoretical results.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
. Details of implementations, data sets and instructions can be found here:
Appendices A, C, E, and our Github repository.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] . Please see Appendices E and C.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] . Error bars have been reported for our
cost comparisons in terms of the 25th and 75ht percentiles of the cost
distribution, see for example Figure 3.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] . All models
are evaluated on an i7-8550U CPU with 16 Gb RAM, running on Windows 10.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] . The data
sets, which are publicly available are appropriately cited in Section 5. We
cite and link to any additional code used, for example [4].

    2. (b)

Did you mention the license of the assets? [Yes] . All assets are publicly
available and attributed.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] . Our implementation and code is accessible through our Github
repository.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A] . We use publicly available data sets without any
personal identifying information.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A] . We use publicly
available data sets without any personal identifying information.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A] . We did not use crowdsourcing or conduct
research with human subjects.

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A] . We did not use
crowdsourcing or conduct research with human subjects.

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A] . We did not use crowdsourcing
or conduct research with human subjects.

##  Appendix A CARLA’s Software Interface

In the following, we introduce our open-source benchmarking software CARLA. we
describe the architecture in more detail and provide examples of different
use-cases and their implementation.

###  A.1 CARLA’s High Level Software Architecture

The purpose of this Python library is to provide a simple and standardized
framework to allow users to apply different state-of-the-art recourse methods
to arbitrary data sets and black-box-models. It is possible to compare
different approaches and save the evaluation results, as described in Section
4.2. For research groups, CARLA provides an implementation interface to
integrate new recourse methods in an easy-to-use way, which allows to compare
their method to already existing methods.

Figure 4: Architecture of the CARLA python library. The silver boxes show the
individual objects that will be created to generate counterfactual
explanations and evaluate recourse methods. Useful explanations to specific
processes are illustrated as yellow notes. The dashed arrows are showing the
different implementation possibilities, either use pre-defined catalog objects
or provide custom implementation. All dependencies between these objects are
visualised by solid arrows with an additional description.

A simplified visualization of the CARLA software architecture is depicted in
Figure 4. For every component (_Data_ , _MLModel_ , and _RecourseMethod_) the
library provides the possibility to use existing methods from our catalog, or
extend the users custom methods and implementations. The components represent
an interface to the key parts in the process of generating counterfactual
explanations. Data provides a common way to access the data across the
software and maintains information about the features. MLModel wraps each
black-box model and stores details on the encoding, scaling and feature order
specific to the model. The primary purpose of RecourseMethod is to provide a
common interface to easily generate counterfactual examples.

Besides the possibility to use pretrained black-box-models and preprocessed
data, CARLA provides an easy way to load and define own data sets and model
structures independent of their framework (e.g., Pytorch, Tensorflow,
sklearn). The following sections will give an overview and provide example
implementations of different use cases.

###  A.2 CARLA for Research Groups

One of the most exciting features of CARLA is, that research groups can make
use of the RecourseMethod-wrapper to implement their own method to generate
counterfactual examples. This opens up a way of standardized and consistent
comparisons between different recourse methods. Strong and weak points of new
algorithms can be stated, benchmarked and analysed in forthcoming publications
with the help of CARLA.

In Figure 5, we show how an implementation of a custom recourse method can be
structured. After defining the recourse method in the shown way, it can be
used with the library to generate counterfactuals for a given data set and
benchmark its results against other methods. Research groups have the choice
to do this using our provided catalog of data sets, recourse methods and
black-box models (Figure 6) or use their own models and data sets (see Figures
7 and 8).

⬇

1

2 from carla import RecourseMethod

3

4 # Custom recourse implementations need to

5 # inherit from the RecourseMethod interface

6 class MyRecourseMethod(RecourseMethod):

7 def __init__(self, mlmodel):

8 super().__init__(mlmodel)

9

10 # Generate and return encoded and

11 # scaled counterfactual examples

12 def get_counterfactuals(self, factuals: pd.DataFrame):

13 [...]

14 return counterfactual_examples

Figure 5: Pseudo-implementation of the CARLA recourse method wrapper

###  A.3 CARLA as a Recourse Library

A common usage of the package is to generate counterfactual examples. This can
be done by loading black-box-models and data sets from our provided catalogs,
or by user-defined models and datasets via integration with the defined
interfaces. Figure 6 shows an implementation example of a simple use-case,
applying a recourse method to a pre-defined data set and model from our
catalog. After importing both catalogs, the only necessary step is to describe
the data set name (e.g., adult, give me some credit, or compas) and the model
type (e.g., ann, or linear) the user wants to load. Every recourse method
contains the same properties to generate counterfactual examples.

⬇

1

2 from carla import DataCatalog, MLModelCatalog

3 from carla.recourse_methods import GrowingSpheres

4

5 # 1. Load data set from the DataCatalog

6 data_name = "adult"

7 dataset = DataCatalog(data_name)

8

9 # 2. Load pre-trained black-box model from the MLModelCatalog

10 model = MLModelCatalog(dataset, "ann")

11

12 # 3. Load recourse model with model specific hyperparameters

13 gs = GrowingSpheres(model)

14

15 # 4. Generate counterfactual examples

16 factuals = dataset.raw.sample(10)

17 counterfactuals = gs.get_counterfactuals(factuals)

Figure 6: Example implementation of CARLA, using the data and model catalog.

To give users the possiblity to explore their own black-box-model on a custom
data set, we implemented in CARLA easy-to-use interfaces, that are able to
wrap every possible model or data set. These interfaces specify particular
properties users have to implement, to be able to work with the library.
Figure 7 shows an example implementation of the data wrapper, and Figure 8
depicts the same for an arbitrary black-box-model. After defining data set and
black-box model classes, users simply need to call the canonical methods and
generate counterfactual examples, similar to the process in Figure 6.

⬇

1

2 from carla import Data

3 from carla.recourse_methods import GrowingSpheres

4

5 # Custom data set implementations need to inherit from the Data interface

6 class MyOwnDataSet(Data):

7 def __init__(self):

8 # The data set can e.g. be loaded in the constructor

9 self._dataset = load_dataset_from_disk()

10

11 # List of all categorical features

12 def categoricals(self):

13 return [...]

14

15 # List of all continous features

16 def continous(self):

17 return [...]

18

19 # List of all immutable features which

20 # should not be changed by the recourse method

21 def immutables(self):

22 return [...]

23

24 # Feature name of the target column

25 def target(self):

26 return "label"

27

28 # Non-encoded and non-normalized, raw data set

29 def raw(self):

30 return self._dataset

Figure 7: Pseudo-implementation of the CARLA data wrapper

⬇

1

2 from carla import MLModel

3

4 # Custom black-box models need to inherit from

5 # the MLModel interface

6 class MyOwnModel(MLModel):

7 def __init__(self, data):

8 super().__init__(data)

9 # The constructor can be used to load or build an

10 # arbitrary black-box-model

11 self._mymodel = load_model()

12

13 # Define a fitted sklearn scaler to normalize input data

14 self.scaler = MySklearnScaler().fit()

15

16 # Define a fitted sklearn encoder for binary input data

17 self.encoder = MySklearnEncoder.fit()

18

19 # List of the feature order the ml model was trained on

20 def feature_input_order(self):

21 return [...]

22

23 # The ML framework the model was trained on

24 def backend(self):

25 return "pytorch"

26

27 # The black-box model object

28 def raw_model(self):

29 return self._mymodel

30

31 # The predict function outputs

32 # the continous prediction of the model

33 def predict(self, x):

34 return self._mymodel.predict(x)

35

36 # The predict_proba method outputs

37 # the prediction as class probabilities

38 def predict_proba(self, x):

39 return self._mymodel.predict_proba(x)

Figure 8: Pseudo-implementation of the CARLA black-box-model wrapper

###  A.4 Benchmarking Recourse Methods

Besides the generation of counterfactual examples, the focus of CARLA lies on
benchmarking recourse methods. Users are able to compute evaluation measures
to make qualitative statements about usability and applicability.

All measurements, which are described in Section 4.2, are implemented in the
Benchmarking class of CARLA and can be used for every wrapped recourse method.
Figure 9 shows an example implementation of a benchmarking process based on
the variables of Figure 6.

⬇

1

2 from carla import Benchmark

3

4 # 1. Initilize the benchmarking class by passing

5 # black-box-model, recourse method, and factuals into it

6 benchmark = Benchmark(model, gs, factuals)

7

8 # 2. Either only compute the distance measures

9 distances = benchmark.compute_distances()

10

11 # 3. Or run all implemented measurements and create a

12 # DataFrame which consists of all results

13 results = benchmark.run_benchmark()

Figure 9: Pseudo-implementation of the CARLA recourse method wrapper

##  Appendix B Additional Experimental Results

In this Section, we depict the missing experiments from the COMPAS data set in
Figure 11 and Table 4. These results underline the trends that we have already
highlighted in Section 5.

Figure 10: COMPAS Data Figure 11: Evaluating the distribution of costs of
counterfactual explanations on the COMPAS dataset. For all instances with a
negative prediction ($\\{x\in\mathcal{D}:f(x)<\theta\\}$), we plot the
distribution of $\ell_{0}$ and $\ell_{1}$ costs of algorithmic recourse as
defined in (1) for a logistic regression and an artificial neural network
classifier. The white dots indicate the medians (lower is better), and the
black boxes indicate the interquartile ranges. We distinguish between
independence based and dependence based methods.

| Artificial Neural Network | Logistic Regression
---|---|---
Data Set | Method | yNN | redund. | violation | success | $\overline{t}(s)$ | yNN | redund. | violation | success | $\overline{t}(s)$
COMPAS | AR(--LIME) | 0.91 | 0.00 | 0.02 | 0.53 | 0.06 | – | – | – | 0.00 | 0.01
CEM | 0.98 | 2.29 | 0.43 | 1.00 | 0.89 | 0.93 | 1.88 | 0.99 | 1.00 | 0.86
DICE | 0.89 | 0.88 | 1.03 | 1.00 | 0.09 | 0.95 | 0.94 | 0.90 | 1.00 | 0.09
GS | 0.44 | 0.97 | 0.03 | 1.00 | 0.01 | 0.60 | 0.64 | 0.02 | 1.00 | 0.01
Wachter | 0.56 | 1.77 | 0.74 | 0.66 | 10.90 | 0.50 | 1.21 | 0.79 | 1.00 | 0.02

(a) Independence based methods

| Artificial Neural Network | Logistic Regression
---|---|---
Data Set | Method | yNN | redund. | violation | success | $\overline{t}(s)$ | yNN | redund. | violation | success | $\overline{t}(s)$
COMPAS | CEM--VAE | 1.00 | 5.59 | 1.98 | 1.00 | 0.89 | 1.00 | 6.91 | 2.14 | 1.00 | 0.87
CLUE | 0.99 | 4.06 | 1.08 | 1.00 | 2.03 | 1.00 | 4.62 | 1.25 | 1.00 | 1.88
FACE--EPS | 0.94 | 3.71 | 1.55 | 0.99 | 0.45 | 0.97 | 3.94 | 1.62 | 0.99 | 0.45
FACE--KNN | 0.94 | 3.83 | 1.63 | 1.00 | 0.44 | 0.97 | 3.86 | 1.57 | 1.00 | 0.44
REVISE | 1.00 | 3.29 | 1.29 | 1.00 | 6.06 | 0.92 | 3.15 | 1.03 | 1.00 | 5.35

(b) Dependence based methods

Table 4: Summary of COMPAS results for independence and dependence based
methods. For all instances with a negative prediction
($\\{x\in\mathcal{D}:f(x)<\theta\\}$), we compute counterfactual explanations
for which we then measure yNN (higher is better), redundancy (lower is
better), violation (lower is better), success rate (higher is better) and time
(lower is better). We distinguish between a logistic regression and an
artificial neural network classifier. Detailed descriptions of these measures
can be found in Section 4. The results are discussed in Appendix B.

##  Appendix C ML Classifiers

In this section, we describe how the black–box models $f$ were fitted. CARLA
supports different ML libraries to estimate these models (e.g., Pytorch,
Tensorflow) as the implementations of the various explanation methods work
with a particular ML library. We note that the various explanation methods
rely on different binary feature encodings. DICE, for example, requires that
binary inputs are supplied as one–hot vectors, while FACE needs binary
features encoded in a single column. If this was the case, we fitted two ML
models, using the same hyperparameters, and generated CEs with respect to the
same set of samples.

To ensure similar behavior between the different ML libraries and encoding
variations, each black-box model type has the same structure (e.g., number of
hidden layer, number of neurons), and training parameters (e.g., learning
rate, epochs, etc.).

The first model is a multi-layer perceptron, consisting of three hidden layers
with 18, 9 and 3 neurons, respectively. We use ReLu activation functions and
binary cross entropy to calculate class probabilities. Optimization of the
loss function is done by RMSProp [56] using a learning rate of 0.002 for every
data set. By performing 25 epochs on COMPAS and 10 epochs on Adult and GMC we
reached acceptable performance. Further increasing epochs gave rise to very
marginal performance increases. For Adult we use a batch–size of 1024, for
COMPAS 25 and for GMC 2048.

To allow a more extensive comparison between CE methods, we choose linear
models as the second black–box model category for which we evaluate the CE
methods. Again, we optimized these models with RMSProp using a binary cross
entropy loss. For Adult, we used 100 epochs and a batch–size of 2048, for
COMPAS we choose 25 epochs and batch–size of 128, and for GMC we chose 10
epochs with a batch–size of 2048. The learning rate on every data set is set
0.002. Table 5 provides an overview of the model’s classification accuracies.

| Adult | COMPAS | Give Me Credit
---|---|---|---
Logistic Regression | 0.83 | 0.84 | 0.92
Neural Network | 0.84 | 0.85 | 0.93
Table 5: Performance of classification models used for generating algortihmic
recourse.

##  Appendix D COMPAS Data Set Description

The COMPAS data set [22] contains data for more than 10,000 criminal
defendants in Florida. It is used by the jurisdiction to score defendant’s
likelihood of reoffending. We kept a small part of the raw data as features
like name, id, casenumbers or _date-time_ were dropped. The classification
task consists of classifying an instance into high risk of recidivism
(score_text is high). By converting the feature race into _white_ and _non-
white_ , we keep the categorical input binary. Similar to Adult, the immutable
features for COMPAS are age, sex and race.

##  Appendix E Hyperparameter Search for the Counterfactual Explanation and
Recourse Methods

We generated counterfactual explanations for instances from $H^{-}$, the set
of factuals with negative class predictions.

#### AR and AR--LIME

It frequently occurred that the action with the lowest cost did not flip the
prediction of the black-box classifier. To overcome this problem, we let AR
compute a flipset of 150 actions per instance, and subsequently search this
set for low–cost CEs. For AR--LIME, we used LIME [51] and required sampling
around the instance to make sure that the coefficients at $x$ were truly
local.

#### CEM

After performing grid search, we set the $\ell_{1}$ weight to 0.9 and the
$\ell_{2}$ weight to 0.1, yielding the strongest performance. For CEM-VAE we
set the $\ell_{2}$ weight to 0.1, and the VAE–weight to 0.9.

#### CLUE

We use the default hyperparameters from [4], which are set as a function of
the data set dimension $d$. Performing hyperparameter search did not yield
results that were improving distances while keeping the same success rate.

#### DICE

Since DICE is able to compute a set of counterfactuals for a given instance,
we only chose to generate one CE per input instance. We use a _grid search_
for the _proximity_ and _diversity_ weights.

#### FACE

To determine the strongest hyperparameters for the graph size we conducted a
_grid search_. We found that values of $k_{FACE}=50$ gave rise to the best
balance of success rate and costs. For the epsilon graph, a radius of 0.25
yields the strongest results to balance between high yNN and low cost.

#### GS

We chose 0.02 as the step size with which the sphere is grown. Lower values
yield similar results at the costs of higher computational time, while higher
values gave worse results.

#### REVISE

The grid search to find an acceptable learning rate and similarity weight
$\lambda$ yielded $\eta=0.1$ and $\lambda=0.5$ for about 1500 iterations.

#### Wachter

For the target loss, we choose the Binary Cross Entropy with a learning rate
of $0.01$ and an initial $\lambda$ of $0.01$. For the distance loss, we use
the $\ell_{1}$\- norm to measure the similarity between the factual input and
the counterfactual point $\check{x}$.