# Policy learning with asymmetric utilities††thanks: We acknowledge the
partial support from Cisco Systems, Inc. (CG #2370386), National Science
Foundation (SES–2051196), and Sloan Foundation (Economics Program;
2020–13946). We also thank the IQSS’s Alexander and Diviya Magaro Peer Pre-
Review Program for feedback.

Eli Ben-Michael Corresponding author. Assistant Professor, Department of
Statistics & Data Science and Heinz College of Information Systems & Public
Policy, Carnegie Mellon University. 4800 Forbes Avenue, Hamburg Hall,
Pittsburgh PA 15213. Email<EMAIL_ADDRESS>URL: ebenmichael.github.io
Kosuke Imai Professor, Department of Government and Department of Statistics,
Harvard University. 1737 Cambridge Street, Institute for Quantitative Social
Science, Cambridge MA 02138. Email<EMAIL_ADDRESS>URL:
https://imai.fas.harvard.edu Zhichao Jiang Professor, School of mathematics,
Sun Yat-sen University, Guangzhou Guangdong 510275. Email:
<EMAIL_ADDRESS>
###### Abstract

Data-driven decision making plays an important role even in high stakes
settings like medicine and public policy. Learning optimal policies from
observed data requires a careful formulation of the utility function whose
expected value is maximized across a population. Although researchers
typically use utilities that depend on observed outcomes alone, in many
settings the decision maker’s utility function is more properly characterized
by the joint set of potential outcomes under all actions. For example, the
Hippocratic principle to “do no harm” implies that the cost of causing death
to a patient who would otherwise survive without treatment is greater than the
cost of forgoing life-saving treatment. We consider optimal policy learning
with asymmetric utility functions of this form. We show that asymmetric
utilities lead to an unidentifiable social welfare function, and so we first
partially identify it. Drawing on statistical decision theory, we then derive
minimax decision rules by minimizing the maximum regret relative to
alternative policies. We show that one can learn minimax decision rules from
observed data by solving intermediate classification problems. We also
establish that the finite sample regret of this procedure is bounded by the
mis-classification rate of these intermediate classifiers. We apply this
conceptual framework and methodology to the decision about whether or not to
use right heart catheterization for patients with possible pulmonary
hypertension.

Keywords: Hippocratic oath, minimax regret, partial identification, policy
learning, principal stratification

##  1 Introduction

The well-known Trolley Problem in ethics goes as follows:

> Edward is the driver of a trolley, whose brakes have just failed. On the
> track ahead of him are five people; the banks are so steep that they will
> not be able to get off the track in time. The track has a spur leading off
> to the right, and Edward can turn the trolley onto it. Unfortunately there
> is one person on the right-hand track. Edward can turn the trolley, killing
> the one; or he can refrain from turning the trolley, killing the five
> (Thomson, 1976, p. 206).

Should Edward turn the trolley? Is killing someone worse than letting them
die? Such ethical dilemmas frequently confront us in moral and legal debates
concerning various issues that range from abortion to self-driving cars (e.g.,
Foot, 1967; Lin, 2016). Similarly, in the ethics of modern medicine, the
Hippocratic principle of “do no harm” remains influential (e.g., Smith, 2005).
In the language of utility theory, a physician may assign a utility loss of
greater magnitude to the case where a new drug harms a patient than to the
case where not providing the new drug leads to the failure to save a patient
(e.g., Bordley, 2009).

These examples illustrate the potential applications of _asymmetric_
utilities, which treat an undesirable outcome differently depending on which
action causes it. Yet, to the best of our knowledge, the existing literature
on data-driven decision making and algorithmic policy learning assumes that
the decision maker’s utility function is _symmetric_ and only depends on the
observed outcome regardless of which action causes it.

In this paper, we develop the methodological framework for optimal policy
learning with asymmetric utilities, which includes standard symmetric
utilities as a special case. We show that in general, asymmetric utilities
lead to an unidentifiable social welfare function. Therefore, we partially
identify the social welfare function and propose to minimize the maximum
regret relative to a particular comparison policy. We consider the maximum
regret relative to constant policies such as always-treat and never-treat
policies as well as the oracle policy that has complete knowledge of the
social welfare function. We demonstrate that one can learn minimax decision
rules from observed data by solving intermediate classification problems. We
also establish that the finite sample regret of this procedure is bounded by
the mis-classification rate of these intermediate classification problems.

We use this framework to re-assess the use of Right Heart Catheterization
(RHC), an invasive diagnostic tool (Connors et al., 1996). We learn decision
rules based on clinical variables as we vary the asymmetry in the utility
function between failing to prevent a patients death and causing it via RHC.
These decision rules differ depending on whether we minimize the worst-case
regret relative to a constant policy (always or never using RHC) or the oracle
policy that uses RHC optimally. We inspect how the choice of utility function
and comparator affect the learned decision rules, finding substantial
variability based on these choices. Finally, we translate these findings into
a constrained symmetric utility maximization framework, where we seek to find
an algorithmic policy that minimizes expected mortality subject to constraints
on the worst-case proportion of patients that the policy harms or fails to
save. We exhibit a trade-off between these goals, showing that expected
mortality will rise with more-stringent constraints.

##### Related Literature.

Recent years have seen an increased interest in developing approaches for
algorithmic policy learning from randomized control trials or observational
data. Many of these approaches follow a similar structure. First, quantify the
social welfare of a policy based on the marginal distribution of the potential
outcomes. Then, show how to identify the social welfare from observable data
and find a policy that maximizes the empirical welfare or minimizes the regret
relative to a set comparison policy. These approaches typically use inverse
propensity score weighting or double-robust methods for the identification and
estimation steps (see Beygelzimer and Langford, 2009; Qian and Murphy, 2011;
Dudik et al., 2011; Zhao et al., 2012; Zhang et al., 2012; Swaminathan and
Joachims, 2015; Kitagawa and Tetenov, 2018; Kallus, 2018; Luedtke and Van Der
Laan, 2016; Athey and Wager, 2021, among others). There is also a related
literature that focuses on identifying and estimating optimal policies in
settings with unmeasured confounding via instrumental variables; see Cui and
Tchetgen Tchetgen (2021); Qiu et al. (2021).

More immediately relevant to our discussion here, there is a parallel set of
recent work that considers scenarios where we cannot point identify the social
welfare function for possible policies. This arises in situations including
the following; (1) there is unmeasured confounding in an observational study
(Kallus and Zhou, 2021), (2) there is an instrumental variable that is
correlated with treatment effects (Pu and Zhang, 2021), (3) there is no
overlap between different treatment conditions (Ben-Michael et al., 2021), or
(4) treatments are dynamic without sequential ignorability (Han, 2021). In
these settings, the social welfare function can only be partially identified
from observable data, and so we cannot find an algorithmic policy that
maximizes it. Instead, these approaches use ideas from robust optimization
(see Bertsimas et al., 2011, for a review) to minimize the worst case regret
relative to a comparison policy. As Cui (2021) discusses, building on
classical ideas in decision theory (Manski, 2005), the choice of comparison
policy is key to determining the minimax optimal policy. We will explore this
below in the setting where the utilities explicitly depend on the joint
potential outcomes or principal strata.

##### Paper outline.

The paper proceeds as follows. Section 2 describes the goal of policy learning
with asymmetric utilities and reviews the standard symmetric case. Section 3
discusses partial identification of the social welfare function and the
minimax population policies relative to different alternatives. Section 4 then
shows how to estimate such policies from data. Finally, Section 5 exhibits the
procedure via a brief simulation study, Section 6 applies this framework to
the use of RHC, and Section 7 concludes.

##  2 Preliminaries

In this section, we introduce the notation and assumptions used throughout
this paper. We also discuss the nature of asymmetric utilities before
providing a brief review of policy learning with symmetric utilities, which is
a special case of our proposed framework.

###  2.1 Notation and assumptions

Suppose that we have a simple random sample of $n$ units from a super
population $\mathcal{P}$ where each unit $i=1,\ldots,n$ has a set of
characteristics $X_{i}\in\mathcal{X}$. We consider a binary treatment
assignment decision $D_{i}\in\\{0,1\\}$, which can be made by either
individual $i$ or a policy maker. We assume that the outcome $Y_{i}$ is binary
with $Y_{i}=1$ indicating a desirable outcome (e.g., survival) and $Y_{i}=0$
representing an undesirable outcome (e.g., death). Under the assumption that
there is only one version of treatment and no interference across units, we
have two binary potential outcomes for each unit $i$ where
$Y_{i}(d)\in\\{0,1\\}$ represents the potential outcome under the scenarios
where the unit receives the decisions $D_{i}=d$ for $d=0,1$ (Neyman, 1923;
Rubin, 1980).

The setup implies that the observed outcome for unit $i$ can be written as
$Y_{i}=D_{i}Y_{i}(1)+(1-D_{i})Y_{i}(0)$ and the tuple of random variables
$\\{X_{i},D_{i},Y_{i}(1),Y_{i}(0)\\}$ is assumed to be independently and
identically distributed. Importantly, under this setting, each unit belongs to
one of the four principal strata defined by the values of the two potential
outcomes, i.e.,
$(Y_{i}(1),Y_{i}(0))=(y_{1},y_{0})\in\\{(0,0),(0,1),(1,0),(1,1)\\}$ (Frangakis
and Rubin, 2002). For example, the principal stratum $(y_{1},y_{0})=(1,0)$
represents a group of units who would yield the desirable outcome only when
they are treated, i.e., $D_{i}=1$, whereas the principal stratum
$(y_{1},y_{0})=(1,1)$ indicates a group of units whose outcome is desirable
regardless of the treatment decision. Since we never observe the two potential
outcomes at the same time for any given unit, it is impossible to know which
principal stratum each unit belongs to without additional assumptions.

Throughout this paper, for the sake of notational simplicity, we will drop the
individual $i$ subscript in expressions involving expectations over the
distribution. We will also assume the following standard strong ignorability
assumption for observational studies and randomized control trials.

######  Assumption 1 (Strong ignorablity (Rosenbaum and Rubin, 1983)).

$\\{Y(1),Y(0)\\}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}D\mid X$ and $0<d(x)<1$ for all
$x\in\mathcal{X}$ where $d(x)\equiv P(D=1\mid X=x)$ represents the propensity
score.

Under this assumption, we can identify the expected potential outcome under
decision $d$ given covariates $x$, which we denote as
$m(d,x)\equiv\mathbb{E}[Y(d)\mid X=x]$. However, it is impossible to identify
the principal score, or the conditional probability of belonging to a
principal stratum given covariates, which is formally defined as
$e_{y_{1}y_{0}}(x)\equiv P(Y(1)=y_{1},Y(0)=y_{0}\mid X=x)$, because we do not
observe the two potential outcomes at the same time for any given unit (Ding
and Lu, 2017; Jiang et al., 2020).

###  2.2 Asymmetric utilities and regret

We focus on deterministic individualized policies
$\pi:\mathcal{X}\to\\{0,1\\}$ that assign a binary treatment decision to
individual units according to their characteristics $X\in\mathcal{X}$. To
learn optimal policies from the observed data, we consider a utility function
$u(d;y_{1},y_{0})$ that encodes the utility for taking treatment decision $d$
for a unit in principal stratum $(y_{1},y_{0})$. Crucially, this utility
function depends on the values of _both_ potential outcomes. This contrasts
with the standard utility function $u(d;y)$, which only depends on the
realized potential outcome $Y_{i}(d)=y$ under the decision $d$.

The key aspect of this utility function is that it allows for asymmetric
utilities with the same realized outcome. For example, under the Hippocratic
oath, the absolute magnitude of the utility loss when a new treatment harms a
patient is greater than that of the utility gain under a scenario where the
same treatment benefits another patient. Specifically, we parameterize the
utility function as follows:

  1. (i)

the utility gain associated with a “useful treatment,” i.e.,
$(y_{1},y_{0})=(1,0)$, is $u_{g}-c_{g}\equiv u(1;1,0)-u(0;1,0)$ (e.g.,
treating with a drug that would benefit the patient)

  2. (ii)

the utility loss associated with a “harmful treatment,” i.e.,
$(y_{1},y_{0})=(0,1)$, is $-u_{l}-c_{l}\equiv u(1;0,1)-u(0;0,1)$ (e.g.,
treating with a drug that would harm the patient)

  3. (iii)

the utility loss of treating with a “harmless treatment,” i.e.,
$(y_{1},y_{0})=(1,1)$, is $-c_{1}=u(1;1,1)-u(0;1,1)$ (e.g., treating with a
drug that would not harm the patient)

  4. (iv)

the utility loss of treating with a “useless treatment,” i.e.,
$(y_{1},y_{0})=(0,0)$, is $-c_{0}=u(1;0,0)-u(0;0,0)$ (e.g., treating with a
drug that would not benefit the patient)

The values $c_{g},c_{l},c_{1},c_{0}$ denote the cost of administering the
treatment $d=1$ relative to not doing so $d=0$ in each of the four principal
strata. The values $u_{g}$ and $u_{l}$ represent the magnitude of the utility
gain and loss for administering a useful and harmful treatment, respectively.
In this setting, these utility values are known and fixed by the decision
maker. If the utility functions were unknown, they would first need to be
elicited in some manner (see, e.g. González-Ortega et al., 2018, for a
review).

| $Y_{i}(0)=1$ | $Y_{i}(0)=0$
---|---|---
$Y_{i}(1)=1$ | Harmless | Useful
$-c$ | $u_{g}-c$
1-3 $Y_{i}(1)=0$ | Harmful | Useless
$-u_{l}-c$ | $-c$
Table 1: Asymmetric utilities based on the four principal strata. Each cell
corresponds to the principal stratum defined by the values of the two
potential outcomes, $Y_{i}(1)$ and $Y_{i}(0)$. Each entry represents the
utility gain/loss of treatment assignment, relative to no treatment ($c$ is
the cost of treatment assignment), for a unit that belongs to the
corresponding principal stratum. We assume that $u_{l},u_{g}>0$ and $c\geq 0$.
Symmetric utilities are a special case with $u_{g}=u_{l}$.

Throughout, we will assume that the costs are identical, i.e.,
$c_{g}=c_{b}=c_{1}=c_{0}=c$. In addition, we assume $u_{g}$ and $u_{l}$ are
positive, and $c$ is non-negative. Table 1 summarizes this asymmetric utility
structure. Fixing the costs to be identical amounts to restricting the utility
loss from a harmless and useless treatment to be equal. Without this
restriction there will be an additional asymmetry due to the different costs,
which would not affect our development, except to make the notation more
cumbersome and results less interpretable. Note that our asymmetric utilities
include symmetric utilities as special cases, thereby generalizing the
standard setting considered in the policy learning literature. In Appendix A,
we show that although it is possible to construct asymmetric utilities without
using principal strata, doing so still implies some restrictions on the
structure of utilities.

Finally, in many scenarios the decision maker’s utility function could depend
on covariates as well. For instance, the cost of administering treatment may
depend on individual characteristics. Our discussion below readily extends to
such utility functions by allowing utility functions to depend on the
covariates $X$. Below we will show that the asymmetric utility functions
indeed induce decision problems where weights on decisions depend on
covariates.

Given this setup, we measure the overall quality of a policy $\pi$ by its
expected utility (also called the _value_ or _social welfare_),

$\displaystyle V(\pi)$
$\displaystyle=\mathbb{E}\left[\sum_{y_{1}=0}^{1}\sum_{y_{0}=0}^{1}\mathbbm{1}\\{Y(1)=y_{1},Y(0)=y_{0}\\}\left\\{u(0;y_{1},y_{0})(1-\pi(X))+u(1;y_{1},y_{0})\pi(X)\right\\}\right]$
(1)
$\displaystyle=\mathbb{E}\left[\sum_{y_{1}=0}^{1}\sum_{y_{0}=0}^{1}e_{y_{1}y_{0}}(X)\left\\{u(0;y_{1},y_{0})(1-\pi(X))+u(1;y_{1},y_{0})\pi(X)\right\\}\right].$

We will primarily be comparing two policies with one another, rather than
considering one policy in isolation. We begin by defining the _regret_ of
policy $\pi$ relative to another policy $\varpi$ as the difference in values,
$V(\varpi)-V(\pi)$. Using the relations $m(1,x)=e_{11}(x)+e_{10}(x)$ and
$m(0,x)=e_{11}(x)+e_{01}(x)$, we show in Appendix D that we can write the
regret in a simplified form:

$R_{e_{01}}(\pi,\varpi)=V(\varpi)-V(\pi)=\mathbb{E}\left[(\varpi(X)-\pi(X))\\{u_{g}\tau(X)+(u_{g}-u_{l})e_{01}(X)-c\\}\right].$
(2)

where $\tau(x)\equiv\mathbb{E}(Y(1)-Y(0)\mid X=x)=m(1,x)-m(0,x)$ is the
conditional average treatment effect (CATE) given the covariates $X=x$.

There are three components that contribute differently to the regret in
Equation (2). First, the difference in the expected treatment effects for
those treated under policy $\varpi$ and policy $\pi$, which is scaled by the
utility gain for a useful treatment $u_{g}$, represents a symmetric component
of the utility, where we compare the marginal benefits of the policies. The
second component is an asymmetric adjustment term, and relates to the
probability of belonging to the group for whom the treatment is harmful
(principal stratum with $(y_{1},y_{0})=(0,1)$). This can counteract the
marginal benefit of treatment and is scaled by the difference between the
utility gain for a useful treatment and the loss for a harmful treatment,
i.e., $u_{g}-u_{l}$. The final term $c$ corresponds to the difference in the
overall costs of the two policies.

The first and third components, the difference in effects and costs are point
identifiable under Assumption 1. The second component, however, is only
_partially identifiable_ due to the unidentifiability of principal score
$e_{01}(\cdot)$. We use the $e_{01}$ subscript for the regret in Equation (2)
to signify this fact. Therefore, we cannot pinpoint whether any policy is
superior to any other policy in general. The remainder of this paper focuses
on handling this ambiguity.

##### Constrained formulation.

While we have arrived at the objective defined in Equation (2) through a
utility-based framework, we can also characterize this decision problem in the
following constrained form,

$\displaystyle\min_{\pi}$ $\displaystyle P(Y(\pi(X))<Y(1))$ (3) subject to
$\displaystyle P(Y(\pi(X))<Y(0))\leq\delta$
$\displaystyle\mathbb{E}[\pi(X)]\leq B$

where $P(Y(\pi(X))<Y(0))=P(Y(1)=0,Y(0)=1,\pi(X)=1)$ and
$P(Y(\pi(X))<Y(1))=P(Y(1)=1,Y(0)=0,\pi(X)=0)$ represent the probabilities that
policy $\pi$ gives a harmful treatment or fails to give a useful treatment for
a randomly selected member of the population, respectively.

In this formulation, the goal is to find a policy $\pi$ that minimizes the
expected proportion of _false negatives_ — failing to give a useful treatment
— subject to a constraint on the expected proportion of _false positives_ —
providing a harmful treatment — and the treatment _budget_ — the proportion of
units treated. Thus, the decision problem given in Equation (3) allows the
policy maker to explicitly state their preferences via the constraint on the
number of false positives and the budget, rather than implicitly through the
utility function in the regret $R_{e_{01}}(\pi,\varpi)$. It is also possible
to swap the constraints and the objective to minimize the proportion of false
positives subject to a constraint on the proportion of false negatives. We can
also interpret Equation (3) through the lens of multiple testing, for each
unit $i$ we have a null hypothesis $H_{0i}:Y_{i}(1)<Y_{i}(0)$, i.e. that unit
$i$ is harmed by treatment. Then, we can view the policy $\pi(X_{i})$ as
determining whether to reject $H_{0i}$ or not. Then, the constraint on the
proportion of false positives in Equation (3) is a scaling of the false
detection rate, where the budget constraint limits the number of rejections,
and the objective is a measure of the average power under the alternative
$H_{1i}:Y_{i}(1)>Y_{i}(0)$.

However, note that $P(Y(\pi(X))<Y(0))=\mathbb{E}[\pi(X)e_{01}(X)]$ and
$P(Y(\pi(X))<Y(1))=\mathbb{E}[(1-\pi(X))(\tau(x)+e_{01}(X))]$. Thus, we can
view the regret $R_{e_{01}}(\pi,\varpi)$ for a constant comparison policy —
either always or never providing treatment — as a Lagrangian relaxation of the
decision problem defined in Equation (3), where some choice of the false-
positive constraint $\delta$ and budget $B$ will correspond to a particular
value of the utility ratio $\frac{u_{g}-u_{l}}{u_{g}}$ and cost ratio
$\frac{c}{u_{g}}$. This is in contrast to the regret relative to the oracle
policy that minimizes the true regret. As we show below, the regret relative
to the oracle involves unidentifiable terms in the relative weights on
$\tau(x)$ and $e_{01}(x)$, so it cannot be written as a Lagrangian relaxation
of Equation (3).

###  2.3 Policy learning with symmetric utilities: A review

Before discussing policy learning under asymmetric utility functions, we
briefly review policy learning with _symmetric_ utilities — a special case of
our framework — where the absolute magnitude of the utility gain when the
treatment leads to a desirable outcome is equal to that of the utility loss
when it leads to an undesirable outcome, i.e., $u_{g}=u_{l}$. In this case, a
policy can make up for the loss from harming some units by the gain from
benefiting other units. This can be seen in the following simplified version
of the regret in Equation (2):

$R_{\text{symm}}(\pi,\varpi)=\mathbb{E}\left[(\varpi(X)-\pi(X))(u_{g}\tau(X)-c)\right].$
(4)

This simplification also occurs under the _monotoncity_ assumption about the
potential outcomes: if the treatment is almost surely never harmful, i.e.,
$P(Y(1)\geq Y(0))=1$, then the principal score $e_{01}(x)$ is zero for all
$x\in\mathcal{X}$.

The symmetric utility does not involve the principal score $e_{10}(\cdot)$,
and is identifiable under Assumption 1. Thus, under this setting, the oracle
policy that minimizes the regret relative to any other policy has a
straightforward form:

$\pi^{\text{symm}}\equiv\underset{\pi}{\text{argmin}}\max_{\varpi}\
R_{\text{symm}}(\pi,\varpi)=\mathbbm{1}\\{u_{g}\tau(\cdot)\geq c\\}.$ (5)

This oracle policy assigns the treatment to all individuals with
characteristics $x$ if their expected utility gain of assigning the treatment
relative to not assigning it at least makes up for its cost, i.e.,
$u_{g}\tau(x)\geq c$. Note that this is equivalent to maximizing the value
$V(\pi)$ directly.

Under Assumption 1, therefore, we can write the symmetric regret in Equation
(4) in terms of the observed data by using a scoring function
$\Gamma_{w}(x,d,y)$ such that $\mathbb{E}\left[\Gamma_{w}(X,D,Y)\mid
X=x\right]=m(w,x)$. For example, the Inverse Probability-of-treatment
Weighting (IPW) scoring function weights the observed outcome by the inverse
probability of receiving the decision $d$:

$\Gamma_{w}^{\text{ipw}}(X,D,Y)=Y\left\\{\frac{wD}{d(X)}+\frac{(1-w)(1-D)}{1-d(X)}\right\\}.$

An alternative is the Doubly Robust (DR) scoring function that combines the
observed outcomes and their conditional expectations:

$\Gamma_{w}^{\text{dr}}(X,D,Y)=m(w,X)+\\{Y-m(w,X)\\}\left\\{\frac{wD}{d(X)}+\frac{(1-w)(1-D)}{1-d(X)}\right\\}.$

With such a scoring rule, we can then write the symmetric regret function as:

$\mathbb{E}\left[(\varpi(X)-\pi(X))\left\\{u_{g}(\Gamma_{1}(X,D,Y)-\Gamma_{0}(X,D,Y))-c\right\\}\right],$

where the observable quantity $\Gamma_{1}(X,D,Y)-\Gamma_{0}(X,D,Y)$ has
replaced the causal quantity $\tau(X)$. See Knaus (2020) for a recent review.

In order to empirically find optimal policies from data, recent approaches
estimate the propensity score $\hat{d}(\cdot)$ and/or the conditional expected
potential outcome $\hat{m}(\cdot,\cdot)$ to create estimated scores
$\widehat{\Gamma}(X_{i},D_{i},Y_{i})$. For example, we can estimate the IPW
scoring function as

$\widehat{\Gamma}_{w}^{\text{ipw}}(X,D,Y)=Y\left\\{\frac{wD}{\hat{d}(X)}+\frac{(1-w)(1-D)}{1-\hat{d}(X)}\right\\}$

and the DR scoring function as

$\widehat{\Gamma}_{w}^{\text{dr}}(X,D,Y)=\hat{m}(w,X)+\\{Y-\hat{m}(w,X)\\}\left\\{\frac{wD}{\hat{d}(X)}+\frac{(1-w)(1-D)}{1-\hat{d}(X)}\right\\}.$

Then, we solve the sample analog of Equation (4). This leads to finding policy
$\hat{\pi}$ that solves the following optimization problem:

$\min_{\pi\in\Pi}\
-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left\\{u_{g}\left(\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})-\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})\right)-c\right\\},$

where $\Pi$ represents the _policy class_ and restricts the functional form of
potential policies. Athey and Wager (2021) establish strong asymptotic
guarantees on the regret of the empirical $\hat{\pi}$ relative to the best-in-
class policy when using the DR approach with appropriately cross-fit models;
see also Zhao et al. (2012); Kitagawa and Tetenov (2018).

##  3 Policy learning with asymmetric utilities

We now turn to the problem of finding optimal policies in the general
asymmetric case where $u_{g}\neq u_{l}$. We will first consider the
identification problems in the population — i.e., with infinite data. We then
show how to find policies empirically from observed data in Section 4.

###  3.1 The oracle policy with an asymmetric utility function

We begin by considering the oracle policy in the general asymmetric case. By
direct computation, the oracle policy that minimizes the regret relative to
any other policy with an asymmetric utility function is given by:

$\pi^{o}\equiv\underset{\pi}{\text{argmin}}\max_{\varpi}R_{e_{01}}(\pi,\varpi)=\mathbbm{1}\left\\{\tau(\cdot)\geq\frac{u_{l}-u_{g}}{u_{g}}e_{01}(\cdot)+\frac{c}{u_{g}}\right\\}.$
(6)

Unlike the oracle policy under a symmetric utility function in Equation (5),
the oracle policy in the asymmetric case includes the principal score
$e_{01}$, which is unidentifiable under Assumption 1. Since $0\leq
e_{01}(x)\leq 1$ for all $x$, the asymmetric oracle policy uses a varying
threshold for assigning the treatment where the threshold depends on the
principal score $e_{01}(X)$.

The way in which the oracle policy depends on the principal score is
characterized in part by the nature of asymmetry in the utility function. To
understand this relation, consider the case where treatment is costless and so
$c=0$. When $u_{l}>u_{g}$, causing the undesirable outcome by assigning the
treatment is considered worse than failing to prevent such an outcome by not
providing the treatment. This is the asymmetry encoded by the principle to “do
no harm” in the Hippocratic oath. This raises the threshold for assigning the
treatment because the expected effect must be larger in order to compensate
for the downside risk of causing the undesirable outcome. As a result, this
biases the oracle policy towards inaction. Conversely, when $u_{l}<u_{g}$, it
is better to cause the undesirable outcome than fail to prevent it by
inaction. In this case, the threshold for the treatment assignment is lower,
which biases the oracle policy towards action.

(a) Setting

(b) Decision rules

Figure 1: Example decision rules in a hypothetical example with a single
covariate and cost $c=0$. The left plot (a) presents the setting of the
example where the values of the principal score $e_{01}(x)=P(Y(1)=0,Y(0)=1\mid
X=x)$ are in red while the conditional expectations
$m(d,x)=\mathbb{E}(Y(d)\mid X=x)$ are in blue for $d=0$ and green for $d=1$,
respectively. The right plot (b) presents the decision rules corresponding to
(i) the oracle in the symmetric case where $u_{g}=u_{l}$ (in green); (ii) the
oracle in the asymmetric case where $u_{g}=0.8$ and $u_{l}=1$ (in orange);
(iii) the minimal regret solution relative to always not assigning the
treatment $d=0$ (in purple); and (iv) the minimax regret solution relative to
the oracle (in pink). All rules have been transformed so that the policy takes
decision 1 when the rule is greater than or equal to zero.

Plot (a) of Figure 1 shows the setup of a one-dimensional example of these
decision rules where the cost is zero, $c=0$. The principal score $e_{01}(x)$
is shown in red whereas the conditional expectations
$m(d,x)=\mathbb{E}(Y(d)\mid X=x)$ are shown in blue ($d=0$) and green ($d=1$).
Plot (b) of the figure shows the functions that make up the decision rules in
this example, centered so that the corresponding policies assign $d=1$ if the
function is positive. The symmetric case is shown in green, the oracle in the
asymmetric case ($u_{g}=0.8$ and $u_{l}=1$) is in orange, the minimal regret
solution relative to always not assigning the treatment is shown in purple,
and the minimal regret solution relative to the oracle is in pink. The last
two solutions are discussed in Section 3.3.

In this example, providing the treatment $d=1$ leads to a higher probability
of the desirable outcome except near zero. Therefore, with a symmetric utility
function, the oracle policy would assign the treatment in most cases (green).
However, the asymmetric case where $u_{g}=0.8$ and $u_{l}=1$ is different
(orange). There is a region of the covariate space where the principal score
$e_{01}(x)$ is relatively high, leading to a relatively high probability that
the treatment causes the undesirable outcome. Therefore, the asymmetric oracle
rule has a higher threshold for the treatment assignment, only providing the
treatment when the CATE $\tau(x)$ is large enough and the principal score
$e_{10}(x)$ is small enough.

###  3.2 Partial identification and minimizing worst-case regret

Recall that due to the fundamental problem of causal inference, we cannot
point identify the principal score $e_{01}(x)$ for any value of
$x\in\mathcal{X}$, leading to the unidentifiability of the regret in Equation
(2) in the general asymmetric case with $u_{g}\neq u_{l}$. However, we can
_partially identify_ the principal score by deriving its sharp upper and lower
bounds, $L(x)$ and $U(x)$. We then take a minimax approach, and find the
policy $\pi^{\ast}$ in the policy class $\Pi$ that minimizes the maximal
regret relative to a fixed policy $\varpi$:

$\pi^{\ast}\in\underset{\pi\in\Pi}{\text{argmin}}\underbrace{\max_{e_{01}(x)\in[L(x),U(x)]}R_{e_{01}}(\pi,\varpi)}_{R_{\text{sup}}(\pi,\varpi)}.$
(7)

Note that the maximum regret $R_{\text{sup}}(\pi,\varpi)$ is relative to a
_particular_ alternative policy $\varpi$, and is maximal over all possible
values for the principal score $e_{01}(x)$. In comparison to Equation (6), we
consider a general alternative policy rather than the only the best-possible
one. As we show below, the choice of this alternative policy will lead to
different objectives and optimal solutions (see Cui, 2021, for a recent
general discussion).

To derive the sharp lower and upper bounds of the principal score, we first
define two classification functions:

$\displaystyle\delta_{+}(x)$ $\displaystyle\ =\
\mathbbm{1}\\{m(0,x)+m(1,x)-1\geq 0\\},$ (8) $\displaystyle\delta_{\tau}(x)$
$\displaystyle\ =\ \mathbbm{1}\\{\tau(x)\geq 0\\}.$

Notice that the difference in the probability that both potential outcomes are
one and zero is given by $e_{11}(x)-e_{00}(x)=m(0,x)+m(1,x)-1$, which is the
decision function for classifier $\delta_{+}(x)$. In other words, we have
$\delta_{+}(x)=1$ if and only if $e_{00}(x)\leq e_{11}(x)$. Thus, we can view
$\delta_{+}(x)$ as classifying whether there is a higher probability that both
potential outcomes are one rather than zero. The other function
$\delta_{\tau}(x)$ classifies whether the CATE is positive, corresponding to
the symmetric oracle rule with cost $c=0$.

With these classifiers, we can find the sharp lower and upper bounds for the
principal score such that $e_{01}(x)\in[L(x),U(x)]$ for all $x$:

$\displaystyle L(x)$ $\displaystyle\ =\ \max\\{0,-\tau(x)\\}\ =\
-\tau(x)\\{1-\delta_{\tau}(x)\\}$ (9) $\displaystyle U(x)$ $\displaystyle\ =\
\min\\{m(0,x),1-m(1,x)\\}\ =\ m(0,x)+\delta_{+}(x)\\{1-m(0,x)-m(1,x)\\}.$ (10)

These lower and upper bounds are point-identifiable from observable data, and
with them we can create a point-identifiable objective. See Kallus (2022) for
similar partial identification bounds for these quantities.

###  3.3 Worst case regret relative to different alternative policies

We now inspect the worst-case regret $R_{\text{sup}}(\pi,\varpi)$ in Equation
(7) for different choices of alternative policy $\varpi$. We will consider
three main alternatives. First, the “never-treat” policy $\pi^{\mathbbm{O}}$
that does not treat anyone, i.e., $\pi^{\mathbbm{O}}(x)=0$ for all $x$.
Second, the “always-treat” policy $\pi^{\mathbbm{1}}$ that treats everyone,
i.e., $\pi^{\mathbbm{1}}(x)=1$ for all $x$. In many cases, the alternative to
algorithmic decision making via a data-driven policy is to take the same
decision for everyone; thus these two policies are reasonable alternatives as
they represent the standard of care in the absence of an individualized
policy. Finally, we will also compare to the oracle policy $\pi^{o}$ defined
in Equation (6). Recall that the oracle is unidentifiable from observable
data, so $R_{\text{sup}}(\pi,\pi^{o})$ represents the worst-case regret
between our policy $\pi$ and the best-possible policy if we could know the
joint distribution of potential outcomes.

The following theorem shows that the worst-case regret relative to each of
these three policies takes a common form.

######  Theorem 1 (Worst case regret).

Let $\pi:\mathcal{X}\to\\{0,1\\}$ be a deterministic policy. For comparison
policy $\varpi\in\\{\pi^{\mathbbm{O}},\pi^{\mathbbm{1}}\\}$ the worst-case
regret of $\pi$ relative to $\varpi$ is

$\displaystyle R_{\sup}(\pi,\varpi)$
$\displaystyle=C-\mathbb{E}\left[\pi(X)\left\\{c_{1}^{\varpi}(X)m(1,X)+c_{0}^{\varpi}(X)m(0,X)+c^{\varpi}(X)\right\\}\right]$
(11)
$\displaystyle=C-\mathbb{E}\left[\pi(X)\left\\{c^{\varpi}_{1}(X)\Gamma_{1}(X,D,Y)+c_{0}^{\varpi}(X)\Gamma_{0}(X,D,Y)+c^{\varpi}(X)\right\\}\right],$

where $C$ is a constant that does not depend on $\pi$. For $u_{g}\geq u_{l}$,

$\displaystyle c_{1}^{\pi^{\mathbbm{O}}}(x)$
$\displaystyle=u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)$ $\displaystyle
c_{0}^{\pi^{\mathbbm{O}}}(x)$
$\displaystyle=-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x)$ $\displaystyle
c^{\pi^{\mathbbm{O}}}(x)$ $\displaystyle=-c,$ $\displaystyle
c^{\pi^{\mathbbm{1}}}_{1}(x)$ $\displaystyle=u_{g}+\delta_{+}(x)(u_{l}-u_{g})$
$\displaystyle c^{\pi^{\mathbbm{1}}}_{0}(x)$
$\displaystyle=-u_{l}-\delta_{+}(x)(u_{g}-u_{l})$ $\displaystyle
c^{\pi^{\mathbbm{1}}}(x)$ $\displaystyle=\delta_{+}(x)(u_{g}-u_{l})-c,$

and for $u_{g}<u_{l}$,

$\displaystyle c^{\pi^{\mathbbm{O}}}_{1}(x)$
$\displaystyle=u_{g}+\delta_{+}(x)(u_{l}-u_{g})$ $\displaystyle
c^{\pi^{\mathbbm{O}}}_{0}(x)$
$\displaystyle=-u_{l}-\delta_{+}(x)(u_{g}-u_{l})$ $\displaystyle
c^{\pi^{\mathbbm{O}}}(x)$ $\displaystyle=\delta_{+}(x)(u_{g}-u_{l})-c,$
$\displaystyle c_{1}^{\pi^{\mathbbm{1}}}(x)$
$\displaystyle=u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)$ $\displaystyle
c_{0}^{\pi^{\mathbbm{1}}}(x)$
$\displaystyle=-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x)$ $\displaystyle
c^{\pi^{\mathbbm{1}}}(x)$ $\displaystyle=-c.$

Now define
$\pi^{\ast}_{\mathbbm{O}}\equiv\text{argmin}_{\pi}R_{\sup}(\pi,\pi^{\mathbbm{O}})$
as the minimax regret solution relative to the never-treat policy and
$\pi^{\ast}_{\mathbbm{1}}\equiv\text{argmin}_{\pi}R_{\sup}(\pi,\pi^{\mathbbm{1}})$
as the minimax regret solution relative to the always-treat policy. The worst-
case regret relative to the oracle policy $\pi^{o}$ is of the form in Equation
(11) where for $u_{g}\geq u_{l}$,

$\displaystyle\left(\begin{array}[]{c}c^{\pi^{o}}_{1}(x)\\\
c^{\pi^{o}}_{0}(x)\\\ c^{\pi^{o}}(x)\end{array}\right)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{l
l}(u_{l}+(u_{g}-u_{l})\delta_{\tau}(x),-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x),-c)&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
(u_{g}-(u_{g}-u_{l})\delta_{+}(x),-u_{l}-(u_{g}-u_{l})\delta_{+}(x),(u_{g}-u_{l})\delta_{+}(x)-c)&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\
(u_{l}+u_{g}+(u_{g}-u_{l})(\delta_{\tau}(x)-\delta_{+}(x)),-2u_{l}-(u_{g}-u_{l})(\delta_{\tau}(x)+\delta_{+}(x)),(u_{g}-u_{l})\delta_{+}(x)-2c)&\pi^{\ast}_{\mathbbm{O}}(x)\neq\pi^{\ast}_{\mathbbm{1}}(x)\\\
\end{array}\right.,$

and for $u_{g}<u_{l}$,

$\displaystyle\left(\begin{array}[]{c}c^{\pi^{o}}_{1}(x)\\\
c^{\pi^{o}}_{0}(x)\\\ c^{\pi^{o}}(x)\end{array}\right)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{l
l}(u_{g}-(u_{g}-u_{l})\delta_{+}(x),-u_{l}-(u_{g}-u_{l})\delta_{+}(x),(u_{g}-u_{l})\delta_{+}(x)-c)&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
(u_{l}+(u_{g}-u_{l})\delta_{\tau}(x),-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x),-c)&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\
(u_{l}+u_{g}+(u_{g}-u_{l})(\delta_{\tau}(x)-\delta_{+}(x)),-2u_{l}-(u_{g}-u_{l})(\delta_{\tau}(x)+\delta_{+}(x)),(u_{g}-u_{l})\delta_{+}(x)-2c)&\pi^{\ast}_{\mathbbm{O}}(x)\neq\pi^{\ast}_{\mathbbm{1}}(x)\\\
\end{array}\right..$

The maximum regret objective in Theorem 1 is a weighted average of the
expected potential outcomes under treatment and no treatment plus a proxy for
the cost. The choice of alternative policy $\varpi$ determines these weights
$c^{\varpi}_{0}(\cdot),c^{\varpi}_{1}(\cdot)$ and cost $c^{\varpi}(\cdot)$,
all of which potentially vary with the covariates $X$ (see Cui, 2021, for
additional discussion on the choice of comparison policy determining the form
of objective in instrumental variables settings).

Note that the symmetric utility case from Section 2.3 is also of this form,
with $c_{1}^{\varpi}(X)=-c_{0}^{\varpi}(X)=u_{g}$ and $c^{\varpi}(X)=c$. This
makes sense because it is a special case of the general asymmetric utilities
we consider. Similarly, the two classifiers in Equation (8) have this form,
with $\delta_{\tau}$ corresponding to $c_{1}^{\varpi}(X)=-c_{0}^{\varpi}(X)=1$
and $c^{\varpi}(X)=0$, and $\delta_{+}$ corresponding to
$c_{1}^{\varpi}(X)=c_{0}^{\varpi}(X)=1$ and $c^{\varpi}(X)=-1$.

The second line of Equation (11) shows how to write the worst-case regret
$R_{\sup}(\pi,\varpi)$ in terms of observable data using the scoring functions
$\Gamma_{w}$ (either IPW or DR) discussed in Section 2.3. So, targeting the
worst-case regret yields an objective function that is identifiable from the
observed data, unlike the true regret. As we will see below, this allows us to
construct decision rules based on observable data that control the true regret
by minimizing the worst-case regret. As such, the principal score $e_{01}(x)$
will not be involved in the remaining estimation strategies and results,
having been replaced with point-identifiable upper and lower bounds.

Next, we compute and inspect the policy that is the unconstrained minimizer of
the maximum regret in the population, relative to each of the three potential
comparison policies in turn. Then we will turn to estimating constrained
minimax policies in finite samples in Section 4 below.

####  3.3.1 Regret relative to a constant decision

We begin by considering the worst-case regret relative to always taking the
same decision: either the always-treat or never-treat policy.

######  Corollary 1 (Minimax regret relative to the never-treat policy).

If $u_{g}\geq u_{l}$, the minimax regret solution to Equation (7),
$\pi^{\ast}_{\mathbbm{O}}\equiv\text{argmin}_{\pi}R_{\sup}(\pi,\pi^{\mathbbm{O}})$
is the symmetric policy,

$\pi^{\ast}_{\mathbbm{O}}(x)=\mathbbm{1}\left\\{\tau(x)\geq\frac{c}{u_{g}}\right\\}=\pi^{\text{symm}}(x).$

Otherwise, if $u_{g}<u_{l}$, it is given by,

$\pi^{\ast}_{\mathbbm{O}}(x)=\left\\{\begin{array}[]{l
r}\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{l}}{u_{g}}m(0,x)+\frac{c}{u_{g}}\right\\},&\delta_{+}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{g}}{u_{l}}m(0,x)+\frac{u_{l}-u_{g}+c}{u_{l}}\right\\},&\delta_{+}(x)=1\end{array}\right..$

Corollary 1 shows that the form of the minimax regret policy depends on the
direction of the asymmetry. To build intuition, consider the case where the
treatment is costless, $c=0$. If $u_{g}>u_{l}$ — so we would rather cause an
undesirable outcome than to fail to prevent it — then the minimax solution
relative to the never-treat policy is the same as the optimal rule under a
symmetric utility function: assign the treatment when the CATE is positive. In
this case, the unit is more likely to be in the $(y_{1},y_{0})=(1,0)$
principal stratum than the $(y_{1},y_{0})=(0,1)$ stratum, and since
$u_{g}>u_{l}$, it will be better to treat the unit than to not. Conversely,
when the CATE is negative it may be possible that it is still better to treat
the unit, but in the worst case it is not. In order to minimize the worst-case
regret relative to never treating, the minimax policy does not treat.

However, the minimax solution is different when $u_{g}<u_{l}$ — i.e. when it
is worse to cause an undesirable outcome than it is to fail to prevent it. In
this case, the oracle rule depends on the value of the classifier
$\delta_{+}(x)=\mathbbm{1}\\{e_{00}(x)\leq e_{11}(x)\\}$. If both potential
outcomes are more likely to be zero than one, the probability that $Y(1)$
equals one must be higher than the probability that $Y(0)$ equals one by a
factor of $\frac{u_{l}}{u_{g}}>1$. Comparing to the decision rule under the
symmetric utility, we see that this raises the threshold for assigning the
treatment. In contrast, if both potential outcomes are more likely to be one
than zero, the threshold is raised by adding a constant cost
$\frac{u_{l}-u_{g}}{u_{l}}>0$, but the multiplicative factor on the
probability that $Y(0)$ equals one is $\frac{u_{g}}{u_{l}}<1$. Overall, this
has the effect of creating a more cautious policy that provides the treatment
less often. Finally, note that the weighting and cost functions depend on the
classifier $\delta_{+}(\cdot)$; if we know it (or can estimate it well), then
we can find the minimax policy relative to the never-treat policy
$\pi^{\mathbbm{O}}$; we discuss this further in Section 4 below.

The right plot (b) of Figure 1 shows the minimax decision rule relative to
$\pi^{\mathbbm{O}}$ (purple) in the one-dimensional example where
$u_{g}<u_{l}$ and $c=0$ — i.e. it is worse to cause than fail to prevent an
undesirable outcome. In this case, we see that the decision function is well
below the symmetric rule shown in green (i.e., CATE), leading to a large part
of the covariate space being assigned no treatment even though the CATE is
positive. In fact, this policy is overly cautious: it does not assign the
treatment even in many cases where the oracle rule that knows the principal
score would provide the treatment. This is because the alternative policy is
to never treat anyone.

######  Corollary 2 (Minimax regret relative to the always-treat policy).

If $u_{g}\geq u_{l}$, the minimax regret solution to Equation (7),
$\pi^{\ast}_{\mathbbm{1}}\equiv\text{argmin}_{\pi}R_{\sup}(\pi,\pi^{\mathbbm{1}})$,
is

$\pi^{\ast}_{\mathbbm{1}}(x)=\left\\{\begin{array}[]{l
r}\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{l}}{u_{g}}m(0,x)\right\\},&\delta_{+}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{g}}{u_{l}}m(0,x)+\frac{u_{l}-u_{g}+c}{u_{l}}\right\\},&\delta_{+}(x)=1\end{array}\right..$

Otherwise, if $u_{g}<u_{l}$, it is given by the symmetric policy,

$\pi^{\ast}_{\mathbbm{1}}(x)=\mathbbm{1}\left\\{\tau(x)\geq\frac{c}{u_{g}}\right\\}=\pi^{\text{symm}}(x).$

Corollary 2 shows that, as expected, the minimax regret policy relative to the
always-treat policy is the mirror image of that relative to the never-treat
policy. The weighting and cost functions are the same, but the relation to
$u_{g}$ and $u_{l}$ is reversed. Now if $u_{g}<u_{l}$, the minimax solution
$\pi^{\ast}_{\mathbbm{1}}$ is the symmetric policy, with similar intuition
that the always-treat policy makes the wrong tradeoff when we are more
concerned with causing an undesirable outcome than failing to prevent it. By
the same token, if $u_{g}\geq u_{l}$, the decision rule for
$\pi^{\ast}_{\mathbbm{1}}$ is equal to the rule for the minimax solution
relative to the never-treat policy, $\pi^{\ast}_{\mathbbm{O}}$, with
$u_{g}<u_{l}$. However, in this case the relation $u_{g}\geq u_{l}$ serves to
_decrease_ the threshold for providing the treatment and so leading to a more
aggressive policy.

####  3.3.2 Regret relative to the oracle

We next consider the regret relative to the oracle $\pi^{o}$ in Equation (6).
Unlike in the previous section, this oracle policy makes different decisions
depending on the value of both the CATE $\tau(x)$ and the principal score
$e_{01}(x)$. Since the principal score is unidentifiable under Assumption 1,
we must consider the worst-case regret across all possible values of the
principal score. The following corollary shows the corresponding minimax
regret policy. For simplicity, we write the policy for when the cost $c=0$;
when $c>0$, there will be further terms, as can be seen from the proof of
Theorem 1 in Appendix D.

######  Corollary 3 (Minimax regret relative to the oracle policy).

When $c=0$, the minimax solution solving Equation (7)
$\pi^{\ast}_{o}\equiv\text{argmin}_{\pi}R_{\sup}(\pi,\pi^{o})$ for $u_{g}\geq
u_{l}$ is given by,

$\pi^{\ast}_{o}(x)=\left\\{\begin{array}[]{c c}1&\delta_{\tau}(x)=1\\\
0&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{2u_{l}}{u_{g}+u_{l}}m(0,x)\right\\}&\delta_{\tau}(x)=0,\delta_{+}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{g}+u_{l}}{2u_{l}}m(0,x)+\frac{u_{l}-u_{g}}{2u_{l}}\right\\}&\delta_{\tau}(x)=0,\delta_{+}(x)=1\end{array}\right.$

And if $u_{g}<u_{l}$ the minimax solution is given by,

$\pi^{\ast}_{o}(x)=\left\\{\begin{array}[]{c
c}1&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\ 0&\delta_{\tau}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{u_{g}+u_{l}}{2u_{g}}m(0,x)\right\\}&\delta_{\tau}(x)=1,\delta_{+}(x)=0\\\
\mathbbm{1}\left\\{m(1,x)\geq\frac{2u_{g}}{u_{g}+u_{l}}m(0,x)+\frac{u_{l}-u_{g}}{u_{g}+u_{l}}\right\\}&\delta_{\tau}(x)=1,\delta_{+}(x)=1\end{array}\right.$

Theorem 1 and Corollary 3 show that we can write the worst-case regret and the
minimax policy relative to the oracle in terms of observable data, just as for
the constant policies above. This, however, requires _four_ classifiers rather
than one: (i) $\delta_{+}$, which classifies whether $e_{00}(x)\leq
e_{11}(x)$; (ii) $\delta_{\tau}$, which classifies whether the CATE is
positive; and (iii) the minimax solution relative to $\pi^{\mathbbm{1}}$ and
(iv) the minimax solution relative to $\pi^{\mathbbm{O}}$. Recall from
Corollaries 1 and 2 that either $\pi^{\mathbbm{O}}$ (when $u_{g}\geq u_{l}$)
or $\pi^{\mathbbm{1}}$ (when $u_{g}<u_{l}$) is the symmetric policy
$\pi^{\text{symm}}$. Therefore, if the cost $c=0$ as in Corollary 3, then we
in fact only need _three_ classifiers to construct the objective, since
$\pi^{\text{symm}}=\delta_{\tau}$ in this case.

Finally, note that here we consider the minimax optimal _deterministic_ policy
relative to the oracle policy. The form of the minimax regret relative to the
oracle in Theorem 1 — and so the form of $\pi^{\ast}_{o}$ in Corollary 3 —
relies on restricting to deterministic policies. However, as Cui (2021)
discusses, unlike with the regret relative to the always treat or never treat
policies, allowing for _stochastic_ policies that randomize between actions
can lead to lower regret relative to the oracle, though this leads to a more
complicated form for the regret. We leave further understanding the
implications for stochastic policies to future work and here continue to focus
on deterministic policies.

Inspecting the minimax solution relative to the oracle policy when $u_{g}\geq
u_{l}$, we see that it assigns the treatment if the symmetric rule does,
whereas it does not provide the treatment if the minimax solution relative to
the always-treat policy does not. In between these two extremes, the decision
rule lowers the threshold for the treatment assignment relative to the
symmetric rule. The opposite is true when $u_{g}<u_{l}$. If the symmetric rule
does not assign treatment, the minimax solution relative to the oracle does
not either, but it does provide the treatment whenever the minimax solution
relative to the always-treat policy does. In between these two cases, the
threshold for treatment assignment is higher than that under the symmetric
rule.

The right plot (b) of Figure 1 shows the decision rule (pink) in our running
one-dimensional example where $u_{g}<u_{l}$. The decision rule is equivalent
to $\pi^{\ast}_{\mathbbm{O}}$ (purple) when the CATE is negative, and is equal
to the CATE decision rule $\delta_{\tau}$ when
$\pi^{\ast}_{\mathbbm{O}}(x)=1$. When there is disagreement between the CATE
rule $\delta_{\tau}$ and $\pi^{\ast}_{\mathbbm{O}}$, the minimax oracle rule
interpolates between them, leading to a more aggressive policy than
$\pi^{\ast}_{\mathbbm{O}}$ that treats individuals in some cases that
$\pi^{\ast}_{\mathbbm{O}}$ does not. Comparing to the oracle rule (orange), we
see that this interpolation causes the decision thresholds for the minimax
oracle rule to be close to the best possible decision thresholds.

##  4 The empirical minimax policy

Having established the behavior and form of the minimax policy $\pi^{\ast}$ in
Equation (7) in the population for an unconstrained policy class, we now turn
to learning a policy $\hat{\pi}$ empirically from observed data using a
constrained policy class $\Pi$.

###  4.1 Estimation algorithms

To begin, note that if we knew the weighting and cost functions
$c^{\varpi}_{1}(\cdot),c^{\varpi}_{0}(\cdot),c^{\varpi}(\cdot)$, then we can
follow existing procedures and minimize the sample analog to
$R_{\text{sup}}(\pi,\varpi)$, using estimated scoring functions
$\widehat{\Gamma}_{w}(X_{i},D_{i},Y_{i})$:

$\tilde{R}_{\text{sup}}(\pi,\varpi)=-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left\\{c^{\varpi}_{1}(X_{i})\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+c^{\varpi}_{0}(X_{i})\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})+c^{\varpi}(X_{i})\right\\},$
(12)

where we drop the constant term that does not depend on the policy $\pi$.

1:Policy classes $\Pi$ and $\Delta_{+}$

2:Empirical minimax policy $\hat{\pi}$ relative to $\varpi$

3:Find $\hat{\delta}_{+}$ by solving
$\min_{\delta\in\Delta_{+}}-\frac{1}{n}\sum_{i=1}^{n}\delta(X_{i})\left\\{\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})-1\right\\}$

4:Compute weighting and cost functions
$\hat{c}^{\varpi}_{1}(x)=u_{g}+\hat{\delta}_{+}(x)(u_{l}-u_{g}),\;\hat{c}^{\varpi}_{0}(x)=-u_{l}-\hat{\delta}_{+}(x)(u_{g}-u_{l})\text{
and }\hat{c}^{\varpi}(x)=\hat{\delta}_{+}(x)(u_{g}-u_{l})$

5:Find the empirical minimax policy
$\hat{\pi}\in\underset{\pi\in\Pi}{\text{argmin}}\;\hat{R}_{\text{sup}}(\pi,\varpi)$

Algorithm 1 Empirical minimax policy $\hat{\pi}$ relative to the always-treat
policy $\pi^{\mathbbm{1}}$ (when $u_{g}\geq u_{l}$) and the never-treat policy
$\pi^{\mathbbm{O}}$ (when $u_{g}<u_{l}$)

However, Theorem 1 shows that the true functions involve unknown classifiers.
In the absence of knowledge about these classifiers, we estimate them and then
compute _estimates_ of the weighting and cost functions
$\hat{c}^{\varpi}_{1}(\cdot),\hat{c}^{\varpi}_{0}(\cdot)$, and
$\hat{c}^{\varpi}(\cdot)$ by plugging in the classifiers that are fit to data.
We then find an empirical minimax policy $\hat{\pi}$ that solves the following
optimization problem:

$\hat{\pi}\in\underset{\pi\in\Pi}{\text{argmin}}\
\underbrace{-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left\\{\hat{c}^{\varpi}_{1}(X_{i})\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}_{0}(X_{i})\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}(X_{i})\right\\}}_{\hat{R}_{\sup}(\pi,\varpi)}.$
(13)

Note that for the intermediate classifiers $\delta_{+}$ and $\delta_{\tau}$,
the corresponding weight and cost functions are known (and constant), so we
can estimate them by minimizing a form of Equation (12).

This leads to the following procedures. Algorithm 1 shows the steps to finding
the empirical minimax policy relative to either the always-treat (if
$u_{g}\geq u_{l}$) or never-treat (if $u_{g}<u_{l}$) policies.111Recall that
the minimax policy relative to the always-treat policy when $u_{g}<u_{l}$ and
relative to the never-treat policy when $u_{g}\geq u_{l}$ reduces to the usual
symmetric policy, so we do not focus on these cases. First, minimize
$\tilde{R}_{\text{sup}}$ in Equation (12) to obtain an estimate
$\hat{\delta}_{+}$, then estimate the weighting and cost functions and solve
Equation (13). This involves estimating one intermediate or nuisance
classifier, $\delta_{+}$. As we show in Section 4.2 below, this nuisance
classifier must be accurate in order for our learned minimax policy
$\hat{\pi}$ to have low worst-case regret; therefore, we must choose a policy
class $\Delta_{+}$ that is sufficient to include the true classifier
$\delta_{+}$.

On the other hand, finding the minimax policy relative to the oracle involves
estimating _three_ or _four_ nuisance classifiers depending on whether $c=0$,
as Algorithm 2 shows. First, estimate $\hat{\delta}_{+}$ as above; then
estimate the minimax policy relative to the always-treat and never treat
policies; then estimate the CATE classifier $\delta_{\tau}$. Finally, use
these nuisance classifiers to estimate the weighting and cost functions via
Theorem 1 and find the minimax policy by solving Equation (13). Again, the
particular choice of policy classes for these nuisance classifiers will be
important: they must be flexible enough to have low classification error.

1:Policy classes $\Pi$, $\Pi^{\prime}$, $\Delta_{+}$, and $\Delta_{\tau}$

2:Empirical minimax policy $\hat{\pi}$ relative to the oracle $\pi^{\ast}$

3:Find $\hat{\delta}_{+}$ by solving
$\min_{\delta\in\Delta_{+}}-\frac{1}{n}\sum_{i=1}^{n}\delta(X_{i})\left\\{\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})-1\right\\}$

4:Find $\hat{\pi}_{\mathbbm{1}}$ (for $u_{g}\geq u_{l}$) or
$\hat{\pi}_{\mathbbm{O}}$ (for $u_{g}<u_{l}$) via Algorithm 1 with policy
class $\Pi^{\prime}$.

5:If $c\neq 0$, find $\hat{\pi}_{\mathbbm{O}}$ (for $u_{g}\geq u_{l}$) or
$\hat{\pi}_{\mathbbm{1}}$ (for $u_{g}<u_{l}$) by solving
$\min_{\pi\in\Pi^{\prime}}-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left[u_{g}\left\\{\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})-\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})\right\\}-c\right]$

6:Find $\hat{\delta}_{\tau}$ by solving
$\min_{\delta\in\Delta_{\tau}}-\frac{1}{n}\sum_{i=1}^{n}\delta(X_{i})\left\\{\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})-\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})\right\\}$

7:Compute weighting and cost functions
$\hat{c}^{\pi^{\ast}}_{1}(x),\hat{c}^{\pi^{\ast}}_{0}(x),\hat{c}^{\pi^{\ast}}(x)$
via Theorem 1

8:Find the empirical minimax policy
$\hat{\pi}\in\underset{\pi\in\Pi}{\text{argmin}}\hat{R}_{\text{sup}}(\pi,\pi^{o})$

Algorithm 2 Empirical minimax policy $\hat{\pi}$ relative to the oracle policy
$\pi^{o}$

The optimization problems in Equation (13) and Algorithms 1 and 2 are not
convex in general. This makes them difficult to solve exactly. Some bespoke
procedures exist for certain policy classes (e.g. Sverdrup et al., 2020, for
finite-depth decision rules). In more general settings we can solve a convex
relaxation of the optimization problems by noting that they are equivalent to
a weighted classification problem and using surrogate losses (see, e.g. Zhao
et al., 2012; Zhang et al., 2012; Kitagawa and Tetenov, 2018; Pu and Zhang,
2021). Below, we use a hinge-loss and approximately solve Equation (13) via a
weighted SVM. See Appendix C for details on the optimization procedure.

###  4.2 Excess worst-case regret

To understand the statistical properties of our empirical minimax policy
$\hat{\pi}$, we will compare it to the policy $\pi^{\ast}$ that minimizes the
worst-case regret in the population among those in the policy class $\Pi$ by
solving Equation (7). For a given alternative policy $\varpi$, we will use the
excess worst-case regret
$R_{\text{sup}}(\hat{\pi},\varpi)-R_{\text{sup}}(\pi^{\ast},\varpi)$ to
measure the quality of the empirical minimax policy $\hat{\pi}$ since
$R_{\text{sup}}(\pi^{\ast},\varpi)$ is the best possible regret in the worst
case.

There are two reasons that the empirical policy $\hat{\pi}$ can differ from
the population policy $\pi^{\ast}$. First, the difference between the
empirical and population regret when the functions
$c^{\varpi}_{1}(\cdot),c^{\varpi}_{0}(\cdot),c^{\varpi}(\cdot)$ are known
(i.e., $\tilde{R}_{\text{sup}}(\pi,\varpi)-R_{\text{sup}}(\pi,\varpi)$).
Second, the error in estimating
$c^{\varpi}_{1}(\cdot),c^{\varpi}_{0}(\cdot),c^{\varpi}(\cdot)$ (i.e.,
$\hat{R}_{\text{sup}}(\pi,\varpi)-\tilde{R}_{\text{sup}}(\pi,\varpi)$). We
formalize this in the following proposition.

######  Proposition 1.

Let $\varpi$ be a fixed policy. For $\hat{\pi}$ solving the empirical worst-
case regret minimization problem in Equation (13) and $\pi^{\ast}$ solving the
population problem in Equation (7), the excess regret of $\hat{\pi}$ relative
to $\pi^{\ast}$ satisfies

$\displaystyle R_{\sup}(\hat{\pi},\varpi)-R_{\sup}(\pi^{\ast},\varpi)$
$\displaystyle\leq\sup_{\pi\in\Pi}|R_{\sup}(\pi,\varpi)-\tilde{R}_{\sup}(\pi,\varpi)|$
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}|\hat{c}^{\varpi}(X_{i})-c^{\varpi}(X_{i})|+\frac{\xi}{n}\sum_{w=0}^{1}\sum_{i=1}^{n}\left|\hat{c}_{w}^{\varpi}(X_{i})-c_{w}^{\varpi}(X_{i})\right|,$

where
$\xi=\max_{i=1,\ldots,n,w=0,1}\left|\widehat{\Gamma}_{w}(X_{i},D_{i},Y_{i})\right|$.

Proposition 1 shows that the excess regret is controlled by two terms. First
is the worst-case difference between the empirical and population regret when
the weighting and cost functions are known. This can further be controlled via
empirical process techniques as we discuss below.

The second term depends on the estimation error for the weighting functions,
scaled by the maximum estimated scoring function.222The constant $\xi$ will
depend on the particular choice of scoring function. For example, for the IPW
function with standardized IP weights, $\xi=1$ because the outcome is binary.
This is related to the classification error for the classifiers above. To see
this, consider the specialization of Proposition 1 to the case where the
alternative policy is the zeros policy $\pi^{\mathbbm{O}}$ and $u_{g}<u_{l}$.

######  Corollary 4.

Let $u_{g}<u_{l}$. For $\hat{\pi}$ solving the empirical worst-case regret
minimization problem in Equation (13) and $\pi^{\ast}$ solving the population
problem in Equation (7), with alternative policy $\varpi=\pi^{\mathbbm{O}}$,
the excess regret of $\hat{\pi}$ relative to $\pi^{\ast}$ satisfies

$\displaystyle
R_{\sup}(\hat{\pi},\pi^{\mathbbm{O}})-R_{\sup}(\pi^{\ast},\pi^{\mathbbm{O}})$
$\displaystyle\leq\sup_{\pi\in\Pi}|R_{\sup}(\pi)-\tilde{R}_{\sup}(\pi)|+3\xi\frac{u_{l}-u_{g}}{n}\sum_{i=1}^{n}\mathbbm{1}\\{\hat{\delta}_{+}(X_{i})\neq\delta_{+}(X_{i})\\},$

where
$\xi=\max_{i=1,\ldots,n,w=0,1}\left|\widehat{\Gamma}_{w}(X_{i},D_{i},Y_{i})\right|$.

In this case, we see that the misclassification rate for $\hat{\delta}_{+}$
controls the component of the error due to estimating $c^{\varpi}_{1}(\cdot)$,
$c^{\varpi}_{0}(\cdot)$, and $c^{\varpi}(\cdot)$. If $\hat{\delta}_{+}$ can
correctly classify all of the units $i$ into whether $e_{00}(X_{i})\leq
e_{11}(X_{i})$, then there will be no error due to this term. Therefore, we
must have an accurate nuisance classifier in order to be able to learn the
minimax optimal policy.

To get a sense of the impact of the accuracy of the classifier(s) on the
excess regret, we can consider the influence of the two terms in the bound.
For policy classes $\Pi$ with finite VC dimension, and when the propensity
score $d(\cdot)$ is known, Kitagawa and Tetenov (2018) show that the IPW score
will lead the first term in Proposition 1 to converge at the optimal
$n^{-1/2}$ rate. Athey and Wager (2021) show the same for an appropriately
cross-fit DR score when the propensity score is not known. This term depends
on the VC dimension of the policy class $\Pi$, which is user-defined: we can
choose the complexity of $\Pi$ based on what we want to implement in practice,
and the results above will be relative to the best possible policy in $\Pi$.

On the other hand, the results from Kitagawa and Tetenov (2018) and Athey and
Wager (2021) imply that we should expect the term due to the classification
error to depend on VC dimension of the nuisance classifier class $\Delta_{+}$.
Unlike for the minimax policy class $\Pi$, unless the nuisance classifier
class contains the _true_ function $\delta_{+}\in\Delta_{+}$, there will be
irreducible approximation error in the misclassification term. Therefore, we
might choose a class $\Delta_{+}$ with a higher VC dimension than $\Pi$, or
one that is increasing with the sample size $n$. In that case, the
misclassification error will primarily control the excess regret, as this will
either have a larger constant term (when $\Delta_{+}$ has a finite VC
dimension) or converge at a slower rate than $n^{1/2}$ (when $\Delta_{+}$ has
a growing VC dimension).

Finally, note that an alternative to finding $\hat{\delta}_{+}$ via regret
minimization is to use a plug-in estimate
$\hat{\delta}_{+}(x)=\mathbbm{1}\\{\hat{m}(0,x)+\hat{m}(1,x)\geq 1\\}$. Plug-
in approaches such as these are typically analyzed under a _margin assumption_
that there is little mass near the decision boundary for the nuisance
functions. These types of margin assumptions are distinct from assumptions on
the policy class that the true decision function $\delta_{+}(x)$ belongs to,
and so cannot be directly compared. Audibert and Tsybakov (2007) show that
with a margin assumption, plug-in decision rules can also lead to fast rates
for the misclassification error, where stronger margin assumptions can lead to
weaker assumptions on the estimators of the underlying nuisance functions. See
Kallus (2022) for an application of these techniques to estimating bounds on
$P(Y(1)<Y(0))$.

##  5 Simulation study

As the results in Section 4.2 above show, the misclassification rates of the
underlying nuisance classifiers are important in controlling the excess regret
due to estimating the weighting and cost functions that make up the worst-case
regret. Additionally, although the minimax policies we consider are designed
to minimize the worst-case regret, in some cases it may be possible that the
true, unidentifiable regret may also be small. To inspect how the
misclassification rates and the true regret behave in finite samples, as the
sample size increase, we now conduct a brief simulation study, where we can
know the true values of the principal scores $e_{y_{1}y_{0}}(x)$.

We generate $n$ 1-dimensional i.i.d. covariates $X_{i}\sim N(0,2)$, then we
construct log-linear principal scores as

$e_{y_{1}y_{0}}(x)=\frac{\exp\left(\alpha_{y_{1}y_{0}}+x\beta_{y_{1}y_{0}}\right)}{\sum_{y_{1}^{\prime}=0}^{1}\sum_{y_{0}^{\prime}=0}^{1}\exp\left(\alpha_{y_{1}^{\prime}y_{0}^{\prime}}+x\beta_{y_{1}^{\prime}y_{0}^{\prime}}\right)},$

where $(\alpha_{00},\alpha_{10},\alpha_{01},\alpha_{11})=(.2,.15,0,0)$,
$\beta_{y_{1}y_{0}}\sim N(0,40)$ for
$(y_{1},y_{0})\in\\{(0,0),(1,0),(0,1)\\}$, and $\beta_{11}=0$. We then jointly
sample potential outcomes $Y_{i}(1),Y_{i}(0)$ according to the principal
scores at covariate value $X_{i}$. In this simulation study, we consider a
randomized control trial with binary treatment $D_{i}$ sampled independently
as Bernoulli random variables with probability one half.

(a) Misclassification rate for nuisance classifiers

(b) Regret of minimax policy relative to oracle

Figure 2: Performance of nuisance classifiers and the minimax optimal policy
relative to the oracle across simulation runs. Panel (a) shows the
misclassification rate for the nuisance classifiers $\hat{\delta}_{+}$ (“More
likely in (1,1) stratum”) and $\hat{\delta}_{\tau}$ (“Positive CATE”), as well
as the minimax policy relative to the never-treat policy for $u_{l}=0.833$.
Panel (b) shows the true regret of the minimax optimal policy relative to the
oracle, in the sample, for $u_{g}=1$ and as $u_{l}$ varies between 0.5 and
1.5.

For each value of sample size $n\in\\{100,500,1000,5000,10000\\}$, we draw
1,000 samples according to the above data generating process. In each
simulation run, we find the minimax optimal policy with respect to the oracle
following Algorithm 2 with zero cost $c=0$, $u_{g}=1$, and $u_{l}$ varying
between 0.6 and 1.4, where the value of $u_{l}$ changes within each simulation
run.

We use the IPW scoring function and restrict all policy classes to be the set
of linear thresholds, solving the optimization problem exactly by direct
search. Figure 2(a) shows the average misclassification rate for the nuisance
classifiers $\hat{\delta}_{+}$ and $\hat{\delta}_{\tau}$, as well as the
misclassification rate for the the minimax policy relative to always treating
for $u_{l}=0.833$. As we expect, we see that these misclassification rates
decrease as the sample size increases.

Figure 2(b) shows the true regret of the minimax policy relative to the oracle
as $u_{l}$ varies. Since the oracle is the best possible policy, this regret
is always positive. The regret does decrease along with the sample size,
reflecting both the decrease in the nuisance misclassification rate and the
decrease in the worst-case excess regret when the nuisance classifiers are
known. Notice, however, that the regret does stop decreasing after a certain
point, flattening out at a different level depending on the asymmetry in the
utility function. In highly asymmetric settings where $u_{\ell}$ is small the
regret is essentially flat. This is due to the fundamental identifiability
problem, and even with infinite data we cannot guarantee that the true regret
will be zero. In contrast, in the symmetric setting the regret continues to
decrease as the sample size increases.

##  6 Application to Right Heart Catheterization

We now apply the proposed framework and procedures to a particular decision
problem: whether or not to use Right Heart Catheterization (RHC) in a clinical
setting. RHC is a diagnostic tool where a catheter is inserted into the
pulmonary artery. In a controversial observational study, Connors et al.
(1996) found that RHC led to an increase in mortality on average. RHC,
however, can lead to life-saving treatment for some patients. In this section,
we will use the data from Connors et al. (1996) to learn policies for using
RHC for certain patients, inspecting how asymmetry in the policy maker’s
utility function can lead to different data driven decision-making processes.

###  6.1 Data and setup

The data from Connors et al. (1996) include $n=5,735$ ICU patients, 2,184 of
whom had RHC applied. We will code the outcome $Y(d)=1$ as survival by thirty
days. In this case, the utility value $u_{g}$ represents the utility gain in
saving a patient’s life under RHC who would otherwise die without RHC, and
$u_{l}$ represents the cost of RHC leading to the death of a patient who would
otherwise survive. In this study, RHC use was not experimentally randomized
and so we will be relying on Assumption 1, using the same set of socioeconomic
and health characteristics as Hirano and Imbens (2001) use in their propensity
score-based analysis.

Throughout, we will use the estimated doubly robust score
$\widehat{\Gamma}^{\text{dr}}_{w}$. To do so, we need estimates of the
conditional expectation function $m(w,\cdot)$ and the propensity score
$d(\cdot)$. We use a cross-fitting procedure to estimate these conditional
expectations. To estimate the conditional probability of survival without RHC,
$m(0,\cdot)$, we split the non-RHC patients into three folds. For each fold,
we estimate $\hat{m}(0,\cdot)$ on the two other folds via gradient boosted
decision stumps (Friedman, 2001) and obtain estimates $\hat{m}(0,X_{i})$ for
the held out fold. For the RHC patients we take the median estimate across the
three folds. We mirror this procedure to estimate the conditional survival
probability with RHC, $\hat{m}(1,X_{i})$. We similarly use a 3-fold cross-
fitting procedure to estimate the conditional probability of receiving RHC
$\hat{d}(X_{i})$.

To fit each of these models, we use the full set of 66 covariates that Hirano
and Imbens (2001) use; below we consider decision rules that use a subset of
these variables, but that are based on the DR scores estimated with all of the
covariates. With the combined DR scores, we estimate that RHC led to an
overall increase in 30-day mortality by 4.6 percentage points, with an
estimated standard error of 1.2 percentage points, consistent with other
analyses.

###  6.2 Decision rule with two variables

(a) Relative to all RHC

(b) Relative to oracle policy

Figure 3: Minimax quadratic decision rules relative to (a) using RHC for all
patients and (b) the oracle policy as $u_{l}$ varies and $u_{g}=1$ using two
variables: the estimated probability of survival and whether a patient has a
Do-Not-Resuscitate (DNR) order. The results are shown separately for patients
with (right) and without (left) an DNR order. The dashed line and shaded area
are the estimated conditional average treatment effect (CATE) and 95%
confidence interval, respectively.

For illustration, we begin by considering low-dimensional decision rules that
only use two clinical variables: the estimated probability of surviving two
months and whether the patient has a Do-Not-Resuscitate (DNR) order.
Throughout we will use decision rules that are quadratic in the estimated
probability of surviving two months, and separate for DNR and non-DNR
patients. First, we will consider minimizing the worst-case regret relative to
using RHC for all patients, following Algorithm 1 with $u_{g}\geq u_{l}$. The
first step in the procedure is learning the classifier $\hat{\delta}_{+}$,
which we do by solving the SVM approximation to Equation (13) (Appendix Figure
B.1 shows the resulting decision rule). Then we create estimates of the
weighting and cost functions
$\hat{c}_{0}^{\mathbbm{1}}(\cdot),\hat{c}_{1}^{\mathbbm{1}}(\cdot),\hat{c}^{\mathbbm{1}}(\cdot)$
and approximately solve Equation (13) again to estimate the minimax quadratic
policy $\hat{\pi}_{\mathbbm{1}}$ relative to always using RHC.

Figure 3(a) shows the resulting decision rules, centered so that the policy
uses RHC when the function is greater than zero, when we set $u_{g}=1$ and
vary $u_{l}\in[0.8,1]$ for patients with (right) and without (left) a DNR
order. We also estimate the CATE $\tau(x)$ conditioned on the estimated
probability of survival and DNR status by regressing the difference in DR
scores $\widehat{\Gamma}_{1}^{\text{dr}}-\widehat{\Gamma}_{0}^{\text{dr}}$ on
a quadratic basis for the estimated probability, interacted by DNR status
(Knaus, 2020). Note that the estimated CATE is only positive for non-DNR
patients with a low probability of survival, and even then the effect is near
zero. In the symmetric case when $u_{l}=u_{g}=1$, this leads to a decision
rule that _never_ uses RHC (yellow line). As $u_{l}$ decreases so that the
utility gain in saving a life becomes greater than the cost of causing death,
the decision rule uses RHC for non-DNR patients with a lower estimated
probability of surviving, with the threshold for non-DNR patients
monotonically increasing as $u_{l}$ decreases. Even though we optimize over
all quadratic decision rules — which can use two thresholds to determine the
RHC policy — each decision rule in fact only uses one threshold, showing that
linear decision rules would have sufficed.

However, the decision rule never uses RHC for DNR patients as indicated by the
fact that all lines are below zero. In the extreme case when $u_{l}=0.8$ (so
it is 25% better to save a life than it is worse to treat a patient who would
die) the decision rule uses RHC for all non-DNR patients and avoids it for all
DNR patients. Even though the CATE is indistinguishable from zero for non-DNR
patients, the asymmetry in the utility function pushes the decision rule
towards using RHC. In contrast, the CATE is so strongly negative for DNR
patients that the rule continues not to use RHC regardless of the asymmetry.
Although we do not include this in Figure 3, when considering the minimax
policy relative to never using RHC, with $u_{l}>u_{g}$, the asymmetry works in
the opposite direction and the minimax rule never uses RHC just as in the
symmetric case. Here, the particular _shape_ of the decision rule — whether it
is convex or concave — is immaterial as all of the decision rules are the
same: do not use RHC at any estimated survival probability for DNR patients.

Next, we consider finding the minimax policy relative to the oracle
$\hat{\pi}_{o}$, following Algorithm 2 and using the estimates of the
classifiers $\hat{\delta}_{+},\hat{\delta}_{\tau},$ and
$\hat{\pi}_{\mathbbm{1}}$ that we have constructed so far. Figure 3(b) shows
the decision functions. Similar to the minimax policy relative to always using
RHC, DNR patients are never assigned RHC, regardless of their estimated
probability of survival and the utility function. But the rules do differ for
non-DNR patients. As $u_{l}$ decreases, the threshold for using RHC based on
the estimated probably of survival rises, though not monotonically. However,
even in the extreme case where $u_{l}=0.8$, not all non-DNR patients are
assigned RHC. Matching our general discussion in Section 3.3, measuring regret
relative to the best possible policy leads to a less aggressive decision rule
than measuring regret relative to always using RHC.

###  6.3 Decision rules with all clinical variables

Next, we move from low dimensional decision rules to high dimensional rules
involving all 56 clinical variables (see Hirano and Imbens, 2001, , Table 1,
for a full list of covariates). With this many variables, we restrict our
investigation to linear policies rather than the quadratic policies above. As
before, we first consider the minimax policies relative to the constant
decisions as the utility asymmetry changes with $u_{g}=1$ and
$u_{l}\in[0.65,1.2]$, where here we use a larger range than the previous
section in order to explore the full range of potential policies with all
clinical variables. Using Algorithm 1, for $u_{l}<u_{g}$, we find the minimax
policy relative to always using RHC, $\hat{\pi}_{\mathbbm{1}}$; for
$u_{l}>u_{g}$, we find the minimax policy relative to never using RHC,
$\hat{\pi}_{\mathbbm{O}}$. Again, we first learn the classifier
$\hat{\delta}_{+}$ via the SVM approximation, and then create estimates of the
weighting and cost functions. Lastly, we learn the minimax linear decision
rules relative to the oracle by using Algorithm 2 with the classifier
$\hat{\delta}_{+}$ and the symmetric and asymmetric policies
$\hat{\delta}_{\tau}$, $\hat{\pi}_{\mathbbm{O}}$, and
$\hat{\pi}_{\mathbbm{1}}$.

Figure 4: A subset of coefficients on minimax linear decision rules relative
to always and never using RHC with $u_{l}\in[0.65,1.2]$ and $u_{g}=1$.
Highlighted column corresponds to the symmetric policy with $u_{l}=u_{g}$.

(a)

(b)

Figure 5: Percent of patients assigned RHC under minimax decision rules
relative to (a) always or never using RHC and (b) the oracle policy, with
$u_{l}\in[0.65,1.2]$ and $u_{g}=1$. The line is a smoothed fit. In both
panels, the symmetric policy is highlighted in red.

Figure 4 shows a subset of the coefficients of the resulting minimax linear
decision rules relative to the constant policies, with the symmetric rule
highlighted. Figure 5(a) shows the percent of patients assigned RHC under the
different decision rules. All covariates have been scaled to be between 0 and
1. The symmetric decision rule uses two key risk measures to determine when
_not_ to use RHC: the estimated probability of survival in two months, and the
APACHE score, a classification system that assigns a higher number of points
to patients with higher risk of mortality (Knaus et al., 1985). Both factors
have negative coefficients in the symmetric linear rule, so that if the
patient’s estimated probability of survival is high or conversely they are
classified as higher risk from the APACHE risk score, the underlying linear
decision function is lower. Other measures work in the opposite direction,
e.g., whether the patient has a trauma diagnosis, was recently transferred
from another hospital, or suffers from chronic renal disease.

As we move away from the symmetric case towards prioritizing using RHC for
patients that will benefit from it, the decision rule changes. For instance,
it begins to place more weight on other factors such as the stomach pH level.
In the extreme case all patients are assigned RHC. In the other direction, as
we increase $u_{l}$ relative to $u_{b}$ and so seek to prevent avoidable
deaths, the decision rule also changes. In particular, there is a sharp
transition where the policy primarily only uses RHC for the few patients with
a trauma diagnosis, chronic renal disease, or lung cancer. We can compare this
to the minimax policies relative to the oracle; Figure 5(b) shows the percent
of patients assigned RHC as we vary $u_{l}$. We see smaller changes to percent
of patients assigned RHC. In particular, while the minimax policy relative to
the always- or never-RHC policies moves between always or never using RHC, the
percentage of patients assigned RHC under the minimax policy relative to the
oracle is relatively stable. As with the two-covariate case in Figure 3 and
our discussion in Section 3.3.2, the minimax regret policy relative to the
oracle is less aggressive, and stays near the symmetric case.

Figure 6: Upper bounds on the proportion of patients that do not receive a
useful treatment — the false negative rate — and the proportion of patients
given a harmful treatment — the false positive rate — for linear minimax
policies relative to always or never using RHC, as $u_{l}$ varies in
$[0.65,1.2]$ and $u_{g}=1$. The line is a convex fit. The red point is the
symmetric linear policy.

Finally, we can measure the impact of these policies in terms of directly
interpretable patient outcomes rather than the expected regret. Recall that we
can view the expected utility as the Lagrangian form of the constrained
optimization problem in Equation (3), where we seek to minimize the proportion
patients that we fail to give a useful treatment to — the false negatives —
while constraining the proportion of patients for which the policy is harmful.
With zero cost $c=0$ in the regret, varying $u_{l}$ relative to $u_{b}$ with a
constant comparison policy $\varpi$ is equivalent to varying the constraint on
the proportion of harmful treatments, without a budget constraint. Following
the argument in Section 3, we can find upper bounds on these probabilities for
a policy $\pi$ as

$\displaystyle\sup_{e_{01}(x)\in[L(x),U(x)]}P(Y(\pi(X))<Y(0))$
$\displaystyle=\mathbb{E}\left(\pi(X)\left[m(0,X)+\delta_{+}(X)\left\\{1-m(1,X)-m(0,X)\right\\}\right]\right)$
$\displaystyle\sup_{e_{01}(x)\in[L(x),U(x)]}P(Y(\pi(X))<Y(1))$
$\displaystyle=\mathbb{E}\left(\left\\{1-\pi(X)\right\\}\left[m(1,X)+\delta_{+}(X)\left\\{1-m(1,X)-m(0,X)\right\\}\right]\right).$

Using the learned classifier $\hat{\delta}_{+}$ and the DR scores
$\widehat{\Gamma}_{w}$, we can get plug-in estimates of these upper bounds for
each of our policies $\pi$.

Figure 6 shows the worst-case proportion of patients for whom $\pi$ fails to
give a useful treatment (vertical axis,
$\sup_{e_{01}(x)\in[L(x),U(x)]}P(Y(\pi(X))<Y(1))$ for $u_{l}<u_{g}$) versus
the worst-case proportion for whom $\pi$ gives a harmful treatment (horizontal
axis, $\sup_{e_{01}(x)\in[L(x),U(x)]}P(Y(\pi(X))<Y(0))$ for $u_{l}>u_{g}$) as
we vary $u_{l}\in[0.65,1.2]$. We observe the trade-off between these two types
of errors, in the worst-case. If $\pi$ treats no one, then in the worst-case
no patients receive a harmful treatment, but $\pi$ could be failing to help up
to 25% of patients. Conversely, if $\pi$ treats everyone, then $\pi$
necessarily treats everyone for whom it is helpful, but could be providing a
harmful treatment for up to 30% of patients. The intermediate points in Figure
6 show the frontier between these two extreme scenarios.

Along with this trade-off between the two types of errors, we can also see the
impact on overall mortality as we move along the frontier. Figure 7 shows the
estimated expected mortality under each policy
$\hat{\mathbb{E}}[\widehat{\Gamma}_{\pi(X)}(X,D,Y)]$ (vertical axis) plotted
against the worst-case proportion of patients (horizontal axis) for whom $\pi$
fails to give a useful treatment or gives a harmful treatment as we vary
$u_{l}\in[0.65,1.2]$. The figure shows the trade-off between reducing
mortality and potentially failing to give a useful treatment to a patient or
harming them (the constraints). Note that because we are only approximately
solving the empirical minimax regret optimization problem in Equation (13),
this is not an exact frontier, so we include the convex function that best
fits the points.

Figure 7: Estimated expected mortality versus the worst case proportion of
cases where the policy fails to give a useful treatment (left panel) and where
the policy gives a harmful treatment (right panel), for linear minimax
policies relative to never using RHC, as $u_{l}$ varies in $[0.65,1.2]$ and
$u_{g}=1$. The line is a convex fit. The red point is the symmetric linear
policy.

At one extreme, the policy that always uses RHC has the highest expected
mortality, because on average RHC is harmful. As we allow for the possibility
of failing to treat a higher percentage of patients for whom RHC would be
useful — and so the policy uses RHC more judiciously — the expected mortality
decreases. The symmetric policy is at the other end of this tradeoff, with a
lower expected mortality but potentially failing to use RHC when it is useful
for over 13% of patients. On the other extreme, the policy that never uses RHC
has a lower expected mortality than always using RHC, but a higher one than
the symmetric policy. Here, there is also a tradeoff between the expected
mortality and the worst-case proportion of patients receiving a harmful
treatment, though it is less stark since RHC is harmful on average. We find
that while the symmetric policy reduces mortality, up to 10% of patients will
receive RHC even though it is harmful for them.

##  7 Discussion

In this paper, we developed a general policy learning framework that allows
for asymmetric utilities, which reflect common ethical principles including
the Hippocratic oath. The proposed framework can also be seen as policy
learning subject to the constraints on the proportion of decisions that are
harmful or not helpful. The asymmetry of utility functions generally leads to
the unidentifiability of expected utilities because we cannot point-identify
the conditional joint probability of the two potential outcomes. We addressed
this problem by employing a partial identification approach and then finding a
policy that minimizes the maximum regret relative to a particular policy,
showing how the resulting objective changes with both the asymmetry in the
utility and the choice of reference policy. We illustrated this framework by
reanalyzing the controversial study of Right Heart Catheterization, finding
that the minimax optimal policy varies substantially with the asymmetry in the
utility and choice of reference policy.

There are several avenues for future research. First, we have restricted our
attention here to binary outcomes and binary treatments, leading to four
principal strata. However, many decision problems involve multiple potential
actions and categorical or continuous outcomes. This leads to many more
principal strata and potential asymmetries in the utility function. For
example, with a single treatment of varying intensity or dosage we might want
to penalize any dosage above that required to prevent a bad outcome. Second,
we may consider decision problems with utility functions that depend on other
post-treatment variables or mediators as well as outcomes, leading to a
different principal stratification structure. Finally, we can consider using
these ideas to incorporate further constraints into the policy learning
problem that encode notions of fairness, such as the concept of principal
fairness that requires decisions to be independent of protected attributes
conditioned on the principal stratum (Imai and Jiang, 2022).

## References

  * Athey and Wager (2021) Athey, S. and S. Wager (2021).  Policy Learning With Observational Data.  Econometrica 89(1), 133–161.
  * Audibert and Tsybakov (2007) Audibert, J. Y. and A. B. Tsybakov (2007).  Fast learning rates for plug-in classifiers.  Annals of Statistics 35(2), 608–633.
  * Ben-Michael et al. (2021) Ben-Michael, E., D. J. Greiner, K. Imai, and Z. Jiang (2021).  Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment.
  * Bertsimas et al. (2011) Bertsimas, D., D. B. Brown, and C. Caramanis (2011).  Theory and applications of robust optimization.  SIAM Review 53(3), 464–501.
  * Beygelzimer and Langford (2009) Beygelzimer, A. and J. Langford (2009).  The offset tree for learning with partial labels.  Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 129–137.
  * Bordley (2009) Bordley, R. F. (2009).  The hippocratic oath, effect size, and utility theory.  Medical Decision Making 29(3), 377–379.  PMID: 19380886.
  * Connors et al. (1996) Connors, A. F., T. Speroff, N. V. Dawson, C. Thomas, F. E. Harrell, D. Wagner, N. Desbiens, L. Goldman, A. W. Wu, R. M. Califf, W. J. Fulkerson, H. Vidaillet, S. Broste, P. Bellamy, J. Lynn, and W. A. Knaus (1996).  The effectiveness of right heart catheterization in the initial care of critically ill patients.  Journal of the American Medical Association 276(11), 889–897.
  * Cui (2021) Cui, Y. (2021).  Individualized decision making under partial identification: three perspectives, two optimality results, and one paradox.  Harvard Data Science Review.  Just accepted.
  * Cui and Tchetgen Tchetgen (2021) Cui, Y. and E. Tchetgen Tchetgen (2021).  A Semiparametric Instrumental Variable Approach to Optimal Treatment Regimes Under Endogeneity.  Journal of the American Statistical Association 116(533), 162–173.
  * Ding and Lu (2017) Ding, P. and J. Lu (2017).  Principal stratification analysis using principal scores.  Journal of the Royal Statistical Society, Series B (Statistical Methodology) 79(3), 757–777.
  * Dudik et al. (2011) Dudik, M., J. Langford, and L. Li (2011).  Doubly Robust Policy Evaluation and Learning.  In Proceedings of the 28th International Conference on Machine Learning.
  * Foot (1967) Foot, P. (1967).  The problem of abortion and the doctrine of the double effect.  Oxford Review 5, 5–15.
  * Frangakis and Rubin (2002) Frangakis, C. E. and D. B. Rubin (2002).  Principal stratification in causal inference.  Biometrics 58(1), 21–29.
  * Friedman (2001) Friedman, J. (2001).  Greedy Function Approximation : A Gradient Boosting Machine.  The Annals of Statistics 29(5), 1189–1232.
  * González-Ortega et al. (2018) González-Ortega, J., V. Radovic, and D. Ríos Insua (2018).  Utility elicitation.  In L. C. Dias, A. Morton, and J. Quigley (Eds.), Elicitation, Volume 261, pp. 241–264.
  * Han (2021) Han, S. (2021).  Optimal Dynamic Treatment Regimes and Partial Welfare Ordering.
  * Hirano and Imbens (2001) Hirano, K. and G. W. Imbens (2001).  Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization.  Health Services and Outcomes Research Methodology 2(3-4), 259–278.
  * Imai and Jiang (2022) Imai, K. and Z. Jiang (2022+).  Principal fairness for human and algorithmic decision-making.  Statistical Science.
  * Jiang et al. (2020) Jiang, Z., S. Yang, and P. Ding (2020).  Multiply robust estimation of causal effects under principal ignorability.  arXiv preprint arXiv:2012.01615.
  * Kallus (2018) Kallus, N. (2018).  Balanced policy evaluation and learning.  Advances in Neural Information Processing Systems 2018-December(1), 8895–8906.
  * Kallus (2022) Kallus, N. (2022).  What’s the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment.
  * Kallus and Zhou (2021) Kallus, N. and A. Zhou (2021).  Minimax-optimal policy learning under unobserved confounding.  Management Science 67(5), 2870–2890.
  * Kitagawa and Tetenov (2018) Kitagawa, T. and A. Tetenov (2018).  Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice.  Econometrica 86(2), 591–616.
  * Knaus (2020) Knaus, M. C. (2020).  Double Machine Learning based program evaluation under unconfoundedness.  arXiv, 1–61.
  * Knaus et al. (1985) Knaus, W. A., E. A. Draper, D. P. Wagner, and J. E. Zimmerman (1985).  APACHE II: A severity of disease classification system.  Critical Care Medicine 13(10), 818–829.
  * Lin (2016) Lin, P. (2016).  Autonomous Driving: Technical, Legal and Social Aspects, Chapter Why Ethnics Matters for Autonomous Cars, pp. 69–85.  Springer Open.
  * Luedtke and Van Der Laan (2016) Luedtke, A. R. and M. J. Van Der Laan (2016).  Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy.  Annals of Statistics 44(2), 713–742.
  * Manski (2005) Manski, C. F. (2005).  Social Choice with Partial Knowledge of Treatment Response.  Princeton University Press.
  * Neyman (1923) Neyman, J. (1990 [1923]).  On the application of probability theory to agricultural experiments. essay on principles. section 9.  Statistical Science 5(4), 465–472.
  * Pu and Zhang (2021) Pu, H. and B. Zhang (2021).  Estimating optimal treatment rules with an instrumental variable: A partial identification learning approach.  Journal of the Royal Statistical Society Series B, 1–28.
  * Qian and Murphy (2011) Qian, M. and S. A. Murphy (2011).  Performance guarantees for individualized treatment rules.  The Annals of Statistics 39(2), 1180–1210.
  * Qiu et al. (2021) Qiu, H., M. Carone, E. Sadikova, M. Petukhova, R. C. Kessler, and A. Luedtke (2021).  Optimal Individualized Decision Rules Using Instrumental Variable Methods.  Journal of the American Statistical Association 116(533), 174–191.
  * Rosenbaum and Rubin (1983) Rosenbaum, P. R. and D. B. Rubin (1983).  The Central Role of the Propensity Score in Observational Studies for Causal Effects.  Biometrika 70(1), 41–55.
  * Rubin (1980) Rubin, D. B. (1980).  Comment on “Randomization Analysis of Experimental Data: The Fisher Randomization Test”.  Journal of the American Statistical Association 75(371), 591–593.
  * Smith (2005) Smith, C. M. (2005).  Origin and uses of primum non nocere — above all, do no harm!  Journal of Clinical Pharmacology 45, 371–377.
  * Sverdrup et al. (2020) Sverdrup, E., A. Kanodia, Z. Zhou, S. Athey, and S. Wager (2020).  policytree: Policy learning via doubly robust empirical welfare maximization over trees.  Journal of Open Source Software 5(50), 2232.
  * Swaminathan and Joachims (2015) Swaminathan, A. and T. Joachims (2015).  Batch learning from logged bandit feedback through counterfactual risk minimization.  Journal of Machine Learning Research 16, 1731–1755.
  * Thomson (1976) Thomson, J. J. (1976, April).  Killing, letting die, and the trolley problem.  The Monist 59(2), 204–217.  Philosophical Problems of Death.
  * Zhang et al. (2012) Zhang, B., A. A. Tsiatis, M. Davidian, M. Zhang, and E. Laber (2012).  Estimating optimal treatment regimes from a classification perspective.  Stat 1(1), 103–114.
  * Zhao et al. (2012) Zhao, Y., D. Zeng, A. J. Rush, and M. R. Kosorok (2012).  Estimating individualized treatment rules using outcome weighted learning.  Journal of the American Statistical Association 107(499), 1106–1118.

Appendix

##  Appendix A Asymmetric utilities based on observed outcomes

Although it is possible to construct asymmetric utilities without relying on
principal strata, doing so places additional restrictions on the structure of
utilities. Consider the following utility function based on observed outcomes
alone, $u_{1}dY_{i}(d)+u_{0}Y_{i}(d)-cd$. This utility function includes the
interaction between the decision and the outcome. Indeed, for binary decision
and outcome, this represents the most general utility that could be specified
using the observed outcome.

| $Y_{i}(0)=1$ | $Y_{i}(0)=0$
---|---|---
$Y_{i}(1)=1$ | Harmless | Useful
$u_{1}-c$ | $u_{1}+u_{0}-c$
1-3 $Y_{i}(1)=0$ | Harmful | Useless
$-u_{0}-c$ | $-c$
Table 2: Asymmetric utilities based on the observed outcomes. The utility
function is given by $u_{1}dY_{i}(d)+u_{0}Y_{i}(d)-cd$. Each cell corresponds
to the principal stratum defined by the values of the two potential outcomes,
$Y_{i}(1)$ and $Y_{i}(0)$. Each entry represents the utility gain/loss of
treatment assignment, relative to no treatment, for a unit that belongs to the
corresponding principal stratum.

Table 2 summarizes the utility gain/loss for a unit that belongs to each
principal stratum. With an interaction term, this utility has different
utility gains/losses in principal strata $(Y(1)=1,Y(0)=0)$ and
$(Y(1)=0,Y(0)=1)$, allowing for the asymmetry in the utilities as required by
the Hippocratic principle. This utility, however, still places restrictions on
its structure. In particular, it requires that the difference between the
utility gains in principal strata $(Y(1)=1,Y(0)=1)$ and $(Y(1)=0,Y(0)=1)$ is
the same as that between the utility losses in principal strata
$(Y(1)=1,Y(0)=0)$ and $(Y(1)=0,Y(0)=0)$. Therefore, it might be violated if
the difference between harmful and harmless decisions is much greater than
that between useful and useless decisions. Thus, a fully general construction
of asymmetric utilities requires the use of principal strata.

##  Appendix B Additional figures

Figure B.1: Estimated quadratic decision rule $\hat{\delta}_{+}$ to classify
whether $e_{11}(x)\geq e_{00}(x)$

##  Appendix C Approximation via weighted SVM

Rather than solving the non-convex empirical welfare minimization problem in
Equation (13), we solve an approximate version via a weighted SVM. To write
the optimization problem, define

$\displaystyle\tilde{y}_{i}$
$\displaystyle=2*\mathbbm{1}\left\\{\hat{c}^{\varpi}_{1}(X_{i})\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}_{0}(X_{i})\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}(X_{i})\geq
0\right\\}-1$ $\displaystyle\gamma_{i}$
$\displaystyle=\left|\hat{c}^{\varpi}_{1}(X_{i})\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}_{0}(X_{i})\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})+\hat{c}^{\varpi}(X_{i})\right|.$

The first value $\tilde{y}_{i}$ is a pseudo outcome that is 1 if the
contribution of treating unit $i$ to the regret is negative and -1 if it is
positive, and $\gamma_{i}$ is a weight corresponding to the magnitude of the
regret increase/decrease. With these values, we solve a weighted soft-margin
SVM problem

$\displaystyle\min_{\beta_{0},\beta,\xi}$ $\displaystyle
C\sum_{i=1}^{n}\xi_{i}\gamma_{i}+\frac{1}{2}\|\beta\|_{2}^{2}$ (14) subject to
$\displaystyle\\{\beta_{0}+\beta\cdot\phi(X_{i})\\}\tilde{y}_{i}\geq
1-\xi_{i}$ $\displaystyle\xi_{i}\geq 0,$

where $\phi(\cdot)$ is a transformation of the covariates. With the solution
$\hat{\beta}_{0},\hat{\beta}$, the resulting policy is

$\hat{\pi}(x)=\mathbbm{1}\left\\{\hat{\beta}_{0}+\hat{\beta}\cdot\phi(x)\geq
0\right\\}.$

The corresponding Lagrangian to optimization problem (14) has Lagrange
multipliers $\alpha\in\mathbb{R}^{n}$:

$\mathcal{L}(\beta_{0},\beta,\xi,\alpha)=C\sum_{i=1}^{n}\xi_{i}\gamma_{i}+\frac{1}{2}\|\beta\|_{2}^{2}-\sum_{i=1}^{n}\left\\{(\beta_{0}+\beta\cdot\phi(X_{i}))\tilde{y}_{i}-1+\xi_{i}\right\\}\alpha_{i}.$

So the Lagrangian dual is

$\displaystyle q(\alpha)$
$\displaystyle=\min_{\beta_{0},\beta,\xi}\mathcal{L}(\beta_{0},\beta,\xi,\alpha)$
$\displaystyle=\min_{\beta}\;\frac{1}{2C}\|\beta\|_{2}^{2}-\sum_{i=1}^{n}\beta\cdot\phi(X_{i})\tilde{y}_{i}\alpha_{i}+\min_{\beta_{0}}-\sum_{i=1}^{n}\beta_{0}\tilde{y}_{i}\alpha_{i}+\sum_{i=1}^{n}\min_{\xi_{i}\geq
0}\xi_{i}(C\gamma_{i}-\alpha_{i})+\sum_{i=1}^{n}\alpha_{i}.$

Solving the inner minimization problems, we find that the Lagrangian dual
problem is

$\displaystyle\min_{\alpha}$
$\displaystyle\frac{1}{2}\sum_{i,j=1}^{n}\alpha_{i}\alpha_{j}\phi(X_{i})\cdot\phi(X_{j})\tilde{y}_{i}\tilde{y_{j}}-\sum_{i=1}^{n}\alpha_{i}$
(15) subject to $\displaystyle\sum_{i=1}^{n}\tilde{y}_{i}\alpha_{i}=0$
$\displaystyle 0\leq\alpha_{i}\leq\gamma_{i}C$

With the solution to the dual problem (15) $\hat{\alpha}$ we can recover the
primal solution via

$\displaystyle\hat{\beta}$
$\displaystyle=\sum_{i=1}^{n}\tilde{y}_{i}\phi(X_{i})\alpha_{i}$
$\displaystyle\hat{\beta}_{0}$
$\displaystyle=\frac{1}{\sum_{\alpha_{i}<C\gamma_{i}}1}\sum_{\alpha_{i}<C\gamma_{i}}\tilde{y}_{i}-\hat{\beta}\cdot\phi(X_{i}).$

##  Appendix D Proofs and derivations

##### Derivation of the regret

The regret of policy $\pi$ relative to policy $\varpi$ is

$\displaystyle V(\varpi)-V(\pi)$
$\displaystyle=\mathbb{E}\left[\sum_{y_{1}=0}^{1}\sum_{y_{0}=0}^{1}(\varpi(X)-\pi(X))e_{y_{1}y_{0}}(X)\left\\{u(1;y_{1},y_{0})-u(0;y_{1},y_{0})\right\\}\right]$
$\displaystyle=\mathbb{E}\left[(\varpi(X)-\pi(X))\left\\{e_{10}(X)(u_{g}-c)-e_{01}(X)(u_{l}+c)-e_{00}(X)c-e_{11}(X)c\right\\}\right]$
$\displaystyle=\mathbb{E}\left[(\varpi(X)-\pi(X))\left\\{e_{10}(X)u_{g}-e_{01}(X)u_{l}-c(e_{10}(X)+e_{01}(X)+e_{00}(X)+e_{11}(X))\right\\}\right]$
$\displaystyle=\mathbb{E}\left[(\varpi(X)-\pi(X))\left\\{(\tau(X)+e_{01}(X))u_{g}-e_{01}(X)u_{l}-c\right\\}\right]$
$\displaystyle=\mathbb{E}\left[(\varpi(X)-\pi(X))\left\\{u_{g}\tau(X)+(u_{g}-u_{l})e_{01}(X)-c\right\\}\right],$

where we have used the fact that $\tau(x)=e_{10}(x)-e_{01}(x)$.

###### Proof of Theorem 1.

Define $b(x)=u_{g}\tau(x)+(u_{g}-u_{l})e_{01}(X)-c$, and

$\displaystyle L_{b}(x)$
$\displaystyle=\min_{e(x)\in[L(x),U(x)]}\left\\{u_{g}\tau(x)+(u_{g}-u_{l})e_{01}(X)-c\right\\}$
$\displaystyle U_{b}(x)$
$\displaystyle=\max_{e(x)\in[L(x),U(x)]}\left\\{u_{g}\tau(x)+(u_{g}-u_{l})e_{01}(X)-c\right\\}.$

Note that the worst-case regret relative to the always and never treat
policies are

$\displaystyle R_{\text{sup}}(\pi,\pi^{\mathbbm{O}})$
$\displaystyle=-\mathbb{E}[\pi(X)L_{b}(X)]$ $\displaystyle
R_{\text{sup}}(\pi,\pi^{\mathbbm{1}})$
$\displaystyle=\mathbb{E}[\\{1-\pi(X)\\}U_{b}(X)]=\mathbb{E}[U_{b}(X)]-\mathbb{E}[\pi(X)U_{b}(X)].$

From this, we can find the unconstrained minimax regret policies

$\displaystyle\pi^{\ast}_{\mathbbm{O}}$
$\displaystyle=\underset{\pi}{\text{argmin}}-\mathbb{E}[\pi(X)L_{b}(X)]=$
$\displaystyle\mathbbm{1}\\{L_{b}(x)\geq 0\\}$
$\displaystyle\pi^{\ast}_{\mathbbm{1}}$
$\displaystyle=\underset{\pi}{\text{argmin}}-\mathbb{E}[\pi(X)U_{b}(X)]=$
$\displaystyle\mathbbm{1}\\{U_{b}(x)\geq 0\\}.$

Now, the oracle policy is $\pi^{o}(x)=\mathbbm{1}\\{b(x)\geq 0\\}$. So if
$L_{b}(x)\geq 0\Leftrightarrow\pi^{\ast}_{\mathbbm{O}}(x)=1$ then
$\pi^{o}(x)=1$ for all possible values of the principal score $e_{01}(x)$. In
this case the upper bound is,

$\max_{e(x)\in[L(x),U(x)]}\\{\pi^{o}(x)-\pi(x)\\}b(x)=(1-\pi(x))U_{b}(x).$

Similarly, if $U_{b}(x)<0\Leftrightarrow\pi^{\ast}_{\mathbbm{1}}(x)=0$ then
$\pi^{o}(x)=0$, and

$\max_{e(x)\in[L(x),U(x)]}\\{\pi^{o}(x)-\pi(x)\\}b(x)=-\pi(x)L_{b}(x).$

Finally, if $L_{b}(x)<0$ and $U_{b}(x)\geq 0$ (so
$\pi^{\ast}_{\mathbbm{O}}(x)=0$ and $\pi^{\ast}_{\mathbbm{1}}(x)=1$), then the
oracle policy can be either 0 or 1, $\pi^{o}(x)\in\\{0,1\\}$. Therefore,

$\max_{e(x)\in[L(x),U(x)]}\\{\pi^{o}(x)-\pi(x)\\}b(x)=\max\\{(1-\pi(x))U_{b}(x),-\pi(x)L_{b}(x)\\}=U_{b}(x)-\pi(x)\\{U_{b}(x)+L_{b}(x)\\}.$

Putting together the pieces, the worst-case regret relative to the oracle is

$\displaystyle R_{\text{sup}}(\pi,\pi^{o})$
$\displaystyle=\mathbb{E}([\pi^{\ast}_{\mathbbm{O}}(X)+\\{1-\pi^{\ast}_{\mathbbm{O}}(X)\\}\pi^{\ast}_{\mathbbm{1}}(X)]U_{b}(X))$
$\displaystyle-\mathbb{E}[\pi(X)\left\\{\pi^{\ast}_{\mathbbm{O}}(X)U_{b}(X)+(1-\pi^{\ast}_{\mathbbm{1}}(X))L_{b}(X)+(1-\pi^{\ast}_{\mathbbm{O}}(X))\pi^{\ast}_{\mathbbm{1}}(X)(U_{b}(X)+L_{b}(X))\right\\}],$

and the unconstrained minimizer is

$\pi^{\ast}_{o}=\underset{\pi}{\text{argmin}}R_{\text{sup}}(\pi,\pi^{o})=\left\\{\begin{array}[]{c
c}\pi^{\ast}_{\mathbbm{1}}(x)&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\
\pi^{\ast}_{\mathbbm{O}}(x)&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
\mathbbm{1}\\{U_{b}(x)\geq-
L_{b}(x)\\}&\pi^{\ast}_{\mathbbm{O}}(x)=0,\pi^{\ast}_{\mathbbm{1}}(x)=1\end{array}\right..$

Now notice that $\pi^{\ast}_{\mathbbm{O}}(x)=1\Leftrightarrow L_{b}(x)\geq
0\Rightarrow U_{b}(x)\geq 0\Leftrightarrow\pi^{\ast}_{\mathbbm{1}}(x)=1$, and
$\pi^{\ast}_{\mathbbm{1}}(x)=0\Leftrightarrow U_{b}(x)<0\Rightarrow
L_{b}(x)<0\Leftrightarrow\pi^{\ast}_{\mathbbm{O}}(x)=0$, so we can simplify
this to

$\pi^{\ast}_{o}=\underset{\pi}{\text{argmin}}R_{\text{sup}}(\pi,\pi^{o})=\left\\{\begin{array}[]{c
c}1&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\ 0&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
\mathbbm{1}\\{U_{b}(x)\geq-
L_{b}(x)\\}&\pi^{\ast}_{\mathbbm{O}}(x)=0,\pi^{\ast}_{\mathbbm{1}}(x)=1\end{array}\right..$

To complete the proof, we need to compute $L_{b}(x)$ and $U_{b}(x)$. First, we
begin with the case where $u_{g}\geq u_{l}$. In this case,

$\displaystyle L_{b}(x)$
$\displaystyle=\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}\tau(x)-c=\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}m(1,x)-\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}m(0,x)-c$
$\displaystyle U_{b}(x)$
$\displaystyle=\\{u_{g}-(u_{g}-u_{l})\delta_{+}(x)\\}m(1,x)-\\{u_{l}+(u_{g}-u_{l})\delta_{+}(x)\\}m(0,x)+(u_{g}-u_{l})\delta_{+}(x)-c.$

This gives the form of the worst-case regret relative to $\pi^{\mathbbm{1}}$
and $\pi^{\mathbbm{O}}$. For the worst-case regret relative to the oracle, we
collect terms to get

$\left(\begin{array}[]{c}c^{\pi^{o}}_{1}(x)\\\ c^{\pi^{o}}_{0}(x)\\\
c^{\pi^{o}}(x)\end{array}\right)=\left\\{\begin{array}[]{l
l}(u_{l}+(u_{g}-u_{l})\delta_{\tau}(x),-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x),-c)&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
(u_{g}-(u_{g}-u_{l})\delta_{+}(x),-u_{l}-(u_{g}-u_{l})\delta_{+}(x),(u_{g}-u_{l})\delta_{+}(x)-c)&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\
(u_{l}+u_{g}+(u_{g}-u_{l})(\delta_{\tau}(x)-\delta_{+}(x)),-2u_{l}-(u_{g}-u_{l})(\delta_{\tau}(x)+\delta_{+}(x)),(u_{g}-u_{l})\delta_{+}(x)-2c)&\pi^{\ast}_{\mathbbm{O}}(x)\neq\pi^{\ast}_{\mathbbm{1}}(x)\\\
\end{array}\right..$

Now for the case where $u_{g}<u_{l}$, the lower and upper bounds switch:

$\displaystyle L_{b}(x)$
$\displaystyle=\\{u_{g}-(u_{g}-u_{l})\delta_{+}(x)\\}m(1,x)-\\{u_{l}+(u_{g}-u_{l})\delta_{+}(x)\\}m(0,x)+(u_{g}-u_{l})\delta_{+}(x)-c$
$\displaystyle U_{b}(x)$
$\displaystyle=\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}\tau(x)-c=\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}m(1,x)-\\{u_{l}+(u_{g}-u_{l})\delta_{\tau}(x)\\}m(0,x)-c.$

For the worst-case regret relative to the oracle, we collect terms to get

$\left(\begin{array}[]{c}c^{\pi^{o}}_{1}(x)\\\ c^{\pi^{o}}_{0}(x)\\\
c^{\pi^{o}}(x)\end{array}\right)=\left\\{\begin{array}[]{l
l}(u_{g}-(u_{g}-u_{l})\delta_{+}(x),-u_{l}-(u_{g}-u_{l})\delta_{+}(x),(u_{g}-u_{l})\delta_{+}(x)-c)&\pi^{\ast}_{\mathbbm{1}}(x)=0\\\
(u_{l}+(u_{g}-u_{l})\delta_{\tau}(x),-u_{l}-(u_{g}-u_{l})\delta_{\tau}(x),-c)&\pi^{\ast}_{\mathbbm{O}}(x)=1\\\
(u_{l}+u_{g}+(u_{g}-u_{l})(\delta_{\tau}(x)-\delta_{+}(x)),-2u_{l}-(u_{g}-u_{l})(\delta_{\tau}(x)+\delta_{+}(x)),(u_{g}-u_{l})\delta_{+}(x)-2c)&\pi^{\ast}_{\mathbbm{O}}(x)\neq\pi^{\ast}_{\mathbbm{1}}(x)\\\
\end{array}\right..$

∎

######  Lemma 1.

For all $\pi$,

$\left|\hat{R}_{\sup}(\pi)-\tilde{R}_{\sup}(\pi)\right|\leq\frac{1}{n}\sum_{i=1}^{n}|\hat{c}(X_{i})-c(X_{i})|+\sum_{w=0}^{1}\max_{i=1,\ldots,n}\left|\widehat{\Gamma}_{w}(X_{i},D_{i},Y_{i})\right|\frac{1}{n}\sum_{i=1}^{n}\left|\hat{c}_{w}(X_{i})-c_{w}(X_{i})\right|$

###### Proof of Lemma 1.

Note that

$\displaystyle\hat{R}_{\sup}(\pi)-\tilde{R}_{\sup}(\pi)$
$\displaystyle=-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left(\left\\{\hat{c}^{\varpi}_{1}(X_{i})-c^{\varpi}_{1}(X_{i})\right\\}\widehat{\Gamma}_{1}(X_{i},D_{i},Y_{i})+\left\\{\hat{c}^{\varpi}_{0}(X_{i})-c^{\varpi}_{0}(X_{i})\right\\}\
\widehat{\Gamma}_{0}(X_{i},D_{i},Y_{i})\right)$
$\displaystyle\qquad-\frac{1}{n}\sum_{i=1}^{n}\pi(X_{i})\left\\{\hat{c}^{\varpi}(X_{i})-c^{\varpi}(X_{i})\right\\}$

Since $\pi(X_{i})\in\\{0,1\\}$, the result follows from further bounding the
expression with the upper bound of the scoring functions. ∎

###### Proof of Proposition 1.

First, note that the excess regret can be decomposed into

$\displaystyle
R_{\text{sup}}(\hat{\pi},\varpi)-R_{\text{sup}}(\pi^{\ast},\varpi)$
$\displaystyle=R_{\text{sup}}(\hat{\pi},\varpi)-\tilde{R}_{\text{sup}}(\hat{\pi},\varpi)+\tilde{R}_{\text{sup}}(\hat{\pi},\varpi)-\hat{R}_{\text{sup}}(\hat{\pi},\varpi)$
$\displaystyle\qquad+\hat{R}_{\text{sup}}(\hat{\pi},\varpi)-\hat{R}_{\text{sup}}(\pi^{\ast},\varpi)+\hat{R}_{\text{sup}}(\pi^{\ast},\varpi)-\tilde{R}_{\text{sup}}(\pi^{\ast},\varpi)$
$\displaystyle\qquad+\tilde{R}_{\text{sup}}(\pi^{\ast},\varpi)-R_{\text{sup}}(\pi^{\ast},\varpi)$

Now since $\hat{\pi}$ minimizes $\hat{R}_{\text{sup}}(\pi,\varpi)$,
$\hat{R}_{\text{sup}}(\hat{\pi},\varpi)-\hat{R}_{\text{sup}}(\pi^{\ast},\varpi)\leq
0$, so the excess regret reduces to

$R_{\text{sup}}(\hat{\pi},\varpi)-R_{\text{sup}}(\pi^{\ast},\varpi)\leq
2\sup_{\pi\in\Pi}\left|\tilde{R}_{\text{sup}}(\pi,\varpi)-R_{\text{sup}}(\pi,\varpi)\right|+2\sup_{\pi\in\Pi}\left|\hat{R}_{\text{sup}}(\pi,\varpi)-\tilde{R}_{\text{sup}}(\pi,\varpi)\right|$

Combining with Lemma 1 gives the result. ∎