# When Personalization Harms Performance:
Reconsidering the Use of Group Attributes in Prediction

Vinith M. Suriyakumar<EMAIL_ADDRESS>
Massachusetts Institute of Technology Marzyeh<EMAIL_ADDRESS>
Massachusetts Institute of Technology Berk Ustun††footnotemark<EMAIL_ADDRESS>
University of California, San Diego Equal Supervision

###### Abstract

Machine learning models are often personalized with categorical attributes
that are protected, sensitive, self-reported, or costly to acquire. In this
work, we show models that are personalized with group attributes can reduce
performance at a group level. We propose formal conditions to ensure the “fair
use" of group attributes in prediction tasks by training one additional model
– i.e., collective preference guarantees to ensure that each group who
provides personal data will receive a tailored gain in performance in return.
We present sufficient conditions to ensure fair use in empirical risk
minimization, and characterize failure modes that lead to fair use violations
due to standard practices in model development and deployment. We present a
comprehensive empirical study of fair use in clinical prediction tasks. Our
results demonstrate the prevalence of fair use violations in practice, and
illustrate simple interventions to mitigate their harm.

Keywords: personalization; clinical decision support; algorithmic fairness;
group attributes; classification.

##  1 Introduction

Machine learning models are often used to support or automate decisions that
affect people. In medicine, for example, models diagnose illnesses [88, 45,
101], estimate survival rates [107, 78, 26], and predict treatment response
[58, 74, 53]. In such tasks, medical decisions follow the ethical principles
of beneficence and non-maleficence [11]. In turn, models that guide medical
decisions are _personalized_ to perform as well as possible without inflicting
harm [22]. In practice, clinical prediction models are routinely personalized
to include all characteristics that are “biologically relevant" to target
heterogeneous subpopulations [62, 96]. In practice, most models use
characteristics that are protected (e.g., sex as in the CHA2DS2-VaSC),
sensitive (e.g., HIV), self-reported (e.g. sexual_practices as in the Denver
HIV Risk Score), or costly to acquire (e.g., Brief Psychiatry Rating Scale).
These models are often built using logistic regression with one-hot encodings
(i.e. score corrections) of these protective, sensitive, or self-reported
characteristics.

The widespread use of group attributes in prediction models is a common form
of _personalization_. However, it represents a unique opportunity to test its
implicit promise. In effect, individuals who report their personal
characteristics expect to receive tailored performance gains in return. In
prediction tasks where models are personalized with group attributes, we can
check this promise. First, by testing that every group expects to receive more
accurate predictions from a _personalized model_ that solicits group
membership compared to a _generic model_ that does not. Next, by testing that
the gains are _tailored_ , meaning that every group prefers their personalized
predictions to the predictions personalized for any other group.

Although many models are personalized with group attributes, there is little
awareness that personalization can reduce performance at a group level [see
e.g., 3, 87]. It is only natural to assume a model fit with group attributes
would perform better at the group level than a model that does not. In
practice, personalized models are trained to use group attributes to improve
performance at a population level. As we will show, this can lead models to
assign unnecessarily inaccurate predictions at the group level due to standard
practices in model specification and model selection. (see Fig. 2).

In many real-world applications, the resulting drop in performance inflicts
tangible harm. In clinical prediction tasks, for example, risk scores have
more unreliable predictions for African American patients, leading to worse
decisions and health outcomes [96]. These effects are silent and avoidable.
Silent because fair use violations would only draw attention if model
developers were to evaluate the _gains_ of personalization for
_intersectional_ groups. Avoidable because a fair use violation shows that a
group could receive better predictions from a generic model or a personalized
model for another group; thus one could resolve a fair use violation by
assigning predictions from this better-performing model.

Our goal in this paper is to expose this effect and lay the foundations to
address it. The main contributions of our work include:

  1. 1.

We propose formal conditions to ensure the fair use of group attributes in
prediction models. Furthermore, we provide a simple procedure for testing
these conditions that only requires training one additional model.

  2. 2.

We characterize sufficient theoretical conditions under which empirical risk
minimization with group attributes ensures or violates fair use. Additionally,
we show “failure modes" that arise during model development, selection, and
deployment that lead to fair use violations. These failure modes can inform
the design of interventions to mitigate these effects.

  3. 3.

We conduct a comprehensive empirical study on the gains of personalization in
clinical prediction tasks. Our study shows that fair use violations are highly
prevalent regardless of model classes, personalization techniques, or
prediction tasks. We show that these violations can reliably be detected with
simple statistical tests. Finally, while our results highlight practical
challenges in avoiding detrimental personalization at the group level, they
also provide best practices for model development to prevent fair use
violations.

  4. 4.

We present a case study on personalization for a model trained to predict
mortality for patients with acute kidney injury. Our study shows how a fair
use audit can safeguard against incorrect “race correction" in clinical
prediction models, and presents targeted interventions that reduce harm.

Group | Size | Training Error | Gain
---|---|---|---
$\bm{g}{}$ | $n_{\bm{g}}$ | ${R(h_{0}{})}$ | ${R_{\bm{g}}(h)}$ | ${\Delta_{\bm{g}}}({h},{h_{0}{}})$
female, <30 | 48 | 38.1% | 26.8% | 11.3%
male, <30 | 49 | 23.9% | 26.7% | -2.8%
female, 30 to 60 | 304 | 30.3% | 29.1% | 1.2%
male, 30 to 60 | 447 | 15.4% | 15.2% | 0.2%
female, 60+ | 123 | 19.3% | 21.9% | -2.6%
male, 60+ | 181 | 11.0% | 8.2% | 2.8%
Total | 1152 | 20.4% | 19.4% | 1.0%

Figure 1: Personalization can reduce performance at the group level. We train
a personalized model $h_{\bm{g}}$ and generic model $h_{0}$ with logistic
regression, personalizing $h_{\bm{g}}$ with a one-hot encoding of sex and
age_group to screen for obstructive sleep apnea (see the apnea dataset in
Section 4). As shown, personalization reduces training error at a population
level from 20.4% to 19.4% yet _increases_ error for two groups: (female, 60+)
and (male, <30). These effects are also present on test data.

### Related Work

Personalization encompasses a broad range of techniques that use personal
data. Here, we use it to describe techniques that target _groups_ rather than
_individuals_ – i.e., “categorization" rather than “individualization" as per
the taxonomy of Fan and Poole [37]. Personalized models that encode personal
characteristics in categorical attributes are widely used in medicine,
consumer finance, and criminal justice. Personalization has also been an
important topic of research in federated learning (FL). In FL, personalization
algorithms aim to adapt a shared model among users to account for the
heterogeneity between individual user data distributions. We note that these
algorithms and this research are orthogonal and not applicable to our setting.

In medicine, many models are trained using logistic regression with a one-hot
encoding of categorical attributes [87, 96, 35]. Works that measure the gains
of personalization do so at a population level rather a group level [see e.g.,
52, 89]. This population-level focus extends to methodological work: recent
methods use categorical attributes to improve population-level performance by
accounting for heterogeneity – e.g., by automatically including higher-order
interaction effects [14, 65, 94] or recursively partitioning data [34, 15, 13,
12].

Our work introduces necessary conditions on group performance for models that
use group attributes to assign more accurate predictions. Several works
discuss the need to account for group membership in such settings [104, 33,
24, 61, 66, 99], observing that it is otherwise impossible for a model to
perform equally well for all groups [47, 103, 105, 38, 2, 73, 21]. Methods to
equalize performance are ill-suited for personalization because they can
equalize performance by reducing performance for groups for whom the model
performs well, rather than by improving performance for groups for whom the
model performs poorly [66, 51, 81, 67, 68].

More broadly, our work highlights a practical application for preference-based
notions of fairness [104, 92, 60, 95, 30], and a recent trend to evaluate
fairness across intersectional groups [c.f., 57, 50, 43, 98], which are more
granular than those considered in the literature yet large enough to estimate
performance [c.f., 32, 7].

Group | Data | Personalized | Generic | Gain
---|---|---|---|---
$\bm{g}{}$ | $n^{+{}}_{\bm{g}}$ |  $n^{-{}}_{\bm{g}}$ |  $h$ |  $R_{\bm{g}}(h)$ | $h_{0}$ | $R_{\bm{g}}(h_{0}{})$ | ${\Delta_{\bm{g}}}({h},{h_{0}{}})$
female, young | $0$ |  $24$ |  $+$ |  $24$ | $-$ | $0$ | $-24$
male, young | $25$ |  $0$ |  $+$ |  $0$ | $-$ | $25$ | $25$
female, old | $25$ |  $0$ |  $+$ |  $0$ | $-$ | $25$ | $25$
male, old | $0$ |  $27$ |  $-$ |  $0$ | $-$ | $0$ | $0$
Total | 50 |  51 |  |  24 |  | 50 | 26

Figure 2: 2D classification task where the best personalized model reduces
performance for a specific group due to model misspecification. There are
$n^{+{}}=50$ positive examples and $n^{-{}}=51$ negative examples. We train a
generic classifier $h_{0}$ (no group information) and a personalized linear
classifier with a one-hot encoding of
$\bm{g}\in\\{\texttt{\footnotesize{male}},\texttt{\footnotesize{female}}\\}\times\\{\texttt{\footnotesize{old}},\texttt{\footnotesize{young}}\\}$.
Personalization reduces overall error from 50 to 24. However, not all groups
gain from personalization: $(\texttt{\footnotesize{young, female}})$ receives
less accurate predictions and $(\texttt{\footnotesize{old, male}})$ receives
no gain.

##  2 Fair Use Conditions

We present formal conditions for the fair use of group attributes in
prediction based on collective preference guarantees.

#### Preliminaries

We consider a supervised learning task with group attributes. We start with a
dataset $(\bm{x}_{i},y_{i},\bm{g}_{i})_{i=1}^{n}$, where example $i$ consists
of a feature vector $\bm{x}_{i}=[x_{i,1},\ldots,x_{i,d}]\in\mathbb{R}^{d}$, a
label $y_{i}\in\mathcal{Y}$, and $k$ categorical _group attributes_
$\bm{g}_{i}=[g_{i,1},\ldots,g_{i,k}]\in\mathcal{G}_{1}\times\ldots\times\mathcal{G}_{k}=\mathcal{G}$.
We refer to $\bm{g}_{i}$ as the _group membership_ of person $i$. For example,
a female over 60 would have
$\bm{g}_{i}=[\texttt{\footnotesize{female}},\,\texttt{\footnotesize{age}}\geq\texttt{\footnotesize{60}}]$.
We let $n_{\bm{g}}:=|\\{i~{}|\bm{g}_{i}=\bm{g}\\}|$ denote the size of group
$\bm{g}$, and $m:=|\mathcal{G}|$ denote the number of intersectional groups.

We use the data to train a _personalized_ model that uses group attributes
$h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$; and a _generic_ model that
does not $h_{0}{}:\mathcal{X}\to\mathcal{Y}$. We train all models via
empirical risk minimization with a loss function
$\ell:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}_{+}$, denoting the empirical
risk and true risk as ${\hat{R}(h)}$ and ${R(h)}$, respectively. We assume
that the personalized and generic models represent the empirical risk
minimizer on a dataset with group attributes
$(\bm{x}_{i},y_{i},\bm{g}_{i})_{i=1}^{n}$ and without them
$(\bm{x}_{i},y_{i})_{i=1}^{n}$:

$\displaystyle h\in\operatorname*{argmin}_{h\in\mathcal{H}}{\hat{R}(h)}\qquad
h_{0}{}\in\operatorname*{argmin}_{h\in\mathcal{H}_{0}}{\hat{R}(h)}$

We measure the _gains of personalization_ for a personalized model $h$ for
each group. As part of this evaluation, we measure how the performance of a
model for group $\bm{g}$ changes when they are assigned the personalized
predictions for a different group – i.e., the predictions that they could
receive by “misreporting" group membership as $\bm{g}^{\prime}$. Given a
personalized model $h$, we denote its _empirical risk_ and _true risk_ for
group $\bm{g}$ when they report $\bm{g}^{\prime}$ as:

$\displaystyle{\hat{R}_{\bm{g}}(h_{\bm{g}^{\prime}})}$
$\displaystyle:=\frac{1}{n_{\bm{g}}}\sum_{i:\bm{g}_{i}=\bm{g}}\ell\left(h(\bm{x}_{i},\bm{g}^{\prime}),y_{i}\right)\qquad{R_{\bm{g}}(h_{\bm{g}^{\prime}})}:=\mathbb{E}\left[\ell\left(h(\bm{x},\bm{g}^{\prime}),y\right)\mid\mathcal{G}=\bm{g}\right].$

Here, we use $h_{\bm{g}{}^{\prime}}:=h(\cdot,\bm{g}{}^{\prime})$ to denote a
personalized model where group attributes are fixed to $\bm{g}{}^{\prime}$. We
denote similar definitions for the generic model $h_{0}$ except that the
empirical risk and true risk are defined without the use of the group
attribute $\bm{g}$ in the loss function.

We assume that each group prefers models that assign more accurate predictions
as measured in terms of true risk, and evaluate the preferences of group
$\bm{g}$ between $h$ and $h^{\prime}$ using the _gain_ measure:
$\Delta_{\bm{g}}({h},{h^{\prime}}):={R_{\bm{g}}(h^{\prime})}-{R_{\bm{g}}(h)}$.

###  2.1 Fair Use Guarantees

In Definition 1, we characterize the fair use of a group attribute in terms of
collective preference guarantees.

######  Definition 1.

A personalized model $h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ guarantees
the fair use of group attributes $\mathcal{G}$ if it obeys:

$\displaystyle\Delta_{\bm{g}}({h_{\bm{g}}},{h_{0}})$ $\displaystyle\geq 0$
$\displaystyle\;\textrm{for all groups}\;\bm{g}\in\mathcal{G},$ (1)
$\displaystyle\Delta_{\bm{g}}({h_{\bm{g}}},{h_{\bm{g}^{\prime}}})$
$\displaystyle\geq 0$ $\displaystyle\;\textrm{for all
groups}\;\bm{g},\bm{g}^{\prime}\in\mathcal{G}$ (2)

These conditions are collective in that performance is measured over
individuals in a group. Here, condition (1) ensures _rationality_ for group
$\bm{g}$ – i.e., that a majority of group $\bm{g}$ prefers a personalized
model $h_{\bm{g}}$ to a generic model $h_{0}{}$. Condition (2) ensures _envy-
freeness_ for group $\bm{g}$ – i.e., that majority of group $\bm{g}$ prefers
their personalized predictions to the personalized predictions for any other
group. These conditions reflect minimal expectations of groups from a
personalized model.

These conditions can be adapted to supervised prediction tasks by choosing a
suitable risk metric. Since fair use conditions reflect the expected gains
from personalization, a suitable metric should be an exact measure of model
performance rather than a surrogate measure optimized during training
(otherwise violations can occur as discussed in Section 3. In classification
tasks where we want accurate predictions, this would be the error rate. In
tasks where we want reliable risk estimates, it would be the expected
calibration error [72].

#### Fair Use as a Prerequisite for Truthful Self-Reporting

In copyright law, fair use conditions characterize when individuals can use
copyrighted material without permission from copyright owners [102, 75]. In
our setting, fair use conditions characterize when practitioners can use
personal data without asking from permission from the owners of that data. In
particular, fair use conditions are necessary for “truthful self-reporting” in
a prediction task [see e.g., 85, 55, 44]. Consider a personalized model where
each person could control to a personalized model – i.e., where each person
could report their group membership, withhold it, or report the group
membership of another group.

######  Proposition 2.

Consider a prediction task where each person reports their group membership to
a personalized model $h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$. Denote
the reported group membership of person $i$ as: lclcl r_i & = g_i ⇔ $i$
reports truthfully
r_i ∈ G∖{g_i} ⇔ $i$ misreports
r_i = ? ⇔ $i$ withholds If a personalized model guarantees the fair use of a
group attribute $\mathcal{G}$ then each person would opt to truthfully report
as this strategy would maximize their expected performance:

$\bm{g}_{i}\in\operatorname*{argmin}_{\bm{r}_{i}\in\mathcal{G}\cup\\{?\\}}\mathbb{E}\left[\ell\left(h(\bm{x},\bm{r}_{i}),y_{i}\right)\mid\mathcal{G}=\bm{g}_{i}\right].$

When a model violates fair use conditions, individuals in a specific group
would not report group membership voluntarily or truthfully. If a model obeys
fair use, however, individuals may still withhold group membership because the
gain is simply not sufficient. In light of this, fair use conditions should be
seen as minimal requirements to flag harm rather than a “rubber stamp" to
ensure that a model is personalized appropriately.

#### Use Cases

Fair use conditions should hold in prediction tasks where individuals are
entitled to control and report their own data.In such tasks, we should ensure
fair use conditions for group attributes that encode:

  * Immutable Characteristics: Models often use group attributes that encode immutable characteristics for personalization like sex [see 79, for a discussion]. In this setting, fair use conditions ensure that no group is assigned unnecessarily inaccurate performance on the basis of immutable characteristics.

  * Sensitive Information: Models that use attributes like hiv_status guarantee a tailored gain in performance for the sensitive group, $\texttt{\footnotesize{hiv\\_status}}=\texttt{\footnotesize{+}}$, especially as the information could inflict harm when leaked [see e.g., 8].

  * Self-Reported Information: Models are often personalized using information that individuals must report themselves – see e.g., self-reported screening tests for mental illnesses [see e.g., 59, 91]. These models should obey fair use conditions to incentivize truthful self-reporting as per Proposition 2.

  * Costly Information: Group attributes can encode data collected at prediction time – e.g., an attribute like tumor_subtype whose value can only be determined by an invasive medical test. Models that ensure fair use with respect to tumor_subtype guarantee that patients with a specific type of tumor will not receive a less accurate prediction by undergoing the procedure.

###  2.2 Evaluating Fair Use in Practice

We can always evaluate the fair use guarantees by simply training a generic
model and a personalized model. Now we how to use these two models to reliably
evaluate the conditions described in Definition 1.

Point estimates of the gains of personalization are unreliable, especially for
small groups. In a classification task where a personalized model performs 5%
worse than a generic model, this may represent 5 mistakes for a group with 100
samples, or 200 mistakes for a group with 4000 samples. We can distinguish
between these cases by measuring the statistical significance of gains. In
some settings, a significant fair use violation could warrant the need for a
new model. In others, we may wish to ensure a significant gain to use a group
attribute in the first place.

In practice, we check for a rationality violation using a one-sided hypothesis
test:

$\displaystyle H_{0}:{R_{\bm{g}}(h_{0})}-{R_{\bm{g}}(h)}\leq 0\quad
H_{A}:{R_{\bm{g}}(h_{0})}-{R_{\bm{g}}(h)}>0$

Here, the null hypothesis $H_{0}$ assumes that group $\bm{g}$ prefers
$h_{\bm{g}}$ to $h_{0}$ by default. Thus, we would reject $H_{0}$ when there
is enough evidence to support a rationality violation for $\bm{g}$ on held-out
data.To test the conditions in Definition 1, we would repeat this test for all
$m$ groups to check rationality across all groups, for all $m(m-1)$ pairs of
groups to check envy-freeness. We can control the false discovery rate using a
standard Bonferroni correction[31], which is suitable even for non-independent
tests. The exact test should be chosen based on the performance metric for the
prediction task. In general, one can test these hypotheses for any performance
metric using a bootstrap hypothesis test [28], and use more powerful tests for
salient performance metrics [see e.g., the McNemar test for accuracy 29]. We
note that there are limits to this hypothesis testing approach as the number
group attributes increases significantly [77].

##  3 Theoretical Guarantees and Failure Modes

In this section, we characterize sufficient conditions for when ERM ensures
fair use conditions on both the empirical risk and true risk. We also describe
failure modes that lead to detrimental personalization at a group level.

###  3.1 Theoretical Guarantees

Consider the setup in Section 2 where we fit a personalized model
$h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ from $\mathcal{H}$ as

$h=\operatorname*{argmin}_{h\in\mathcal{H}}{\hat{R}(h)}.$

Additionally, we train a generic model $h_{0}{}:\mathcal{X}\to\mathcal{Y}$
from a hypothesis class $\mathcal{H}_{0}\subset\mathcal{H}$.

######  Definition 3 (Opt-Out Compatibility).

A personalized model class $\mathcal{H}$ is opt-out compatible if for every
$h_{0}\in\mathcal{H}_{0}$, there exists a personalized model $h\in\mathcal{H}$
such that $h_{0}(\bm{x})=h(\bm{x},\bm{g})$ for all $\bm{x}\in\mathcal{X}$ and
all groups $\bm{g}\in\mathcal{G}$.

To ensure rationality and envy-freeeness, we should select $\mathcal{H}$ such
that it is an opt-out compatible model class. Furthermore, $\mathcal{H}$
should be rich enough to personalize for each group who reports personalized
data. If you are going to include an intercept for group $\bm{g}$, include it
for $\bm{g}^{\prime}$ (this doesn’t hold in one-hot encoding). Likewise, for
interaction terms.

######  Proposition 4 (Gains of Personalization for Empirical Risk).

A personalized model $h\in\mathcal{H}$ will guarantee envy-freeness and
rationality if
${\hat{R}_{\bm{g}}(h)}={\hat{R}_{\bm{g}}(h_{\bm{g}})}\;\textrm{for all
groups}\;\bm{g}\in\mathcal{G}$. Here, $h_{\bm{g}}$ is a model trained for each
group

$h_{\bm{g}}\in\operatorname*{argmin}_{h\in\mathcal{H}_{0}}{\hat{R}_{\bm{g}}(h)}.$

We present a proof in Appendix B. The conditions in Proposition 4 are met
when, for example, we train a decoupled classifier. In practice, the result
implies that we should fit personalized models from hypothesis classes that
account for group membership for each intersectional group – i.e., if we are
going to personalize a linear classifier via “score correction" [93].
Otherwise, we may violate fair use due to model misspecification when using a
one-hot encoding as in Fig. 2.

######  Proposition 5 (Generalization of Gains of Personalization).

Consider a personalized model $h_{\bm{g}}\in\mathcal{H}$ that obeys
rationality and envy-freeness on the empirical risk. Denote the gains
associated with rationality and envy-freeness for group $\bm{g}$ on the
empirical risk as:

$\displaystyle\hat{\epsilon}_{\bm{g}}:=\hat{\Delta}_{\bm{g}}({h_{\bm{g}}},{h_{0}{}}).\qquad\hat{\gamma}_{\bm{g}}:=\min_{\bm{g}^{\prime}\in\mathcal{G}/\\{\bm{g}\\}}\hat{\Delta}_{\bm{g}}({h_{\bm{g}}},{h_{\bm{g}^{\prime}}})$
(3)

If $\hat{\epsilon}_{\bm{g}}>0$, then the rationality gains for group $\bm{g}$
will generalize with probability at least $1-\delta$ as long as:

$n_{\bm{g}}\geq\frac{4\textrm{VC}({\mathcal{H}})\ln\left(\frac{2n_{\bm{g}}}{\textrm{VC}({\mathcal{H}})}+1\right)+\ln\left(\tfrac{8}{\delta}\right)}{\hat{\epsilon}_{\bm{g}}^{2}}$

If $\hat{\gamma}_{\bm{g}}>0$, then the envy-freeness gains for group $\bm{g}$
will generalize with probability at least $1-\delta$ as long as:

$\displaystyle n_{\bm{g}}$
$\displaystyle\geq\frac{4\textrm{VC}({\mathcal{H}})\ln\left(\frac{2n_{\bm{g}}}{\textrm{VC}({\mathcal{H}})}+1\right)+\ln\left(\tfrac{8m}{\delta}\right)}{\hat{\gamma}_{\bm{g}}^{2}}$

Proposition 5 characterizes the sample complexity of generalizing rationality
and envy-freeness gains from training data for a general class of personalized
models. Stronger bounds could be determined by assuming a finite hypothesis
class and for decoupled classifiers [e.g, in 92], or accounting for
distributional differences between groups [e.g., 100]. The result show that as
the gains increases, we require fewer samples. These conditions can be checked
and used to easily flag models where personalization is “weird." If two
classifiers assign the same predictions, this flags a potential to revisit
categorization and limit data collection. If the generic model and
personalized model assign the same predictions to each group, there will be no
gain (and thus an opportunity to limit data collection). If the personalized
model assigns the same prediction to groups $\bm{g}^{\prime}$ and $\bm{g}$,
then one could personalize for a larger category $\bm{g},\bm{g}^{\prime}$.

###  3.2 Failure Modes

We now characterize failure modes of personalizing with group attributes –
i.e., common practices that lead personalization to reduce performance at a
group level. We present failure modes of model misspecification and selection
to motivate potential interventions for model development in Section 4. Other
failure modes are presented in Appendix C. These include: training with a
surrogate loss function; generalization; and dataset shifts.

#### Misspecification

We start with misspecification – i.e., when we fit models that cannot capture
the role of group membership in the data distribution. A common form of
misspecification occurs when we personalize simple models with a one-hot
encoding [93]. In such cases, models exhibit fair use violations on data
distributions that exhibit intersectionality. Consider, for example, a
logistic regression model with a one-hot encoding that assigns higher risk to
patients who are young, and to patients who are female. This model would
exhibit a fair use violation for patients who are young _and_ female if their
true risk were lower than either group alone (see Fig. 2). We can eliminate
these violations by training a model which satisfies Proposition 4.

Misspecification can stem from group-specific interaction effects – e.g.,
instances where group attributes act as mediators or moderators [see e.g.,
10]. In Example 1, we show an instance that exhibits the hallmarks of
personalization: a generic model performs poorly on “heterogeneous" groups $A$
and $C$, and a personalized model that accounts for group membership improves
performance.

######  Example 1.

Consider a 2D classification task with three groups $\mathcal{G}=\\{A,B,C\\}$
where each group has two data points (1 positive and 1 negative). The Bayes
optimal personalized model should assign a personalized intercept to each
group _and_ a personalized slope to group $B$.

In this case, a personalized model with a one-hot encoding would only fit a
personalized intercept, but fail to fit a personalized slope for group $B$
(e.g. $t_{\bm{g}}+\mathbf{w_{g}}^{T}\mathbf{x}$ for each group
$\bm{g}\in\mathcal{G}$ where $\mathbf{w_{g}}$ is the same across all groups).
The personalized model would improve overall performance by assigning more
accurate predictions to groups $A$ and $C$. However, it would result in a fair
use violation by performing _worse_ for group $B$.

In practice, we can avoid these issues by fitting models that are rich enough
to capture these effects, or by training a separate model for each group. Both
are challenging in tasks with multiple groups as we must specify interactions
for each group, or fit models using a limited amount of data for each group.

#### Model Selection

Model development often involves choosing a model from a family of candidate
models – e.g., when we set a regularization penalty to avoid overfitting, or
choose a subset of variables to improve usability. Common criteria for model
selection choose models on the basis of population-level performance [e.g.,
mean K-CV test error 5]. In tasks where there exist unavoidable trade-offs
between groups, these criteria lead to models that improve performance for one
group while reducing performance for another group. We demonstrate this effect
in Example 2.

######  Example 2.

Consider a classification task where we have two groups $A$ and $B$ who have
$40$ and $35$ data points respectively. In this task, a model must use at most
one of features $(x_{1},x_{2})\in\\{0,1\\}^{2}$. We perform variable selection
to minimize the overall error rate. We then train a personalized model with a
one-hot encoding of $\mathcal{G}=\\{0,1\\}$ and a generic model.

|  | Generic - uses $x_{1}$ or $x_{2}$
---
| Personalized – uses $x_{1}$
---
| Personalized – uses $x_{2}$
---
$g$ | $(x_{1},x_{2})$ | $n^{+}$ | $n^{-}$ | $h_{0}{}$ | $R(h_{0}{})$ | $h_{1}$ | $R(h_{1})$ | $\Delta$ | $h_{2}$ | $R(h_{2})$ | $\Delta$
$A$ | $(0,0)$ | $0$ | $30$ | $-$ | $0$ | $-$ | $0$ | $0$ | $+$ | $30$ | $-30$
$A$ | $(0,1)$ | $0$ | $0$ | $-$ | $0$ | $-$ | $0$ | $0$ | $-$ | $0$ | $0$
$A$ | $(1,0)$ | $0$ | $20$ | $-$ | $0$ | $+$ | $20$ | $-20$ | $+$ | $20$ | $-20$
$A$ | $(1,1)$ | $0$ | $0$ | $-$ | $0$ | $+$ | $0$ | $0$ | $-$ | $0$ | $0$
$B$ | $(0,0)$ | $25$ | $0$ | $-$ | $25$ | $+$ | $0$ | $25$ | $+$ | $0$ | $25$
$B$ | $(0,1)$ | $0$ | $0$ | $-$ | $0$ | $+$ | $0$ | $0$ | $-$ | $0$ | $0$
$B$ | $(1,0)$ | $15$ | $0$ | $-$ | $15$ | $+$ | $0$ | $15$ | $+$ | $0$ | $15$
$B$ | $(1,1)$ | $0$ | $0$ | $-$ | $0$ | $-$ | $0$ | $0$ | $-$ | $0$ | $0$
Total (A + B) |  | $40$ | $50$ |  | $40$ |  | $35$ | $5$ |  | $50$ | $-10$
$g=A$ |  | $0$ | $50$ |  | $0$ |  | $20$ | $-20$ |  | $50$ | $-50$
$g=B$ |  | $40$ | $0$ |  | $40$ |  | $15$ | $25$ |  | $0$ | $40$

The generic model $h_{0}$ is the same whether it uses $x_{1}$ or $x_{2}$. Yet,
the personalized model exhibits differences based on its selection. Each group
performs better under different choices, defaulting to the choice that
benefits the majority group. In both settings, the best personalized model
that uses $x_{1}$ and the best personalized model that uses $x_{2}$ exhibit
fair use violations.

In practice, such a constraint could arise when developing a medical
diagnostic where, for example, $x_{1}$ and $x_{2}$ encode the outcome of a
medical test, and a diagnostic should only use the test that optimizes
accuracy and the data distribution exhibit heterogeneity across groups. More
broadly, Example 2 highlights how fair use violations may be unavoidable when
we must assign predictions with a single model – as there may not exist a
model that ensure fair use for all groups. In these settings, decoupling
classifiers is a safeguard that developers should evaluate. In the example, if
we decoupled the classifiers for group $A$ and $B$ we could have captured the
effect that different variables are better for different groups.

##  4 Experimental Results

In this section, we investigate fair use in clinical prediction models – i.e.
models that routinely include group attributes where fair use violations
inflict harm. Our goals are to measure the prevalence of fair use violations
and to evaluate our ability to mitigate them through in model development. We
attach software to reproduce these results to our submission, and include
additional results in Appendix E.

|  | Test AUC | Test ECE | Test Error
---|---|---|---|---
Dataset | Metrics |  | 1Hot | All | DCP | 1Hot | All | DCP | 1Hot | All | DCP
| apnea
---
$n={1152},d={26}$
$\mathcal{G}=\\{\texttt{\small{age}},\texttt{\small{sex}}\\}$
$m={6}$
[90]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.750
---
0.001
0.002 / -0.001
1/2
0/0
| 0.750
---
0.000
0.001 / -0.016
1/4
0/0
| 0.803
---
0.053
0.119 / -0.005
4/0
3/0
| 7.5%
---
-1.5%
0.7% / -7.1%
1/3
0/3
| 5.5%
---
0.6%
0.7% / -4.6%
1/3
0/3
| 7.2%
---
-1.1%
1.7% / -6.6%
2/2
4/0
| 34.2%
---
-1.0%
0.0% / -9.9%
0/4
0/6
| 33.8%
---
-0.7%
1.8% / -7.8%
1/3
0/5
| 26.2%
---
7.0%
21.7% / -7.8%
4/1
4/1
| cardio_eicu
---
$n={1341},d={49}$
$\mathcal{G}=\\{\texttt{\small{age}},\texttt{\small{sex}}\\}$
$m={4}$
[83]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.768
---
0.000
0.002 / -0.001
2/2
0/0
| 0.767
---
-0.001
0.001 / -0.001
2/1
0/0
| 0.762
---
-0.007
0.094 / -0.099
1/2
3/1
| 4.4%
---
0.4%
1.6% / -1.5%
2/1
0/2
| 4.6%
---
0.2%
0.9% / -0.2%
1/0
0/2
| 8.9%
---
-4.1%
-1.1% / -6.3%
0/4
1/1
| 29.1%
---
-0.4%
0.0% / -3.1%
0/2
0/3
| 29.1%
---
-0.4%
0.2% / -3.1%
1/2
0/3
| 29.5%
---
-0.9%
12.9% / -8.9%
2/2
3/1
| cardio_mimic
---
$n={5289},d={49}$
$\mathcal{G}=\\{\texttt{\small{age}},\texttt{\small{sex}}\\}$
$m={4}$
[54]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.854
---
0.001
0.001 / -0.000
2/1
0/0
| 0.854
---
0.001
0.001 / -0.000
2/1
0/0
| 0.870
---
0.017
0.051 / 0.006
4/0
4/0
| 2.1%
---
-0.4%
0.5% / 0.4%
4/0
1/3
| 2.3%
---
-0.5%
0.6% / -0.2%
3/0
0/1
| 2.3%
---
-0.6%
0.6% / -2.3%
1/2
3/1
| 23.3%
---
0.3%
0.9% / -0.1%
3/0
0/3
| 23.4%
---
0.3%
0.9% / -0.1%
3/0
0/3
| 21.4%
---
2.2%
7.6% / -0.2%
3/0
4/0
| heart
---
$n={181},d={26}$
$\mathcal{G}=\\{\texttt{\small{sex}},\texttt{\small{age}}\\}$
$m={4}$
[27]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.870
---
-0.007
0.007 / -0.031
1/1
0/0
| 0.846
---
-0.030
0.024 / -0.050
1/1
0/0
| 0.817
---
-0.060
0.039 / -0.190
0/3
1/2
| 8.4%
---
2.8%
4.4% / -0.6%
2/1
0/2
| 17.8%
---
-6.6%
-1.8% / -3.1%
0/4
0/3
| 17.5%
---
-6.3%
10.1% / -4.6%
2/1
2/2
| 19.7%
---
-1.3%
0.0% / -6.1%
0/1
0/1
| 19.7%
---
-1.3%
0.0% / -12.1%
0/3
0/1
| 15.8%
---
2.6%
10.6% / -8.4%
3/1
2/1
| mortality
---
$n={25366},d={468}$
$\mathcal{G}=\\{\texttt{\small{age}},\texttt{\small{sex}}\\}$
$m={6}$
[54]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.848
---
0.000
0.005 / -0.001
3/3
0/0
| 0.848
---
0.001
0.005 / -0.000
3/2
0/0
| 0.880
---
0.033
0.111 / 0.012
6/0
6/0
| 2.0%
---
0.2%
1.5% / 0.1%
5/0
1/1
| 2.1%
---
0.1%
2.6% / -0.3%
5/1
3/2
| 2.5%
---
-0.3%
11.2% / -2.4%
3/2
5/1
| 23.6%
---
-0.2%
0.8% / -2.5%
2/4
0/4
| 23.4%
---
-0.0%
2.1% / -0.4%
3/2
1/4
| 20.2%
---
3.2%
20.1% / -0.5%
5/1
6/0
| saps
---
$n={7797},d={36}$
$\mathcal{G}=\\{\texttt{\small{hiv}},\texttt{\small{age}}\\}$
$m={4}$
[4]
| Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.890
---
0.001
0.014 / -0.000
1/1
0/0
| 0.890
---
0.001
0.014 / -0.001
1/1
0/0
| 0.888
---
-0.001
0.017 / -0.246
2/2
2/1
| 1.5%
---
0.1%
2.8% / -1.5%
2/0
2/2
| 1.5%
---
0.1%
2.4% / -0.6%
2/1
2/2
| 2.0%
---
-0.4%
9.4% / -19.1%
2/2
3/1
| 18.9%
---
0.0%
19.0% / -10.4%
2/1
1/1
| 18.9%
---
0.0%
0.8% / -10.4%
1/3
2/2
| 18.5%
---
0.4%
3.5% / -23.3%
2/1
2/2

Table 1: Performance of personalized logistic regression models on all
datasets. We show the gains of personalization in terms of test AUC, ECE, and
error. We report: model performance at the population level, the overall gain
of personalization, the range of gains over $m$ intersectional groups, and the
number of rationality and envy-freeness gains/violations (evaluated using a
bootstrap hypothesis test at a 10% significance level).

###  4.1 Setup

We work with 6 datasets for clinical prediction tasks listed in Table 1 (see
also Appendix D). We split each dataset into a training sample (80%) to fit
models, and a test sample (20%) to evaluate the gains of personalization. We
train 9 kinds of personalized models for each dataset. Each model belongs to
one of 3 model classes: _logistic regression_ (LR), _random forests_ (RF), and
_neural nets_ (NN), and is personalized using one of 3 techniques to encode
group attributes:

  * $\bullet$

_One-hot Encoding_ (1Hot): We train a model with dummy variables for each
group attribute.

  * $\bullet$

_Intersectional Encoding_ (All): We train a model with dummy variables for
each intersectional group.

  * $\bullet$

_Decoupling_ (DCP): We train a model for each intersectional group using its
own data.

These techniques represent increasingly complex ways to account for group
membership where complexity is measured by the interactions between group
attributes and other features: 1Hot reflect no interactions; All reflect
interactions between group attributes; and DCP reflects all possible
attributes between group attributes and features.

We evaluate the gains of personalization for each model in terms of three
performance metrics: (1) _error rate_ , which reflects the accuracy of yes-or-
no predictions, e.g., for a diagnostic test [36]; (2) _expected calibration
error_ (ECE), which measures the reliability of risk predictions for a risk
score [16]; (3) _area under ROC curve_ (AUC), which measures accuracy in
ranking, e.g., for triage [e.g., 106].

###  4.2 Results

We summarize our results for logistic regression in Table 1 and for other
model classes (neural networks and random forests) in Appendix E.

#### On Prevalence

Our results show that personalized models can improve performance at a
population level yet reduce performance for specific groups. These fair use
violations arise across datasets, personalization techniques, and model
classes. Consider the standard configuration used to develop clinical
prediction models [87] – i.e., a logistic regression model with a one-hot
encoding of group attributes (LR+1Hot). We find that at least one group
experiences a statistically significant fair use violation in terms of error
on 4/6 datasets (5/6 for AUC and ECE).

#### On Personalization Techniques

Our results show there is no one personalization technique that minimizes fair
use violations. There are many phenomena (see Section 3 for examples) that
result in violations and these can co-occur. In Table 1, for example, the best
personalization technique for cardio_eicu is intersectional encoding while the
best personalization technique for mortality was decoupling. These strategies
change across model classes – as the corresponding strategies for neural
networks for cardio_eicu and mortality are decoupling and using an
intersectional encoding, respectively (see Appendix E). In general, even
strategies that exhibit few violations can fail critically. For example,
LR+DCP for saps leads to a 10% increase in error for HIV+ & >30. Overall,
these results suggest that the most consistent way to avoid the harm from a
fair use violation is to check.

#### On Resolving Violations

Our results show that routine decisions in model development can produce
considerable differences in group-level performance. This suggests that when
we spot fair use violations, we could reduce them through “interventions" in
model development. Based on this, we sought to reduce fair use violations by
rerunning our experiments considering interventions that would address the
failure modes in Section 3 – e.g., using an intersectional one-hot encoding,
training decoupled models, and equalizing sample sizes. We show that these
interventions can often reduce fair use violations. For example, we can
eliminate all fair use violations for cardio_mimic in our standard
configuration by training decoupled models. However, there is no “silver
bullet" intervention that resolves fair use across all datasets and model
classes. The right intervention varies across model classes and datasets. This
is often because an intervention that resolves a violation for one group will
result in a violation for others. In cardio_eicu, for instances, a logistic
regression model fit with a one-hot encoding will exhibit a violation on old
males. Switching an intersectional encoding will fix this violation but
introduce a new one for old females.

#### On Detecting Violations

Our results underscore the need for reliable procedures to discover fair use
violations or claim gains from personalization. We can often find detectable
instances of benefit or harm. For example, we find that on saps in our
standard configuration that we detect a gain from personalization for patients
over 30 who are HIV negative. In cardio_eicu when training LR+All we detect a
fair use violation for patients who are old females (see e.g., Rat
Gains/Violations in Table 1). One actionable finding that may result an
evaluation of the gains of personalization is a group does not experience a
meaningful gain nor harm due to personalization. In such cases, one may wish
to intervene to avoid soliciting unnecessary data: when group attributes
encode information that is sensitive or must be collected at prediction time
(e.g., HIV), we may prefer to avoid soliciting information unless it is
demonstrably useful for prediction.

Group | Test AUC | Interventions
---|---|---
$\bm{g}$ |  | ${R_{\bm{g}}(h_{\bm{g}})}$ | $\Delta_{\bm{g}}$ |  | Assign $h_{0}{}$
---
| Assign $h^{\textrm{dcp}}_{\bm{g}}$
---
female, black |  | 0.463 | 0.024 | 0.024 | 0.334
female, white |  | 0.846 | 0.004 | 0.004 | 0.004
female, other |  | 0.860 | -0.003 | 0.000 | 0.057
male, black |  | 0.767 | -0.001 | 0.000 | 0.104
male, white |  | 0.767 | 0.004 | 0.004 | 0.038
male, other |  | 0.836 | -0.002 | 0.000 | 0.017
Total |  | 0.800 | 0.006 | - | -
Test Error | Intervention
---|---
| ${R_{\bm{g}}(h_{\bm{g}})}$ | $\Delta_{\bm{g}}$ |  | Assign $h_{0}{}$
---
| Assign $h^{\textrm{dcp}}_{\bm{g}}$
---
| 52.2% | 6.8% | 6.8% | 37.3%
| 21.7% | 2.0% | 2.0% | 2.0%
| 25.5% | 1.3% | 1.3% | 14.8%
| 34.0% | -5.2% | 0.0% | 15.6%
| 29.2% | 1.3% | 1.3% | 3.7%
| 27.9% | -5.0% | 0.0% | 1.3%
| 28.3% | 0.3% | - | -
Test ECE | Interventions
---|---
| ${R_{\bm{g}}(h_{\bm{g}})}$ | $\Delta_{\bm{g}}$ |  | Assign $h_{0}$
---
| Assign $h^{\textrm{dcp}}_{\bm{g}}$
---
| 31.6% | 2.3% | 2.3% | 12.3%
| 10.2% | 1.9% | 1.9% | 2.1%
| 15.5% | 0.9% | 0.9% | 5.0%
| 20.1% | -2.0% | 0.0% | 4.9%
| 10.3% | 1.2% | 1.2% | 1.2%
| 15.4% | -1.6% | 0.0% | 0.0%
| 4.7% | 0.2% | - | -

Table 2: Fair use evaluation of a personalized logistic regression model with
a one-hot encoding of group attributes. As shown, personalization can improve
overall performance while reduces performance for specific groups (red). This
result holds across all performance metrics. In such cases, we can resolve
fair use violations and improve the gains from personalization by assigning
personalized predictions to each group with multiple models. By this we mean
selecting from one of three available models which provides the most accurate
predictions for a group: a generic model $h_{0}{}$, the personalized model
$h_{\bm{g}}$, or a decoupled model $h^{\textrm{dcp}}$. We highlight cases
where assigning predictions from one of these models led to a gain in green,
and where it resolved a violation in yellow.

##  5 Mortality in Acute Kidney Injury

In this section, we evaluate the gains of personalization in model to predict
mortality for patients with acute kidney injury. Our results demonstrate that
fair use violations are informative for model auditing [69] and for guiding
simple interventions to mitigate harm.

###  5.1 Setup

We consider a classification task to predict mortality for patients who
receive continuous renal replacement therapy while in the ICU. The data
consists of records for $n=2066$ patients from MIMIC III and IV [54]. Here,
$y_{i}=+1$ if patient $i$ dies in the ICU and $\textrm{Pr}(y_{i}=+1)=51.1\%$.
Each patient has $k=2$ group attributes:
$\texttt{\footnotesize{sex}}\in\\{\texttt{\footnotesize{male}},\texttt{\footnotesize{female}}\\}$
and
$\texttt{\footnotesize{race}}\in\\{\texttt{\footnotesize{white}},\texttt{\small{black}},\texttt{\footnotesize{other}}\\}$
and $d=78$ features related to their health, lab tests, length of stay, and
potential for organ failure. We train and evaluate personalized models using
the setup from Table 1.

###  5.2 Results

We show the performance of a personalized logistic regression model with a
one-hot encoding in Table 2, and present results for other configurations and
model classes in Appendix E. Overall, our findings show that personalization
yields uneven gains at a group level. As in Section 4.2, we observe fair use
violations across performance metrics and model classes. In this case, the
gains in error across range from -5.2% to 6.8%, and two groups experience
statistically significant fair use violations: (male, black) and (male,
other).

#### On the Use of Race

Clinical prediction models include group attributes when there is an assumed
relationship between group membership and the outcome of interest. These norms
have led to development of widely-used clinical prediction models that use
race or ethnicity [35, 96, 40, 49, 64, 71, 41, 56]. Recently, Vyas et al. [96]
discuss how these models can inflict harm and urge physicians to check if
“race correction is based on robust [statistical] evidence." Our results
highlight how fair use evaluation can provide evidence that serves as a
barrier to “race correction" for such cases. Here, checking rationality shows
that a race-specific model can reduce performance for specific groups – e.g.,
(male, black) and (male, other). Checking envy-freeness reveals that certain
groups expect better performance by misreporting their group membership –
e.g., (male,other) would experience 5.6% gain in test error by reporting any
other race.

#### On Race as a Proxy and Proxies of Race

In cases where race improve performance, we note that race itself may act as a
proxy for broader social determinants of health. Thus, a model that includes
race may act as a “smoke screen" in that it attributes differences in health
outcomes to an immutable factor, and thereby perpetuates inaction on the root
causes of health disparities [80].

In cases where race does not improve model performance, there may still be
considerable differences in model performance across racial groups – as health
data is generated from a complex human system that can encode proxies for
self-reported race in redacted notes [1], or even band-pass filtered medical
images [42]. In the same way that height, weight, and bone mass are proxies
for sex [76], models can use other features to account for race leading to a
lack of gains across groups. Given these uncertainties, we advocate that race
should only be included in clinical model when there is evidence of gain.
Regardless of its use in prediction, collecting information about race and
ethnicity is necessary to measure model performance across these groups. In
such cases, one should be careful to disclose the purposes of data collection
– stating that it will be used to evaluate performance but not to assign
personalized predictions.

#### Interventions

We use our results to discuss interventions to resolve fair use violations by
assigning predictions from different models at prediction time. These
interventions are admittedly simple, but they are broadly applicable to any
model class and dataset. Thus we can apply them whenever there is a fair use
violation.

_Assigning a Generic Model_. We assign groups who experience a fair use
violation the predictions from a generic model $h_{0}$. This intervention is
guaranteed to resolve all fair use violations in a way that strictly improves
performance. in this case, it resolves all rationality violations (2/3/2 in
terms of error/AUC/ECE respectively). We also observe a potential to reduce
data use: seeing how both (male, black) and (male, other) experience a fair
use violation in terms of error, we see that we could avoid soliciting race
for all male patients and reduce test error by 1% (as the loss in accuracy for
(white, male) are offset by the gain in accuracy for (male,black) and (male,
other).

_Assigning a Decoupled Model_. We assign groups who experience a fair use
violation the predictions from the best of a generic model, personalized
model, or a _decoupled model_ $h^{\textrm{dcp}}_{\bm{g}{}}$– i.e., a model
trained using only data from their group. While this approach may not resolve
fair use violations, it can produce surprisingly large gains as decoupling
effectively personalizes the entire model development pipeline. Our results in
Table 2 show the potential gains of this intervention across all performance
metrics. Focusing on error, we see that one can: (1) eliminate fair use
violations for (male,black) and (male,other); (2) greatly improve the gains
for (female,black) who experience a gain of 37.3%; and (3) improve overall
gains by $6.2\%$. These effects are noted across other configurations and
model classes as well.

##  6 Concluding Remarks

Our work shows prediction models that use group attributes for personalization
can fail to improve performance for all groups. We show how this effect
arises, derive theoretical sufficient conditions for when personalization
improves performance for all gorups, demonstrated its prevalence, and proposed
simple interventions to mitigate harm.

Our results underscore the need to evaluate the gains of personalization
[e.g., 86, 97, 23, 25, 17, 19, 58]. Evaluating the gains of personalization is
a routine procedure that can be performed as part of model development, and
integrated into model reports [70, 6, 9, 18, 20]. The information produced
through this evaluation can help practitioners reduce instances of
unnecessarily poor performance, and guide them to broadening the gains of
personalization. In all cases, fair use conditions should be viewed as a
safeguard rather than a rubber stamp.

#### Acknowledgements

We wish to thank the following individuals for helpful discussions: Flavio
Calmon, Katherine Heller, Sanmi Koyejo, Ziad Obermeyer, Charlie Marx, Stephen
Pfohl, Emma Pierson, Kush Varshney, and Haoran Zhang. This work was supported
by funding from the Wellcome Trust, and the National Science Foundation IIS
2040880.

## References

  * Adam et al. [2022] Adam, Hammaad, Ming Ying Yang, Kenrick Cato, Ioana Baldini, Charles Senteio, Leo Anthony Celi, Jiaming Zeng, Moninder Singh, and Marzyeh Ghassemi.  Write it like you see it: Detectable differences in clinical notes by race lead to differential model recommendations.  In _AIES 2022: Conference on Artificial Intelligence, Ethics and Society_ , 2022.
  * Agarwal et al. [2018] Agarwal, Alekh, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach.  A Reductions Approach to Fair Classification.  In _Proceedings of the 35th International Conference on Machine Learning_ , Proceedings of Machine Learning Research. PMLR, 2018.
  * Agresti [2018] Agresti, Alan.  _An introduction to categorical data analysis_.  John Wiley & Sons, 2018.
  * Allyn et al. [2016] Allyn, Jérôme, Cyril Ferdynus, Michel Bohrer, Cécile Dalban, Dorothée Valance, and Nicolas Allou.  Simplified acute physiology score ii as predictor of mortality in intensive care units: a decision curve analysis.  _PloS one_ , 11(10):e0164828, 2016.
  * Arlot and Celisse [2010] Arlot, Sylvain and Alain Celisse.  A survey of cross-validation procedures for model selection.  _Statistics surveys_ , 4:40–79, 2010.
  * Arnold et al. [2019] Arnold, Matthew, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al.  Factsheets: Increasing trust in ai services through supplier’s declarations of conformity.  _IBM Journal of Research and Development_ , 63(4/5):6–1, 2019.
  * Balcan et al. [2018] Balcan, Maria-Florina, Travis Dick, Ritesh Noothigattu, and Ariel D Procaccia.  Envy-free classification.  _arXiv preprint arXiv:1809.08700_ , 2018.
  * Bansal et al. [2010] Bansal, Gaurav, David Gefen, et al.  The impact of personal dispositions on information sensitivity, privacy concern and trust in disclosing health information online.  _Decision support systems_ , 49(2):138–150, 2010\.
  * Barocas et al. [2021] Barocas, Solon, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, and Hanna Wallach.  Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs.  _arXiv preprint arXiv:2103.06076_ , 2021.
  * Baron and Kenny [1986] Baron, Reuben M and David A Kenny.  The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.  _Journal of personality and social psychology_ , 51(6):1173, 1986.
  * Beauchamp et al. [2001] Beauchamp, Tom L, James F Childress, et al.  _Principles of Biomedical Ethics_.  Oxford University Press, USA, 2001.
  * Bertsimas and Kallus [2020] Bertsimas, Dimitris and Nathan Kallus.  From predictive to prescriptive analytics.  _Management Science_ , 66(3):1025–1044, 2020\.
  * Bertsimas et al. [2019] Bertsimas, Dimitris, Jack Dunn, and Nishanth Mundru.  Optimal prescriptive trees.  _INFORMS Journal on Optimization_ , 1(2):164–183, 2019.
  * Bien et al. [2013] Bien, Jacob, Jonathan Taylor, and Robert Tibshirani.  A lasso for hierarchical interactions.  _Annals of statistics_ , 41(3):1111, 2013.
  * Biggs et al. [2020] Biggs, Max, Wei Sun, and Markus Ettl.  Model distillation for revenue optimization: Interpretable personalized pricing.  _arXiv preprint arXiv:2007.01903_ , 2020.
  * Blaha [2016] Blaha, Michael J.  The critical importance of risk score calibration: time for transformative approach to risk score validation?, 2016.
  * Bouwmeester et al. [2012] Bouwmeester, Walter, Nicolaas PA Zuithoff, Susan Mallett, Mirjam I Geerlings, Yvonne Vergouwe, Ewout W Steyerberg, Douglas G Altman, and Karel GM Moons.  Reporting and methods in clinical prediction research: a systematic review.  _PLoS medicine_ , 9(5):e1001221, 2012.
  * Bynum et al. [2021] Bynum, Lucius, Joshua Loftus, and Julia Stoyanovich.  Disaggregated interventions to reduce inequality.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_ , pages 1–13. 2021.
  * Cabitza and Campagner [2021] Cabitza, Federico and Andrea Campagner.  The need to separate the wheat from the chaff in medical informatics, 2021\.
  * Cabrera et al. [2019] Cabrera, Ángel Alexander, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau.  Fairvis: Visual analytics for discovering intersectional bias in machine learning.  In _2019 IEEE Conference on Visual Analytics Science and Technology (VAST)_ , pages 46–56. IEEE, 2019.
  * Celis et al. [2019] Celis, L Elisa, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi.  Classification with fairness constraints: A meta-algorithm with provable guarantees.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 319–328. ACM, 2019.
  * Chen et al. [2021] Chen, Irene Y, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi.  Ethical machine learning in healthcare.  _Annual review of biomedical data science_ , 4:123–144, 2021.
  * Collins et al. [2015] Collins, Gary S, Johannes B Reitsma, Douglas G Altman, and Karel GM Moons.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): the tripod statement.  _Journal of British Surgery_ , 102(3):148–158, 2015.
  * Corbett-Davies and Goel [2018] Corbett-Davies, Sam and Sharad Goel.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _arXiv preprint arXiv:1808.00023_ , 2018.
  * Cowley et al. [2019] Cowley, Laura E, Daniel M Farewell, Sabine Maguire, and Alison M Kemp.  Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature.  _Diagnostic and prognostic research_ , 3(1):1–23, 2019.
  * Dai et al. [2020] Dai, Jessica, Sina Fazelpour, and Zachary C Lipton.  Fairness under partial compliance.  _arXiv preprint arXiv:2011.03654_ , 2020.
  * Detrano et al. [1989] Detrano, Robert, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher.  International application of a new probability algorithm for the diagnosis of coronary artery disease.  _The American journal of cardiology_ , 64(5):304–310, 1989.
  * DiCiccio and Efron [1996] DiCiccio, Thomas J and Bradley Efron.  Bootstrap confidence intervals.  _Statistical science_ , pages 189–212, 1996.
  * Dietterich [1998] Dietterich, Thomas G.  Approximate statistical tests for comparing supervised classification learning algorithms.  _Neural computation_ , 10(7):1895–1923, 1998\.
  * Do et al. [2021] Do, Virginie, Sam Corbett-Davies, Jamal Atif, and Nicolas Usunier.  Online certification of preference-based fairness for personalized recommender systems.  _arXiv preprint arXiv:2104.14527_ , 2021.
  * Dunn [1961] Dunn, Olive Jean.  Multiple comparisons among means.  _Journal of the American statistical association_ , 56(293):52–64, 1961.
  * Dwork et al. [2012] Dwork, Cynthia, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In _Proceedings of the 3rd innovations in theoretical computer science conference_ , pages 214–226, 2012.
  * Dwork et al. [2018] Dwork, Cynthia, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson.  Decoupled classifiers for group-fair and efficient machine learning.  In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ , volume 81 of _Proceedings of Machine Learning Research_ , pages 119–133. PMLR, 2018.
  * Elmachtoub et al. [2018] Elmachtoub, Adam N, Vishal Gupta, and Michael Hamilton.  The value of personalized pricing.  _Available at SSRN 3127719_ , 2018.
  * Eneanya et al. [2019] Eneanya, Nwamaka Denise, Wei Yang, and Peter Philip Reese.  Reconsidering the consequences of using race to estimate kidney function.  _Jama_ , 322(2):113–114, 2019.
  * Eusebi [2013] Eusebi, Paolo.  Diagnostic accuracy measures.  _Cerebrovascular Diseases_ , 36(4):267–272, 2013\.
  * Fan and Poole [2006] Fan, Haiyan and Marshall Scott Poole.  What is personalization? perspectives on the design and implementation of personalization in information systems.  _Journal of Organizational Computing and Electronic Commerce_ , 16(3-4):179–202, 2006.
  * Feldman et al. [2015] Feldman, Michael, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 259–268. ACM, 2015.
  * Finlayson et al. [2021] Finlayson, Samuel G, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane, and Suchi Saria.  The clinician and dataset shift in artificial intelligence.  _The New England journal of medicine_ , 385(3):283–286, 2021.
  * Flamm and Geiger [1997] Flamm, Bruce L and Ann M Geiger.  Vaginal birth after cesarean delivery: an admission scoring system.  _Obstetrics & Gynecology_, 90(6):907–910, 1997\.
  * Gail et al. [1989] Gail, Mitchell H, Louise A Brinton, David P Byar, Donald K Corle, Sylvan B Green, Catherine Schairer, and John J Mulvihill.  Projecting individualized probabilities of developing breast cancer for white females who are being examined annually.  _JNCI: Journal of the National Cancer Institute_ , 81(24):1879–1886, 1989.
  * Gichoya et al. [2022] Gichoya, Judy Wawira, Imon Banerjee, Ananth Reddy Bhimireddy, John L Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, et al.  Ai recognition of patient race in medical imaging: a modelling study.  _The Lancet Digital Health_ , 2022.
  * Globus-Harris et al. [2022] Globus-Harris, Ira, Michael Kearns, and Aaron Roth.  An algorithmic framework for bias bounties.  2022\.
  * Gneiting and Raftery [2007] Gneiting, Tilmann and Adrian E Raftery.  Strictly proper scoring rules, prediction, and estimation.  _Journal of the American statistical Association_ , 102(477):359–378, 2007.
  * Gulshan et al. [2016] Gulshan, Varun, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al.  Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  _Jama_ , 316(22):2402–2410, 2016.
  * Guo et al. [2021] Guo, Lin Lawrence, Stephen R Pfohl, Jason Fries, Jose Posada, Scott Lanyon Fleming, Catherine Aftandilian, Nigam Shah, and Lillian Sung.  Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine.  _Applied Clinical Informatics_ , 12(04):808–815, 2021.
  * Hardt et al. [2016] Hardt, Moritz, Eric Price, Nati Srebro, et al.  Equality of opportunity in supervised learning.  In _Advances in Neural Information Processing Systems_ , pages 3315–3323, 2016.
  * Harutyunyan et al. [2019] Harutyunyan, Hrayr, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan.  Multitask learning and benchmarking with clinical time series data.  _Scientific data_ , 6(1):1–18, 2019.
  * Haukoos et al. [2012] Haukoos, Jason S, Michael S Lyons, Christopher J Lindsell, Emily Hopkins, Brooke Bender, Richard E Rothman, Yu-Hsiang Hsieh, Lynsay A MacLaren, Mark W Thrun, Comilla Sasson, et al.  Derivation and validation of the denver human immunodeficiency virus (hiv) risk score for targeted hiv screening.  _American journal of epidemiology_ , 175(8):838–846, 2012.
  * Hébert-Johnson et al. [2018] Hébert-Johnson, Úrsula, Michael Kim, Omer Reingold, and Guy Rothblum.  Multicalibration: Calibration for the (computationally-identifiable) masses.  In _Proceedings of the International Conference on Machine Learning_ , pages 1944–1953, 2018.
  * Hu and Chen [2019] Hu, Lily and Yiling Chen.  Fair Classification and Social Welfare.  _arXiv preprint arXiv:1905.00147_ , 2019.
  * Jaques et al. [2016] Jaques, N., Taylor S. Taylor, Nosakhare E. Nosakhare, Sano A. Sano, and & Picard R. Picard R.  Multi-task learning for predicting health, stress, and happiness.  _Neural Information Processing Systems (NeurIPS) Workshop on Machine Learning for Healthcare_ , 2016.
  * Jaworska et al. [2019] Jaworska, Natalia, Sara SalleDe la , Mohamed-Hamza Ibrahim, Pierre Blier, and Verner Knott.  Leveraging machine learning approaches for predicting antidepressant treatment response using electroencephalography (eeg) and clinical data.  _Frontiers in psychiatry_ , 9:768, 2019.
  * Johnson et al. [2016] Johnson, Alistair EW, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Jovanovic [1982] Jovanovic, Boyan.  Truthful disclosure of information.  _The Bell Journal of Economics_ , pages 36–44, 1982.
  * Kanis et al. [2007] Kanis, John Anthony et al.  Assessment of osteoporosis at the primary health-care level.  Technical report, Technical report, 2007.
  * Kearns et al. [2018] Kearns, Michael, Seth Neel, Aaron Roth, and Zhiwei Steven Wu.  Preventing fairness gerrymandering: Auditing and learning for subgroup fairness.  In _Proceedings of the 35th International Conference on Machine Learning_ , Proceedings of Machine Learning Research. PMLR, 2018.
  * Kent et al. [2020] Kent, David M, Jessica K Paulus, David Van Klaveren, Ralph D’Agostino, Steve Goodman, Rodney Hayward, John PA Ioannidis, Bray Patrick-Lake, Sally Morton, Michael Pencina, et al.  The predictive approaches to treatment effect heterogeneity (path) statement.  _Annals of internal medicine_ , 172(1):35–45, 2020.
  * Kessler et al. [2005] Kessler, Ronald C, Lenard Adler, Minnie Ames, Olga Demler, Steve Faraone, EVA Hiripi, Mary J Howes, Robert Jin, Kristina Secnik, Thomas Spencer, et al.  The world health organization adult adhd self-report scale (asrs): a short screening scale for use in the general population.  _Psychological medicine_ , 35(2):245–256, 2005\.
  * Kim et al. [2019] Kim, Michael P, Aleksandra Korolova, Guy N Rothblum, and Gal Yona.  Preference-informed fairness.  _arXiv preprint arXiv:1904.01793_ , 2019.
  * Kleinberg et al. [2018] Kleinberg, Jon, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan.  Algorithmic Fairness.  In _AEA Papers and Proceedings_ , volume 108, pages 22–27, 2018.
  * Kravitz et al. [2004] Kravitz, Richard L, Naihua Duan, and Joel Braslow.  Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.  _The Milbank Quarterly_ , 82(4):661–687, 2004\.
  * Le Gall et al. [1993] Le Gall, Jean-Roger, Stanley Lemeshow, and Fabienne Saulnier.  A new simplified acute physiology score (saps ii) based on a european/north american multicenter study.  _Jama_ , 270(24):2957–2963, 1993.
  * Levey et al. [2009] Levey, Andrew S, Lesley A Stevens, Christopher H Schmid, Yaping Zhang, Alejandro F Castro III, Harold I Feldman, John W Kusek, Paul Eggers, Frederick Van Lente, Tom Greene, et al.  A new equation to estimate glomerular filtration rate.  _Annals of internal medicine_ , 150(9):604–612, 2009.
  * Lim and Hastie [2015] Lim, Michael and Trevor Hastie.  Learning interactions via hierarchical group-lasso regularization.  _Journal of Computational and Graphical Statistics_ , 24(3):627–654, 2015.
  * Lipton et al. [2018] Lipton, Zachary, Julian McAuley, and Alexandra Chouldechova.  Does mitigating ml’s impact disparity require treatment disparity?  In _Advances in Neural Information Processing Systems 31_ , pages 8135–8145, 2018.
  * Martinez et al. [2019] Martinez, Natalia, Martin Bertran, and Guillermo Sapiro.  Fairness with minimal harm: A pareto-optimal approach for healthcare.  _arXiv preprint arXiv:1911.06935_ , 2019.
  * Martinez et al. [2020] Martinez, Natalia, Martin Bertran, and Guillermo Sapiro.  Minimax pareto fairness: A multi objective perspective.  In _International Conference on Machine Learning_ , pages 6755–6764. PMLR, 2020.
  * Metaxa et al. [2021] Metaxa, Danaë, Joon Sung Park, Ronald E Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, Christian Sandvig, et al.  Auditing algorithms: Understanding algorithmic systems from the outside in.  _Foundations and Trends® in Human–Computer Interaction_ , 14(4):272–344, 2021.
  * Mitchell et al. [2019] Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 220–229, 2019.
  * Moore et al. [2014] Moore, Christopher L, Scott Bomann, Brock Daniels, Seth Luty, Annette Molinaro, Dinesh Singh, and Cary P Gross.  Derivation and validation of a clinical prediction rule for uncomplicated ureteral stone—the stone score: retrospective and prospective observational cohort studies.  _Bmj_ , 348, 2014.
  * Naeini et al. [2015] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015\.
  * Narasimhan [2018] Narasimhan, Harikrishna.  Learning with complex loss functions and constraints.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1646–1654, 2018.
  * Nasief et al. [2019] Nasief, Haidy, Cheng Zheng, Diane Schott, William Hall, Susan Tsai, Beth Erickson, and X Allen Li.  A machine learning based delta-radiomics process for early prediction of treatment response of pancreatic cancer.  _NPJ precision oncology_ , 3(1):25, 2019.
  * Netanel [2011] Netanel, Neil Weinstock.  Making sense of fair use.  _Lewis & Clark L. Rev._, 15:715, 2011.
  * Nieves et al. [2005] Nieves, Jeri W, Carmelo Formica, Jamie Ruffing, Marsha Zion, Patricia Garrett, Robert Lindsay, and Felicia Cosman.  Males have larger skeletal size and bone mass than females, despite comparable body size.  _Journal of Bone and Mineral Research_ , 20(3):529–535, 2005.
  * Paes et al. [2022] Paes, Lucas Monteiro, Carol Xuan Long, Berk Ustun, and Flavio Calmon.  On the epistemic limits of personalized prediction.  In Oh, Alice H., Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=Snp3iEj7NJ.
  * Panahiazar et al. [2015] Panahiazar, Maryam, Vahid Taslimitehrani, Naveen Pereira, and Jyotishman Pathak.  Using ehrs and machine learning for heart failure survival analysis.  _Studies in health technology and informatics_ , 216:40, 2015.
  * Paulus et al. [2016] Paulus, Jessica K, Benjamin S Wessler, Christine Lundquist, Lana LY Lai, Gowri Raman, Jennifer S Lutz, and David M Kent.  Field synopsis of sex in clinical prediction models for cardiovascular disease.  _Circulation: Cardiovascular Quality and Outcomes_ , 9(2_suppl_1):S8–S15, 2016.
  * Perez-Rodriguez and de la Fuente [2017] Perez-Rodriguez, Javier and Alejandro Fuentede la .  Now is the time for a postracial medicine: Biomedical research, the national institutes of health, and the perpetuation of scientific racism.  _The American Journal of Bioethics_ , 17(9):36–47, 2017.
  * Pfohl et al. [2019] Pfohl, Stephen, Ben Marafino, Adrien Coulet, Fatima Rodriguez, Latha Palaniappan, and Nigam H Shah.  Creating fair models of atherosclerotic cardiovascular disease risk.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 271–278, 2019.
  * Platt et al. [1999] Platt, John et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3):61–74, 1999.
  * Pollard et al. [2018] Pollard, Tom J, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi.  The eicu collaborative research database, a freely available multi-center database for critical care research.  _Scientific data_ , 5(1):1–13, 2018.
  * Quiñonero-Candela et al. [2008] Quiñonero-Candela, Joaquin, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  _Dataset shift in machine learning_.  Mit Press, 2008.
  * Savage [1971] Savage, Leonard J.  Elicitation of personal probabilities and expectations.  _Journal of the American Statistical Association_ , 66(336):783–801, 1971.
  * Steyerberg et al. [2010] Steyerberg, Ewout W, Andrew J Vickers, Nancy R Cook, Thomas Gerds, Mithat Gonen, Nancy Obuchowski, Michael J Pencina, and Michael W Kattan.  Assessing the performance of prediction models: a framework for some traditional and novel measures.  _Epidemiology (Cambridge, Mass.)_ , 21(1):128, 2010.
  * Steyerberg et al. [2019] Steyerberg, Ewout W et al.  _Clinical prediction models_.  Springer, 2019.
  * Suresh et al. [2017] Suresh, Harini, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi.  Clinical intervention prediction and understanding with deep neural networks.  In _Machine Learning for Healthcare Conference_ , pages 322–337. PMLR, 2017.
  * Taylor et al. [2017] Taylor, Sara, Natasha Jaques, Ehimwenma Nosakhare, Akane Sano, and Rosalind Picard.  Personalized multitask learning for predicting tomorrow’s mood, stress, and health.  _IEEE Transactions on Affective Computing_ , 11(2):200–213, 2017.
  * Ustun et al. [2016] Ustun, Berk, M Brandon Westover, Cynthia Rudin, and Matt T Bianchi.  Clinical prediction models for sleep apnea: the importance of medical history over symptoms.  _Journal of Clinical Sleep Medicine_ , 12(02):161–168, 2016.
  * Ustun et al. [2017] Ustun, Berk, Lenard A Adler, Cynthia Rudin, Stephen V Faraone, Thomas J Spencer, Patricia Berglund, Michael J Gruber, and Ronald C Kessler.  The world health organization adult attention-deficit/hyperactivity disorder self-report screening scale for dsm-5.  _Jama psychiatry_ , 74(5):520–527, 2017.
  * Ustun et al. [2019] Ustun, Berk, Yang Liu, and David Parkes.  Fairness without harm: Decoupled classifiers with preference guarantees.  In _International Conference on Machine Learning_ , pages 6373–6382, 2019.
  * van den Goorbergh et al. [2022] Goorbergh, Rubenvan den , Maarten Smedenvan , Dirk Timmerman, and Ben Van Calster.  The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression.  _Journal of the American Medical Informatics Association_ , 29(9):1525–1534, 06 2022.  ISSN 1527-974X.  doi: 10.1093/jamia/ocac093.  URL https://doi.org/10.1093/jamia/ocac093.
  * Vaughan et al. [2020] Vaughan, Gregory, Robert Aseltine, Kun Chen, and Jun Yan.  Efficient interaction selection for clustered data via stagewise generalized estimating equations.  _Statistics in Medicine_ , 39(22):2855–2868, 2020\.
  * Viviano and Bradic [2020] Viviano, Davide and Jelena Bradic.  Fair policy targeting.  _arXiv preprint arXiv:2005.12395_ , 2020.
  * Vyas et al. [2020] Vyas, Darshali A, Leo G Eisenstein, and David S Jones.  Hidden in plain sight—reconsidering the use of race correction in clinical algorithms, 2020.
  * Wallace et al. [2011] Wallace, Emma, Susan M Smith, Rafael Perera-Salazar, Paul Vaucher, Colin McCowan, Gary Collins, Jan Verbakel, Monica Lakhanpaul, and Tom Fahey.  Framework for the impact analysis and implementation of clinical prediction rules (cprs).  _BMC medical informatics and decision making_ , 11(1):1–7, 2011.
  * Wang et al. [2022] Wang, Angelina, Vikram V Ramaswamy, and Olga Russakovsky.  Towards intersectionality in machine learning: Including more identities, handling underrepresentation, and performing evaluation.  _arXiv preprint arXiv:2205.04610_ , 2022.
  * Wang et al. [2019] Wang, Hao, Berk Ustun, and Flavio P Calmon.  Repairing without retraining: Avoiding disparate impact with counterfactual distributions.  In _Proceedings of the 36th International Conference on Machine Learning_ , Proceedings of Machine Learning Research. PMLR, 2019.
  * Wang et al. [2021] Wang, Hao, Hsiang Hsu, Mario Diaz, and Flavio P Calmon.  To split or not to split: The impact of disparate treatment in classification.  _IEEE Transactions on Information Theory_ , 67(10):6733–6757, 2021.
  * Yala et al. [2019] Yala, Adam, Constance Lehman, Tal Schuster, Tally Portnoi, and Regina Barzilay.  A deep learning mammography-based model for improved breast cancer risk prediction.  _Radiology_ , 292(1):60–66, 2019.
  * Yankwich [1954] Yankwich, Leon R.  What is fair use?  _The University of Chicago Law Review_ , 22(1):203–215, 1954.
  * Zafar et al. [2017a] Zafar, Muhammad Bilal, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.  Fairness beyond disparate treatment and disparate impact: Learning classification without disparate mistreatment.  In _Proceedings of the 26th International Conference on World Wide Web_ , pages 1171–1180. International World Wide Web Conferences Steering Committee, 2017a.
  * Zafar et al. [2017b] Zafar, Muhammad Bilal, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller.  From parity to preference-based notions of fairness in classification.  In _Advances in Neural Information Processing Systems_ , pages 228–238, 2017b.
  * Zafar et al. [2017c] Zafar, Muhammad Bilal, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P. Gummadi.  Fairness Constraints: Mechanisms for Fair Classification.  In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , volume 54 of _Proceedings of Machine Learning Research_ , pages 962–970. PMLR, 20–22 Apr 2017c.
  * Zhan et al. [2020] Zhan, Qiansheng, ET Sierra, Jonas Malmsten, Zhen Ye, Zev Rosenwaks, and Nikica Zaninovic.  Blastocyst score, a blastocyst quality ranking tool, is a predictor of blastocyst ploidy and implantation potential.  _F &S Reports_, 1(2):133–141, 2020.
  * Zhu et al. [2016] Zhu, Xinliang, Jiawen Yao, and Junzhou Huang.  Deep convolutional neural network for survival analysis with pathological images.  In _2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 544–547. IEEE, 2016.

##  Appendix A Notation

Below we provide a table that consolidates and describes the notation used
throughout the paper.

Symbol |  Meaning
---|---
$\bm{x}_{i}=(x_{i,1},x_{i,2},\ldots,x_{i,d})$ |  feature vector of example $i$
$y_{i}\in\mathcal{Y}$ |  label of example $i$
$\bm{g}_{i}\in\\{g_{i,1},g_{i,2},\ldots,g_{i,k}\\}$ |  group membership of example $i$
$\mathcal{G}=\mathcal{G}_{1}\times\mathcal{G}_{2}\times\ldots\times\mathcal{G}_{k}$ |  space of group attributes
$m=|\mathcal{G}|$ |  number of intersectional groups
$n_{\bm{g}}:=\sum 1[\bm{g}_{i}=\bm{g}]$ |  number of examples of group $\bm{g}\in\mathcal{G}$
$n^{+{}}_{\bm{g}}:=\sum 1[\bm{g}_{i}=\bm{g}{},\,y_{i}=+1]$ |  number of examples of group $\bm{g}\in\mathcal{G}$ with $y_{i}=+1$
$n^{-{}}_{\bm{g}}:=\sum 1[\bm{g}_{i}=\bm{g}{},\,y_{i}=-1]$ |  number of examples of group $\bm{g}\in\mathcal{G}$ with $y_{i}=-1$
$\mathcal{H}_{0}$ |  hypothesis class of generic model
$\mathcal{H}$ |  hypothesis class of personalized models
$h_{0}\in:\mathcal{X}\to\mathcal{Y}$ |  generic model
$h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ |  personalized model
$h_{\bm{g}}:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ |  personalized classifier where group membership is reported truthfully (as $\bm{g}{})$
$R_{\bm{g}}(h_{\bm{g}^{{}^{\prime}}})$ |  true risk of model $h$ of group $\bm{g}$ if they report $\bm{g}^{{}^{\prime}}$
$\hat{R}_{\bm{g}}(h_{\bm{g}^{{}^{\prime}}})$ |  empirical risk of model $h$ of group $\bm{g}$ if they report $\bm{g}^{{}^{\prime}}$
${\Delta_{\bm{g}}}({h},{h^{\prime}})$ |  gain (i.e., reduction in true risk) for group $\bm{g}$ when using $h$ rather than $h^{\prime}$
${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{0}})$ |  rationality gap for group $\bm{g}$ (performance gain when reporting $\bm{g}$ as opposed to concealing it)
${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{\bm{g}^{\prime}}})$ |  rationality gap for group $\bm{g}$ (performance gain when reporting $\bm{g}$ as opposed to concealing it)

Table 3: Notation

##  Appendix B Theoretical Guarantees: Proofs

We present proofs of Proposition 4 and 5. Proposition 4 follows from our
definitions. For Proposition 5 we first present a generalization bound for
gains generally. Then we use this result to prove the proposition for
rationality and envy-freeness.

Proof [Proposition 4]

Given the conditions in Proposition 4 we have that for all groups
$\bm{g}\in\mathcal{G}$,
${\hat{R}_{\bm{g}}(h_{\bm{g}}^{*})}\leq{\hat{R}_{\bm{g}}(h_{)}},\forall
h_{\in}\mathcal{H}$. From this fact, we prove that rationality is satisfied
for all groups $\bm{g}\in\mathcal{G}$ because
${\hat{R}_{\bm{g}}(h_{\bm{g}}^{*})}\leq{\hat{R}_{\bm{g}}(h_{0})}$ since
$h_{0}\in\mathcal{H}$. Similarly, we prove that envy-freeness is satisfied for
all groups $\bm{g}\in\mathcal{G}$ because
${\hat{R}_{\bm{g}}(h_{\bm{g}}^{*})}\leq{\hat{R}_{\bm{g}}(h_{\bm{g}^{{}^{\prime}}})}$
since $h_{\bm{g}^{{}^{\prime}}}\in\mathcal{H}$. Therefore both envy-freeness
and rationality are satisfied on the training data if the initial conditions
are satisfied.

######  Lemma 6 (Generalization of Gains).

Given two classifiers $h_{a}$ & $h_{b}$ over group $\bm{g}$ which satisfy
${\hat{\Delta}_{\bm{g}}}({h_{a}},{h_{b}})>0$ then with probability at least
$1-\delta$, ${\Delta_{\bm{g}}}({h_{a}},{h_{b}})>0$ if:

$\displaystyle\sqrt{\frac{4VC(\mathcal{H})\left(\ln{\frac{2n_{\bm{g}}}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8}{\delta})}}{n_{\bm{g}}}}\leq{\hat{\Delta}_{\bm{g}}}({h_{a}},{h_{b}})$
(4)

.

where $VC(\mathcal{H})$ is the VC-dimension of class $\mathcal{H}$.

Proof

For any group $\bm{g}\in\mathcal{G}$ and any classifier $h_{a}\in\mathcal{H}$,
with probability at least $1-\delta/2$, we have that

$\displaystyle\bigl{|}\hat{R}_{\bm{g}}(h_{a})-R_{\bm{g}}(h_{a})\bigr{|}\leq\sqrt{\frac{VC(\mathcal{H})\left(\ln{\frac{2n}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8}{\delta})}}{n}}.$
(5)

where $VC(\mathcal{H})$ is the VC-dimension of class $\mathcal{H}$.

The bound in (5) holds for the classifiers $h_{a}$ and $h_{b}$ with
probability at least $1-\delta$. Thus, we know that:

$\displaystyle R_{\bm{g}}(h_{b})-R_{\bm{g}}(h_{a})=$
$\displaystyle(R_{\bm{g}}(h_{b})-\hat{R}_{\bm{g}}(h_{b}))+(R_{\bm{g}}(h_{a}))-\hat{R}_{\bm{g}}(h_{a}))+{\hat{R}_{\bm{g}}(h^{\prime})}-{\hat{R}_{\bm{g}}(h)}$
$\displaystyle\geq-\left(\sqrt{\frac{4VC(\mathcal{H})\left(\ln{\frac{2n}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8}{\delta})}}{n}}\right)+\hat{\Delta}_{\bm{g}}({h_{a}},{h_{b}})$
$\displaystyle\geq 0$

if the condition specified in (4) holds.

We are now ready to prove Proposition 5 which will follow from the application
of Lemma 6.

Proof [Proof of Proposition 5]

First, given that $\hat{\epsilon}_{\bm{g}}>0$, we can apply Lemma 6 to the
personalized classifier $h_{\bm{g}}$ and the generic classifier $h_{0}$. We
prove the sample complexity of generalizing rationality gains by directly
applying this lemma and rearranging for $n_{g}$.

$\displaystyle\sqrt{\frac{4VC(\mathcal{H})\left(\ln{\frac{2n_{\bm{g}}}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8}{\delta})}}{n_{\bm{g}}}}\leq\
\hat{\epsilon}_{\bm{g}}$ (6) $\displaystyle
n_{\bm{g}}\geq\frac{4VC(\mathcal{H})\left(\ln{\frac{2n_{\bm{g}}}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8}{\delta})}}{{\hat{\epsilon}_{\bm{g}}}^{2}}$
(7)

First, given that $\hat{\gamma}_{\bm{g}}>0$, we can apply Lemma 6 to the two
personalized classifier $h_{\bm{g}}$ and $h_{\bm{g}^{{}^{\prime}}}$ for all
$\bm{g},\bm{g}^{{}^{\prime}}\in\mathcal{G}$. This results in $m-1$ preferences
to generalize. Given that $m-1\leq m$, we can apply Lemma 6 with probability
$1-\frac{\delta}{m}$. Doing so and rearranging for $n_{\bm{g}}$ proves the
result.

$\displaystyle\sqrt{\frac{4VC(\mathcal{H})\left(\ln{\frac{2n_{\bm{g}}}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8m}{\delta})}}{n_{\bm{g}}}}\leq\
\hat{\gamma}_{\bm{g}}$ (8) $\displaystyle
n_{\bm{g}}\geq\frac{4VC(\mathcal{H})\left(\ln{\frac{2n_{\bm{g}}}{VC(\mathcal{H})}}+1\right)+\ln{(\frac{8m}{\delta})}}{{\hat{\gamma}_{\bm{g}}}^{2}}$
(9)

Together these two results make Proposition 5.

##  Appendix C Failure Modes of Personalization

In this Appendix, we describe additional mechanisms that lead personalized
models to exhibit fair use violations. The mechanisms below reflect failure
modes that arise in later stages of the machine learning pipeline, and that
are more difficult to address through interventions.

###  C.1 ERM with a Surrogate Loss Function

Consider a setting where we want a personalized model that maximizes
classification accuracy – i.e., one that minimizes the 0–1 loss. If we fit
this classifier using a linear SVM – e.g., by solving an ERM problem that
optimizes the hinge loss – the approximation error between the 0-1 loss and
the hinge loss can produce a fair use violation (see Figure 3). This example
is specifically designed to avoid fair use violations that stem from model
misspecification.

|  Data for Group $A$
---
|  Data for Group $B$
---
|

Figure 3: Fair use violations resulting from the use of surrogate loss
function in ERM. Here, we are given data for classification task with features
$\bm{x}=(x_{1},x_{2})$ and a group attribute $\bm{g}=\\{A,B\\}$. We fit a
linear SVM $h_{\bm{g}}$ by optimizing the hinge loss for a prediction task
where evaluate the gains of personalization in terms of the error rate (i.e.,
0-1 loss). In this case, the personalized model produces a fair use violation
for Group $B$ due to an outlier $\bm{x}_{O}$. We plot the data for group $A$
and group $B$ separately. Each plot shows the generic classifier ($h_{0}$;
grey) and the personalized classifiers for the corresponding group ($h_{A}$ or
$h_{B}$; black). As a baseline for comparison, we show the personalized models
that we would obtain by optimizing an exact loss function (i.e., 0-1 loss,
which matches the performance metric that we use to evaluate the gains for
personalization). As shown, we would expect to avoid this violation had we fit
a model by optimizing the 0–1 loss directly.

###  C.2 Generalization & Dataset Shifts

Fair use violations can arise in deployment. Small samples may significantly
distort the relative prevalence of each group, leading standard empirical risk
minimization to fit a suboptimal generic model or personalized model (see
Figure 4). Fair use violations can also arise as a result of changes in the
data distribution [i.e., dataset shift 84, 39, 46] (see Figure 5)

| Group
---
| Training Data
---
| Data Distribution
---
| Model Predictions
---
| Observed Performance
---
| True Performance
---
$\bm{g}_{1}$ | $\bm{g}_{2}$ | $n^{+{}}$ | $n^{-{}}$ | $n^{+{}}$ | $n^{-{}}$ | $h_{0}(\bm{x})$ | $h_{\bm{g}}(\bm{x},\bm{g})$ | $R_{\bm{g}}(h_{0})$ | $R_{\bm{g}}(h_{\bm{g}})$ | ${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{0}})$ | $R_{\bm{g}}(h_{0})$ | $R_{\bm{g}}(h_{\bm{g}})$ | ${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{0}})$
$0$ | $0$ | $65$ | $60$ | $130$ | $120$ | $+$ | $+$ | $60$ | $60$ | $0$ | $120$ | $120$ | $0$
$1$ | $0$ | $60$ | $65$ | $120$ | $130$ | $+$ | $-$ | $65$ | $60$ | $5$ | $130$ | $120$ | $10$
$0$ | $1$ | $60$ | $65$ | $130$ | $120$ | $+$ | $-$ | $65$ | $60$ | $5$ | $120$ | $130$ | $-10$
$1$ | $1$ | $70$ | $55$ | $140$ | $110$ | $+$ | $+$ | $55$ | $55$ | $0$ | $110$ | $110$ | $0$
| Total | $255$ | $245$ | $520$ | $480$ | $-$ | N/A | $245$ | $235$ | $10$ | $480$ | $470$ | $0$

Figure 4: Fair use violations can arise when personalizing models on small
samples. Here, we show a 2D classification task in which a personalized model
only exhibits fair use violations in deployment. Here, group $(1,0)$
experiences an gain once the model is deployment. In contrast, group $(0,1)$
experiences a fair use violation as a result of sampling error.

| Group
---
| Training Data
---
| True Distribution
---
| Model Predictions
---
| Train Performance
---
| True Performance
---
$\bm{g}_{1}$ | $\bm{g}_{2}$ | $n^{+{}}$ | $n^{-{}}$ | $n^{+{}}$ | $n^{-{}}$ | $h_{0}(\bm{x})$ | $h_{\bm{g}}(\bm{x},\bm{g})$ | $R_{\bm{g}}(h_{0})$ | $R_{\bm{g}}(h_{\bm{g}})$ | ${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{0}})$ | $R_{\bm{g}}(h_{0})$ | $R_{\bm{g}}(h_{\bm{g}})$ | ${\Delta_{\bm{g}}}({h_{\bm{g}}},{h_{0}})$
$0$ | $0$ | $20$ | $0$ | $20$ | $0$ | $+$ | $+$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$
$1$ | $0$ | $5$ | $25$ | $5$ | $25$ | $+$ | $-$ | $25$ | $5$ | $20$ | $25$ | $5$ | $20$
$0$ | $1$ | $5$ | $25$ | $30$ | $25$ | $+$ | $-$ | $25$ | $5$ | $20$ | $20$ | $30$ | $-10$
$1$ | $1$ | $20$ | $0$ | $20$ | $0$ | $+$ | $+$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$
| Total | $50$ | $50$ | $75$ | $45$ | $+$ | N/A | $50$ | $10$ | $40$ | $45$ | $35$ | $10$

Figure 5: Label shift produces a fair use violation. Here, we train a linear
classifier on a dataset with [one binary feature and one binary group
attribute]. As shown, personalization leads to overall improvement reducing
aggregate reduce from 50 to 24 and group-specific improvement on the training
data. However, not all groups perform equally well in deployment. While groups
$(0,1)$ and $(1,1)$ see improvements, a violation (red) occurs for group
$(1,0)$ due to the label shift where positive examples are no longer present
meanwhile they were the majority in the training data.

##  Appendix D Details on Datasets used in Section 4 & Section 5

In this Appendix, we provide additional information on the datasets used in
Section 4 and Section 5

Dataset | $n$ | $d$ | $\mathcal{G}$ | Prediction Task | Reference
---|---|---|---|---|---
apnea | $1,152$ | $26$ |  |  $\texttt{\small{Age}}\times\texttt{\small{Sex}}=\\{$<30$,30~{}\texttt{\small{to}}~{}60,60+\\}\times\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}$
---
patient has obstructive sleep apnea | Ustun et al. [90]
cardio_eicu | $1,341$ | $49$ |  | $\texttt{\small{Age}}\times\texttt{\small{Sex}}=\\{\texttt{\small{Young}},\texttt{\small{Old}}\\}\times\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}$
---
patient with cardiogenic shock dies | Pollard et al. [83]
cardio_mimic | $5,289$ | $49$ |  | $\texttt{\small{Age}}\times\texttt{\small{Sex}}=\\{\texttt{\small{Young}},\texttt{\small{Old}}\\}\times\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}$
---
patient with cardiogenic shock dies | Johnson et al. [54]
heart | $181$ | $26$ |  | $\texttt{\small{Age}}\times\texttt{\small{Sex}}=\\{\texttt{\small{Young}},\texttt{\small{Old}}\\}\times\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}$
---
patient has heart disease | Detrano et al. [27]
kidney | $2066$ | $78$ |  | $\texttt{\small{Sex}}\times\texttt{\small{Race}}=\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}\times\\{\texttt{\small{White, Black, Other}}\\}$
---
mortality of patient on CRRT | Johnson et al. [54]
mortality | $21,139$ | $484$ |  | $\texttt{\small{Age}}\times\texttt{\small{Sex}}=\\{<30,30~{}\texttt{\small{to}}~{}60,60+\\}\times\\{\texttt{\small{Male}},\texttt{\small{Female}}\\}$
---
mortality of patient in ICU | Harutyunyan et al. [48]
saps | $7,797$ | $36$ |  | $\texttt{\small{Age}}\times\texttt{\small{HIV}}=\\{\leq 30,30+\\}\times\\{\texttt{\small{Positive}},\texttt{\small{Negative}}\\}$
---
mortality of patient in ICU | Le Gall et al. [63]

Table 4: Overview of classification datasets used to train clinical prediction
models in Sections 4 and 5. We describe the conditions that lead to $y_{i}$
under each prediction task. All datasets used are publicly available, have
been deidentified, and inspected to ensure that they contain no offensive
content. In cases where data access requires consent or approval from the data
holders, we have followed the proper procedure to obtain such consent.
Datasets based on MIMIC-III [54] (kidney, mortality) and eICU [83] (cardio)
are hosted on PhysioNet under the PhysioNet Credentialed Health Data License.
The heart dataset is hosted on the UCI ML Repository under an Open Data
license. The apnea and saps datasets must be requested from the authors of the
papers listed above [63, 90]. We minimally process each dataset to impute the
values of missing points (using mean value imputation), and repair class
imbalances across intersectional groups (to eliminate “trivial" fair use
violations that occur due to class imbalance).

#### apnea

We use the obstructive sleep apnea (OSA) dataset outlined in Ustun et al.
[90]. In this dataset, we have a cohort of 1152 patients where 23% have OSA.
We use all available features (e.g. BMI, comobordities, age, and sex) and
binarize them, resulting in 26 binary features.

####  cardio_eicu & cardio_mimic

Cardiogenic shock is a serious acute condition where the heart cannot provide
sufficient blood to the vital organs. Using the eICU Collaborative Research
Database V2.0 [83] and MIMIC-III database [54], we create a cohort of patients
who have cardiogenic shock during the course of their intensive care unit
(ICU) stay. We use an exhaustive set of clinical criteria based on the
patient’s labs and vitals (i.e. presence of hypotension and organ
hypoperfusion). The goal is to predict whether a patient with cardiogenic
shock will die in hospital. As features, we summarize (minimums and maximums)
relevant labs and vitals (e.g. systolic BP, heart rate, hemoglobin count) of
each patient from the period of time prior to the onset of cardiogenic shock
up to 24 hours. This results in a dataset containing 8,815 patients, 13.5% of
whom die in hospital.

#### heart

We use the Heart dataset from the UCI Machine Learning Repository, where the
goal is to predict the presence of heart disease from clinical features. It
consists of 303 patients, 54.5% of which have heart disease. We use all
available features, treating cp, thal, ca, slope and restecg as categorical,
and all remaining features as continuous.

#### kidney

Using MIMIC-III and MIMIC-IV [54], we create a cohort of patients who were
given Continuous Renal Replacement Therapy (CRRT) at any point during their
ICU stay. For patients with multiple ICU stays, we select their first one. We
define the target as whether the patient dies during the course of their
selected hospital admission. As features, we select the most recent instances
of relevant lab measurements (e.g. sodium, potassium, creatinine) prior to the
CRRT start time, along with the patient’s age, the number of hours they have
been in ICU when CRRT was administered, and their Sequential Organ Failure
Assessment (SOFA) score at admission. We treat all variables as continuous
with the exception of the SOFA score, which we treat as ordinal. This results
in a dataset of 1,722 CRRT patients, 51.1% of which die in-hospital. We define
protected groups based on the patient’s sex and self-reported race and
ethnicity.

#### mortality

We follow the cohort creation steps outlined by Harutyunyan et al. [48] for
their in-hospital mortality prediction task. We select the first ICU stay
longer than 48 hours of patients in MIMIC-III [54], and aim to predict whether
they will die in-hospital during their corresponding hospital admission. As
features, we bin the time-series lab and vital measurements provided by
Harutyunyan et al. [48] into four 12-hour time-bins, and compute the mean in
each time-bin. We additionally include the patient’s age and sex as features.
This results in a cohort of 21,139 patients, 13.2% of whom die in hospital.

#### saps

The Simplified Acute Physiology Score II (SAPS II) is a risk score that was
developed for predicting mortality in the ICU [63]. This study was conducted
in 137 medical centers across 12 countries contains 7,797 patients. For each
patient we have access to demographics, comorbidities, and vitals which are
used to predict the risk of mortality in the ICU. For group attributes we use
age and HIV status. The percentage of patients in the dataset who experience
mortality is 21.8%.

##  Appendix E Experimental Results for Neural Nets & Random Forests

In this Appendix, we provide additional results showing the gains of
personalization when training personalized neural nets and random forests. We
present tables that summarize the gains of personalization for neural networks
and random forests. The following tables are analogous to Table 1, except that
they also include results for the kidney dataset in Section 5.

###  E.1 Neural Nets

We trained neural networks with two hidden layers of size 5 and 2 and learning
rate of $1^{-3}$. To ensure that they were calibrated, we applied Platt
scaling [82] the outputs of the neural network model. As in Section 4.2 and
Section 5, we can identify statistically significant fair use violations and
gains as noted by the gains and violations.

|  | Test AUC | Test ECE | Test Error
---|---|---|---|---
Dataset | Metrics |  | 1Hot | All | DCP | 1Hot | All | DCP | 1Hot | All | DCP
apnea |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.705
---
-0.012
0.114 / -0.051
3/3
1/0
| 0.524
---
-0.193
0.029 / -0.501
5/5
0/0
| 0.622
---
-0.095
-0.068 / -0.328
6/6
2/1
| 6.3%
---
-0.7%
10.9% / -8.2%
3/3
2/4
| 2.5%
---
3.2%
24.0% / 1.5%
5/5
5/5
| 5.3%
---
0.4%
9.8% / -5.7%
3/3
5/5
| 36.7%
---
-3.3%
8.4% / -5.0%
1/2
6/6
| 50.6%
---
-17.2%
7.1% / -43.5%
1/1
6/6
| 41.5%
---
-8.1%
-2.2% / -50.5%
0/0
6/6
cardio_eicu |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.739
---
0.001
0.067 / -0.003
0/0
0/0
| 0.738
---
-0.001
0.029 / -0.012
1/1
0/0
| 0.687
---
-0.051
0.007 / -0.090
3/3
2/2
| 4.5%
---
2.3%
2.6% / -1.2%
3/3
1/1
| 5.5%
---
1.4%
2.4% / -1.9%
3/3
2/2
| 5.4%
---
1.5%
4.9% / -3.0%
2/2
2/2
| 31.5%
---
1.6%
8.4% / -0.5%
2/3
4/4
| 31.8%
---
1.3%
5.5% / -1.3%
2/3
4/4
| 36.6%
---
-3.5%
0.1% / -10.2%
0/0
4/4
cardio_mimic |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.849
---
0.004
0.018 / -0.005
1/1
0/0
| 0.849
---
0.004
0.012 / -0.000
0/0
1/1
| 0.836
---
-0.009
0.004 / -0.014
2/2
4/4
| 3.1%
---
1.1%
2.1% / -0.4%
2/2
4/4
| 4.7%
---
-0.4%
1.4% / -2.3%
2/2
3/3
| 3.3%
---
1.0%
2.5% / -0.3%
2/2
3/3
| 23.7%
---
0.6%
2.0% / -1.1%
3/3
4/4
| 24.0%
---
0.2%
2.3% / -2.4%
2/2
4/4
| 23.9%
---
0.4%
1.3% / -1.4%
2/2
4/4
heart |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.457
---
-0.090
0.061 / -0.392
2/2
2/1
| 0.736
---
0.189
0.317 / 0.023
0/0
1/0
| 0.554
---
0.007
0.257 / -0.023
2/1
3/1
| 22.7%
---
-9.1%
4.8% / -29.8%
1/1
1/1
| 17.2%
---
-3.6%
9.8% / -9.7%
1/1
4/4
| 18.1%
---
-4.5%
6.2% / -14.8%
2/2
0/0
| 52.6%
---
-1.3%
1.6% / -9.2%
0/2
4/4
| 27.6%
---
23.7%
38.0% / 4.6%
4/4
4/4
| 38.2%
---
13.2%
28.1% / 7.1%
3/4
4/4
kidney |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.774
---
0.003
0.039 / -0.057
2/2
1/0
| 0.774
---
0.004
0.026 / -0.095
3/3
1/0
| 0.762
---
-0.009
0.033 / -0.152
4/4
4/4
| 6.0%
---
-0.1%
2.8% / -1.6%
2/2
3/5
| 6.2%
---
-0.4%
3.7% / -2.5%
2/2
3/3
| 7.3%
---
-1.4%
0.8% / -5.4%
1/1
2/2
| 29.2%
---
-2.1%
-0.6% / -7.5%
0/0
6/6
| 31.0%
---
-3.8%
4.7% / -5.4%
1/1
6/6
| 30.9%
---
-3.7%
-1.5% / -18.9%
0/0
6/6
mortality |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.870
---
-0.004
0.025 / -0.019
3/3
6/2
| 0.869
---
-0.004
-0.001 / -0.015
5/5
4/4
| 0.895
---
0.022
0.039 / 0.005
0/0
6/6
| 2.8%
---
0.6%
2.7% / -1.7%
3/3
1/5
| 4.3%
---
-0.9%
0.5% / -1.2%
2/2
4/4
| 3.0%
---
0.5%
8.0% / 0.1%
5/5
0/0
| 20.9%
---
-0.4%
5.1% / -2.2%
3/3
6/6
| 21.5%
---
-1.0%
-0.3% / -2.3%
0/0
6/6
| 17.7%
---
2.8%
12.6% / 0.1%
6/6
6/6
saps |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.157
---
-0.037
0.101 / -0.041
3/2
3/1
| 0.872
---
0.678
0.745 / 0.657
0/0
3/1
| 0.758
---
0.565
0.743 / -0.273
1/1
0/0
| 37.8%
---
7.5%
27.1% / 1.5%
4/4
3/3
| 7.8%
---
37.5%
43.2% / -3.5%
3/3
3/3
| 31.5%
---
13.9%
49.9% / 6.4%
4/4
3/3
| 63.4%
---
-1.8%
0.0% / -5.8%
0/1
4/4
| 21.7%
---
39.9%
53.9% / 1.4%
3/4
4/4
| 48.9%
---
12.7%
22.0% / 0.0%
3/4
4/4

Table 5: Performance of personalized neural network models on all datasets. We
show the gains of personalization in terms of test AUC, ECE, and error. We
report: model performance at the population level, the overall gain of
personalization, the range of gains over $m$ intersectional groups, and the
number of rationality and envy-freeness gains/violations (evaluated using a
bootstrap hypothesis test at a 10% significance level).

###  E.2 Random Forests

We trained random forests with the following hyperparameters: 100 estimators,
max depth of 20, minimum samples per split is 5, and minimum number of samples
in each leaf is 2. We expect these models to perform well in terms of error
rare but not necessarily in terms in terms of AUC or risk calibration. We
observe this effect in the Table below. For example, using an intersectional
encoding with random forests minimizing fair use violations in terms of error
rate as measured on mltiple datasets (e.g. apnea, kidney). As noted with other
model classes, we can find statistically significant violations.

|  | Test AUC | Test ECE | Test Error
---|---|---|---|---
Dataset | Metrics |  | 1Hot | All | 1Hot | All | 1Hot | All
apnea |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.757
---
0.002
0.064 / -0.001
1/1
4/0
| 0.759
---
0.004
0.020 / -0.009
2/2
4/2
| 7.7%
---
-0.4%
4.4% / -2.7%
2/2
4/5
| 7.1%
---
0.6%
6.8% / -0.3%
3/3
3/3
| 30.5%
---
0.8%
9.8% / -2.8%
5/5
6/6
| 31.2%
---
-0.5%
4.3% / -1.5%
2/4
6/6
cardio_eicu |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.764
---
0.001
0.009 / -0.022
1/1
1/1
| 0.772
---
-0.000
0.014 / -0.028
2/2
3/3
| 7.8%
---
-1.3%
3.7% / -0.8%
2/2
3/3
| 8.7%
---
-0.7%
1.0% / -3.5%
1/1
3/3
| 30.8%
---
1.0%
4.9% / -1.6%
3/3
4/4
| 30.2%
---
0.4%
1.4% / -1.8%
1/3
4/4
cardio_mimic |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.847
---
-0.003
-0.001 / -0.004
4/4
2/2
| 0.847
---
0.001
0.002 / -0.001
2/2
2/1
| 9.2%
---
0.4%
1.2% / 0.2%
4/4
1/1
| 9.7%
---
-0.4%
0.3% / -0.9%
1/1
3/3
| 24.0%
---
-0.2%
0.8% / -1.1%
1/1
4/4
| 24.3%
---
-0.3%
0.3% / -1.4%
1/1
4/4
heart |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.897
---
-0.006
0.006 / -0.025
3/1
4/0
| 0.896
---
-0.000
0.000 / -0.026
4/1
2/0
| 12.0%
---
4.6%
5.5% / -1.8%
2/2
2/2
| 14.4%
---
-2.7%
9.3% / -5.1%
1/1
2/2
| 17.1%
---
-2.6%
9.6% / -10.8%
1/2
4/4
| 22.4%
---
-2.6%
5.6% / -10.7%
0/2
4/4
kidney |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.775
---
0.001
0.010 / -0.017
3/2
3/0
| 0.778
---
0.003
0.009 / -0.019
3/3
3/2
| 8.8%
---
-0.7%
2.0% / -1.9%
1/1
4/6
| 9.0%
---
-1.1%
1.2% / -3.8%
1/1
3/3
| 29.2%
---
1.2%
5.3% / -0.9%
4/5
6/6
| 29.1%
---
1.0%
2.0% / -3.1%
2/5
6/6
mortality |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.806
---
0.002
0.005 / 0.001
0/0
6/0
| 0.806
---
-0.001
0.012 / -0.004
2/2
3/1
| 10.9%
---
-0.6%
1.4% / -1.2%
1/1
3/5
| 10.8%
---
-0.0%
1.5% / -3.1%
2/2
2/6
| 27.2%
---
0.3%
0.9% / -0.5%
3/3
6/6
| 27.1%
---
-0.1%
0.7% / -1.8%
3/4
6/6
saps |  | Personalized
---
Gain
Best/Worst Gain
Rat. Gains/Viols
EF Gains/Viols
|  | 0.879
---
-0.002
0.000 / -0.050
3/2
3/1
| 0.878
---
-0.002
0.050 / -0.002
2/1
4/2
| 4.6%
---
0.0%
11.2% / -1.6%
2/2
4/4
| 4.9%
---
0.1%
0.2% / -3.5%
2/2
4/4
| 20.1%
---
-0.4%
0.0% / -10.0%
0/1
4/4
| 20.0%
---
-0.4%
0.2% / -5.4%
0/2
4/4

Table 6: Performance of personalized random forest models on all datasets. We
show the gains of personalization in terms of test AUC, ECE, and error. We
report: model performance at the population level, the overall gain of
personalization, the range of gains over $m$ intersectional groups, and the
number of rationality and envy-freeness gains/violations (evaluated using a
bootstrap hypothesis test at a 10% significance level).