prunednode/.style=text=gray, draw, gray, fill=white, font=, dashed, outer
sep=.3em, inner sep=0.8em, genericnode/.style=text=black, draw,
fill=lightgray, font=, inner sep=0.8em, outer sep=0.2em, personnode/.style n
args=2text=black, draw, fill=cyan!#1, font=, outer sep=1.2em, inner sep=0.8em,
groupattr/.style=text=red,font=, s sep=2mm, l sep=15mm,
leaflabel/.style=edge=white, edge label=node[text=black,
above,font=,fill=white,inner sep=2pt], leaflabelpruned/.style=edge=white, edge
label=node[text=gray, above,font=,fill=white,inner sep=2pt]#1,
grouplabel/.style=edge label=node[above,font=,fill=white,inner sep=2pt]#1,
grouplabelpruned/.style=edge=dashed, gray, edge
label=node[above,font=,fill=white,inner sep=2pt, text=gray]#1

# Participatory Personalization in Classification

Hailey Joren  UC San Diego  Chirag Nagpal  Google  Katherine Heller  Google
Berk Ustun  UC San Diego

###### Abstract

Machine learning models are often personalized with information that is
protected, sensitive, self-reported, or costly to acquire. These models use
information about people but do not facilitate nor inform their _consent_.
Individuals cannot opt out of reporting personal information to a model, nor
tell if they benefit from personalization in the first place. We introduce a
family of classification models, called _participatory systems_ , that let
individuals opt into personalization at prediction time. We present a model-
agnostic algorithm to learn participatory systems for personalization with
categorical group attributes. We conduct a comprehensive empirical study of
participatory systems in clinical prediction tasks, benchmarking them with
common approaches for personalization and imputation. Our results demonstrate
that participatory systems can facilitate and inform consent while improving
performance and data use across all groups who report personal data.

###  1 Introduction

Machine learning models routinely assign predictions to _people_ – be it to
screen a patient for a mental illness [35], their risk of mortality in an ICU
[46], or their likelihood of responding to treatment [1]. Many models in such
applications are designed to target heterogeneous subpopulations using
features that explicitly encode personal information. Typically, models are
_personalized_ with categorical attributes that define groups [i.e.,
“categorization” as per 27]. In medicine, for example, clinical prediction
models use _group attributes_ that are _protected_ (e.g., sex in the CHA2DS2
Score for Stroke Risk), _sensitive_ (e.g., HIV status in the VA COVID-19
Mortality Score), _self-reported_ (e.g., alcohol_use in the HAS-BLED Score for
Major Bleeding Risk), or _costly_ to acquire (e.g., leukocytosis in the
Alvarado Appendicitis Score).

---
| Group
---
| Data
---
| Personalized
---
| Generic
---
| $\bm{g}{}$
---
$n^{+{}}_{\bm{g}}$ |  $n^{-{}}_{\bm{g}}$ |  $h$ |  $R_{\bm{g}}(h)$ |  $h_{0}{}$ |  $R_{\bm{g}}(h_{0})$
female, old |  $0$ |  $24$ |  $+$ |  24 |  $-$ |  0
female, young |  $25$ |  $0$ |  $+$ |  0 |  $-$ |  25
male, old |  $25$ |  $0$ |  $+$ |  0 |  $-$ |  25
male, young |  $0$ |  $27$ |  $-$ |  0 |  $-$ |  0
Total |  $50$ |  $51$ |  |  24 |  |  50
| Traditional Personalization
---
| groups receive predictions from $h$
---
| Model
---
| Data Use
---
| Gain
---
| $\bm{r}$ | $\Delta R_{\bm{g}}(h,h_{0}{})$
$h$ |  female, old | $-24$
$h$ |  female, young | $25$
$h$ |  male, old | $25$
$h$ |  male, young | $0$
|  | $26$
| Minimal Participatory System
---
| groups opt into predictions from $h$ or $h_{0}{}$
---
| Model
---
| Data Use
---
| Gain
---
| $\bm{r}$ | $\Delta R_{\bm{g}}(h,h_{0}{})$
$h_{0}{}$ | $\varnothing{}$ | $0$
$h$ |  female, young | $25$
$h$ |  female, young | $25$
$h_{0}{}$ | $\varnothing{}$ | $0$
|  | $50$

Figure 1: Classification task where participation improves accuracy and
minimizes data use. Under traditional personalization, individuals report
group membership to receive personalized predictions from $h$. As shown, this
benefits the population as a whole by improving overall error from 50 to 24
($\Delta R_{\bm{g}}(h,h_{0}{})=$$~{}26$). However, it has no effect on
$[{\footnotesize{\texttt{male}}},{\footnotesize{\texttt{young}}}]$, who
receive the same predicitons ($\Delta R_{\bm{g}}(h,h_{0}{})=$$~{}0$) and a
detrimental effect
to$[{\footnotesize{\texttt{female}}},{\footnotesize{\texttt{old}}}]$, who
receive less accurate predictions ($\Delta R_{\bm{g}}(h,h_{0}{})=~{}$$-24$).
In a minimal participatory system, individuals can _opt in_ to personalization
to receive predictions from $h$ or $h_{0}{}$. Here, individuals in groups
$[{\footnotesize{\texttt{female}}},{\footnotesize{\texttt{old}}}]$ and
$[{\footnotesize{\texttt{male}}},{\footnotesize{\texttt{young}}}]$ would opt
out of personalization, leading to an overall error of 0 ($\Delta
R_{\bm{g}}(h,h_{0}{})=~{}$$50$) and a reduction in unnecessary data collection
($\varnothing$).

Websites that solicit personal data are often required to support _informed
consent_ : individuals can opt out of providing personal data and understand
how it will be used [see, e.g., personal data guidelines in GDPR, OECD privacy
guidelines 26, 42]. Outside of these contexts, personalized models do not
provide such functionality: individuals cannot opt out of reporting data used
to personalize their predictions nor tell if it would improve their
predictions. Practitioners assume that data available for training will be
available at inference time. In practice, this assumption has led to a
proliferation of models that use information that individuals may be unwilling
or unable to report at prediction time [see e.g., the Denver HIV Risk Score
29, which asks patients to report age, gender, race, and sexual_practices]. In
tasks where individuals self-report, they may not voluntarily report
information that could improve their predictions or may report incorrect
information.

The broader need to facilitate and inform consent in personalization stems
from the fact that it may not improve performance for each group that reports
personal data [54]. In practice, a personalized model can perform _worse_ or
the same as a _generic model_ fit without personal information for a group
with specific characteristics. Such models violate the implicit promise of
personalization as individuals report personal information without receiving a
tailored performance gain in return. These instances of _worsenalization_ are
prevalent, hard to detect, and hard to resolve [see 54, 44]. However, they
would be resolved if individuals could opt out of personalization and
understand its expected gains (see Fig. 1).

In this work, we introduce a family of classification models that
operationalize informed consent called _participatory systems_. Participatory
systems _facilitate consent_ by allowing individuals to report personal
information at prediction time. Moreover, they _inform consent_ by showing how
reporting personal information will change their predictions. Models that
facilitate consent operate as _markets_ in which individuals trade personal
information for performance gains. This work seeks to develop systems that (i)
perform as well as possible when individuals opt-in (to incentivize voluntary
reporting) and opt out (to safeguard against abstention); (ii) provide
opportunities for individuals to make informed decisions about data provision.
The resulting systems can improve performance and curb data use for all groups
who report personal data, maximizing the gains of personal information when it
improves performance and limiting data collection when it does not.

##### Related Work

Participatory systems support modern principles of responsible data use such
as _informed consent_ and _collection limitation_ – i.e., data should be
collected with a data subject’s consent and restricted to only what is
necessary. These principles are articulated in, e.g., OECD privacy guidelines
[42], the GDPR [26], and the California Consumer Privacy Act [16]. These
principles stem from extensive work on the right to data privacy [33]. They
are motivated, in part, by research showing that individuals care deeply about
their ability to control personal data [10, 4, 8] and differ considerably in
their desire or capacity to share it [see e.g. 9, 43, 17, 18, 7, 39, 5]. We
follow an interpretation of these principles that ties data minimization to
system performance, and our work complements efforts to understand the trade-
off between personalization and performance [13, 48].

We consider models that are personalized with categorical attributes that
encode personal characteristics [i.e., “categorization” rather than
“individualization” as per 27]. Modern techniques for learning with
categorical attributes [see e.g., 2, 50] use them to improve performance at a
population level – e.g., by accounting for higher-order interaction effects
[14, 38, 60] or recursive partitioning [24, 15, 12, 11]. Our proposed methods
can be used to achieve these goals in tasks where models use features that are
optional or costly to acquire [see e.g., 6, 7, 63, 55].

Our work is broadly related to algorithmic fairness in that we seek to improve
model performance at a group level. Recent work shows that personalization
with group attributes does not uniformly improve performance and can result in
less accurate predictions at a group level [see 54, 44, 59]. The proposed
systems can mitigate this effect by allowing individuals to opt out of these
instances of “ _worsenalization_.” This line of work complements research on
preference-based fairness [64, 59, 36, 61, 22], on ensuring fairness across
complex group structures [34, 30, 28], and on the study of privacy across
subpopulations [56, 13].

###  2 Participatory Systems

We consider a classification task where we personalize a model with
categorical attributes. We start with a dataset
$\\{(\bm{x}_{i},y_{i},\bm{g}_{i})\\}_{i=1}^{n}$ where each example consists of
a feature vector $\bm{x}_{i}\in\mathbb{R}^{d}$, a label $y_{i}\in\mathcal{Y}$,
and a vector of $k$ categorical attributes
$\bm{g}_{i}=[g_{i,1},\ldots,g_{i,k}]\in\mathcal{G}_{1}\times\ldots\times\mathcal{G}_{k}=\mathcal{G}$.
We refer to $\mathcal{G}$ as _group attributes_ and $\bm{g}_{i}$ as the _group
membership_ of person $i$. We let $n_{\bm{g}}:=|\\{i~{}|\bm{g}_{i}=\bm{g}\\}|$
denote the size of group $\bm{g}$, and $|\mathcal{G}_{i}|$ denote the number
of categories for group attribute $i$.

We use the dataset to fit a model
$h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ by empirical risk minimization
with a loss $\ell:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}_{+}$. Given a
model $h$, we denote its true risk and empirical risk ${R(h)}$ and
${\hat{R}(h)}$. We denote the true risk and empirical risk of $h$ for group
$\bm{g}\in\mathcal{G}$ when they are assigned the personalized predictions for
group $\bm{g}^{\prime}\in\mathcal{G}$ as:

$\displaystyle{R_{\bm{g}}(h(\cdot,\bm{g}^{\prime}))}:=\mathbb{E}\left[\ell\left(h(\cdot,\bm{r}^{\prime}),y\right)\mid\mathcal{G}=\bm{g}^{\prime}\right],\qquad{\hat{R}_{\bm{g}}(h(\cdot,\bm{g}^{\prime}))}:=\frac{1}{n_{\bm{g}}}\sum_{i:\bm{r}_{i}=\bm{g}}\ell\left(h(\cdot,\bm{r}^{\prime}),y_{i}\right).$

We assume that groups prefer more accurate predictions. Given models $h$ and
$h^{\prime}$, users in group $\bm{g}{}$ prefer $h$ when
${R_{\bm{g}}(h)}<{R_{\bm{g}}(h^{\prime})}$. This assumption holds in settings
where individuals prefer more accurate predictions – e.g., when predicting the
risk of an illness [see, e.g., 52, 37, 51] or recommending content [40]. It
does not hold in applications where some individuals prefer inaccurate
predictions – e.g., “polar” clinical prediction tasks such as predicting the
risk of organ failure for a transplant [45].

We consider models where individuals consent to personalization by reporting
group attributes at prediction time. We let $\varnothing{}$ denote an
attribute that was not reported, and let
$\bm{r}_{i}=[r_{i,1},\ldots,r_{i,k}]\in\mathcal{R}\subseteq\mathcal{G}\times\bm{\varnothing}{}$
denote the _reported group membership_ of person $i$. For example, a person
with
$\bm{g}_{i}=[{\footnotesize{\texttt{female}}},{\footnotesize{\texttt{HIV}}}={\footnotesize{\texttt{+}}}]$
would have $\bm{r}_{i}=[{\footnotesize{\texttt{female}}},\varnothing{}]$ if
they only report sex, and
$\bm{r}_{i}=\bm{\varnothing}{}:=[\varnothing,\ldots,\varnothing]$ if they opt
out of personalization entirely.

We associate each model with a set of _reporting options_ $\mathcal{R}$. A
traditional model that requires each person to report group attributes has
$\mathcal{R}=\mathcal{G}$. A model where each person could report any subset
of group attributes has $\mathcal{R}=\mathcal{G}\times\bm{\varnothing}{}$. The
truthful reporting options of a group $\bm{g}$ include all
$\bm{r}\in\dim{\bm{g}}:=\bm{g}\times\bm{\varnothing}$. We use $n_{\bm{r}}$ to
denote the number of individuals who could truthfully report
$\bm{r}\in\mathcal{R}{}$. We represent individual decisions to opt into
personalization at prediction time through a _reporting interface_ , defined
below.

######  Definition 1.

Given a personalized classification task with group attributes $\mathcal{G}$,
a _reporting interface_ is a tree $T$ whose
$\textnormal{nodes}(T)=\mathcal{R}\subseteq\mathcal{G}\times\bm{\varnothing}{}$
represent attributes reported at prediction time. The tree is rooted at
$\textnormal{root}(T)=\bm{\varnothing}{}$ and branches as an individual
reports group attributes for personalization. Given a node $\bm{r}$, we denote
its child $\bm{r}^{\prime}$ as $\bm{r}=\textnormal{pa}(\bm{r}^{\prime})$. Each
parent-child pair represents a _reporting decision_ , and the tree’s height
represents the maximum number of reporting decisions.

We say that a tree supports _consent_ to personalization if users can opt out
of personalization entirely $\bm{\varnothing}\in\textrm{nodes}(T)$.

######  Definition 2.

Given a personalized classification task with group attributes $\mathcal{G}$,
a _participatory system_ with reporting interface $T$ is a prediction model
$f_{T}:\mathcal{X}\times\mathcal{R}\to\mathcal{Y}$ that obeys the following
properties:

  1. (R1)

_Incentive Compatibility_ : Opting into personalization improve expected
performance

$\displaystyle{R_{\bm{r}}(f_{T}({\cdot,\bm{r}}))}<{R_{\bm{r}}(f_{T}({\cdot,\bm{r}}^{\prime}))}\textrm{
for all reporting groups }\bm{r},\bm{r}^{\prime}\textrm{ such that
}\bm{r}^{\prime}=\textnormal{pa}(\bm{r})$

  2. (R2)

_Baseline Performance_ : Opting out of personalization should guarantee the
expected performance of a _generic model_ $h_{0}{}:\mathcal{X}\to\mathcal{Y}$
where $h_{0}\in\operatorname*{argmin}_{h\in\mathcal{H}}{R(h)}$.

${R_{\bm{g}}(f_{T}(\cdot,\bm{\varnothing}{}))}\leq{R_{\bm{g}}(h_{0})}\textrm{
for all groups }\bm{g}\in\mathcal{G}.$

We denote the gains of personalization for group $\bm{g}$ in terms of true
risk and empirical risk as
$\Delta_{\bm{g}}({\bm{r}},{\bm{\varnothing}{}}):={R_{\bm{g}}(f_{T}(\cdot,\bm{\varnothing}{}))}-{R_{\bm{g}}(f_{T}(\cdot,\bm{r}))}$
and
$\hat{\Delta}_{\bm{g}}({\bm{r}},{\bm{\varnothing}{}}):={\hat{R}_{\bm{g}}(f_{T}(\cdot,\bm{\varnothing}{}))}-{\hat{R}_{\bm{g}}(f_{T}(\cdot,\bm{r}))}$.

Here, the _Incentive Compatibility_ property guarantees that opting into
personalization will improve performance – i.e. when individuals provide their
personal information, the system can effectively leverage that data to deliver
more accurate predictions in expectation. The _Baseline Performance_ property
safeguards performance for individuals who opt out of personalization – i.e.,
it ensures that individuals who choose not to share their personal information
receive the performance of a generic model trained on a dataset without group
attributes. Together, these properties uphold the principle of _rationality_ ,
in which users always receive gains from personalization such that
$\Delta_{\bm{g}}({\bm{r}},{\bm{\varnothing}{}})>0$ for all
$\bm{g}\in\mathcal{G},\bm{r}\in\dim{\bm{g}}$.

| Interface
---
| Sample Interface
---
Description
| Minimal
---
|  _Minimal systems_ let individuals opt out of personalization from existing
personalized model $h$. Individuals who opt out receive predictions from a
generic model $h_{0}{}$ trained without group attributes. These systems can be
built by training one additional model.
| Flat
---
|  _Flat systems_ support _partial personalization_ by allowing each person to
report any subset of group attributes. Thus, a person with
$\bm{g}_{i}=[\texttt{old},\texttt{female}]$ can report
$\bm{r}_{i}=[\texttt{old},\varnothing]$. These systems can improve performance
by using a distinct model to assign personalized predictions to each reporting
group.
| Sequential
---
|  _Sequential systems_ let support partial personalization while allowing
individuals to report one attribute at a time. This interface is better suited
to inform consent as users can make reporting decisions by comparing two
models simultaneously. They are also well-suited for personalization tasks
with information that must be acquired at prediction time (e.g., the outcome
of a test result).

Figure 2: Participatory systems for personalized classification task with
group attributes
${\footnotesize{\texttt{sex}}}\times{\footnotesize{\texttt{age}}}=[{\footnotesize{\texttt{male}}},{\footnotesize{\texttt{female}}}]\times[{\footnotesize{\texttt{old}}},{\footnotesize{\texttt{young}}}]$.
Each system allows a person to opt out of personalization by reporting
$\varnothing$ and informs this choice through the gains of personalization
(e.g., +0.2% gain in accuracy). Systems minimize data use by removing
reporting options that do not lead to gain (e.g.,
$[{\footnotesize{\texttt{young}}},{\footnotesize{\texttt{female}}}]$ is pruned
in all systems as it leads to a gain $\leq$ 0.0%. We show reporting options
pruned using grey-striped boxes.

The simplest way to facilitate consent is to impute the group membership of
individuals who opt out of personalization at prediction time. Imputation
allows individuals to opt out of personalization but does not guarantee the
accuracy of predictions for individuals who opt out. Formally, imputation
violates the baseline performance guarantee of a participatory system. That
is, if users opt out of personalization ($\bm{r}=\bm{\varnothing}$), they may
receive a less accurate prediction than they would have from a generic model
${R_{\bm{g}}(f_{T}(\cdot,\hat{\bm{g}}))}>{R_{\bm{g}}(h_{0})}$ where
$\hat{\bm{g}}$ denote the imputed attributes. For example, in a setting where
we could perfectly impute group membership, a group might be assigned better
predictions from a generic model (see Fig. 1). In the worst case, imputation
may be incorrect, leading to even more inaccurate predictions than those of
the generic or personalized model.

####  2.1 Agent Model for Individual Disclosure

The performance of participatory systems will depend on individual reporting
decisions. In what follows, we characterize how participatory systems will
perform under a generalized model of individual disclosure. Given a
participatory system $h:\mathcal{X}\times\mathcal{R}{}\to\mathcal{Y}$, we
assume that each person will report group membership as
$\bm{r}_{i}\in\operatorname*{argmax}_{\bm{r}\in\mathcal{R}}{u_{i}({\bm{r};h})}$.
Here, the utility function can be

$u_{i}({\bm{r};h})=b_{i}({\bm{r};h})-c_{i}({\bm{r}}),$

where $c_{i}({\cdot})$ and $b_{i}({\cdot})$ denote their cost and benefit of
disclosure, respectively. We assume that costs increase monotonically with
information that is disclosed so that $c_{i}({\bm{r}})\geq 0$ for all
$\bm{r}\in\mathcal{R}$ and $c_{i}({\bm{r}})\leq c_{i}({\bm{r}^{\prime}})$ for
$\bm{r}\subseteq\bm{r}^{\prime}$. We assume that benefits increase
monotonically with true risk so that
$b_{i}({\bm{r},h})>b_{i}({\bm{r}^{\prime},h})$ when
${R_{\bm{r}}(h(\bm{x}_{i},\bm{r}))}<{R_{\bm{r}}(h(\bm{x}_{i},\bm{r}^{\prime}))}.$

The following remarks apply to any participatory system
$f:\mathcal{X}\times\mathcal{R}\to Y$ that include a personalized model
$h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}$ and a generic model
$h_{0}{}:\mathcal{X}\to\mathcal{Y}$ as its components.

  * $\bullet$

Every participatory system $f$ will perform as well as a generic model
$h_{0}$. When a personalized model $h$ requires users to report information
detrimental to performance (see Fig. 1), individuals incur a cost of
disclosure without receiving a benefit. In such instances, a minimal system
$f:\mathcal{X}\times\mathcal{R}^{\min}\to Y$ would allow individuals to opt
out of detrimental personalization and receive predictions from a generic
model.

  * $\bullet$

Every participatory system $f$ with more reporting options will perform
better. Since utility can only increase with the number of reporting options,
the maximum utility for each person will exceed that of a minimal system.
Thus, flat and sequential systems will perform better than a minimal system.

  * $\bullet$

The best-case performance of any participatory system will exceed the
performance of any of its components. Thus, we are guaranteed that any
participatory system will outperform a traditional personalized model so long
as it is considered a component.

####  2.2 Profiling System Performance with Respect to Participation

We can use the models for individual disclosure to evaluate how a
participatory system will perform once it is deployed. Given a participatory
system, we can evaluate this by simulating the parameters in the individual
disclosure model shown above. We can then summarize the results from this
evaluation for each intersectional group through a performance profile that
shows how the system performance will vary across different levels of
participation.

[<30, HIV+] [>30, HIV-]

[<30, HIV-] [>30, HIV-]

Figure 3: Performance profiles of the simulations performed for each
intersectional group in the saps dataset. The sequential system outperforms
static personalized systems when all group attributes are reported. When the
cost of reporting is high, the sequential system still outperforms minimally
personalized systems, as evidenced by higher accuracy at varying reporting
cost thresholds.

###  3 Learning Participatory Systems

This section describes a model-agnostic algorithm to learn participatory
systems that ensure incentive compatibility and baseline performance in
deployment. We outline our procedure in Algorithm 1 to learn the three kinds
of participatory systems in Fig. 2. The procedure takes as input a pool of
candidate models $\mathcal{M}{}$ and datasets for assignment and pruning
$\mathcal{D}^{\textrm{train}}$ and $\mathcal{D}^{\textrm{valid}}$. It outputs
a collection of participatory systems that obey the desired properties on test
data. The procedure combines three routines to (1) generate viable reporting
interfaces (4); (2) assign models over the interface (6); (3) prune the system
to limit unnecessary data collection (7). We discuss each routine below and
present a complete procedure in Appendix A.

1:Input:
$\mathcal{M}{}:\\{h:\mathcal{X}\times\mathcal{G}\to\mathcal{Y}\\}$pool of
candidate models

2:Input:
$\mathcal{D}^{\textrm{assign}}=\\{(\bm{x}_{i},\bm{g}_{i},y_{i})\\}_{i=1}^{n^{\textrm{assign}}}$training
dataset

3:Input:
$\mathcal{D}^{\textrm{prune}}=\\{(\bm{x}_{i},\bm{g}_{i},y_{i})\\}_{i=1}^{n^{\textrm{prune}}}$validation
dataset

4:$\mathbb{T}\leftarrow{\small\textsf{ViableTrees}}(\mathcal{G},\mathcal{D}^{\textrm{assign}})$
$|\mathbb{T}|=1$ for minimal & flat systems

5:for $T\in\mathbb{T}{}$ do

6:
$T\leftarrow{\small\textsf{AssignModels}}(T,\mathcal{M},\mathcal{D}^{\textrm{assign}})$
assign models

7: $T\leftarrow{\small\textsf{PruneLeaves}}(T,\mathcal{D}^{\textrm{prune}})$
prune models

8:end for

9:$\mathbb{T}{}$, collection of participatory systems

Algorithm 1 Learning Participatory Systems

##### Model Pool

Our procedure takes as input a _pool of personalized models_ $\mathcal{M}{}$
to assign over a reporting interface. At a minimum, a pool should contain two
models: a personalized model $h$ for individuals who opt into personalization
and a generic model $h_{0}{}$ for individuals who opt out of personalization.
A single personalized model can perform unreliably across reporting groups due
to differences in the data distribution or trade-offs between groups. Using a
pool safeguards against these effects by drawing on models from different
model classes that have been personalized using different techniques for each
reporting group. By default, we include a model for each reporting group
trained using only data for that group, as such models can perform well on
heterogeneous subgroups [59, 54].

##### Enumerating Reporting Interfaces

We call the ViableTrees routine in 4 to enumerate _viable_ reporting
interfaces. We only call this routine for sequential systems since
$\mathbb{T}{}$ consists of a single tree known a priori for minimal and flat
systems. ViableTrees takes as input a group attributes $\mathcal{G}$ and a
dataset $\mathcal{D}^{\textrm{assign}}$. It returns all $m$-ary trees that
obey constraints on sample size and reporting (e.g., users who report male
should report age before HIV). By default, we only generate trees so that we
have sufficient data to estimate gains at each node of the reporting interface
(e.g., trees whose leaves contain at least one positive sample, one negative
sample, and $n_{\bm{r}}\geq d+1$ samples to avoid overfitting). In general,
ViableTrees scales to tasks with $\leq 8$ group attributes. Beyond this limit,
one can reduce the enumeration size by specifying ordering constraints or a
threshold number of trees to enumerate before stopping. For a task with three
binary group attributes, $\mathbb{T}{}$ contains $24$ 3-ary trees of depth 3.
Given a complete ordering of all $3$ group attributes, $\mathbb{T}$ would have
$1$ tree. We can also consider a greedy algorithm (see Appendix A.4), which
may be practical for large-scale problems.

##### Model Assignment

We assign each reporting group a model using the AssignModels routine in 6.
Given a reporting group $\bm{r}$, we consider all models that could use group
membership $\bm{r}_{i}\in\dim{\bm{r}}$. Thus, a group that reports age and sex
could be assigned predictions from a model that requires age, sex, both, or
neither. This implies that we can always assign the generic model to any
reporting group, ensuring that the model at each node performs as well as the
generic model on out-of-sample data (i.e., _baseline performance_ in
Definition 2).

##### Pruning

ViableTrees may output trees that violate incentive compatibility by
requesting personal information that does not reliably improve performance.
This can happen when a system assigns the same model or a model that performs
equally well to nested reporting groups – see, e.g., Fig. 2 where the Flat
system assigns $h_{0}{}$ to $[{\footnotesize{\texttt{female}}},\varnothing{}]$
and $[{\footnotesize{\texttt{female}}},{\footnotesize{\texttt{young}}}]$. The
Prune routine in 7. This routine takes as input a participatory system $f_{T}$
and a validation sample $\mathcal{D}^{\textrm{prune}}$ and outputs a system
$f_{T^{\prime}}$ with a pruned interface $T^{\prime}\subseteq T$ that will not
request data in such cases. The routine uses a bottom-up pruning procedure
that calls a one-sided hypothesis test at each node:

$\displaystyle H_{0}:\Delta_{\bm{r}}({\bm{r}},{\textnormal{pa}(\bm{r})})\leq
0\qquad H_{A}:\Delta_{\bm{r}}({\bm{r}},{\textnormal{pa}(\bm{r})})>0$

The test checks if each reporting group $\bm{r}$ receives more accurate
predictions using the personalized model assigned by the current interface $T$
in comparison to its parent $f_{T}({\cdot,\textnormal{pa}(\bm{r})})$. Here,
$H_{0}$ assumes a reporting group prefers the parent model. Thus, we reject
$H_{0}$ when evidence suggests that $f_{T}({\cdot,\bm{r}})$ performs better
for $\bm{r}$ on pruning data. The exact test should be chosen based on the
performance metric for the underlying prediction task. In general, we can use
a bootstrap hypothesis test [20] and draw on more powerful tests for salient
performance metrics [e.g., 21, 19, 53, for accuracy and AUC].

##### On Computation

Our approach provides several options for learning participatory systems. For
example, one can train only two models and build a minimal system or train a
flat or sequential system with a limited number of models in the pool.
Nevertheless, the primary bottleneck when learning participatory systems is
_data_ rather than _compute_. Given a finite sample dataset, we are limited in
the number of categorical attributes used for personalization. This is because
we require a minimum number of samples for each intersectional group to train
a personalized model and evaluate its performance. Given that the number of
intersectional groups increases exponentially with each attribute, we quickly
enter a regime where we cannot train models for a given group (e.g., because
we lack sufficient labels) or reliably evaluate its gain for assignment and
pruning [see 44].

###  4 Experiments

We benchmark participatory systems on real-world clinical prediction tasks.
Our goal is to evaluate these approaches in terms of performance, data usage,
and consent in applications where individuals have a low reporting cost. We
include code to reproduce these results in an public repository.

####  4.1 Setup

We consider six classification tasks for clinical decision support where we
personalize a model with group attributes that are protected or sensitive (see
Table 2 and Appendix B). Each task pertains to an application where we would
expect individuals to have a low cost of reporting – and therefore report
personal information so long as there is any benefit. This is because the
information used for personalization is readily available, relevant to the
prediction task, and likely to be disclosed given laws surrounding the privacy
and confidentiality of health data [57]. [10, 4]. One exception is cardio_eicu
and cardio_mimic, which are personalized based on race and ethnicity. We note
that the use of race in clinical risk scores should be approached with caution
[62]; participatory systems offer one way to safeguard against inappropriate
use. We split each dataset into a test sample (20% for evaluating out-of-
sample performance) and a training sample (80% for training, pruning,
assignment, and estimating gains to show users). We train three kinds of
personalized models for each dataset:

  * $\bullet$

_Static_ : These models are personalized using a one-hot encoding of group
attributes (1Hot) and a one-hot encoding of intersectional groups (mHot)

  * $\bullet$

_Imputed_ : These are variants of static models where we impute the values of
group attributes (KNN-1Hot, KNN-mHot). In practice, the performance of
imputation will range between the values reported for 1Hot and KNN-mHot (100%
opt-in) and KNN-1Hot, KNN-mHot(100% opt-out).

  * $\bullet$

_Participatory_ : These are participatory systems built using our approach.
These include Minimal, a minimal system built from 1Hot and its generic
counterpart; and Flat and Seq, flat and sequential systems built from 1Hot,
mHot and their generic counterparts.

We train all models – personalized models and the components of participatory
systems – from the same model class and evaluate them using the metrics in
Table 1. We repeat the experiments four times, varying the model class
(logistic regression, random forests) and the salient performance metric
(error rate for decision-making and AUC for ranking) to evaluate the
sensitivity of our findings with respect to model classes and classification
tasks.

Metric |  Definition |  Description
---|---|---
Overall Performance |  $\displaystyle\sum_{\bm{g}\in\mathcal{G}}\tfrac{n_{\bm{g}}}{n}{\hat{R}_{\bm{g}}(h_{\bm{g}})}$ |  Population-level performance of a personalized system/model, computed as a weighted average over all groups
Overall Gain |  $\displaystyle\sum_{\bm{g}\in\mathcal{G}}\tfrac{n_{\bm{g}}}{n}\hat{\Delta}_{\bm{g}}({\bm{g}},{\bm{\varnothing}{}})$ |  Population-level gain in performance of a personalized system/model over its generic counterpart
Group Gains |  $\displaystyle\min_{\bm{g}\in\mathcal{G}}/\max_{\bm{g}\in\mathcal{G}}\hat{\Delta}_{\bm{g}}({\bm{g}},{\bm{\varnothing}{}})$ |  Range of gains of a personalized system/model over its generic counterpart across all groups
Rationality Violations |  $\displaystyle\sum_{\bm{g}\in\mathcal{G}}\mathbb{I}[\textrm{reject}~{}H_{0}]$ |  Number of rationality violations detected using a bootstrap hypothesis test with 100 resamples and significance level of 10% where $H_{0}:\Delta_{\bm{g}}({\bm{g}},{\bm{\varnothing}{}})\geq 0$ and $H_{A}:\Delta_{\bm{g}}({\bm{g}},{\bm{\varnothing}{}})<0$
Imputation Risk |  $\displaystyle\min_{\bm{g}\in\mathcal{G}}\hat{\Delta}_{\bm{g}}({\bm{g}},{\bm{g}^{\prime}})$ |  Worst-case loss in performance due to a result of incorrect imputation. This measure can only be computed for static models only.
Options Pruned |  $\displaystyle\frac{|\mathcal{R}|-|\mathcal{R}(h)|}{|\mathcal{R}|}$ |  The number of reporting options pruned from a system/model. Here, $\mathcal{R}(h)$ denotes the options available after $h$ has been pruned, and $\mathcal{R}$ denotes options available before pruning.
Data Use |  $\displaystyle\sum_{\bm{g}\in\mathcal{G}}\tfrac{n_{\bm{g}}}{n}\tfrac{\textrm{requested}({h,\bm{g}})}{\textrm{dim}(\mathcal{G})}$ |  Proportion of group attributes $k$ requested by $h$ from each group, averaged over all groups in $\mathcal{G}$

Table 1: Metrics used to evaluate performance, data use, and consent. We
report performance on a held-out test sample. We assume that individuals
report group membership to static models, do not report group membership to
imputed models, and only report to participatory systems when reporting leads
to a positive gain. In the latter case, we estimate the gain shown to
individuals using a validation set in the training sample.

####  4.2 Results

We show results for logistic regression models and error rate in Table 2 and
for other model classes and classification tasks in Appendix C. In what
follows, we discuss these results.

| Static | Imputed | Participatory
---|---|---|---
Dataset | Metrics | 1Hot | mHot | KNN-1Hot | KNN-mHot | Minimal | Flat | Seq
| apnea
---
$n={1152},d={26}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Ustun et al. [58]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 29.1%
---
0.1%
-1.1% – 1.2%
1
-4.9%
0/6
100.0%
| 29.3%
---
-0.1%
-0.8% – 0.4%
1
-5.2%
0/6
100.0%
| 29.0%
---
0.2%
-1.1% – 1.2%
1
0/12
0.0%
| 27.9%
---
1.3%
-0.8% – 0.4%
1
0/12
0.0%
| 28.9%
---
0.3%
0.0% – 1.2%
0
4/7
33.3%
| 24.1%
---
5.1%
0.0% – 13.8%
0
5/12
83.3%
| 24.3%
---
4.9%
-0.4% – 13.8%
0
6/12
58.3%
| cardio_eicu
---
$n={1341},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Pollard et al. [46]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 21.4%
---
0.4%
-1.3% – 2.6%
1
-4.6%
0/8
100.0%
| 21.5%
---
0.3%
-2.7% – 3.0%
1
-5.4%
0/8
100.0%
| 21.6%
---
0.3%
-1.3% – 2.6%
1
0/27
0.0%
| 22.1%
---
-0.2%
-2.7% – 3.0%
1
0/27
0.0%
| 21.6%
---
0.3%
0.0% – 2.6%
0
6/9
25.0%
| 10.2%
---
11.7%
3.1% – 20.9%
0
10/27
100.0%
| 10.2%
---
11.7%
3.1% – 20.9%
0
9/27
83.3%
| cardio_mimic
---
$n={5289},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Johnson et al. [32]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 19.4%
---
-0.1%
-0.9% – 0.4%
3
-1.1%
0/8
100.0%
| 19.3%
---
-0.0%
-0.9% – 0.5%
2
-1.1%
0/8
100.0%
| 19.3%
---
-0.0%
-0.9% – 0.4%
3
0/27
0.0%
| 20.1%
---
-0.8%
-0.9% – 0.5%
2
0/27
0.0%
| 19.2%
---
0.1%
0.0% – 0.4%
0
6/9
25.0%
| 15.7%
---
3.5%
-1.6% – 9.8%
1
6/27
100.0%
| 15.7%
---
3.5%
-1.6% – 9.8%
1
8/27
91.7%
| coloncancer
---
$n={29211},d={72}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 37.0%
---
0.1%
-0.4% – 0.3%
1
-1.4%
0/6
100.0%
| 36.7%
---
0.4%
-0.1% – 1.1%
0
-0.9%
0/6
100.0%
| 37.0%
---
0.1%
-0.4% – 0.3%
1
0/12
0.0%
| 36.9%
---
0.2%
-0.1% – 1.1%
0
0/12
0.0%
| 37.0%
---
0.1%
0.0% – 0.3%
0
5/7
16.7%
| 36.6%
---
0.5%
0.0% – 1.7%
0
7/12
50.0%
| 36.1%
---
1.0%
0.2% – 1.7%
0
5/12
75.0%
| lungcancer
---
$n={120641},d={84}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 19.6%
---
-0.1%
-0.4% – 0.2%
4
-0.5%
0/6
100.0%
| 19.6%
---
-0.1%
-0.3% – 0.2%
4
-0.5%
0/6
100.0%
| 19.9%
---
-0.3%
-0.4% – 0.2%
4
0/12
0.0%
| 19.8%
---
-0.2%
-0.3% – 0.2%
4
0/12
0.0%
| 19.5%
---
0.0%
0.0% – 0.0%
0
6/7
0.0%
| 18.9%
---
0.6%
0.0% – 0.9%
0
3/12
83.3%
| 18.9%
---
0.6%
0.3% – 0.9%
0
7/12
58.3%
| saps
---
$n={7797},d={36}$
$\mathcal{G}=\\{{\footnotesize{\texttt{HIV}}},{\footnotesize{\texttt{age}}}\\}$
$|\mathcal{G}|={4}$ groups
Allyn et al. [3]
| Overall Performance
---
Overall Gain
Group Gains
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 20.4%
---
1.3%
0.0% – 3.6%
0
0.0%
0/4
100.0%
| 20.7%
---
1.0%
0.0% – 2.7%
0
-2.4%
0/4
100.0%
| 20.4%
---
1.3%
0.0% – 3.6%
0
0/9
0.0%
| 29.4%
---
-7.7%
0.0% – 2.7%
0
0/9
0.0%
| 20.4%
---
1.3%
0.0% – 3.6%
0
1/5
75.0%
| 11.1%
---
10.6%
4.3% – 17.2%
0
1/9
100.0%
| 11.1%
---
10.6%
4.3% – 17.2%
0
3/9
75.0%

Table 2: Participatory systems and personalized models for all datasets. We
summarize metrics in Table 1 and present results for other model classes and
prediction tasks in Appendix C.

##### On Performance

Our results in Table 2 show that participatory systems can improve performance
across reporting groups. Here, Flat and Seq achieve the best overall
performance on 6/6 datasets and improve the gains from personalization for
every reporting group on 5/6 datasets. In contrast, traditional models improve
overall performance while reducing performance at a group level (see
rationality violations on five datasets for 1Hot, mHot). The performance
benefits from participatory systems stem from (i) allowing users to opt out of
these instances of “worsenalization” and (ii) assigning personalized
predictions with multiple models. Using Table 2, we can measure the impact of
(i) by comparing the performance of Minimal vs. 1Hot, and the impact of (ii)
by comparing the performance of Minimal to Flat (or Seq). For example, on
apnea, 1Hot exhibits a significant rationality violation for group
$[{\footnotesize{\texttt{30\\_to\\_60}}},{\footnotesize{\texttt{male}}}]$,
meaning they would have been better off with a generic model that did not use
their personal data. By comparing the performance of 1Hot to Minimal, we see
that allowing users to opt out of worsenalization reduces test error from
29.1% to 28.9%. By comparing the performance on Minimal to Flat and Seq, we
see that using multiple models can further reduce test error from 28.9% to
24.1%.

##### On Informed Consent

Our results show how Flat and Seq systems can inform consent by allowing users
to report a subset of group attributes (e.g., by including reporting options
such as $[{\footnotesize{\texttt{30+}}},\varnothing{}]$ or
$[\varnothing{},{\footnotesize{\texttt{HIV+}}}]$). Although both Flat and Seq
systems allow for partial personalization, their capacity to inform consent
differs. In a flat system, users may inaccurately gauge the marginal benefit
of reporting an attribute by comparing the gains between reporting options.
For example, in Fig. 4, users who are HIV positive would see a gain of $3.7\%$
for reporting $[\varnothing{},{\footnotesize{\texttt{HIV+}}}]$, and $16.7\%$
for reporting $[{\footnotesize{\texttt{30+}}},{\footnotesize{\texttt{HIV+}}}]$
and may mistakenly conclude that the gain of reporting age is
$16.7\%-3.7\%=13.0\%$. This estimate incorrectly presumes that the gains of
$3.7\%$ were distributed equally across age groups. Sequential systems
directly inform users of the gains for partial reporting. In the sequential
system, group $[{\footnotesize{\texttt{30+}}},{\footnotesize{\texttt{HIV+}}}]$
is informed that they would see a marginal gain of 21.5% for reporting age,
while group $[{\footnotesize{\texttt{<30}}},{\footnotesize{\texttt{HIV+}}}]$
is informed they would see a marginal gain of reporting age of $0.0\%$.

##### On Data Minimization

Our results show that participatory systems perform better across all groups
while requesting less personal data on 6/6 datasets. For example, on
cardio_eicu, Seq reduces error by $11.3\%$ compared to 1Hot while requesting,
on average, $83.3\%$ of the data needed by 1Hot. In general, participatory
systems can limit data use where personalization does not improve performance,
e.g., on lungcancer. Even as attributes like sex or age may be readily
reported by patients for any performance benefit, limiting data use is
valuable when there is a tangible cost associated with data collection – e.g.,
when models make use of rating scale for a mental disorder that must be
administered by a clinician [49]. The potential for data minimization varies
substantially across prediction tasks. On apnea, for example, we can prune 6
reporting options when building a Seq for decision making (which optimizes
error) but 4 options for Seq for ranking (which optimizes AUC; see Section
C.1). Overall, participatory systems satisfy “global data minimization” as
proposed in [13], in that they minimize the amount of per-user data requested
while achieving the quality of a system with access to the full data on
average.

Figure 4: Participatory systems for the saps dataset. These models predict ICU
mortality for groups defined by
$\mathcal{G}={\footnotesize{\texttt{HIV}}}\times{\footnotesize{\texttt{age}}}={\footnotesize{\texttt{[+,-]}}}\times{\footnotesize{\texttt{[<30,
30+]}}}$. Here, $h_{0}{}$ is a generic model, $h_{1}$ is a 1Hot model fit with
a one-hot encoding of $\mathcal{G}$, and $h_{2}\cdots h_{m}$ are 1Hot and mHot
models fit for each reporting group. We show the gains of each reporting
option above each box, and highlight pruned options in grey. For example, in
Seq, group (HIV+, 30+) sees an estimated $21.5\%$ error reduction for age
after reporting HIV. In contrast, group (HIV+, <30) sees no gain from
reporting age in addition to HIV status, so this option is pruned.

##### On the Benefits of a Model-Agnostic Approach

Our findings highlight some of the benefits of a model-agnostic approach, in
which we can draw on a rich set of models to achieve better performance while
mitigating harm. The resulting system can balance training costs with
performance benefits. For example, on the cardio_eicu dataset, training a pool
of $126$ candidate models takes 6.5 seconds for logistic regression and 27.5
seconds for random forests on a single CPU. We can also ensure generalization
across reporting groups – e.g., by a generic model fit from a complex model
class, and personalized models fit from a simpler model class. As expected,
fitting for a complex model class can lead to considerable changes in overall
accuracy – e.g., we can reduce overall test error for a personalized model
from 20.4% to 14.1% on saps by fitting a random forest rather than a logistic
regression model (see Appendix C). However, a gain in overall performance does
not always translate to gains at the group level. On saps, for example, using
a random forest also introduces a rationality violation for one group.

##### On the Pitfalls of Imputation

One of the simplest approaches to allow individuals to opt out of
personalization is to pair a personalized model with an imputation technique.
Although this approach can facilitate consent, it may violate the requirements
in 2. Consider a personalized model that exhibits “worsenalization” in Fig. 1.
Even if one could correctly impute the group membership for every person,
individuals may receive more accurate predictions from a generic model
$h_{0}{}$. In practice, imputation is imperfect – as individuals who opt out
of reporting their group membership to a personalized model may be assigned
“worse” predictions because they are imputed the group membership of a
different group. In such cases, opting out may be beneficial, making it
difficult for model developers to promote participation while informing
consent. Our results highlight the prevalence of these effects across in
practice. For example, on cardio_eicu the estimated “risk of imputation” is
$-4.6\%$ , indicating that every intersectional group can experience an
additional $4.6\%$ error rate as a result of incorrect imputation. The results
for KNN-1Hot show that this estimate loss in performance can be realized in
practice using KNN-imputation, as we find that the imputed system leads to
rationality violations on 5/6 datasets.

###  5 Limitations and Future Work

In this work, we introduced a family of prediction models that allow
individuals to report personal data at prediction time. These systems can
inform consent while producing large improvements in performance and data use
for each group that reports personal data.

##### Limitations

One limitation of this work includes the ability to retrain models on data
collection during deployment. This setting arises because data collected post-
deployment could exhibit “non-response” bias as individuals opt out, and it
would be misleading to use it to refine estimates or retrain models. If we
retrain using only data from deployment, it is possible and even likely to
lose granularity for evaluating certain groups. This setting, however, may
violate the principle of _purpose specification_ in data collection [42]. If
the purpose of data collection is to monitor or improve a model, then
individuals could be asked to report information voluntarily for this purpose.
If data collection aims to improve the model, however, then individuals are
within their right to opt-out and should, at minimum, not be harmed by opting
in. Solutions could include asking individuals to report their data to improve
the system voluntarily or reweighting data collected under non-response bias
by comparing the prevalence of groups in deployment to the prevalence in the
baseline dataset (though this is an open problem). In addition, reaping the
benefits of personalization hinges on the ability to effectively inform users
of the impacts of disclosure [4]. Participatory systems are designed to not
impose restrictions on what we would present to users – specifically because
the information we’d want to show to inform consent should change across
applications and target audiences to inform consent [23]. There are good
“default practices” that avoid pitfalls that would most affect a large
collection of end-users (i.e., the kind of topic that would be pursued in a
user study). In general, we can draw on a broad set of literature in numeracy
and uncertainty quantification to promote understanding among end-users [25,
41].

##### Future Work

One exciting avenue of future research is to evaluate participatory systems in
the context of user studies. While this work lays the foundation for building
and evaluating participatory systems, future research should include work on
the user interface design of participatory systems (e.g., how to communicate
gains) and evaluation of these choices (e.g., how well do users choices align
with their personal costs and benefits of data provision and performance).
Another avenue of future work could including exploring participatory systems
in the context of continuous attributes and unstructured data. The key
requirement for applying this framework to other settings is the ability to
estimate the gains of personalization. We could estimate the gains in this
work because we had “groups.” When tasked with unstructured data, we may be
able to accomplish this when we have multiple observations per person (e.g.,
in speech recognition). On the other hand, continuous attributes may prove
more difficult and may require a different approach.

### References

  * Abajian et al. [2018] Abajian, A., Murali, N., Savic, L. J., Laage-Gaupp, F. M., Nezami, N., Duncan, J. S., Schlachter, T., Lin, M., Geschwind, J.-F., and Chapiro, J.  Predicting treatment response to intra-arterial therapies for hepatocellular carcinoma with the use of supervised machine learning—an artificial intelligence concept.  _Journal of Vascular and Interventional Radiology_ , 29(6):850–857, 2018.
  * Agresti [2018] Agresti, A.  _An introduction to categorical data analysis_.  John Wiley & Sons, 2018.
  * Allyn et al. [2016] Allyn, J., Ferdynus, C., Bohrer, M., Dalban, C., Valance, D., and Allou, N.  Simplified acute physiology score ii as predictor of mortality in intensive care units: a decision curve analysis.  _PloS one_ , 11(10):e0164828, 2016.
  * Anderson & Agarwal [2011] Anderson, C. L. and Agarwal, R.  The digitization of healthcare: boundary risks, emotion, and consumer willingness to disclose personal health information.  _Information Systems Research_ , 22(3):469–490, 2011.
  * Arellano et al. [2018] Arellano, A. M., Dai, W., Wang, S., Jiang, X., and Ohno-Machado, L.  Privacy policy and technology in biomedical data science.  _Annual review of biomedical data science_ , 1:115, 2018\.
  * Atan et al. [2016] Atan, O., Whoiles, W., and Schaar, M.  Data-driven online decision making with costly information acquisition.  _Arxiv_ , 02 2016.
  * Auer et al. [2008] Auer, P., Jaksch, T., and Ortner, R.  Near-optimal regret bounds for reinforcement learning.  _Advances in neural information processing systems_ , 21, 2008.
  * Auxier et al. [2019] Auxier, B., Rainie, L., Anderson, M., Perrin, A., Kumar, M., and Turner, E.  Americans and privacy: Concerned, confused and feeling lack of control over their personal information.  _Pew Research Center: Internet, Science and Tech_ , 2019.
  * Awad & Krishnan [2006] Awad, N. F. and Krishnan, M. S.  The personalization privacy paradox: an empirical evaluation of information transparency and the willingness to be profiled online for personalization.  _MIS quarterly_ , pp. 13–28, 2006.
  * Bansal et al. [2010] Bansal, G., Gefen, D., et al.  The impact of personal dispositions on information sensitivity, privacy concern and trust in disclosing health information online.  _Decision support systems_ , 49(2):138–150, 2010\.
  * Bertsimas & Kallus [2020] Bertsimas, D. and Kallus, N.  From predictive to prescriptive analytics.  _Management Science_ , 66(3):1025–1044, 2020\.
  * Bertsimas et al. [2019] Bertsimas, D., Dunn, J., and Mundru, N.  Optimal prescriptive trees.  _INFORMS Journal on Optimization_ , 1(2):164–183, 2019.
  * Biega et al. [2020] Biega, A. J., Potash, P., Daumé, H., Diaz, F., and Finck, M.  Operationalizing the legal principle of data minimization for personalization.  In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_ , pp. 399–408, 2020.
  * Bien et al. [2013] Bien, J., Taylor, J., and Tibshirani, R.  A lasso for hierarchical interactions.  _Annals of statistics_ , 41(3):1111, 2013.
  * Biggs et al. [2020] Biggs, M., Sun, W., and Ettl, M.  Model distillation for revenue optimization: Interpretable personalized pricing.  _arXiv preprint arXiv:2007.01903_ , 2020.
  * Bukaty [2019] Bukaty, P.  _The California Consumer Privacy Act (CCPA): An implementation guide_.  IT Governance Publishing, 2019.  ISBN 9781787781337.  URL https://books.google.com/books?id=vGWfDwAAQBAJ.
  * Campbell & Kracaw [1980] Campbell, T. S. and Kracaw, W. A.  Information production, market signalling, and the theory of financial intermediation.  _the Journal of Finance_ , 35(4):863–882, 1980\.
  * Chemmanur [1993] Chemmanur, T. J.  The pricing of initial public offerings: A dynamic model with information production.  _The Journal of Finance_ , 48(1):285–304, 1993\.
  * DeLong et al. [1988] DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.  _Biometrics_ , pp. 837–845, 1988.
  * DiCiccio & Efron [1996] DiCiccio, T. J. and Efron, B.  Bootstrap confidence intervals.  _Statistical science_ , pp. 189–212, 1996.
  * Dietterich [1998] Dietterich, T. G.  Approximate statistical tests for comparing supervised classification learning algorithms.  _Neural computation_ , 10(7):1895–1923, 1998\.
  * Do et al. [2021] Do, V., Corbett-Davies, S., Atif, J., and Usunier, N.  Online certification of preference-based fairness for personalized recommender systems.  _arXiv preprint arXiv:2104.14527_ , 2021.
  * Edwards et al. [2013] Edwards, A. G., Naik, G., Ahmed, H., Elwyn, G. J., Pickles, T., Hood, K., and Playle, R.  Personalised risk communication for informed decision making about taking screening tests.  _Cochrane database of systematic reviews_ , Cochrane database of systematic reviews(2), 2013.
  * Elmachtoub et al. [2018] Elmachtoub, A. N., Gupta, V., and Hamilton, M.  The value of personalized pricing.  _Available at SSRN 3127719_ , 2018.
  * Estrada et al. [1999] Estrada, C., Barnes, V., Collins, C., and Byrd, J. C.  Health literacy and numeracy.  _Jama_ , 282(6):527–527, 1999.
  * European Parliament and of the Council [2016] European Parliament and of the Council.  Regulation 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation), 2016.  URL https://eur-lex.europa.eu/eli/reg/2016/679/oj.  Official Journal of the European Union.
  * Fan & Poole [2006] Fan, H. and Poole, M. S.  What is personalization? perspectives on the design and implementation of personalization in information systems.  _Journal of Organizational Computing and Electronic Commerce_ , 16(3-4):179–202, 2006.
  * Globus-Harris et al. [2022] Globus-Harris, I., Kearns, M., and Roth, A.  An algorithmic framework for bias bounties.  _2022 ACM Conference on Fairness, Accountability, and Transparency_ , Jun 2022.  doi: 10.1145/3531146.3533172.  URL http://dx.doi.org/10.1145/3531146.3533172.
  * Haukoos et al. [2012] Haukoos, J. S., Lyons, M. S., Lindsell, C. J., Hopkins, E., Bender, B., Rothman, R. E., Hsieh, Y.-H., MacLaren, L. A., Thrun, M. W., Sasson, C., et al.  Derivation and validation of the denver human immunodeficiency virus (hiv) risk score for targeted hiv screening.  _American journal of epidemiology_ , 175(8):838–846, 2012.
  * Hébert-Johnson et al. [2018] Hébert-Johnson, Ú., Kim, M., Reingold, O., and Rothblum, G.  Multicalibration: Calibration for the (computationally-identifiable) masses.  In _Proceedings of the International Conference on Machine Learning_ , pp. 1944–1953, 2018.
  * Hollenberg [2003] Hollenberg, S.  Cardiogenic shock.  In _Intensive Care Medicine_ , pp. 447–458. Springer, 2003.
  * Johnson et al. [2016] Johnson, A. E., Pollard, T. J., Shen, L., Li-Wei, H. L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Kaminski [2019] Kaminski, M. E.  The right to explanation, explained.  _Berkeley Tech. LJ_ , 34:189, 2019.
  * Kearns et al. [2018] Kearns, M., Neel, S., Roth, A., and Wu, Z. S.  Preventing fairness gerrymandering: Auditing and learning for subgroup fairness.  In _International Conference on Machine Learning_ , pp. 2564–2572, 2018.
  * Kessler et al. [2005] Kessler, R. C., Adler, L., Ames, M., Demler, O., Faraone, S., Hiripi, E., Howes, M. J., Jin, R., Secnik, K., Spencer, T., et al.  The world health organization adult adhd self-report scale (asrs): a short screening scale for use in the general population.  _Psychological medicine_ , 35(2):245–256, 2005\.
  * Kim et al. [2019] Kim, M. P., Korolova, A., Rothblum, G. N., and Yona, G.  Preference-informed fairness.  _arXiv preprint arXiv:1904.01793_ , 2019.
  * Le Gall et al. [1993] Le Gall, J.-R., Lemeshow, S., and Saulnier, F.  A new simplified acute physiology score (saps ii) based on a european/north american multicenter study.  _Jama_ , 270(24):2957–2963, 1993.
  * Lim & Hastie [2015] Lim, M. and Hastie, T.  Learning interactions via hierarchical group-lasso regularization.  _Journal of Computational and Graphical Statistics_ , 24(3):627–654, 2015.
  * Lundberg et al. [2019] Lundberg, I., Narayanan, A., Levy, K., and Salganik, M. J.  Privacy, ethics, and data access: A case study of the fragile families challenge.  _Socius_ , 5:2378023118813023, 2019.
  * McAuley [in press] McAuley, J.  _Personalized Machine Learning_.  Cambridge University Press, in press.
  * Nanayakkara et al. [2022] Nanayakkara, P., Bater, J., He, X., Hullman, J., and Rogers, J.  Visualizing privacy-utility trade-offs in differentially private data releases.  _arXiv preprint arXiv:2201.05964_ , 2022.
  * OECD [2013] OECD.  Recommendation of the council concerning guidelines governing the protection of privacy and transborder flows of personal data, 2013.  URL https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0188.
  * Ortlieb & Garner [2016] Ortlieb, M. and Garner, R.  Sensitivity of personal data items in different online contexts.  _it-Information Technology_ , 58(5):217–228, 2016\.
  * Paes et al. [2022] Paes, L. M., Long, C. X., Ustun, B., and Calmon, F.  On the epistemic limits of personalized prediction.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=Snp3iEj7NJ.
  * Paulus & Kent [2020] Paulus, J. K. and Kent, D. M.  Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities.  _NPJ digital medicine_ , 3(1):1–8, 2020.
  * Pollard et al. [2018] Pollard, T. J., Johnson, A. E., Raffa, J. D., Celi, L. A., Mark, R. G., and Badawi, O.  The eicu collaborative research database, a freely available multi-center database for critical care research.  _Scientific data_ , 5(1):1–13, 2018.
  * Scosyrev et al. [2012] Scosyrev, E., Messing, J., Noyes, K., Veazie, P., and Messing, E.  Surveillance epidemiology and end results (seer) program and population-based research in urologic oncology: an overview.  In _Urologic Oncology: Seminars and Original Investigations_ , volume 30, pp. 126–132. Elsevier, 2012.
  * Shanmugam et al. [2022] Shanmugam, D., Diaz, F., Shabanian, S., Finck, M., and Biega, A.  Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 839–849, 2022.
  * Sharp [2015] Sharp, R.  The hamilton rating scale for depression.  _Occupational Medicine_ , 65(4):340–340, 2015\.
  * Steyerberg et al. [2019] Steyerberg, E. W. et al.  _Clinical prediction models_.  Springer, 2019.
  * Struck et al. [2017] Struck, A. F., Ustun, B., Ruiz, A. R., Lee, J. W., LaRoche, S. M., Hirsch, L. J., Gilmore, E. J., Vlachy, J., Haider, H. A., and Rudin, C.  Association of an electroencephalography-based risk score with seizure probability in hospitalized patients.  _JAMA neurology_ , 74(12):1419–1424, 2017.
  * Struck et al. [2020] Struck, A. F., Tabaeizadeh, M., Schmitt, S. E., Ruiz, A. R., Swisher, C. B., Subramaniam, T., Hernandez, C., Kaleem, S., Haider, H. A., and Cissé, A. F.  Assessment of the validity of the 2helps2b score for inpatient seizure risk prediction.  _JAMA neurology_ , 77(4):500–507, 2020.
  * Sun & Xu [2014] Sun, X. and Xu, W.  Fast implementation of delong’s algorithm for comparing the areas under correlated receiver operating characteristic curves.  _IEEE Signal Processing Letters_ , 21(11):1389–1393, 2014.  doi: 10.1109/LSP.2014.2337313.
  * Suriyakumar et al. [2023] Suriyakumar, V. M., Ghassemi, M., and Ustun, B.  When personalization harms: Reconsidering the use of group attributes in prediction.  In _International Conference on Machine Learning_ , 2023.
  * Tran & Fioretto [2023] Tran, C. and Fioretto, F.  Personalized privacy auditing and optimization at test time.  _arXiv preprint arXiv:2302.00077_ , 2023.
  * Tran et al. [2021] Tran, C., Dinh, M., and Fioretto, F.  Differentially private empirical risk minimization under the fairness lens.  _Advances in Neural Information Processing Systems_ , 34:27555–27565, 2021.
  * U.S. Congress [1996] U.S. Congress.  Health insurance portability and accountability act of 1996, 1996.  URL https://www.hhs.gov/hipaa/for-professionals/privacy/index.html.  Public Law 104-191.
  * Ustun et al. [2016] Ustun, B., Westover, M. B., Rudin, C., and Bianchi, M. T.  Clinical prediction models for sleep apnea: the importance of medical history over symptoms.  _Journal of Clinical Sleep Medicine_ , 12(02):161–168, 2016.
  * Ustun et al. [2019] Ustun, B., Liu, Y., and Parkes, D.  Fairness without harm: Decoupled classifiers with preference guarantees.  In _International Conference on Machine Learning_ , pp. 6373–6382, 2019.
  * Vaughan et al. [2020] Vaughan, G., Aseltine, R., Chen, K., and Yan, J.  Efficient interaction selection for clustered data via stagewise generalized estimating equations.  _Statistics in Medicine_ , 39(22):2855–2868, 2020\.
  * Viviano & Bradic [2020] Viviano, D. and Bradic, J.  Fair policy targeting.  _arXiv preprint arXiv:2005.12395_ , 2020.
  * Vyas et al. [2020] Vyas, D. A., Eisenstein, L. G., and Jones, D. S.  Hidden in plain sight—reconsidering the use of race correction in clinical algorithms, 2020.
  * Yu et al. [2009] Yu, S., Krishnapuram, B., Rosales, R., and Rao, R. B.  Active sensing.  In _AISTATS_ , 2009.
  * Zafar et al. [2017] Zafar, M. B., Valera, I., Rodriguez, M., Gummadi, K., and Weller, A.  From parity to preference-based notions of fairness in classification.  In _Advances in Neural Information Processing Systems_ , pp. 228–238, 2017.

parttocoff parttoc-2.5em

##

Supplementary Material

###  Appendix A Supporting Material for Section 3

####  A.1 Enumeration Routine for Algorithm 1

We summarize the Enumeration routine in Algorithm 2. Algorithm 2 takes as
input a set of group attributes $\mathcal{G}$ and a dataset $\mathcal{D}$ and
outputs a collection of reporting interfaces $\mathbb{T}{}{}$ that obey
ordering and plausibility constraints.

1:procedure ViableTrees($\mathcal{G},\mathcal{D}$)

2: if dim($\mathcal{G}{}$) = 1 return [$T_{\mathcal{G}}{}$] base case: we are
left with only a single attribute on which to branch

3: $\mathbb{T}{}\leftarrow[\ ]$

4: for each group attribute
$\mathcal{A}\in[\mathcal{G}_{1},\ldots,\mathcal{G}_{k}]$ do

5: $T_{\mathcal{A}}{}\leftarrow$ reporting tree of depth 1 with
$|\mathcal{A}|$ leaves

6:
$\mathcal{S}\leftarrow\textsf{ViableTrees}(\mathcal{G}\setminus\mathcal{A},\mathcal{D})$
all subtrees using all attributes except $\mathcal{A}$

7: for $\Pi{}$ in ValidAssignments($\mathcal{S}{},\mathcal{A},\mathcal{D}$)
do: each assignment is a permutation of $|\mathcal{A}|$ to leaves of
$T_{\mathcal{A}}{}$

8: $\mathbb{T}{}{}\leftarrow\mathbb{T}{}{}\cup
T_{\mathcal{A}}.\textsf{assign}(\Pi{})$ extends the tree by assigning subtrees
to each leaf

9: end for

10: end for

11: return $\mathbb{T}{}{}$, reporting interfaces for group attributes
$\mathcal{G}{}$ that obey plausibility and ordering constraints

12:end procedure

Algorithm 2 Enumerate All Possible Reporting Trees for Reporting Options
$\mathcal{G}{}$

The routine enumerates all possible reporting interfaces for a given set of
group attributes $\mathcal{G}$ through a recursive branching process. Given a
set of group attributes, the routine is called for each attribute that has yet
to be considered in the tree 4, ensuring a complete enumeration. We note that
the routine is only called for building Sequential systems since there is only
one possible reporting interface for Minimal and Flat systems.

Enumerating all possible trees ensures we can recover the best tree given the
selection criteria and allows practitioners to choose between models based on
other criteria. We generate trees that meet plausibility constraints based on
the dataset, such as having at least one negative and one positive sample and
at least $s$ total samples at each leaf. In settings constrained by
computational resources, we can impose additional stopping criteria and modify
the ordering to enumerate more plausible trees first or exclusively (e.g., by
changing the ordering of $\mathcal{G}$ or imposing constraints in
ValidAssignments).

####  A.2 Assignment Routine for Algorithm 1

We summarize the routine for AssignModels procedure in Algorithm 3.

1:procedure AssignModels($T,\mathcal{M}{},\mathcal{D}$)

2: $Q\leftarrow[T.\textsf{root}]$ initialize with the root of the tree,
reporting group $\bm{\varnothing}{}$

3: while $Q$ is not empty do

4: $\bm{r}\leftarrow Q.\textsf{pop}()$

5: $\mathcal{M}_{\bm{r}}\leftarrow\textsf{ViableModels}(\mathcal{M},\bm{r})$
filter $\mathcal{M}$ to models that can be assigned to $\bm{r}$

6: $\displaystyle
h^{*}\leftarrow\operatorname*{argmin}_{h\in\mathcal{M}_{\bm{r}}}{{\hat{R}_{\bm{r}}(h,\mathcal{D})}}$assign
the model with the best training performance

7: $T.\textsf{set\\_model}(\bm{r},h^{*})$

8: for $\bm{r}^{\prime}\in T.\textsf{get\\_subgroups}(\bm{r})$ do iterate
through the children reporting groups of $\bm{r}$

9: $Q.\textsf{enqueue}(\bm{r}^{\prime})$

10: end for

11: end while

12: return $T$ that maximizes gain for each reporting group

13:end procedure

Algorithm 3 Assigning Models

Algorithm 3 takes as inputs a reporting tree $T{}$, a pool candidate models
$\mathcal{M}$, and an assignment (training) dataset $\mathcal{D}$ and outputs
a tree $T$ that maximizes the gains of reporting group information. The pool
of candidate models is filtered to viable models for each reporting group.
Since the pool of candidate models includes the generic model $h_{0}{}$, each
reporting group will have at least one viable model. We assign each reporting
group the best-performing model on the training set and default to the generic
model $h_{0}{}$ when a better-performing personalized model is not found. We
assign performance on the training set and then prune using performance on the
validation set to avoid biased gain estimations.

####  A.3 Pruning Routine for Algorithm 1

We summarize the routine used for the PruneLeaves procedure in Algorithm 1.
The PruneLeaves routine

1:procedure PruneLeaves($T,\mathcal{D}$)

2: $Stack\leftarrow[T.\textsf{leaves}]$ initialize stack with all leaves

3: repeat

4: $\bm{r}\leftarrow Stack.pop()$

5: $h\leftarrow T.\textsf{get\\_model}(\bm{r})$

6: $h^{\prime}\leftarrow T.\textsf{get\\_model}(\textnormal{pa}(\bm{r}))$

7: if not ${\small\textsf{Test}}(\bm{r},h,h^{\prime},\mathcal{D})$ thentest
gains to see if parent model is as good as leaf model

8: $T.\textsf{prune}(\bm{r})$

9: end if

10: if $T.\textsf{get\\_children}(\textnormal{pa}(\bm{r}))$ is empty then
consider pruning the parent if the parent has become a leaf

11: $Stack.\textsf{enqueue}(\textnormal{pa}(\bm{r}))$

12: end if

13: until $Stack$ is empty

14: return $T$, reporting interface that ensures data collection leads to gain

15:end procedure

Algorithm 4 Pruning Participatory Systems

Algorithm 1 takes as input a reporting interface $T$ and a validation sample
$\mathcal{D}$, and performs a bottom-up pruning to output a reporting
interface $T$ that asks individuals to report attributes that are expected to
lead to a gain. The pruning decision at each leaf is based on a hypothesis
test that evaluates the gains of reporting for a reporting group on a
validation dataset. This test has the form:

$\displaystyle
H_{0}:{R_{\bm{g}}(h)}\leq{R_{\bm{g}}(h^{\prime})}\quad\text{vs.}\quad
H_{A}:{R_{\bm{g}}(h)}>{R_{\bm{g}}(h^{\prime})}$

This procedure evaluates the gains of reporting by comparing the performance
of a model assigned at a leaf node $h$ and a model assigned at a parent node
$h^{\prime}$ which does not use the reported information. Here, the null
hypothesis $H_{0}$ assumes that the parent model performs as well as the leaf
model – and thus, we reject the null hypothesis when there is sufficient
evidence to suggest that reporting will improve performance in deployment. Our
routine allows practitioners to specify the hypothesis test to compute the
gains. By default, we use the McNemar test for accuracy [21] and the Delong
test for AUC [19, 53]. In general, we can use a bootstrap hypothesis test
[20].

####  A.4 Greedy Induction of Sequential Reporting Interface

We present an additional routine to construct reporting interfaces for
sequential systems in Algorithm 5. We include this routine as an alternative
option that can be used to construct a reporting interface in settings where
it may be impractical or undesirable to enumerate all possible reporting
interfaces. The procedure results in a valid reporting interface that ensures
gains. However, it does not guarantee an optimal tree in terms of maximizing
the overall gain and does not allow to practitioners to choose between
reporting interfaces after training.

1:procedure GreedyTree($\mathcal{R}$)

2: $T{}\leftarrow$ empty reporting interface

3: repeat

4: for $\bm{r}{}\in\textrm{leaves}(T{})$ do

5: $\\{\mathcal{A}_{\bm{r}{}}\\}\leftarrow{G_{i}:\bm{r}{}[i]=\varnothing}$
$\\{\mathcal{A}_{\bm{r}{}}\\}$ contains all heretofore unused attributes

6:
$\mathcal{A}^{*}\leftarrow\operatorname*{argmax}_{\mathcal{A}\in\\{\mathcal{A}_{\bm{r}{}}\\}}{\min_{\bm{r}^{\prime}\in\bm{r}{}.\textsf{split}(\mathcal{A})}{\Delta_{\bm{r}^{\prime}}({\bm{r}^{\prime}},{\bm{r}})}}$

7: $\bm{r}{}.\textsf{split}(\mathcal{A}^{*})$ Split on attribute that
maximizes worse-case gain

8: end for

9: until no splits are added

10: return $T{}$, reporting interface that ensures gains for reporting each
$\mathcal{R}{}$.

11:end procedure

Algorithm 5 Greedy Induction Routine for Sequential Reporting Interfaces

Algorithm 5 takes as input a collection of reporting options $\mathcal{R}$ and
outputs a single reporting interface using a greedy tree induction routine
that chooses the attribute to report to maximize the minimum gain at each
step. The procedure uses the reporting options to iteratively construct a
reporting tree that branches on all of the attributes in $\mathcal{R}$. The
procedure considers each unused attribute for each splitting point and splits
on the attribute that provides the greatest minimum gain for the groups
contained at that node.

###  Appendix B Description of Datasets used in Section 4 – Experiments

We include additional information about the datasets used in Section 4.

| Dataset
---
| Reference
---
| Outcome Variable
---
| $n$
---
| $d$
---
| $m$
---
$\mathcal{G}$
apnea | Ustun et al. [58] | patient has obstructive sleep apnea | 1,152 | 28 | 6 |  {age, sex}
cardio_eicu | Pollard et al. [46] | patient with cardiogenic shock dies | 1,341 | 49 | 8 |  {age, sex, race}
cardio_mimic | Johnson et al. [32] | patient with cardiogenic shock dies | 5,289 | 49 | 8 |  {age, sex, race}
coloncancer | Scosyrev et al. [47] | patient dies within 5 years | 29,211 | 72 | 6 |  {age, sex}
lungcancer | Scosyrev et al. [47] | patient dies within 5 years | 120,641 | 84 | 6 |  {age, sex}
saps | Allyn et al. [3] | ICU mortality | 7,797 | 36 | 4 |  {age, HIV}

Table 3: Datasets used to fit clinical prediction models in Section 4. Here:
$n$ denotes the number of examples in each dataset; $d$ denotes the number of
features; $\mathcal{G}$ denotes the group attributes that are used for
personalization; and $m=|\mathcal{G}|$ denotes the number of intersectional
groups. Each dataset is de-identified and available to the public. The
cardio_eicu, cardio_mimic, lungcancer datasets require access to public
repositories listed under the references. The saps and apnea datasets must be
requested from the authors. The support dataset can be downloaded directly
from the URL below.

##### apnea

We use the obstructive sleep apnea (OSA) dataset outlined in Ustun et al.
[58]. This dataset includes a cohort of 1,152 patients where 23% have OSA. We
use all available features (e.g. BMI, comorbidities, age, and sex) and
binarize them, resulting in 26 binary features.

#####  cardio_eicu & cardio_mimic

Cardiogenic shock is an acute condition in which the heart cannot provide
sufficient blood to the vital organs [31]. These datasets are designed to
predict cardiogenic shock for patients in intensive care. Each dataset
contains the same features, group attributes, and outcome variables for
patients in different cohorts. The cardio_eicu dataset contains records for a
cohort of patients in the Collaborative Research Database V2.0 [46]. The
cardio_eicu dataset contains records for a cohort of patients in the MIMIC-III
[32] database. Here, the outcome variable indicates whether a patient in the
ICU with cardiogenic shock will die while in the ICU. The features encode the
results of vital signs and routine lab tests (e.g. systolic BP, heart rate,
hemoglobin count) that were collected up to 24 hours before the onset of
cardiogenic shock.

##### lungcancer

We consider a cohort of 120,641 patients who were diagnosed with lung cancer
between 2004-2016 and monitored as part of the National Cancer Institute SEER
study [47]. Here, the outcome variable indicates if a patient dies within five
years from any cause, and 16.9% of patients died within the first five years
from diagnosis. The cohort includes patients from Greater California, Georgia,
Kentucky, New Jersey, and Louisiana, and does not cover patients who were lost
to follow-up (censored). Age and Sex were considered as group attributes. The
features reflect the morphology and histology of the tumor (e.g., size,
metastasis, stage, node count and location, number and location of notes) as
well as interventions that were administered at the time of diagnosis (e.g.,
surgery, chemo, radiology).

##### coloncancer

We consider a cohort of 120,641 patients who were diagnosed with colorectal
cancer between 2004-2016 and monitored as part of the National Cancer
Institute SEER study [47]. Here, the outcome variable indicates if a patient
dies within five years from any cause, and 42.1% of patients die within the
first five years from diagnosis. The cohort includes patients from Greater
California. Age and Sex were considered as group attributes. The features
reflect the morphology and histology of the tumor (e.g., size, metastasis,
stage, node count and location, number and location of notes) as well as
interventions that were administered at the time of diagnosis (e.g., surgery,
chemo, radiology).

##### saps

The Simplified Acute Physiology Score II (SAPS II) score predicts the risk of
mortality of critically-ill patients in intensive care [37]. The data contains
records of 7,797 patients from 137 medical centers in 12 countries. Here, the
outcome variable indicates whether a patient dies in the ICU, with 12.8%
patient of patients dying. The features reflect comorbidities, vital signs,
and lab measurements.

###  Appendix C Results for Different Model Classes and Prediction Tasks

In this Appendix, we present experimental results for additional model classes
and prediction tasks. We produce these results using the setup in Section 4.1,
and summarize them in the same way as Table 2. We refer to them in our
discussion in Section 4.2.

####  C.1 Logistic Regression for Ranking (AUC)

| Static | Imputed | Participatory
---|---|---|---
Dataset | Metrics | 1Hot | mHot | KNN-1Hot | KNN-mHot | Minimal | Flat | Seq
| apnea
---
$n={1152},d={26}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Ustun et al. [58]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.774
---
-0.002
-0.002 – 0.002
0.004
2
-0.002
0/6
100.0%
| 0.774
---
-0.002
-0.002 – 0.003
0.005
2
-0.002
0/6
100.0%
| 0.776
---
0.000
-0.002 – 0.002
0.004
2
0/12
0.0%
| 0.776
---
-0.000
-0.002 – 0.003
0.005
2
0/12
0.0%
| 0.776
---
0.000
0.000 – 0.002
0.002
0
5/7
16.7%
| 0.851
---
0.074
0.004 – 0.115
0.111
0
4/12
100.0%
| 0.851
---
0.074
0.004 – 0.115
0.111
0
4/12
83.3%
| cardio_eicu
---
$n={1341},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Pollard et al. [46]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.864
---
0.002
-0.005 – 0.003
0.009
3
-0.005
0/8
100.0%
| 0.863
---
0.001
-0.010 – 0.010
0.019
3
-0.010
0/8
100.0%
| 0.863
---
0.000
-0.005 – 0.003
0.009
3
0/27
0.0%
| 0.862
---
-0.001
-0.010 – 0.010
0.019
3
0/27
0.0%
| 0.865
---
0.002
0.000 – 0.003
0.003
0
6/9
25.0%
| 0.966
---
0.103
0.010 – 0.180
0.170
0
13/27
100.0%
| 0.966
---
0.103
0.010 – 0.180
0.170
0
11/27
95.8%
| cardio_mimic
---
$n={5289},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Johnson et al. [32]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.881
---
0.000
-0.001 – 0.001
0.002
3
-0.001
0/8
100.0%
| 0.881
---
0.000
-0.001 – 0.001
0.002
3
-0.001
0/8
100.0%
| 0.882
---
0.002
-0.001 – 0.001
0.002
3
0/27
0.0%
| 0.880
---
-0.000
-0.001 – 0.001
0.002
3
0/27
0.0%
| 0.881
---
0.000
0.000 – 0.001
0.001
0
6/9
25.0%
| 0.914
---
0.034
0.008 – 0.057
0.049
0
9/27
100.0%
| 0.914
---
0.034
0.008 – 0.057
0.049
0
8/27
91.7%
| coloncancer
---
$n={29211},d={72}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.685
---
0.001
-0.001 – 0.002
0.003
3
-0.001
0/6
100.0%
| 0.685
---
0.002
-0.001 – 0.001
0.002
2
-0.002
0/6
100.0%
| 0.683
---
-0.000
-0.001 – 0.002
0.003
3
0/12
0.0%
| 0.683
---
-0.000
-0.001 – 0.001
0.002
2
0/12
0.0%
| 0.685
---
0.001
0.000 – 0.001
0.001
0
5/7
16.7%
| 0.700
---
0.016
0.001 – 0.021
0.020
0
2/12
100.0%
| 0.700
---
0.016
0.001 – 0.021
0.020
0
5/12
75.0%
| lungcancer
---
$n={120641},d={84}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.855
---
0.001
-0.000 – 0.000
0.001
2
-0.000
0/6
100.0%
| 0.855
---
0.001
-0.000 – 0.000
0.001
2
-0.000
0/6
100.0%
| 0.852
---
-0.002
-0.000 – 0.000
0.001
2
0/12
0.0%
| 0.854
---
0.000
-0.000 – 0.000
0.001
2
0/12
0.0%
| 0.855
---
0.001
0.000 – 0.000
0.000
1
4/7
33.3%
| 0.861
---
0.006
0.001 – 0.012
0.011
0
2/12
100.0%
| 0.861
---
0.006
0.001 – 0.012
0.011
0
2/12
91.7%
| saps
---
$n={7797},d={36}$
$\mathcal{G}=\\{{\footnotesize{\texttt{HIV}}},{\footnotesize{\texttt{age}}}\\}$
$|\mathcal{G}|={4}$ groups
Allyn et al. [3]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.875
---
0.010
-0.000 – 0.016
0.017
1
-0.000
0/4
100.0%
| 0.877
---
0.011
-0.002 – 0.019
0.021
1
-0.002
0/4
100.0%
| 0.875
---
0.010
-0.000 – 0.016
0.017
1
0/9
0.0%
| 0.857
---
-0.008
-0.002 – 0.019
0.021
1
0/9
0.0%
| 0.875
---
0.009
0.000 – 0.016
0.016
0
1/5
75.0%
| 0.960
---
0.095
0.035 – 0.141
0.106
0
2/9
100.0%
| 0.960
---
0.095
0.035 – 0.141
0.106
0
3/9
87.5%

Table 4: Overview of performance, data use, and consent for all personalized
models and systems on all datasets as measured by test auc. We show the
performance of models and systems built using logistic regression.

####  C.2 Random Forests for Decision-Making (Error)

| Static | Imputed | Participatory
---|---|---|---
Dataset | Metrics | 1Hot | mHot | KNN-1Hot | KNN-mHot | Minimal | Flat | Seq
| apnea
---
$n={1152},d={26}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Ustun et al. [58]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 26.3%
---
1.5%
-0.8% – 4.2%
5.0%
1
-1.2%
0/6
100.0%
| 26.0%
---
1.8%
0.4% – 3.8%
3.4%
0
-1.2%
0/6
100.0%
| 25.9%
---
1.9%
-0.8% – 4.2%
5.0%
1
0/12
0.0%
| 27.4%
---
0.4%
0.4% – 3.8%
3.4%
0
0/12
0.0%
| 26.3%
---
1.5%
0.0% – 4.2%
4.2%
0
2/7
66.7%
| 12.2%
---
15.6%
5.3% – 22.2%
16.9%
0
1/12
100.0%
| 12.2%
---
15.6%
5.3% – 22.2%
16.9%
0
2/12
91.7%
| cardio_eicu
---
$n={1341},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Pollard et al. [46]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 18.6%
---
-0.2%
-3.5% – 1.4%
4.9%
2
-3.5%
0/8
100.0%
| 17.8%
---
0.6%
-2.2% – 3.0%
5.3%
2
-2.2%
0/8
100.0%
| 18.2%
---
0.2%
-3.5% – 1.4%
4.9%
2
0/27
0.0%
| 18.6%
---
-0.2%
-2.2% – 3.0%
5.3%
2
0/27
0.0%
| 18.4%
---
0.0%
0.0% – 0.0%
0.0%
0
8/9
0.0%
| 5.7%
---
12.7%
6.0% – 14.9%
8.9%
0
11/27
100.0%
| 6.0%
---
12.4%
6.0% – 14.9%
8.9%
0
8/27
91.7%
| cardio_mimic
---
$n={5289},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Johnson et al. [32]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 19.9%
---
-0.3%
-1.1% – 1.3%
2.4%
5
-1.1%
0/8
100.0%
| 20.1%
---
-0.5%
-1.3% – 0.5%
1.7%
6
-1.3%
0/8
100.0%
| 19.9%
---
-0.3%
-1.1% – 1.3%
2.4%
5
0/27
0.0%
| 20.2%
---
-0.6%
-1.3% – 0.5%
1.7%
6
0/27
0.0%
| 19.6%
---
0.0%
0.0% – 0.0%
0.0%
0
8/9
0.0%
| 11.5%
---
8.1%
1.0% – 14.9%
13.8%
0
6/27
100.0%
| 11.4%
---
8.1%
1.0% – 14.9%
13.8%
0
5/27
87.5%
| coloncancer
---
$n={29211},d={72}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 37.2%
---
-0.2%
-0.7% – 0.1%
0.7%
4
-0.7%
0/6
100.0%
| 37.0%
---
0.0%
-0.3% – 0.2%
0.5%
1
-0.3%
0/6
100.0%
| 37.2%
---
-0.2%
-0.7% – 0.1%
0.7%
4
0/12
0.0%
| 37.0%
---
-0.0%
-0.3% – 0.2%
0.5%
1
0/12
0.0%
| 37.0%
---
0.0%
0.0% – 0.0%
0.0%
0
6/7
0.0%
| 35.9%
---
1.0%
0.1% – 3.2%
3.1%
0
3/12
100.0%
| 35.9%
---
1.0%
0.1% – 3.2%
3.1%
0
5/12
75.0%
| lungcancer
---
$n={120641},d={84}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 20.0%
---
0.1%
-0.3% – 0.2%
0.6%
1
-0.3%
0/6
100.0%
| 20.2%
---
-0.1%
-0.5% – 0.0%
0.5%
4
-0.5%
0/6
100.0%
| 20.0%
---
0.1%
-0.3% – 0.2%
0.6%
1
0/12
0.0%
| 20.3%
---
-0.2%
-0.5% – 0.0%
0.5%
4
0/12
0.0%
| 20.0%
---
0.1%
0.0% – 0.2%
0.2%
0
3/7
50.0%
| 19.3%
---
0.8%
0.0% – 2.3%
2.3%
0
1/12
100.0%
| 19.3%
---
0.7%
0.0% – 2.2%
2.1%
0
3/12
83.3%
| saps
---
$n={7797},d={36}$
$\mathcal{G}=\\{{\footnotesize{\texttt{HIV}}},{\footnotesize{\texttt{age}}}\\}$
$|\mathcal{G}|={4}$ groups
Allyn et al. [3]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 14.1%
---
0.9%
-0.8% – 3.4%
4.2%
1
-0.8%
0/4
100.0%
| 15.0%
---
-0.0%
-0.5% – 0.3%
0.8%
1
-0.7%
0/4
100.0%
| 14.1%
---
0.9%
-0.8% – 3.4%
4.2%
1
0/9
0.0%
| 15.7%
---
-0.7%
-0.5% – 0.3%
0.8%
1
0/9
0.0%
| 13.9%
---
1.1%
0.0% – 3.4%
3.4%
0
2/5
50.0%
| 9.8%
---
5.2%
0.0% – 16.4%
16.4%
0
1/9
75.0%
| 9.8%
---
5.2%
0.0% – 16.4%
16.4%
0
1/9
87.5%

Table 5: Overview of performance, data use, and consent for all personalized
models and systems on all datasets as measured by test error. We show the
performance of models and systems built using random forests.

####  C.3 Random Forests for Ranking (AUC)

| Static | Imputed | Participatory
---|---|---|---
Dataset | Metrics | 1Hot | mHot | KNN-1Hot | KNN-mHot | Minimal | Flat | Seq
| apnea
---
$n={1152},d={26}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Ustun et al. [58]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.825
---
0.008
-0.004 – 0.009
0.012
2
-0.004
0/6
100.0%
| 0.824
---
0.006
-0.005 – 0.012
0.017
3
-0.005
0/6
100.0%
| 0.822
---
0.004
-0.004 – 0.009
0.012
2
0/12
0.0%
| 0.806
---
-0.012
-0.005 – 0.012
0.017
3
0/12
0.0%
| 0.823
---
0.005
0.000 – 0.009
0.009
0
3/7
50.0%
| 0.944
---
0.126
0.058 – 0.157
0.098
0
2/12
100.0%
| 0.942
---
0.124
0.058 – 0.157
0.098
0
4/12
75.0%
| cardio_eicu
---
$n={1341},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Pollard et al. [46]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.896
---
0.003
-0.008 – 0.011
0.020
3
-0.008
0/8
100.0%
| 0.896
---
0.003
-0.005 – 0.011
0.016
4
-0.005
0/8
100.0%
| 0.897
---
0.004
-0.008 – 0.011
0.020
3
0/27
0.0%
| 0.886
---
-0.007
-0.005 – 0.011
0.016
4
0/27
0.0%
| 0.894
---
0.001
0.000 – 0.004
0.004
0
7/9
12.5%
| 0.987
---
0.094
0.010 – 0.132
0.122
0
10/27
100.0%
| 0.987
---
0.094
0.010 – 0.130
0.120
0
10/27
87.5%
| cardio_mimic
---
$n={5289},d={49}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}},{\footnotesize{\texttt{race}}}\\}$
$|\mathcal{G}|={8}$ groups
Johnson et al. [32]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.884
---
0.000
-0.005 – 0.006
0.011
3
-0.005
0/8
100.0%
| 0.883
---
-0.001
-0.006 – 0.013
0.019
7
-0.006
0/8
100.0%
| 0.884
---
0.001
-0.005 – 0.006
0.011
3
0/27
0.0%
| 0.881
---
-0.002
-0.006 – 0.013
0.019
7
0/27
0.0%
| 0.885
---
0.001
0.000 – 0.006
0.006
0
5/9
37.5%
| 0.955
---
0.071
0.016 – 0.108
0.092
0
6/27
100.0%
| 0.954
---
0.071
0.016 – 0.107
0.090
0
6/27
83.3%
| coloncancer
---
$n={29211},d={72}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.684
---
0.002
-0.002 – 0.004
0.006
0
-0.002
0/6
100.0%
| 0.682
---
0.000
-0.004 – 0.002
0.007
0
-0.004
0/6
100.0%
| 0.681
---
-0.001
-0.002 – 0.004
0.006
0
0/12
0.0%
| 0.680
---
-0.002
-0.004 – 0.002
0.007
0
0/12
0.0%
| 0.683
---
0.001
0.000 – 0.004
0.004
0
3/7
50.0%
| 0.696
---
0.014
0.004 – 0.035
0.030
0
2/12
100.0%
| 0.696
---
0.014
0.004 – 0.031
0.026
0
5/12
75.0%
| lungcancer
---
$n={120641},d={84}$
$\mathcal{G}=\\{{\footnotesize{\texttt{age}}},{\footnotesize{\texttt{sex}}}\\}$
$|\mathcal{G}|={6}$ groups
Scosyrev et al. [47]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.849
---
0.002
-0.001 – 0.003
0.004
1
-0.001
0/6
100.0%
| 0.849
---
0.001
-0.001 – 0.002
0.003
1
-0.001
0/6
100.0%
| 0.848
---
0.001
-0.001 – 0.003
0.004
1
0/12
0.0%
| 0.849
---
0.001
-0.001 – 0.002
0.003
1
0/12
0.0%
| 0.848
---
0.000
0.000 – 0.003
0.003
0
2/7
66.7%
| 0.856
---
0.008
0.002 – 0.020
0.018
0
1/12
100.0%
| 0.856
---
0.008
0.002 – 0.020
0.018
0
2/12
91.7%
| saps
---
$n={7797},d={36}$
$\mathcal{G}=\\{{\footnotesize{\texttt{HIV}}},{\footnotesize{\texttt{age}}}\\}$
$|\mathcal{G}|={4}$ groups
Allyn et al. [3]
| Overall Performance
---
Overall Gain
Group Gains
Max Disparity
Rat. Violations
Imputation Risk
Options Pruned
Data Use
| 0.921
---
0.003
-0.002 – 0.010
0.012
2
-0.002
0/4
100.0%
| 0.922
---
0.004
-0.002 – 0.013
0.015
2
-0.002
0/4
100.0%
| 0.922
---
0.003
-0.002 – 0.010
0.012
2
0/9
0.0%
| 0.906
---
-0.012
-0.002 – 0.013
0.015
2
0/9
0.0%
| 0.921
---
0.002
0.000 – 0.010
0.010
0
2/5
50.0%
| 0.966
---
0.048
0.009 – 0.109
0.100
0
2/9
100.0%
| 0.966
---
0.048
0.009 – 0.109
0.100
0
2/9
87.5%

Table 6: Overview of performance, data use, and consent for all personalized
models and systems on all datasets as measured by test auc. We show the
performance of models and systems built using random forests.