# From Biased Selective Labels to Pseudo-Labels:
An Expectation-Maximization Framework for Learning from Biased Decisions

Trenton Chang  Jenna Wiens

###### Abstract

Selective labels occur when label observations are subject to a decision-
making process; _e.g._ , diagnoses that depend on the administration of
laboratory tests. We study a clinically-inspired selective label problem
called disparate censorship, where labeling biases vary across subgroups and
unlabeled individuals are imputed as “negative” (i.e., no diagnostic test = no
illness). Machine learning models naïvely trained on such labels could amplify
labeling bias. Inspired by causal models of selective labels, we propose
Disparate Censorship Expectation-Maximization (DCEM), an algorithm for
learning in the presence of disparate censorship. We theoretically analyze how
DCEM mitigates the effects of disparate censorship on model performance. We
validate DCEM on synthetic data, showing that it improves bias mitigation
(area between ROC curves) without sacrificing discriminative performance (AUC)
compared to baselines. We achieve similar results in a sepsis classification
task using clinical data.

disparate censorship, selective labels, machine learning in healthcare, semi-
supervised learning, healthcare

Figure 1: Top: Causal model of disparate censorship ($\mathbf{x}$: covariates,
$y$: ground truth, $\tilde{y}$: observed label, $t$: testing/labeling
indicator, $a$: sensitive attribute). Shaded variables are fully observed.
Bottom: Disparate Censorship Expectation-Maximization (DCEM). Dashed nodes are
probabilistic estimates.

##  1 Introduction

Selective labels occur when a decision-making process determines access to
ground truth (Lakkaraju et al., 2017). We study a practical case of selective
labels: disparate censorship (Chang et al., 2022). Disparate censorship
introduces two challenges: different labeling biases across subgroups and the
assumption that unlabeled individuals have a negative label. For example, in
healthcare, labels may depend on laboratory test results only available in
some patients.

Past work has trained ML models to predict outcomes based on laboratory test
results (e.g., sepsis (Seymour et al., 2016; Rhee & Klompas, 2020)). In this
setting, patients with no test result are defined as negative (Hartvigsen et
al., 2018; Teeple et al., 2020; Jehi et al., 2020; McDonald et al., 2021;
Adams et al., 2022; Kamran et al., 2022). However, laboratory testing
decisions may be biased. For example, women are undertested and underdiagnosed
for cardiovascular disease (Beery, 1995; Schulman et al., 1999). ML models
trained on such data may recommend women less often for diagnostic testing
than men, reinforcing inequity.

To address this bias, one option is to train only on tested individuals. Such
an approach may discard a large subset of the data and may not generalize to
untested patients. Another option is semi-supervised approaches that do not
assume untested patients are negative, such as label propagation (Zhu &
Ghahramani, 2002; Lee, 2013) or filtering (Li et al., 2020; Nguyen et al.,
2020), or noisy-label learning methods (Blum & Stangl, 2020; Wang et al.,
2021; Zhu et al., 2021). However, such methods do not leverage causal models
of label bias, a potential source of additional information. We aim to develop
an approach that leverages all available signal while accounting for labeling
biases.

Inspired by causal models of selective labeling (Laine et al., 2020; Chang et
al., 2022; Guerdan et al., 2023a), we propose a simple method for mitigating
bias when training models under disparate censorship: Disparate Censorship
Expectation-Maximization (DCEM; Fig. 1). First, we show that DCEM regularizes
model estimates to counterbalance disparate censorship. We validate DCEM in a
simulation study and a sepsis classification task on clinical data. We find
that our method mitigates bias (area between ROC curves) while maintaining
competitive discriminative performance (AUC), and is generally more robust
than baselines to changes in the data generation process.

##  2 Preliminaries: Disparate Censorship

We consider a dataset
$\\{\mathbf{x}^{(i)},\tilde{y}^{(i)},t^{(i)},a^{(i)}\\}_{i=1}^{N}$, with
covariates $\mathbf{x}^{(i)}\in\mathbb{R}^{d}$, labeling/testing decision
$t^{(i)}\in\\{0,1\\}$, sensitive attribute $a^{(i)}$, and observed label
$\tilde{y}^{(i)}\in\\{0,1\\}$, a proxy for ground truth $y^{(i)}\in\\{0,1\\}$.
The proxy label $\tilde{y}^{(i)}=y^{(i)}$ when $t^{(i)}=1$, and
$\tilde{y}^{(i)}=0$ otherwise (_i.e._ , $\tilde{y}^{(i)}=y^{(i)}t^{(i)}$).

#### What is disparate censorship?

Disparate censorship models “double standards” in label collection decisions
(Fig. 1, top). It is a variation of selective labeling or outcome measurement
error (Lakkaraju et al., 2017; Guerdan et al., 2023b). Disparate censorship
uniquely assumes that untested individuals are imputed as _negative._

We consider disparate censorship in the context of binary classification
(Chang et al., 2022) (Fig. 1, top). We justify the model by example. Consider
a patient in an emergency room with characteristics $\mathbf{x}$ and sensitive
attribute $a$. This patient may have some condition $y$ (currently unobserved)
caused by $\mathbf{x}$ but _not_ $a$. A clinician may order a diagnostic test
(set $t$ to 1) to determine $y$. The decision is based on $\mathbf{x}$, but
could be swayed by biases in $a$.

To simplify, suppose that tests are perfectly sensitive.111If not, we can
define $T$ to indicate whether a label is confirmed correct. This definition
captures differences in test sensitivity across groups (_i.e._ , spectrum bias
(Mulherin & Miller, 2002)). Then, we observe ground truth for tested
individuals ($t=1\implies\tilde{y}=y$). Otherwise, the patient’s label is
imputed as negative ($t=0\implies\tilde{y}=0$; _i.e._ , untested patients are
presumed healthy). However, due to biases in testing decisions $t$, $y$ may
only be available in a biased subset of the data. The causal model of
disparate censorship (Fig. 1, top) encodes this decision-making pipeline.
Beyond healthcare, disparate censorship may arise whenever potentially biased
decisions affect data labeling.

#### Learning under disparate censorship.

We aim to learn a mapping $f_{\theta}:\mathbf{x}\to y$ parameterized by
$\theta$ optimized for discriminative performance (_i.e._ , AUC), but only
observe proxy labels $\tilde{y}$. The default approach for learning under
disparate censorship is to assume $y=\tilde{y}$ and proceed using supervised
learning. However, such an $f_{\theta}$ may encode labeling biases: estimates
of $P(\tilde{Y}\mid X)$ may be inflated compared to $P(Y\mid X)$ for those
more likely to be labeled. Thus, biased labeling could yield disproportionate
impacts on performance across different subgroups of the data.

Note that we can interpret the estimand of interest as the causal effect of
testing on the observed label, since, in the language of do-calculus (Pearl,
2009),

$\displaystyle\mathbb{E}[Y\mid X]$ $\displaystyle=\mathbb{E}[Y\mid
X,T=1]=\mathbb{E}[\tilde{Y}\mid X,T=1]$
$\displaystyle=\mathbb{E}[\tilde{Y}\mid X,do(T=1)],$ (1)

which follows from standard causal identifiability derivations given the
causal graph of Fig. 1 (Imbens & Rubin, 2015). Intuitively, testing an
individual ($do(T=1)$) reveals their outcome. Thus, a model trained only on
tested individuals could consistently estimate $P(Y|X)$, but may not correct
for labeling bias. We discuss other approaches in semi-supervised learning in
Section 6.

##  3 Methodology

We propose Disparate Censorship Expectation-Maximiza-tion (DCEM) as an
approach for learning in the presence of disparate censorship. We first build
intuition for how one could mitigate disparate censorship based on the causal
model (Section 3.1). We then derive DCEM (Section 3.2) and show that it
mitigates disparate censorship via a form of regularization (Section 3.3). We
consider alternative designs and their limitations (Section 3.4). Detailed
proofs and definitions are in Appendix B.

###  3.1 Towards mitigating disparate censorship

Recalling the causal model of disparate censorship, suppose that we are
naively training a model $f_{\theta}$ to predict $\tilde{y}$. Define groups
$a$ and $a^{\prime}$ and $\mathbf{x}\sim\mathcal{X}$. Consider some
$\mathcal{X}^{\prime}\subseteq\mathcal{X}$ so that

$\underset{\mathbf{x}\in\mathcal{X}^{\prime}}{\mathbb{P}}[T\mid
X,A=a]<<\underset{\mathbf{x}\in\mathcal{X}^{\prime}}{\mathbb{P}}[T\mid
X,A=a^{\prime}]$ (2)

for all $\mathbf{x}\in\mathcal{X}^{\prime}$. Define $\hat{t}\triangleq P(T\mid
X=\mathbf{x},A=a)$ (_e.g._ , probability of receiving a laboratory test) and
$\hat{y}\triangleq P(Y\mid X=\mathbf{x})$. By assumption, $\mathbf{x}$ is
sufficient for predicting $y$ (_i.e._ , as in Fig. 1, top), such that the
optimal $\hat{y}$ should be similar across $a$ (within
$\mathcal{X}^{\prime}$). However, Eq. 2 states that group $a$ is undertested
relative to group $a^{\prime}$: they have a lower $\hat{t}$ within
$\mathcal{X}^{\prime}$. Equivalently, labeling bias _favors_ group
$a^{\prime}$. Thus, our naive model would underestimate $\hat{y}$ in group $a$
(lower $\hat{t}$ than group $a^{\prime}$ in $\mathcal{X}^{\prime}$) relative
to group $a^{\prime}$.

To counterbalance this bias, one could increase $\hat{y}$ (within
$\mathcal{X}^{\prime}$) where group $a$ is more prevalent than group
$a^{\prime}$; _i.e._ , lower-$\hat{t}$ regions. Since we are interested in
discriminative performance, this is analogous to decreasing $\hat{y}$ where
$\hat{t}$ is higher, from which the proposed method follows. More broadly,
variables associated with labeling bias ($A$ causally affects $T$) but not the
outcome of interest ($A$ does not causally affect $Y$) may be useful for
mitigating labeling bias.

Given our causal model with latent variable $Y$ (Fig. 1, top), we base our
approach on expectation-maximization (EM) (Dempster et al., 1977). We can
write:

$\displaystyle P(\tilde{Y},Y,T,X,A,U)$ $\displaystyle=P(\tilde{Y}\mid
Y,T)P(Y\mid X)P(T\mid X,A)P(X,A,U).$ (3)

Since $y$ is not fully observed, Eq. 3 cannot be optimized via standard
supervised objectives. Dropping terms that do not involve $Y$, we can write
the maximization of Eq. 3 as

$\displaystyle\max\;P(\tilde{Y}\mid Y,T)P(Y\mid X).$ (4)

Optimizing Eq. 4 proceeds via EM. We show that the resulting objectives align
with reducing $\hat{y}$ in higher-$\hat{t}$ regions and maintain
discriminative performance on tested individuals.

Algorithm 1 Disparate Censorship Expectation-Maximiza-tion. $\mathcal{L}$:
binary cross-entropy loss.

Input: $\\{(\mathbf{x}^{(i)},\tilde{y}^{(i)},t^{(i)},a^{(i)})\\}_{i=1}^{N}$

Output:
$f_{\theta}:\mathcal{X}\to[0,1]$$f_{\theta}\leftarrow\underset{f_{\theta}}{\arg\min}\frac{1}{|\\{i:t^{(i)}=1\\}|}\sum_{i:t^{(i)}=1}\mathcal{L}(\tilde{y}^{(i)},f_{\theta}(\mathbf{x}^{(i)}))$$g_{\zeta^{*}}\leftarrow\underset{g_{\zeta}}{\arg\min}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(t^{(i)},g_{\zeta}(\mathbf{x}^{(i)},a^{(i)}))$$\hat{t}^{(i)}\leftarrow
g_{\zeta^{*}}(\mathbf{x}^{(i)},a^{(i)})$

while not converged do

$Q(y^{(i)})\leftarrow
t^{(i)}\tilde{y}^{(i)}+(1-t^{(i)})f_{\theta}(\mathbf{x}^{(i)})$ // E-step

$f_{\theta}\leftarrow\underset{f_{\theta}}{\arg\min}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(Q(y^{(i)}),f_{\theta}(\mathbf{x}^{(i)}))$

$\quad+Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},f_{\theta}(\mathbf{x}^{(i)})\cdot\hat{t}^{(i)})$
// M-step

end while

return $f_{\theta}$

###  3.2 Disparate Censorship Expectation-Maximization

#### Informal overview.

EM alternates an expectation step (E-step), which imputes guesses for the
latent variable(s) (_i.e._ , $Y$ in Eq. 4), and a maximization step that
optimizes likelihood given the imputed estimates (M-step, _i.e._ , Eq. 4). Our
E-step imputes preliminary estimates of $P(Y\mid X)$ for untested individuals.
Our M-step updates the estimates to counteract labeling biases when
$t^{(i)}=0$, and is equivalent to full supervision when $t^{(i)}=1$. The E-
and M-steps alternate until convergence. Fig. 1 (bottom) shows a schematic of
DCEM, with pseudocode in Algorithm 1.

#### E-step.

The posterior of $y^{(i)}$ given the observed data is:

$Q(y^{(i)})\triangleq\mathbb{E}[y^{(i)}\mid\tilde{y}^{(i)},t^{(i)},\mathbf{x}^{(i)},a^{(i)}].~{}$
(5)

We can rewrite Eq. 5 as follows:

######  Theorem 3.1 (E-step).

The posterior distribution of $y^{(i)}$ given the observed data is equivalent
to

$\displaystyle Q(y^{(i)})$
$\displaystyle=\begin{cases}\tilde{y}^{(i)}&t^{(i)}=1\\\
\mathbb{E}[y^{(i)}=1\mid\mathbf{x}^{(i)}]&\text{otherwise}\end{cases}$ (6)

Intuitively, the E-step uses $\tilde{y}$ as the label when we have complete
label information (recall $t^{(i)}=1\implies\tilde{y}^{(i)}=y^{(i)}$);
otherwise, we use the posterior estimate $f_{\theta}(\mathbf{x}^{(i)})$ as a
smoothed label. Equivalently, the E-step imputes _soft pseudo-labels_ for
unlabeled data, _i.e._ , probabilistic estimates $\hat{y}^{(i)}\in[0,1]$.
Motivated by approaches that train a pseudo-labeling model on labeled data
(Arazo et al., 2020; Rizve et al., 2021), we pre-train $f_{\theta}$ on tested
individuals.

#### M-step.

The M-step maximizes the log-likelihood of Eq. 4 given E-step estimates
$Q(y^{(i)})$ (Eq. 6). There are two terms to model, which is done via an
estimator for $y^{(i)}$ trained using $Q(y^{(i)})$ and an estimator for
$\tilde{y}^{(i)}$. The latter is obtained by combining an estimate of
$t^{(i)}$ with $Q(y^{(i)})$. Concretely, let $\hat{y}^{(i)}\triangleq
f_{\theta}(\mathbf{x}^{(i)})$, and let $h_{\phi}$ be a model of
$P(\tilde{Y}\mid Y,T)$. Maximizing the log-likelihood of Eq. 4 reduces to

$\displaystyle\underset{\theta}{\max}\;\sum_{i=1}^{N}Q(y^{(i)})\log\hat{y}^{(i)}+(1-Q(y^{(i)}))\log(1-\hat{y}^{(i)})$
$\displaystyle\quad+Q(y^{(i)})\left[\tilde{y}^{(i)}\log
h_{\phi}(\hat{y}^{(i)},\hat{t}^{(i)})\right.$
$\displaystyle\quad\left.+(1-\tilde{y}^{(i)})\log(1-h_{\phi}(\hat{y}^{(i)},\hat{t}^{(i)}))\right].$
(7)

This leads to the following result:

######  Theorem 3.2 (M-step, informal).

Maximizing Eq. 7 also maximizes the evidence-based lower bound of Eq. 3.

In practice, we set
$h_{\phi}(\hat{y}^{(i)},\hat{t}^{(i)})\triangleq\hat{y}^{(i)}\hat{t}^{(i)}$, a
smoothed version of the assumption $\tilde{y}=yt$. Defining $\mathcal{L}$ as
binary cross-entropy loss, we can rewrite Eq. 7:

$\underset{\theta}{\min}\;\sum_{i=1}^{N}\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})\\!+\\!Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)}).$
(8)

Eq. 8 can be interpreted as a regularized cross-entropy loss with respect to
pseudo-label $Q(y^{(i)})$. The first term pushes $\hat{y}^{(i)}$ towards
$Q(y^{(i)})$, while the second “encourages” $\hat{y}^{(i)}$ to be consistent
with the causal model. To obtain $\hat{t}$, we pre-train and freeze a binary
classifier for $t$, and take the probabilistic estimates as $\hat{t}$.

###  3.3 DCEM counterbalances disparate censorship

We show that DCEM imposes a form of “causal regularization” that lowers
$\hat{y}$ in untested individuals.

#### DCEM is a form of causal regularization.

By analogy to regularized risk minimization, consider an objective

$\mathcal{L}(\theta)+\lambda r(\theta),$ (9)

for $\lambda\in\mathbb{R}_{+}$ (regularization strength) and a regularizer
$r:\Theta\to\mathbb{R}$, where $\Theta$ is the parameter space of $\theta$.

Without loss of generality, setting $\lambda=1$ and matching terms between Eq.
9 and Eq. 8 yields
$r(\theta)=Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)})$.
While $\hat{t}^{(i)}$ affects the optimization of Eq. 8, it is not a
multiplier (_e.g._ , $\lambda$ in Eq. 9). To interpret the effect of
$\hat{t}^{(i)}$, we propose a definition of causal regularization strength
based how the optimal $\hat{y}^{(i)}$ changes.222“Causal regularization” has
been defined in the context of causal discovery (Bahadori et al., 2017;
Janzing, 2019). Our usage is unrelated: we use a causal model to regularize an
estimator.

######  Definition 3.3 (Causal regularization strength, informal).

Let $\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ be the minimizer of
Eq. 8. For $\mathcal{L}$ finite & convex on $\hat{y}^{(i)}$ in [0, 1], the
causal regularization strength is
$R(\cdot)\triangleq|Q({y}^{(i)})-\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})|$.

Definition 3.3 quantifies the tradeoff between matching $\hat{y}^{(i)}$ to the
E-step estimates and optimizing Eq. 8. While $\hat{y}^{(i)}$ is not an
optimization parameter, analyzing the optimal $\hat{y}^{(i)}$ can clarify the
inductive bias of the M-step. We proceed by considering how causal
regularization impacts untested vs. tested individuals. When $t^{(i)}=0$, the
M-step is

$\underset{\theta}{\min}\;\sum_{i=1}^{N}\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})-Q(y^{(i)})\log(1-\hat{y}^{(i)}\hat{t}^{(i)}).$
(10)

Since $-\log(1-\hat{y}^{(i)}\hat{t}^{(i)})$ increases in $\hat{y}^{(i)}$, the
regularization term “encourages” $\hat{y}^{(i)}$ to decrease when
$\hat{t}^{(i)}>0$. The regularization term is constant if $\hat{t}^{(i)}=0$,
such that the M-step would not change the E-step estimate. This matches the
intuition that one cannot learn about $y^{(i)}$ from individuals that are very
different from labeled individuals (_i.e._ , when the _overlap_ assumption in
causal inference is violated). The regularization strength depends on
$\hat{t}^{(i)}$ as follows:

######  Theorem 3.4 (informal).

If $t^{(i)}=0$, causal regularization strength increases in $\hat{t}^{(i)}$.

The result implies that causal regularization counterbalances disparate
censorship. Recall that lowering $\hat{y}^{(i)}$ in regions where
$\hat{t}^{(i)}$ is higher can mitigate bias. Equivalently, causal
regularization must strengthen as $\hat{t}^{(i)}$ increases, which follows
from Theorem 3.4.

#### Causal regularization aligns with full supervision in tested individuals.

When $t^{(i)}=1$, the M-step is

$\displaystyle\underset{\theta}{\min}$
$\displaystyle\;\sum_{i=1}^{N}\mathcal{L}(y^{(i)},\hat{y}^{(i)})+y^{(i)}\mathcal{L}(y^{(i)},\hat{y}^{(i)}\hat{t}^{(i)}),~{}$
(11)

substituting $y^{(i)}$ for $Q(y^{(i)})$ and $\tilde{y}^{(i)}$. Thus:

######  Proposition 3.5.

Eq. 11 is minimized when $\hat{y}^{(i)}=y^{(i)}$.

Proposition 3.5 states that causal regularization does not change the M-step
optimum from matching ground truth when $t^{(i)}=1$ (_i.e._ , regularization
strength = 0). Thus, the M-step objective aligns with fully-supervised loss.

Thus, the M-step (Eq. 8) counterbalances disparate censorship by regularizing
$\hat{y}^{(i)}$ towards 0 as $\hat{t}^{(i)}$ increases. For $t^{(i)}=1$, the
M-step optimum stays constant, and DCEM should maintain discriminative
performance.

###  3.4 Alternative designs and their limitations

We consider two alternative designs and their limitations: directly using
$t^{(i)}$ in DCEM and propensity score adjustment.

#### Why not use $t^{(i)}$ directly?

We substitute $\hat{t}^{(i)}=t^{(i)}$ into Eq. 8 and analyze one summand
(without loss of generality):

$\displaystyle\mathcal{L}(y^{(i)},\hat{y}^{(i)})+y^{(i)}\mathcal{L}(y^{(i)},\hat{y}^{(i)})$
$\displaystyle\quad t^{(i)}=1$ (12)
$\displaystyle\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})$ $\displaystyle\quad
t^{(i)}=0$ (13)

Both losses use the E-step estimate $Q(y^{(i)})$ as supervision. When
$t^{(i)}=1$ (Eq. 12), the M-step adds
$y^{(i)}\mathcal{L}(y^{(i)},\hat{y}^{(i)})$, penalizing false negatives 2x as
heavily as false positives. This does not affect ranking metrics (_e.g._ ,
AUC). When $t^{(i)}=0$ (Eq. 13), the M-step drops causal regularization, and
thus cannot counterbalance disparate censorship. Directly using $t^{(i)}$
would only help if counterbalancing disparate censorship is unnecessary for
good estimation, _i.e._ , when tested individuals are representative of the
population.

#### Why not propensity score adjustment/related causal approaches?

Recall that estimating the effect of $T$ on the observed label yields a
consistent estimate of $P(Y\mid X)$ (Eq. 1, Section 2). Indeed,
$\hat{t}^{(i)}$ is an estimate of $P(T\mid X,A)$, _i.e._ , a propensity score,
motivating the usage of causal effect estimators that leverage
$\hat{t}^{(i)}$. However, propensity score adjustment (_e.g._ , IPW (Rosenbaum
& Rubin, 1983) or doubly-robust variations (Robins et al., 1994; Van Der Laan
& Rubin, 2006; Hu et al., 2022)) require an “overlap” assumption
$\eta<\hat{t}^{(i)}<1-\eta$ for some $\eta=(0,\frac{1}{2})$ and have
asymptotic variance of order $O(1/(\eta\cdot(1-\eta))$, which is sensitive to
extreme $\hat{t}^{(i)}$ (_e.g._ , as in AIPW (Glynn & Quinn, 2010)).

However, in finite-sample settings, “sharp” testing decisions lead to weak
overlap. Such extreme $\hat{t}^{(i)}$ may arise in threshold-based decisions
(Djulbegovic et al., 2014; Pierson et al., 2018). For example, a patient
either exhibits or does not exhibit the requisite symptoms to warrant testing.
This is analogous to inducing covariate shift between tested and untested
individuals. In other words, “holes” in the training data emerge when using
only labeled examples. Thus, systematic testing bias could exacerbate model
performance gaps across population subgroups. While low overlap still impacts
DCEM (since DCEM cannot learn when $\hat{t}^{(i)}=0$), our method instead
leverages an evidence-based lower bound to model $y$ under disparate
censorship. We further discuss potential improvements in overlap-robustness of
the proposed approach in Appendix B.

##  4 Experimental Setup

We validate DCEM with synthetic data across different data-generation
processes on simulated binary classification tasks (Section 4.1) and in a
pseudo-synthetic sepsis classification task using real clinical data (MIMIC-
III) (Johnson et al., 2016), across potential laboratory testing policies
(Section 4.2). We then discuss our chosen baselines (Section 4.3) and
evaluation metrics (Section 4.4).

###  4.1 Synthetic Datasets

By definition, $y$ is not fully observed under disparate censorship. Thus, we
design a simulation study in order to evaluate various methods with respect to
ground truth. The data generation process follows from the assumed causal
model of disparate censorship (Fig. 1, top):

$\displaystyle\quad a^{(i)}\sim
Ber(0.5),\mathbf{x}^{(i)}\sim\mathcal{N}(\mu_{a}\cdot\mathbf{1}_{2},0.03^{2}\mathbf{I}_{2\times
2})$ $\displaystyle\quad t^{(i)}\sim Ber(\sigma(30\cdot
s_{T}(\mathbf{x}^{(i)},a^{(i)})))$ $\displaystyle\quad y^{(i)}\sim
Ber(\sigma(10\cdot
s_{Y}(\mathbf{x}^{(i)})-c_{y})),\;\tilde{y}^{(i)}=y^{(i)}t^{(i)}$

where $\mathbf{I}_{2\times 2}$ is the identity matrix, and
$s_{T}:\mathbf{x},a\to\mathbb{R}$, $s_{Y}:\mathbf{x}\to\mathbb{R}$, and
$\mu_{a}\in\mathbb{R},c_{y}\in\mathbb{R}$ are simulation parameters. We set
$P(A=0)=0.5$ and induce confounding between $\mathbf{x}^{(i)}$ and $a^{(i)}$
by setting $u^{(i)}=a^{(i)}$. We draw $\mathbf{x}^{(i)}\in\mathbb{R}^{2}$ from
group-specific Gaussians, and assume Bernoulli-distributed $t^{(i)}$ and
$y^{(i)}$ with parameters defined via $s_{T}:\mathbf{x},a\to\mathbb{R}$ and
$s_{Y}:\mathbf{x}\to\mathbb{R}$, respectively. Intuitively, $s_{T}$ ($s_{Y}$)
is a soft “decision boundary” for $T$ ($Y$). Inspired by observations that
clinician testing is can be represented by simpler functions than observed
outcomes (Mullainathan & Obermeyer, 2022), we choose a non-linear $s_{Y}$ and
a linear $s_{T}$.

We simulate $N=20,000$ individuals for training, validation, and testing each
(_i.e._ , 60,000 total). We define settings in terms of testing disparity
$q_{t}=\frac{P(T\mid A=0)}{P(T\mid A=1)}$, prevalence disparity
$q_{y}=\frac{P(Y\mid A=0)}{P(Y\mid A=1)}$, and testing multiple
$k=\frac{P(T=1)}{P(Y=1)}$. Intuitively, $q_{t}$ controls labeling biases,
$q_{y}$ controls differences between groups, and $k$ controls testing rate. We
consider $q_{t}\in\\{1/4,1/3,1/2,1,2,3,4\\}$, $q_{y}\in\\{1/4,1/3,1/2,1\\}$,
and $k\in\\{1/4,1/3,1/2,1,2,3,4\\}$, and set simulation parameters to yield
the desired $q_{t},q_{y},k$.333We skip settings where $q_{t},q_{y},k$ yield
infeasible testing rates.

Since $s_{Y}$ is unknown in practice, we replicate the main experiments across
various $s_{Y}$ as a robustness check. The simulation makes simplifying
assumptions (_e.g._ , low dimensionality and $P(A=0)=0.5$) but allows full
control over $y$ and $t$. Additional simulation details are in Appendix C.1.

###  4.2 Clinical data: MIMIC-III

Multiple sepsis definitions, such as Sepsis-3 (Singer et al., 2016), are based
on laboratory tests (blood culture) such that patients without a test result
are _by definition_ negative. Thus, sepsis classification is a potential real-
world case of disparate censorship. We curate a sepsis classification task
using the MIMIC-III Sepsis-3 cohort (Johnson et al., 2016, 2018), an
electronic health record dataset.

We aim to distinguish patients who never develop sepsis from those who develop
sepsis within 8 hours of an initial 3-hour observation period. If a patient
met the Sepsis-3 criteria between 3-11 hours of the first chart measurement,
we set $y=1$, and $y=0$ if the patient never develops sepsis during their
hospital stay. We exclude patients with onset times outside this range and
include only White and Black patients to simplify the analysis of bias
mitigation. We choose $\mathbf{x}\in\mathbb{R}^{13}$ following an existing
sepsis prediction model (Delahanty et al., 2019), and exclude patients where
all features are missing. This yields $N=5,301$ patients, from which we create
a 60-20-20 train-validation-test split. This is a simplified version of a real
clinical task, since we exclude patients who develop sepsis later during their
hospitalization. Nonetheless, it is helpful for probing the strengths and
weaknesses of the proposed approach.

To evaluate model performance, we assume that the observed $y$ reflects ground
truth, since $\approx 90\%$ of patients were tested (_i.e._ , received a blood
culture) in our cohort. To generate label proxies $\tilde{y}$, we simulate
multiple potential labeling biases via a clinically-inspired testing function
$s_{T}$. We specify a linear $s_{T}$ based on qSOFA, a score for triaging
patients at risk of sepsis (Seymour et al., 2016). Inspired by observations
that clinicians over-weight representative symptoms in diagnostic test
decisions (Mullainathan & Obermeyer, 2022), we create different versions of
$s_{T}$ via different weightings of qSOFA features. We examine
$k\in\\{1/4,1/3,1/2,1,2,3,4,5\\}$ and $q_{t}\in\\{1/2,2/3,1,3/2,2\\}$. Details
of the sepsis cohort are in Appendix C.2.

###  4.3 Models

As naive baselines, we test a $y$-obs model (training on $\tilde{y}$) and
training on group $a$ only. We select similarly-motivated or applicable
baselines from related settings:

  * •

Noisy-label learning: Group peer loss (Wang et al., 2021) (Appendix: Peer loss
(Liu & Guo, 2020), truncated $\ell_{q}$ loss (Zhang & Sabuncu, 2018) and
generalized Jensen-Shannon loss (Englesson & Azizpour, 2021)),

  * •

Semi-supervised learning: SELF (Nguyen et al., 2020) (Appendix: DivideMix (Li
et al., 2020)),

  * •

Causal inference: tested-only (training on examples where $t=1$), and
DragonNet (Shi et al., 2019), using the treatment effect of the sensitive
attribute on testing to correct disparate censorship (_i.e._ , learn a
correction for $P(Y\mid X)-P(\tilde{Y}\mid X,A)$),

  * •

Positive-unlabeled learning: Selected-At-Random EM (SAREM) (Bekker et al.,
2020).

As an oracle, we compare to training on $y$ (“$y$-model”). We use neural
networks for all approaches. Training details, such as hyperparameters, are in
Appendix D.

Figure 2: Comparison of ROC gap (left) and AUC (right) of selected models at
$q_{y}=0.5,k=1,q_{t}=2$. Each point represents a different $s_{Y}$. Our method
(DCEM, magenta) mitigates bias while maintaining competitive AUC compared to
baselines, with a tighter range and improved empirical worst-case for both
metrics. “-”: median, “$\bigtriangleup$”: worst-case ROC gap,
“$\bigtriangledown$”: worst-case AUC.

###  4.4 Evaluation metrics

We consider bias mitigation and discriminative performance metrics with
respect to $y$, and measure the robustness of both metrics to changes in the
data-generation process.

#### Discriminative performance.

We use the area under the receiver operating characteristic curve (AUC), a
standard discriminative performance metric.

#### Mitigating bias.

We use the ROC gap (also called ROC fairness (Vogel et al., 2021) or ABROCA
(Gardner et al., 2019)), the absolute area between the ROC curves for each
group $a$. The ROC gap is in [0, 1]. Lower values indicate better bias
mitigation. Intuitively, the ROC gap is zero when a classifier with some fixed
false positive rate in each group obtains equal true positive rates across
groups. Under disparate censorship, a zero ROC gap is achievable if a model
perfectly predicts $y$ from $\mathbf{x}$.

#### Robustness.

We consider the median AUC and ROC gap over all $s_{Y}$ (synthetic data
setting) or $s_{T}$ (sepsis classification) and the empirical worst-case (AUC:
min.; ROC gap: max.) and range.

Figure 3: Relative frequencies of ROC gaps for DCEM vs. tested-only models at
similar AUC (increasing to the right), pooling models across all
$k,q_{y},q_{t}$ tested. Dashed lines = mean ROC gap by model. DCEM improves
bias mitigation among models with similar AUC.

##  5 Experiments & Discussion

Our experiments aim to substantiate our main claims:

  * •

In synthetic data, DCEM mitigates bias, maintains competitive discriminative
performance and improves robustness, while achieving better tradeoffs between
performance and bias mitigation compared to baselines (Section 5.1).

  * •

On a sepsis classification task, DCEM improves discriminative performance
while maintaining good tradeoffs with bias mitigation, and is more robust
compared to baselines (Section 5.2).

We also report full results (Appendix E.1) and an ablation study of DCEM
(Appendix E.2). We also benchmark causal effect estimators (_i.e._ , as
alternatives to the tested-only model) and their overlap robustness compared
to DCEM (Appendix E.3). Further sensitivity analyses can be found in Appendix
E.4 (smoothed $\hat{t}^{(i)}$) and E.5 (E-step initialization).

###  5.1 Results on simulated disparate censorship

Fig. 2 shows ROC gaps (left) and AUCs (right) of the baselines most
competitive with our approach (DCEM, magenta) at $q_{y}=0.5,k=1,q_{t}=2$. In
this setting, 25% (_i.e._ , $k\cdot P(Y=1)$) of individuals are tested, and
the base rate of $Y$ in group $a=0$ is $1/2$ that of group $a=1$, but group
$a=0$ is twice as likely to be tested. Each point is an ROC gap/AUC value
achieved under one decision boundary $s_{Y}$. Results for the remaining
baselines are in Appendix E.1. The takeaways align with the main results.

#### DCEM mitigates bias more effectively than baselines.

DCEM achieves a median ROC gap of 0.030 (2nd-best, SELF: 0.034), suggesting
that it mitigates bias more effectively than baselines (Fig. 2, left). We show
similar trends for $q_{t}\geq 1$, $q_{y}$, and $k\in[1/3,2]$ (Appendix E.1).
At low testing rates, all models mitigate bias poorly. At high testing rates,
the tested-only model is sufficient.

For $q_{t}<1$ (Appendix E.1), DCEM mitigates bias compared to the default
approach ($y$-obs model) but no longer dominates the baselines. We hypothesize
that DCEM has similar bias mitigation capabilities as baselines, since there
is less bias to mitigate. Recalling that $q_{y}<1$, since $q_{t}<1$, testing
probability, $P(Y\mid X)$ _and_ $P(\tilde{Y}\mid X)$ are correlated. Learning
to predict $\tilde{Y}$ would preserve ordering in $P(Y\mid X)$, reducing
impacts on ranking metrics (_e.g._ , ROC gap).444Such settings are related to
boundary-consistent noise; see Proposition 1 of (Menon et al., 2018).

Figure 4: ROC gaps (left) and AUC (right) of baselines and DCEM on sepsis
classification task at $q_{t}=1.5,k=4$. Each dot represents a different
$s_{T}$. Our method (DCEM, magenta) maintains competitive or better bias
mitigation and discriminative performance compared to baselines. “-”: median,
“$\bigtriangleup$”: worst-case ROC gap, “$\bigtriangledown$”: worst-case AUC.

#### DCEM is more robust than baselines to changes in the data-generating
process.

Fig. 2 (left) shows that the maximum ROC gap is lower for DCEM compared to
baselines (ours: 0.060 vs. 2nd-best, tested-only: 0.083). We report similar
results for the minimum AUC (Fig. 2, right; ours: 0.768 vs. 2nd-best, tested-
only: 0.623). DCEM also achieves a tighter ROC gap and AUC range. Fig. 2 also
shows that our method has the tightest ROC gap range (left, DCEM: 0.048 vs.
tested-only, 2nd-tightest: 0.063) and AUC range (right, DCEM: 0.055 vs.
DragonNet: 0.199).

The results suggest that DCEM maintains robust bias mitigation and
discriminative performance across different data-generation processes
($s_{Y}$). This is expected, as DCEM optimizes likelihood under the disparate
censorship data-generation process by design. In contrast, the baselines may
experience selection bias or misspecification, since they discard data or
assume certain noise structures/variable dependencies that disparate
censorship violates.

#### DCEM maintains competitive discriminative performance.

Fig. 2 (right) shows that DCEM outperforms all baselines except for the
tested-only model, which our approach lags by 0.028 AUC (DCEM: 0.787 vs.
tested-only: 0.815). Other causal estimators achieve similar discriminative
performance to the tested-only approach (Appendix E.3). However, our method
improves on the “default” $y$-obs model, increasing the median AUC by 0.130
($y$-obs: 0.657). SELF, which has a similar median ROC gap to DCEM,
underperforms DCEM by 0.110 AUC (SELF: 0.677 vs. DCEM: 0.787). Other baselines
also underperform. This is expected, since some methods ignore label bias:
training on $\tilde{Y}$ alone is misspecified for $E(Y\mid X)$, since it
incorrectly assumes that if $T=0$, then $Y=0$. The same argument applies to
Group 0/1 only approaches.

Some baselines account for label noise/bias, but are mis-specified under
disparate censorship since they make different independence assumptions. Group
peer loss assumes $T\perp\\!\\!\\!\perp X\mid(Y,A)$, and SELF assumes
$T\perp\\!\\!\\!\perp X\mid Y$, ignoring the dependence of biased selective
labeling on $X$. DragonNet accounts for $X$ by adding $P(T\mid X,A=1)-P(T\mid
X,A=0)$ to the default model’s estimates (i.e., $P(\tilde{Y}\mid X)$) as a
“correction factor.” However, the correction factor may be biased for true
negatives: the oracle is zero, because $P(\tilde{Y}\mid X)=P(Y\mid X)$, but
$P(T\mid X,A=1)-P(T\mid X,A=0)\neq 0$ in general under systematic labeling
bias.

SAR-EM, which is most similar to the proposed approach, models missingness at
random (i.e., $T\not\perp\\!\\!\\!\perp X,Y$), but discards reliable
negatives. In contrast, the proposed approach incorporates reliable negatives
in alignment with our assumptions about labeling bias, allowing it to
counterbalance biased selective labeling. Trends are similar for other
$q_{t},q_{y}$ and $k\in[1/2,2]$. Since the tested-only model is a strong
baseline, we now compare it directly to DCEM.

#### DCEM achieves better tradeoffs between discriminative performance and
bias mitigation.

Among models with similar AUC where $\text{AUC}<0.875$, DCEM reduces ROC gaps
compared to the tested-only model (Fig. 3). For example, for
$\text{AUC}\in(0.825,0.875)$ (Fig. 3, 2nd from right), DCEM improves the
average ROC gap by 0.022 (0.028 vs. 0.050), with similar trends at lower AUCs.
Among the best-performing models ($\text{AUC}>0.875$; Fig. 3, 1st from right),
both methods have similar ROC gaps.

The results suggest that DCEM is not trading discriminative performance for
bias mitigation. At a given AUC, DCEM more often yields models with a lower
ROC gap than the tested-only model. Since the tested-only approach does not
account for label bias, it can achieve relatively high AUC without mitigating
bias. In contrast, DCEM explicitly counteracts disparate censorship. A similar
comparison to SELF shows that, at low ROC gaps, DCEM likewise finds higher-AUC
solutions than SELF (Appendix E.6).

###  5.2 Results on sepsis classification in MIMIC-III

#### DCEM has better discriminative performance than baselines.

Fig. 4 compares the ROC gap and AUC of DCEM to selected baselines at testing
disparity $q_{t}=1.5$, and testing rate multiplier $k=4$. Each dot corresponds
to one variation of $s_{T}$ (laboratory testing policy). Our method has the
highest median AUC among baselines (ours: 0.620 vs. DragonNet: 0.593), nearing
the oracle ($y$-model, 0.633). Note that DCEM has better discriminative
performance than the tested-only approach, suggesting that extrapolation from
tested to untested individuals is more difficult on the sepsis classification
task than the fully synthetic tasks.

#### DCEM achieves good tradeoffs with bias mitigation.

DCEM achieves a smaller median ROC gap compared to five of eight baselines
tested. Group peer loss, DragonNet and the Group 0 approach achieve lower
median ROC gaps of 0.070, 0.088 and 0.082, respectively (DCEM: 0.105).
However, the Group 0 approach catastrophically fails (median AUC: 0.342).
Models may perform arbitrarily poorly under disparate censorship if labeling
biases sufficiently “conceal” the true decision boundary. Group peer loss
(among many other baselines) also exhibits a much wider AUC range than the
proposed approach (Group peer loss: 0.182 vs. DCEM: 0.065), suggesting that
its discriminative utility may be limited. DragonNet appears competitive
(0.027 AUC lower than DCEM), but would only perform well when the effect of
race on testing is close to $|P(Y\mid X)-P(\tilde{Y}\mid X,A)|$, which is
violated if labeling biases (large effect of race on testing) are present in
negative patients ($P(Y\mid X)\approx P(\tilde{Y}\mid X,A)$).

Many approaches, including DCEM, obtain a lower ROC gap than training on $y$.
Although the oracle obtains the highest median AUC, optimizing discriminative
performance on $y$ is not always guaranteed to mitigate bias. DCEM uses
labeling probabilities to mitigate bias via causal regularization, while
DragonNet directly uses an estimate of the labeling bias as a correction
factor. Thus, the results validate that the labeling bias can provide signal
for bias mitigation.

#### DCEM is more robust than most baselines to changes in $s_{T}$.

DCEM maintains robust bias mitigation capabilities across $s_{T}$; _i.e._ ,
differences in how labelers weigh features in their decisions. Fig. 4 shows
that DCEM attains a maximum ROC gap of 0.133 (left; DragonNet: 0.144), and a
minimum AUC of 0.584 (right; DragonNet: 0.574). Fig. 4 also shows that DCEM
achieves the tightest ROC gap range (left; DCEM: 0.094 vs. 2nd-best: 0.102)
and 2nd-tightest AUC range (right; DCEM: 0.065 vs. DragonNet: 0.018). Many
baselines also exhibit a bimodal empirical AUC distribution and only perform
well under specific labeling behaviors. We examine the sensitivity of
baselines to $s_{T}$ by plotting AUC and ROC gaps against coefficients of
$s_{T}$ (Appendix E.7).

While DragonNet is competitive on this dataset, its robustness and performance
capabilities may not generalize (_e.g._ , simulation results, Fig. 2). DCEM is
the only approach tested that achieved competitive discriminative performance
and bias mitigation on both datasets. Trends in performance and robustness are
similar for other $k,q_{t}$ (Appendix E.1).

#### Overall takeaways.

In a simulation study of disparate censorship, DCEM mitigates bias while
achieving similar or better discriminative performance compared to baselines.
The proposed approach is empirically more robust than baselines to changes in
the data-generating process. On a sepsis classification task, DCEM mitigates
bias while improving or maintaining discriminative performance compared to
baselines across different labeling behaviors. Thus, DCEM can potentially
mitigate bias with less impact on discriminative performance than existing
methods.

##  6 Related Work

#### Selective labeling/disparate censorship.

Disparate censorship is a variation of selective labeling (Lakkaraju et al.,
2017; Kleinberg et al., 2018) and outcome measurement error (Guerdan et al.,
2023b). Selective labeling problems have been studied in clinical settings
(Farahani et al., 2020; Shanmugam et al., 2024; Chang et al., 2022;
Mullainathan & Obermeyer, 2022; Balachandar et al., 2024), social/public
policy (Saxena et al., 2020; Kontokosta & Hong, 2021; Laufer et al., 2022; Liu
& Garg, 2022; Kiani et al., 2023), and finance (Björkegren & Grissen, 2020;
Henderson et al., 2023), among other domains. For an extended literature
review of selective labeling problems, see Appendix A.

Past work has trained ML models under disparate censorship, directly encoding
untested individuals as negative (Henry et al., 2015; Jehi et al., 2020;
McDonald et al., 2021; Adams et al., 2022; Kamran et al., 2022). Previous
approaches for learning under selective labels leverage heterogeneity in human
decisions to recover outcome estimates (Lakkaraju et al., 2017; Kleinberg et
al., 2018; Chen et al., 2023), or use domain-specific adjustments (Gholami et
al., 2018; Balachandar et al., 2024). We propose DCEM, a complementary
approach for mitigating bias under disparate censorship without such
restrictions.

#### Semi-supervised learning.

Semi-supervised approaches do not assume labels for untested individuals.
However, many causally-motivated methods diverge from the causal model of
disparate censorship (Madras et al., 2019; Yao et al., 2021; Garg et al.,
2023; Guerdan et al., 2023a; Gong et al., 2021; Kato et al., 2023; Sportisse
et al., 2023) via different independence/causal relationships between
variables. Filtering methods (Han et al., 2018; Li et al., 2020; Nguyen et
al., 2020; Chen et al., 2020; Zhang et al., 2021; Zhao et al., 2022) assume
specific model behavior on noisy examples (_e.g.,_ , noise is learned late in
training (Arpit et al., 2017)) or labeling bias (randomness/class-dependence),
which disparate censorship violates. We also highlight historical expectation-
maximization approaches for learning with missing data (Ghahramani & Jordan,
1993; Ghahramani et al., 1996; Ambroise & Govaert, 2000), which place
parametric assumptions on the data-generation process. We use neural networks
to target the estimands of interest to circumvent parametric assumptions.

Other alternatives include positive-unlabeled learning approaches that assume
labeling depends on covariates (_e.g._ , missing not at random) (Bekker et
al., 2020; Furmańczyk et al., 2022; Gerych et al., 2022; Wang et al., 2024).
However, these methods do not leverage correctly-labeled negatives, and
naively-incorporating negative examples without causal assumptions may
potentially harm model performance or bias mitigation. Other methods for
noisy-label learning make assumptions incompatible with our setting, _e.g._
uniform noise within subgroups (Wang et al., 2021), almost-surely clean &
noisy examples (Liu & Tao, 2015; Patrini et al., 2017; Tjandra & Wiens, 2023),
different variable independence/directionality relationships (Wu et al.,
2022), that noisy (_i.e._ , out of distribution) examples are rare (Wald &
Saria, 2023), or other noise constraints (Li et al., 2021; Zhu et al., 2021).
Our approach complements existing work by jointly modeling selective and
biased labeling via causal assumptions tailored to a biased decision-making
pipeline.

##  7 Conclusion

When biased human decisions affect observations of ground truth, applying
standard supervised learning techniques to data exhibiting disparate
censorship can amplify the harm of ML models to marginalized groups. We
propose Disparate Censorship Expectation-Maximization (DCEM), a novel approach
to classification, to mitigate such harm. In a simulation study and a sepsis
classification task, DCEM mitigates bias and maintains competitive
discriminative performance compared to baselines. Limitations of DCEM include
potential slow convergence, since EM is iterative. Model evaluation under
disparate censorship is also inherently difficult due to the difficulty of
obtaining ground truth, motivating future work in dataset curation.
Furthermore, DCEM does not learn a full generative model with all variables.
While such a model could target a wider range of estimands, it would also
increase the number of terms that need to be modeled. Ultimately, DCEM is a
step towards mitigating the disproportionate impacts of disparate censorship.
Our work aims to raise awareness of disparate censorship and motivate the
study of bias mitigation methods.

## Impact Statement

This paper addresses disparate censorship, a realistic source of label bias in
ML, and proposes a method that mitigates its harms. Since the goal of the
paper is aligned with reducing inequity in decision-making, practical use
cases of DCEM are inherently high-stakes settings. Thus, we believe that the
ethical usage of DCEM (or any bias mitigation approach) in the real-world
requires prospective model evaluation in the context of use (_e.g._ ,
shadowing human decision-makers) to assess unforeseen negative impacts. Our
work provides a general choice of bias mitigation (area between ROC curves)
and discriminative performance metrics (AUC), which are motivated by clinical
tasks where equitably ranking individuals in terms of resource needs is
important. Practitioners should ensure their evaluation metrics align with
domain-specific criteria for bias mitigation and performance.

## Acknowledgements

We thank (in alphabetical order) Donna Tjandra, Divya Shanmugan, Fahad Kamran,
Jung Min Lee, Maggie Makar, Meera Krishnamoorthy, Michael Ito, Sarah Jabbour,
Shengpu Tang, Stephanie Shepard, and Winston Chen for helpful conversations
and proofreading, and the anonymous reviewers for their constructive feedback.
T.C. and J.W. are supported by the U.S. National Heart, Lung, and Blood
Insitute of the National Institutes of Health (Grant No. 5R01HL158626-03). The
views and conclusions in this document are those of the authors and should not
be interpreted as necessarily representing the official policies, either
expressed or implied of the U.S. National Institutes of Health.

## References

  * Adams et al. (2022) Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al.  Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis.  _Nature Medicine_ , pp. 1–6, 2022.
  * Ambroise & Govaert (2000) Ambroise, C. and Govaert, G.  EM algorithm for partially known labels.  In _Data Analysis, Classification, and Related Methods_ , pp. 161–166. Springer, 2000.
  * Arazo et al. (2020) Arazo, E., Ortego, D., Albert, P., O’Connor, N. E., and McGuinness, K.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8. IEEE, 2020.
  * Arpit et al. (2017) Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., and Lacoste-Julien, S.  A closer look at memorization in deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 233–242, 2017.
  * Bahadori et al. (2017) Bahadori, M. T., Chalupka, K., Choi, E., Chen, R., Stewart, W. F., and Sun, J.  Causal regularization.  _arXiv preprint arXiv:1702.02604_ , 2017.
  * Balachandar et al. (2024) Balachandar, S., Garg, N., and Pierson, E.  Domain constraints improve risk prediction when outcome data is missing.  In _12th International Conference on Learning Representations_ , 2024\.
  * Beery (1995) Beery, T. A.  Gender bias in the diagnosis and treatment of coronary artery disease.  _Heart & Lung_, 24(6):427–435, 1995.
  * Bekker et al. (2020) Bekker, J., Robberechts, P., and Davis, J.  Beyond the selected completely at random assumption for learning from positive and unlabeled data.  In _Machine Learning and Knowledge Discovery in Databases_ , pp. 71–85, Cham, 2020.
  * Bergman et al. (2021) Bergman, P., Kopko, E., and Rodriguez, J. E.  A seven-college experiment using algorithms to track students: Impacts and implications for equity and fairness.  Technical report, National Bureau of Economic Research, 2021.
  * Binns et al. (2017) Binns, R., Veale, M., Van Kleek, M., and Shadbolt, N.  Like trainer, like bot? Inheritance of bias in algorithmic content moderation.  In _Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Proceedings, Part II 9_ , pp. 405–415, 2017.
  * Björkegren & Grissen (2020) Björkegren, D. and Grissen, D.  Behavior revealed in mobile phone usage predicts credit repayment.  _The World Bank Economic Review_ , 34(3):618–634, 2020.
  * Blum & Stangl (2020) Blum, A. and Stangl, K.  Recovering from biased data: Can fairness constraints improve accuracy?  In _1st Symposium on Foundations of Responsible Computing_ , 2020\.
  * Chang et al. (2022) Chang, T., Sjoding, M. W., and Wiens, J.  Disparate censorship & undertesting: A source of label bias in clinical machine learning.  In _Proceedings of the 7th Machine Learning for Healthcare Conference_ , volume 182 of _Proceedings of Machine Learning Research_ , pp. 343–390, Aug 2022.
  * Chen et al. (2023) Chen, J., Li, Z., and Mao, X.  Learning under selective labels with heterogeneous decision-makers: An instrumental variable approach.  _arXiv preprint arXiv:2306.07566_ , 2023.
  * Chen et al. (2020) Chen, P., Ye, J., Chen, G., Zhao, J., and Heng, P.-A.  Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 2020.
  * Delahanty et al. (2019) Delahanty, R. J., Alvarez, J., Flynn, L. M., Sherwin, R. L., and Jones, S. S.  Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis.  _Annals of Emergency Medicine_ , 73(4):334–344, 2019.
  * Dempster et al. (1977) Dempster, A. P., Laird, N. M., and Rubin, D. B.  Maximum likelihood from incomplete data via the EM algorithm.  _Journal of the Royal Statistical Society: Series B (methodological)_ , 39(1):1–22, 1977.
  * Djulbegovic et al. (2014) Djulbegovic, B., Elqayam, S., Reljic, T., Hozo, I., Miladinovic, B., Tsalatsanis, A., Kumar, A., Beckstead, J., Taylor, S., and Cannon-Bowers, J.  How do physicians decide to treat: an empirical evaluation of the threshold model.  _BMC Medical Informatics and Decision Making_ , 14:1–10, 2014.
  * Englesson & Azizpour (2021) Englesson, E. and Azizpour, H.  Generalized jensen-shannon divergence loss for learning with noisy labels.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 30284–30297, 2021.
  * Farahani et al. (2020) Farahani, N. Z., Sundaram, D. S. B., Enayati, M., Arunachalam, S. P., Pasupathy, K., and Arruda-Olson, A. M.  Explanatory analysis of a machine learning model to identify hypertrophic cardiomyopathy patients from EHR using diagnostic codes.  In _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pp. 1932–1937, 2020.
  * Furmańczyk et al. (2022) Furmańczyk, K., Mielniczuk, J., Rejchel, W., and Teisseyre, P.  Joint estimation of posterior probability and propensity score function for positive and unlabelled data.  _arXiv preprint arXiv:2209.07787_ , 2022.
  * Gardner et al. (2019) Gardner, J., Brooks, C., and Baker, R.  Evaluating the fairness of predictive student models through slicing analysis.  In _Proceedings of the 9th International Conference on Learning Analytics & Knowledge_, pp. 225–234, 2019.
  * Garg et al. (2023) Garg, A., Nguyen, C., Felix, R., Do, T.-T., and Carneiro, G.  Instance-dependent noisy label learning via graphical modelling.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pp. 2288–2298, 2023.
  * Gerych et al. (2022) Gerych, W., Hartvigsen, T., Buquicchio, L., Agu, E., and Rundensteiner, E.  Recovering the propensity score from biased positive unlabeled data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 6694–6702, 2022.
  * Ghahramani & Jordan (1993) Ghahramani, Z. and Jordan, M.  Supervised learning from incomplete data via an EM approach.  _Advances in Neural Information Processing Systems_ , 6, 1993.
  * Ghahramani et al. (1996) Ghahramani, Z., Hinton, G. E., et al.  The EM algorithm for mixtures of factor analyzers.  Technical report, Technical Report CRG-TR-96-1, University of Toronto, 1996.
  * Gholami et al. (2018) Gholami, S., Mc Carthy, S., Dilkina, B., Plumptre, A., Tambe, M., Driciru, M., Wanyama, F., Rwetsiba, A., Nsubaga, M., Mabonga, J., et al.  Adversary models account for imperfect crime data: Forecasting and planning against real-world poachers.  In _International Conference on Autonomous Agents and Multiagent Systems_ , 2018.
  * Glynn & Quinn (2010) Glynn, A. N. and Quinn, K. M.  An introduction to the augmented inverse propensity weighted estimator.  _Political Analysis_ , 18(1):36–56, 2010.
  * Gong et al. (2021) Gong, C., Wang, Q., Liu, T., Han, B., You, J. J., Yang, J., and Tao, D.  Instance-dependent positive and unlabeled learning with labeling bias estimation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44:4163–4177, 2021.
  * Guerdan et al. (2023a) Guerdan, L., Coston, A., Holstein, K., and Wu, Z. S.  Counterfactual prediction under outcome measurement error.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 1584–1598, 2023a.
  * Guerdan et al. (2023b) Guerdan, L., Coston, A., Wu, Z. S., and Holstein, K.  Ground(less) truth: A causal framework for proxy labels in human-algorithm decision-making.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 688–704, 2023b.
  * Han et al. (2018) Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I. W., and Sugiyama, M.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 8536–8546, 2018.
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, September 2020\.
  * Hartvigsen et al. (2018) Hartvigsen, T., Sen, C., Brownell, S., Teeple, E., Kong, X., and Rundensteiner, E. A.  Early Prediction of MRSA Infections using Electronic Health Records.  In _HEALTHINF_ , pp. 156–167, 2018.
  * Henderson et al. (2023) Henderson, P., Chugg, B., Anderson, B., Altenburger, K., Turk, A., Guyton, J., Goldin, J., and Ho, D. E.  Integrating reward maximization and population estimation: Sequential decision-making for internal revenue service audit selection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pp. 5087–5095, 2023.
  * Henry et al. (2015) Henry, K. E., Hager, D. N., Pronovost, P. J., and Saria, S.  A targeted real-time early warning score (trewscore) for septic shock.  _Science Translational Medicine_ , 7(299):299ra122–299ra122, 2015.
  * Hu et al. (2022) Hu, X., Niu, Y., Miao, C., Hua, X.-S., and Zhang, H.  On non-random missing labels in semi-supervised learning.  In _10th International Conference on Learning Representations_ , 2022\.
  * Imbens & Rubin (2015) Imbens, G. W. and Rubin, D. B.  _Causal inference in statistics, social, and biomedical sciences_.  Cambridge University Press, 2015.
  * Janzing (2019) Janzing, D.  Causal regularization.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Jehi et al. (2020) Jehi, L., Ji, X., Milinovich, A., Erzurum, S., Rubin, B. P., Gordon, S., Young, J. B., and Kattan, M. W.  Individualizing risk prediction for positive coronavirus disease 2019 testing: results from 11,672 patients.  _Chest_ , 158(4):1364–1375, 2020.
  * Johnson et al. (2016) Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3(1):1–9, 2016.
  * Johnson et al. (2018) Johnson, A. E., Aboab, J., Raffa, J. D., Pollard, T. J., Deliberato, R. O., Celi, L. A., and Stone, D. J.  A comparative analysis of sepsis identification methods in an electronic database.  _Critical Care Medicine_ , 46(4):494–499, 2018\.
  * Kamran et al. (2022) Kamran, F., Tang, S., Ötleş, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., Valley, T. S., West, L. R., Singh, K., Blumberg, S., Donnelly, J. P., Shenoy, E. S., Ayanian, J. Z., Nallamothu, B. K., Sjoding, M. W., and Wiens, J.  Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study.  _The BMJ_ , 376, 2022.
  * Kato et al. (2023) Kato, M., Wu, S., Kureishi, K., and Yasui, S.  Automatic debiased learning from positive, unlabeled, and exposure data.  _arXiv preprint arXiv:2303.04797_ , 2023.
  * Kennedy (2023) Kennedy, E. H.  Towards optimal doubly robust estimation of heterogeneous causal effects.  _Electronic Journal of Statistics_ , 17(2):3008–3049, 2023.
  * Kiani et al. (2023) Kiani, S., Barton, J., Sushinsky, J., Heimbach, L., and Luo, B.  Counterfactual prediction under selective confounding.  _arXiv preprint arXiv:2310.14064_ , 2023.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations_ , 2015\.
  * Kleinberg et al. (2018) Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., and Mullainathan, S.  Human decisions and machine predictions.  _The Quarterly Journal of Economics_ , 133(1):237–293, 2018.
  * Kontokosta & Hong (2021) Kontokosta, C. E. and Hong, B.  Bias in smart city governance: How socio-spatial disparities in 311 complaint behavior impact the fairness of data-driven decisions.  _Sustainable Cities and Society_ , 64:102503, 2021.
  * Laine et al. (2020) Laine, R., Hyttinen, A., and Mathioudakis, M.  Evaluating decision makers over selectively labelled data: A causal modelling approach.  In _Discovery Science: 23rd International Conference, DS 2020, Thessaloniki, Greece, October 19–21, 2020, Proceedings 23_ , pp. 3–18, 2020\.
  * Lakkaraju et al. (2017) Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., and Mullainathan, S.  The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 275–284, 2017.
  * Laufer et al. (2022) Laufer, B., Pierson, E., and Garg, N.  End-to-end auditing of decision pipelines.  In _ICML Workshop on Responsible Decision-Making in Dynamic Environments._ , pp. 1–7, 2022.
  * Lee (2013) Lee, D.-H.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3(2), pp. 896, 2013.
  * Li et al. (2020) Li, J., Socher, R., and Hoi, S. C. H.  Dividemix: Learning with noisy labels as semi-supervised learning.  In _8th International Conference on Learning Representations, ICLR_ , 2020.
  * Li et al. (2021) Li, X., Liu, T., Han, B., Niu, G., and Sugiyama, M.  Provably end-to-end label-noise learning without anchor points.  In _Proceedings of the 38th International Conference on Machine Learning_ , pp. 6403–6413, 2021.
  * Liu & Tao (2015) Liu, T. and Tao, D.  Classification with noisy labels by importance reweighting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(3):447–461, 2015.
  * Liu & Guo (2020) Liu, Y. and Guo, H.  Peer loss functions: Learning from noisy labels without knowing noise rates.  In _Proceedings of the 37th International Conference on Machine Learning_ , 2020.
  * Liu & Garg (2022) Liu, Z. and Garg, N.  Equity in resident crowdsourcing: Measuring under-reporting without ground truth data.  In _Proceedings of the 23rd ACM Conference on Economics and Computation_ , pp. 1016–1017, 2022.
  * Madras et al. (2019) Madras, D., Creager, E., Pitassi, T., and Zemel, R.  Fairness through causal awareness.  _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , 2019.
  * McDonald et al. (2021) McDonald, S. A., Medford, R. J., Basit, M. A., Diercks, D. B., and Courtney, D. M.  Derivation with internal validation of a multivariable predictive model to predict covid-19 test results in emergency department patients.  _Academic Emergency Medicine_ , 28(2):206–214, 2021.
  * Menon et al. (2018) Menon, A. K., Van Rooyen, B., and Natarajan, N.  Learning from binary labels with instance-dependent noise.  _Machine Learning_ , 107(8):1561–1595, 2018.
  * Mulherin & Miller (2002) Mulherin, S. A. and Miller, W. C.  Spectrum bias or spectrum effect? subgroup variation in diagnostic test evaluation.  _Annals of Internal Medicine_ , 137(7):598–602, 2002.
  * Mullainathan & Obermeyer (2022) Mullainathan, S. and Obermeyer, Z.  Diagnosing physician error: A machine learning approach to low-value health care.  _The Quarterly Journal of Economics_ , 137(2):679–727, 2022.
  * Nguyen et al. (2020) Nguyen, D. T., Mummadi, C. K., Ngo, T., Nguyen, T. H. P., Beggel, L., and Brox, T.  SELF: Learning to Filter Noisy Labels with Self-Ensembling.  In _8th International Conference on Learning Representations_ , 2020\.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library, 2019\.
  * Patrini et al. (2017) Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L.  Making deep neural networks robust to label noise: A loss correction approach.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1944–1952, 2017.
  * Pearl (2009) Pearl, J.  _Causality_.  Cambridge university press, 2009.
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Peng et al. (2019) Peng, A., Nushi, B., Kıcıman, E., Inkpen, K., Suri, S., and Kamar, E.  What you see is what you get? the impact of representation criteria on human bias in hiring.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 7, pp. 125–134, 2019.
  * Pierson et al. (2018) Pierson, E., Corbett-Davies, S., and Goel, S.  Fast threshold tests for detecting discrimination.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 96–105, 2018.
  * Pierson et al. (2020) Pierson, E., Simoiu, C., Overgoor, J., Corbett-Davies, S., Jenson, D., Shoemaker, A., Ramachandran, V., Barghouty, P., Phillips, C., Shroff, R., et al.  A large-scale analysis of racial disparities in police stops across the united states.  _Nature Human Behaviour_ , 4(7):736–745, 2020\.
  * Rambachan & Roth (2020) Rambachan, A. and Roth, J.  Bias in, bias out? Evaluating the folk wisdom.  In _1st Symposium on Foundations of Responsible Computing_ , 2020\.
  * Rhee & Klompas (2020) Rhee, C. and Klompas, M.  Sepsis trends: increasing incidence and decreasing mortality, or changing denominator?  _Journal of Thoracic Disease_ , 12(Suppl 1):S89, 2020.
  * Rizve et al. (2021) Rizve, M. N., Duarte, K., Rawat, Y. S., and Shah, M.  In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.  In _9th International Conference on Learning Representations_ , 2021\.
  * Robins et al. (1994) Robins, J. M., Rotnitzky, A., and Zhao, L. P.  Estimation of regression coefficients when some regressors are not always observed.  _Journal of the American Statistical Association_ , pp. 846–866, 1994.
  * Rockafellar (1970) Rockafellar, R. T.  _Convex analysis_.  Princeton University Press, Princeton, N. J., 1970.
  * Rosenbaum & Rubin (1983) Rosenbaum, P. R. and Rubin, D. B.  The central role of the propensity score in observational studies for causal effects.  _Biometrika_ , 70(1):41–55, 1983.
  * Saxena et al. (2020) Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., and Guha, S.  A human-centered review of algorithms used within the u.s. child welfare system.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pp. 1–15, 2020.
  * Schulman et al. (1999) Schulman, K. A., Berlin, J. A., Harless, W., Kerner, J. F., Sistrunk, S., Gersh, B. J., Dube, R., Taleghani, C. K., Burke, J. E., Williams, S., et al.  The effect of race and sex on physicians’ recommendations for cardiac catheterization.  _New England Journal of Medicine_ , 340(8):618–626, 1999.
  * Seymour et al. (2016) Seymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al.  Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3).  _JAMA_ , 315(8):762–774, 2016.
  * Shalev-Shwartz & Ben-David (2014) Shalev-Shwartz, S. and Ben-David, S.  _Understanding machine learning: From theory to algorithms_.  Cambridge University Press, 2014.
  * Shanmugam et al. (2024) Shanmugam, D., Hou, K., and Pierson, E.  Quantifying disparities in intimate partner violence: a machine learning method to correct for underreporting.  _npj Women’s Health_ , 2(1), 2024.
  * Shi et al. (2019) Shi, C., Blei, D., and Veitch, V.  Adapting neural networks for the estimation of treatment effects.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Singer et al. (2016) Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al.  The third international consensus definitions for sepsis and septic shock (Sepsis-3).  _Jama_ , 315(8):801–810, 2016.
  * Sportisse et al. (2023) Sportisse, A., Schmutz, H., Humbert, O., Bouveyron, C., and Mattei, P.-A.  Are labels informative in semi-supervised learning? estimating and leveraging the missing-data mechanism.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.
  * Sühr et al. (2021) Sühr, T., Hilgard, S., and Lakkaraju, H.  Does fair ranking improve minority outcomes? Understanding the interplay of human and algorithmic biases in online hiring.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 989–999, 2021.
  * Teeple et al. (2020) Teeple, E., Hartvigsen, T., Sen, C., Claypool, K. T., and Rundensteiner, E. A.  Clinical performance evaluation of a machine learning system for predicting hospital-acquired clostridium difficile infection.  In _HEALTHINF_ , pp. 656–663, 2020.
  * The pandas development team (2020) The pandas development team.  pandas-dev/pandas: Pandas, 2020.
  * Tjandra & Wiens (2023) Tjandra, D. and Wiens, J.  Leveraging an alignment set in tackling instance-dependent label noise.  In _Proceedings of the Conference on Health, Inference, and Learning_ , 2023.
  * Van Der Laan & Rubin (2006) Van Der Laan, M. J. and Rubin, D.  Targeted maximum likelihood learning.  _The International Journal of Biostatistics_ , 2(1), 2006\.
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.
  * Vogel et al. (2021) Vogel, R., Bellet, A., and Clémençon, S.  Learning fair scoring functions: Bipartite ranking under ROC-based fairness constraints.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 784–792, 2021.
  * Wager (2020) Wager, S.  Stats 361: Causal inference.  Technical report, Stanford University, 2020.
  * Wald & Saria (2023) Wald, Y. and Saria, S.  Birds of an odd feather: Guaranteed out-of-distribution (OOD) novel category detection.  In _Uncertainty in Artificial Intelligence_ , pp. 2179–2191, 2023\.
  * Wang et al. (2021) Wang, J., Liu, Y., and Levy, C.  Fair classification with group-dependent label noise.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 526–536, 2021.
  * Wang et al. (2024) Wang, X., Chen, H., Guo, T., and Wang, Y.  Pue: Biased positive-unlabeled learning enhancement by causal inference.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wu et al. (2022) Wu, S., Gong, M., Han, B., Liu, Y., and Liu, T.  Fair classification with instance-dependent label noise.  In _Proceedings of the First Conference on Causal Learning and Reasoning_ , volume 177 of _Proceedings of Machine Learning Research_ , pp. 927–943, Apr 2022.
  * Yao et al. (2021) Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., and Zhang, K.  Instance-dependent label-noise learning under a structural causal model.  In _Advances in Neural Information Processing Systems_ , volume 34, pp. 4409–4420, 2021.
  * Zhang et al. (2021) Zhang, Y., Zheng, S., Wu, P., Goswami, M., and Chen, C.  Learning with feature-dependent label noise: A progressive approach.  In _9th International Conference on Learning Representations_ , 2021\.
  * Zhang & Sabuncu (2018) Zhang, Z. and Sabuncu, M. R.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 8792–8802, 2018.
  * Zhao et al. (2022) Zhao, G., Li, G., Qin, Y., Liu, F., and Yu, Y.  Centrality and consistency: Two-stage clean samples identification for learning with instance-dependent noisy labels.  In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), _Computer Vision – ECCV 2022_ , pp. 21–37, 2022.
  * Zhu & Ghahramani (2002) Zhu, X. and Ghahramani, Z.  Learning from labeled and unlabeled data with label propagation.  Technical report, Center for Automated Learning and Discovery, Carnegie Mellon University, 2002.
  * Zhu et al. (2021) Zhu, Z., Song, Y., and Liu, Y.  Clusterability as an alternative to anchor points when learning with noisy labels.  In _Proceedings of the 38th International Conference on Machine Learning_ , pp. 12912–12923, 2021.

##  Appendix A Selective labeling in the literature

We enumerate domains in which our literature review found instances of
selective label problems in the ML methods and applications literature:

  * •

Healthcare: Laboratory/diagnostic testing (Chang et al., 2022; Mullainathan &
Obermeyer, 2022) and diagnosis (Farahani et al., 2020; Shanmugam et al., 2024;
Balachandar et al., 2024)

  * •

Social & public policy: Child welfare assessment (Saxena et al., 2020; Kiani
et al., 2023), urban planning/policy (Kontokosta & Hong, 2021; Laufer et al.,
2022; Liu & Tao, 2015), hiring pipelines (Peng et al., 2019; Sühr et al.,
2021), student placement (Bergman et al., 2021), and bias in policing
(Rambachan & Roth, 2020; Pierson et al., 2020)

  * •

Finance: Credit repayment (Björkegren & Grissen, 2020) and financial auditing
(Henderson et al., 2023)

  * •

Others/miscellaneous: Wildlife conservation (Gholami et al., 2018), social
media content moderation (Binns et al., 2017)

We note that this is not an exhaustive list of all papers in the selective
labeling literature or related problem settings. However, this list
illustrates the broad applicability and relevance of our problem setting.

##  Appendix B Omitted Proofs

For convenience, we restate all theorems and propositions here.

###  B.1 Theorem 3.1

######  Theorem (E-step derivation).

The posterior conditional mean of $y^{(i)}=1$ given the observed data,
$Q(y^{(i)})\triangleq\mathbb{E}[y^{(i)}=1\mid
t^{(i)},\tilde{y}^{(i)},a^{(i)},\mathbf{x}^{(i)}]$, is equal to

$\displaystyle Q(y^{(i)})$
$\displaystyle=\begin{cases}\tilde{y}^{(i)}&t^{(i)}=1\\\
\mathbb{E}[y^{(i)}=1\mid\mathbf{x}^{(i)}]&\text{otherwise}\end{cases}.$ (14)

###### Proof.

We drop superscripts $(\cdot)^{(i)}$ in the proof for clarity. Denote $Q(y)$
as posterior distribution of $y$ given the observed data, $\mathbb{E}[y=1\mid
t,\tilde{y},a,\mathbf{x}]$ (_i.e._ , the E-step estimate). First, we can write

$\displaystyle Q(y)$ $\displaystyle\triangleq\mathbb{E}[y=1\mid
t,\tilde{y},a,\mathbf{x}]=\mathbb{E}[y=1\mid\tilde{y},x,t]=P(y=1\mid\tilde{y},x,t)$
(15)

for simplicity, where we use the fact that $Y\perp\\!\\!\\!\perp A\mid(T,X)$
and the fact that $\mathbb{E}[y=1\mid\tilde{y},x,t]$ is binary. Proceeding, we
can write:

$\displaystyle=t\cdot P(y=1\mid\tilde{y},x,t=1)+(1-t)\cdot
P(y=1\mid\tilde{y},x,t=0)$ (16)
$\displaystyle=t\tilde{y}+(1-t)[\tilde{y}P(y=1\mid\tilde{y}=1,x,t=0)+(1-\tilde{y})P(y=1\mid\tilde{y}=0,x,t=0)]$
(17) $\displaystyle=t\tilde{y}+(1-t)(1-\tilde{y})P(y=1\mid\tilde{y}=0,x,t=0)$
(18) $\displaystyle=t\tilde{y}+(1-t)(1-\tilde{y})\cdot\frac{P(\tilde{y}=0\mid
y=1,x,t=0)P(y=1\mid x,t=0)}{P(\tilde{y}=0\mid x,t=0)}$ (19)
$\displaystyle=t\tilde{y}+(1-t)(1-\tilde{y})P(y=1\mid x).$ (20)

The second equality follows since $t=1\implies\tilde{y}=y$. The third equality
holds since $P(y=1\mid\tilde{y}=1,x,t=0)=0$ by construction, since
$\tilde{y}=yt$ under disparate censorship. The final step follows from three
facts: (1) $P(\tilde{y}=0\mid y=1,x,t=0)=1$ for all $x$, (2)
$P(\tilde{y}=0\mid x,t=0)=1$ for all $x$, and (3) $Y\perp\\!\\!\\!\perp T\mid
X$. This is more succinctly rewritten as E-step is:

$Q(y)=\begin{cases}\tilde{y}&t=1\\\ \mathbb{E}[y=1\mid x]&\text{otherwise
(i.e., $\tilde{y}=0\land t=0$)}\end{cases},$ (21)

which is what we wanted to show. ∎

######  Remark B.1.

Since $y^{(i)}$ is binary by assumption, this result fully determines the
posterior distribution since
$\mathbb{E}[y=1\mid\tilde{y},x,t]=1-\mathbb{E}[y=0\mid\tilde{y},x,t]$.

###  B.2 Theorem 3.2

######  Theorem (M-step derivation).

Let $P(U,A,X,T,\tilde{Y};\theta)$ be a model for the joint data distribution
parameterized by some arbitrary $\theta\in\Theta$ in some parameter space
$\Theta$, which factorizes according to the disparate censorship DAG =(Fig.
1). Let $Q(y)\triangleq\mathbb{E}[y=1\mid t,\tilde{y},a,\mathbf{x}]$ be the
posterior expectation that $y=1$ given the observed data. Then (replacing
random variables with their realized counterparts), we have

$\displaystyle\max_{\theta}\;\frac{1}{N}\sum_{i=1}^{N}\log
P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)};\theta)\geq$
$\displaystyle\max_{\theta}\frac{1}{N}\sum_{i=1}^{N}Q(y^{(i)})\log
P(y^{(i)}\mid\mathbf{x}^{(i)};\theta_{Y\mid X})$
$\displaystyle\quad+(1-Q(y^{(i)}))\log(1-P(y^{(i)}\mid\mathbf{x}^{(i)};\theta_{Y\mid
X}))$ $\displaystyle\quad+Q(y^{(i)})\log P(\tilde{y}^{(i)}\mid
y^{(i)},t^{(i)};\theta_{\tilde{Y}\mid Y,T})$ (22)

where $\theta=[\theta_{Y\mid X}\;\theta_{\tilde{Y}\mid Y,T}]^{\top}$.

###### Proof.

We first construct the evidence-based lower bound (ELBO) of the LHS in the
theorem statement. First, for a single example indexed by $i$, we can write:

$\displaystyle\log
P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)};\theta)$
$\displaystyle=\log\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\frac{P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)},y^{(i)};\theta)}{Q(y^{(i)})}$
(23)
$\displaystyle\geq\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\log\frac{P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)},y^{(i)};\theta)}{Q(y^{(i)})}$
(24)

via Jensen’s inequality. Then, we note that

$\displaystyle\underset{\theta}{\max}\frac{1}{N}\sum_{i=1}^{N}\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\log\frac{P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)},y^{(i)};\theta)}{Q(y^{(i)})}$
$\displaystyle=\;$
$\displaystyle\underset{\theta}{\max}\frac{1}{N}\sum_{i=1}^{N}\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\log
P(u^{(i)},a^{(i)},\mathbf{x}^{(i)},t^{(i)},\tilde{y}^{(i)},y^{(i)};\theta),$
(25)

dropping $Q(y^{(i)})\log Q(y^{(i)})$, which is constant with respect to
$\theta$, after expanding the $\log$ term. We can then use the DAG to
factorize the joint distribution of all variables (including latent variable
$Y$), which is given by

$\displaystyle P(\tilde{Y},Y,T,X,A,U)$ $\displaystyle=P(\tilde{Y}\mid
Y,T)P(Y\mid X)P(T\mid X,A)P(X,A,U).$ (26)

Note that we need only model the first two terms for estimation of $y^{(i)}$.
The first two terms do not involve $y^{(i)}$, are not parameterized, and can
be dropped from the maximization problem. Hence, we proceed to write

$=\;\underset{\theta}{\max}\;\frac{1}{N}\sum_{i=1}^{N}\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\log
P(y^{(i)}\mid\mathbf{x}^{(i)};\theta_{\tilde{Y}\mid X})P(\tilde{y}^{(i)}\mid
t^{(i)},y^{(i)};\theta_{\tilde{Y}\mid Y,T})$ (27)

where $\theta=[\theta_{Y\mid X}\;\theta_{\tilde{Y}\mid Y,T}]^{\top}$. This can
be rewritten as

$=\;\underset{\theta}{\max}\;\frac{1}{N}\sum_{i=1}^{N}\sum_{y^{(i)}\in\\{0,1\\}}Q(y^{(i)})\log
P(y^{(i)}\mid\mathbf{x}^{(i)};\theta_{\tilde{Y}\mid
X})+Q(y^{(i)})P(\tilde{y}^{(i)}\mid t^{(i)},y^{(i)};\theta_{\tilde{Y}\mid
Y,T}),$ (28)

at which point we note that it is sufficient to show that

$(1-Q(y^{(i)}))\log(1-P(\tilde{y}^{(i)}\mid
y^{(i)},t^{(i)};\theta_{\tilde{Y}\mid Y,T}))$ (29)

is constant in $\theta$. We can rewrite the above as

$(1-Q(y^{(i)}))[\tilde{y}^{(i)}\log(P(\tilde{y}^{(i)}=1\mid
y^{(i)}=0,t^{(i)};\theta_{\tilde{Y}\mid
Y,T}))+(1-\tilde{y}^{(i)})\log(P(\tilde{y}^{(i)}=0\mid
y^{(i)}=0,t^{(i)};\theta_{\tilde{Y}\mid Y,T}))].$ (30)

First, note that the event $\\{\tilde{y}^{(i)}=1\mid y^{(i)}=0\\}$ occurs with
probability zero by definition (recall $\tilde{y}^{(i)}=y^{(i)}t^{(i)}$).
Thus, $P(\tilde{y}^{(i)}=1\mid y^{(i)}=0,t^{(i)};\theta_{\tilde{Y}\mid Y,T}))$
cannot change with respect to $\theta$; we drop it from the maximization
problem. Similarly, $P(\tilde{y}^{(i)}=0\mid y^{(i)}=0,t^{(i)})=0$ by
definition, so $(1-Q(y^{(i)}))\log(1-P(\tilde{y}^{(i)}\mid
y^{(i)},t^{(i)}))=0$ which is constant as needed, from which the theorem
follows. ∎

######  Remark B.2.

In the theorem statement, replacing
$P(y^{(i)}\mid\mathbf{x}^{(i)};\theta_{Y\mid X})$ with $\hat{y}^{(i)}$ and
$P(\tilde{y}^{(i)}\mid y^{(i)},t^{(i)};\theta_{\tilde{Y}\mid Y,T})$ with
$h_{\phi}(\cdot)$, assuming $y^{(i)}$ and $\tilde{y}^{(i)}$ are binary, and
writing the explicit form of negative binary cross-entropy (_e.g._ , $y\log
y+(1-y)\log\hat{y}$) recovers the form of the M-step objective seen in Eq. 7.
Note that the optimization problem flips from a maximization to a minimization
due to the relationship between maximizing log-likelihood of binary
variable(s) and minimizing cross-entropy loss.

###  B.3 Theorem 3.4

######  Theorem (Strength of the causal regularizer in $\hat{t}^{(i)}$).

For an example indexed by $i$, $Q(y^{(i)})\in[0,1)$, and $J^{(i)}$ defined as
in Eq. 88, $R(J^{(i)})$ is monotonically increasing in $\hat{t}^{(i)}$ on
$(0,1]$.

###### Proof.

As a proof outline, we first show the closed-form of
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ by solving the first-
order optimality condition of Eq. 88. Then, we show that
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ is decreasing in
$\hat{t}^{(i)}$, and attains a maximum of $Q(y^{(i)})$ as $\hat{t}^{(i)}\to
0^{+}$. We conclude by showing that the preceding implies that $R(J^{(i)})$ is
monotonically increasing in $\hat{t}^{(i)}$ on $(0,1]$, as desired.

The first-order optimality condition of Eq. 88 is

$-\left(\frac{Q(y^{(i)})}{\hat{y}^{(i)}}-\frac{1-Q(y^{(i)})}{1-\hat{y}^{(i)}}\right)+\frac{\hat{t}^{(i)}Q(y^{(i)})}{1-\hat{y}^{(i)}\hat{t}^{(i)}}=0.$
(31)

By assumption (convexity of $\mathcal{L}$), the minimizer is unique. Some
algebra yields

$\displaystyle-Q(y^{(i)})(1-\hat{y}^{(i)})(1-\hat{y}^{(i)}\hat{t}^{(i)})+(1-Q(y^{(i)}))\hat{y}^{(i)}(1-\hat{y}^{(i)}\hat{t}^{(i)})+\hat{t}^{(i)}Q(y^{(i)})\hat{y}^{(i)}(1-\hat{y}^{(i)})$
$\displaystyle=0$ (32) $\displaystyle\iff$
$\displaystyle(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})\hat{y}^{(i)2}-(2Q(y^{(i)})\hat{t}^{(i)}+1)\hat{y}^{(i)}+Q(y^{(i)})$
$\displaystyle=0,$ (33)

from which we can apply the quadratic formula. Define
$B(Q(y^{(i)}),\hat{t}^{(i)})\triangleq 2Q(y^{(i)})\hat{t}^{(i)}+1$ and
$D(Q(y^{(i)}),\hat{t}^{(i)})\triangleq(-B(Q(y^{(i)}),\hat{t}^{(i)}))^{2}-4Q(y^{(i)})(Q(y^{(i)})\hat{t}^{(i)}+\hat{t}^{(i)})$.555We
use letter $B(\cdot)$ because it corresponds to coefficient $b$ in the
conventional quadratic formula:  $x=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}$ for a
quadratic polynomial $ax^{2}+bx+c=0$. We choose letter $D(\cdot)$ since it
corresponds to the discriminant. The quadratic formula yields solutions

$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})=\frac{B(Q(y^{(i)}),\hat{t}^{(i)})\pm\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}.$
(34)

We use the fact that $\hat{y}^{(i)}$ must be in [0, 1] and the constraints
that $\hat{t}^{(i)}\in(0,1]$ and $Q(y^{(i)})\in[0,1)$ to determine which
branch of Eq. 34 yields real solutions in $[0,1]$. By Lemma 1,
$D(Q(y^{(i)},\hat{t}^{(i)})\geq 0$, so the solutions are real. Then, by Lemma
58,

$\frac{B(Q(y^{(i)}),\hat{t}^{(i)})+\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}\geq
1,$ (35)

eliminating that branch. By elimination,

$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})=\frac{B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}.$
(36)

Lemma 66 verifies that the resulting
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ are in [0, 1], as
needed. To proceed, it suffices to show that
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ is decreasing in
$\hat{t}^{(i)}$ and attains a maximum of $Q(y^{(i)})$ as $\hat{t}^{(i)}\to
0^{+}$.

Applying Lemma 4, to prove that
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ decreases in
$\hat{t}^{(i)}$, it is sufficient to show

$1-2\hat{t}^{(i)}Q(y^{(i)})^{2}-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})<0$ (37)

because $1-2\hat{t}^{(i)}Q(y^{(i)})^{2}-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}$ has
the same sign as the derivative of
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ for the values of
$(\hat{t}^{(i)},Q(y^{(i)}))$ of interest.

For values of $\hat{t}^{(i)}\in(0,1]$ and $Q(y^{(i)})\in[0,1)$ such that
$1-2\hat{t}^{(i)}Q(y^{(i)})^{2}<0$, Lemma 1 yields the desired result. For the
remaining values of $(\hat{t}^{(i)},Q(y^{(i)}))$, we can write

$\displaystyle
1-4Q(y^{(i)})^{2}\hat{t}^{(i)}+4Q(y^{(i)})^{4}\hat{t}^{(i)2}<D(Q(y^{(i)}),\hat{t}^{(i)})$
(38) $\displaystyle\iff$ $\displaystyle
1-4Q(y^{(i)})^{2}\hat{t}^{(i)}+4Q(y^{(i)})^{4}\hat{t}^{(i)2}<1-4Q(y^{(i)})^{2}\hat{t}^{(i)}+4Q(y^{(i)})^{2}\hat{t}^{(i)2}$
(39) $\displaystyle\iff$ $\displaystyle Q(y^{(i)})^{2}<1$ (40)

which holds for all feasible values of $Q(y^{(i)})\in[0,1)$. Lastly, due to
the monotonicity of $\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ in
$\hat{t}^{(i)}$, the following one-sided limit is the maximum:

$\underset{\hat{t}^{(i)}\to
0^{+}}{\lim}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})=\underset{\hat{t}^{(i)}\in(0,1]}{\max}\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)}).$
(41)

We want to show that the limit is $Q(y^{(i)})$. Note that, since $\mathcal{L}$
is finite and convex, it is continuous (Corollary 10.1.1, (Rockafellar,
1970)); hence, this limit exists. Since substituting $\hat{t}^{(i)}=0$ yields
the indeterminate form $0/0$, we appeal to L’Hôpital’s rule:

$\underset{\hat{t}^{(i)}\to
0^{+}}{\lim}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})=\underset{\hat{t}^{(i)}\to
0^{+}}{\lim}\;\frac{2Q(y^{(i)})-\frac{4\hat{t}^{(i)}Q(y^{(i)})^{2}-2Q(y^{(i)})^{2}}{\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}}}{2(Q(y^{(i)})+1)}=\frac{2Q(y^{(i)})+2Q(y^{(i)})^{2}}{2(Q(y^{(i)})+1)}=Q(y^{(i)}).$
(42)

Note that
$\left.\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}\;\right|_{\hat{t}^{(i)}=0}=1$. Since
$Q(y^{(i)})-\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})>0$,

$R(J^{(i)})=|Q({y}^{(i)})-\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})|=Q({y}^{(i)})-\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)}).$
(43)

Furthermore, since $\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ is
decreasing in $\hat{t}^{(i)}$, $R(J^{(i)})$ must increase in $\hat{t}^{(i)}$,
from which the theorem follows. ∎

######  Remark B.3.

We comment on the potential for DCEM to improve robustness to low overlap. To
do so, we analyze the sensitivity of the M-step optimum to extreme
$\hat{t}^{(i)}$. While analyzing the asymptotic variance of consistent
estimators is a common approach, asymptotic guarantees for DCEM are unclear
due to the inherently non-convex (with respect to the parameters) objective
function. Thus, we analyze the Lipschitzness of the M-step optimum versus
other causal effect estimators. First, note that

$\frac{d}{d\hat{t}}\hat{y}_{OPT}(Q(y),\hat{t})=\frac{1-2Q(y)^{2}\hat{t}^{2}-\sqrt{C}}{2(Q(y)+1)\hat{t}^{2}\sqrt{C}}$
(44)

where $C=4Q(y)^{2}\hat{t}^{2}-4Q(y)^{2}\hat{t}+1$. For all $Q(y^{(i)})<1$ and
all $\hat{t^{(i)}}$, this derivative is bounded (_e.g._ , see Figure 5), and
is $O(1)-$Lipschitz in $\hat{t}^{(i)}$. However, consider the expression for
an inverse-propensity-weighted estimator, which sums terms of the form

$\frac{y^{(i)}t^{(i)}}{\hat{t}^{(i)}}-\frac{y^{(i)}(1-t^{(i)})}{1-\hat{t}^{(i)}}$
(45)

to obtain a final estimate. Eq. 45 has $O(\hat{t}^{(i)2})$-Lipschitz terms
with respect to $\hat{t}^{(i)}$. Thus, in a Lipschitz sense, DCEM may be less
sensitive to extreme propensity scores than causal effect estimators such as
IPW.

######  Corollary B.4.

For an example indexed by $i$, $Q(y^{(i)})=1$, and $J^{(i)}$ defined as in Eq.
88, $R(J^{(i)})$ is monotonically non-decreasing in $\hat{t}^{(i)}$ on
$(0,1]$.

###### Proof.

The proof is identical to that of Theorem 3.4, except we find that

$\frac{\partial}{\partial\hat{t}^{(i)}}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})\leq
0,$ (46)

instead of being strictly less than zero, from which the corollary follows. ∎

######  Remark B.5.

For intuition, we show a contour plot of
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ in Fig. 5. We verify
the result in CVXPY.

Figure 5: Contour plot of
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ with respect to
$\hat{t}^{(i)}$ ($x$-axis) and $Q(y^{(i)})$ ($y$-axis).
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ scales with
$Q(y^{(i)})$ but decreases in $\hat{t}^{(i)}$.

###  B.4 Proposition 3.5

######  Proposition (Minimizer of M-step when $t^{(i)}=1$).

Suppose that $t^{(i)}=1$, let $Q(y^{(i)})\triangleq\mathbb{E}[y=1\mid
t,\tilde{y},a,\mathbf{x}]$, and let $\hat{y}^{(i)}$ be some estimate of
$y^{(i)}$. Use $\mathcal{L}:[0,1]^{2}\to\mathbb{R}_{+}$ be shorthand for
binary cross-entropy loss. Then, the minimization problem

$\underset{\hat{y}}{\min}\;\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})+Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)})$
(47)

admits the solution $\hat{y}^{(i)}=y^{(i)}$ for all $i\in\\{1,\dots,N\\}$.

###### Proof.

We briefly verify the convexity of the objective, which follows from the
convexity of binary cross-entropy loss and the closure of convexity under
addition and positive scalar multiplication ($Q(y^{(i)})\geq 0$). Thus, any
minimizer of the objective is unique.

We proceed by cases. First, suppose that $y^{(i)}=1$. Substituting the
definition of $Q(y^{(i)})$, and using the fact that $t^{(i)}=1\implies
y^{(i)}=\tilde{y}^{(i)})$, the objective function for a single example becomes

$\mathcal{L}(1,\hat{y}^{(i)})+\mathcal{L}(1,\hat{y}^{(i)}\hat{t}^{(i)}),$ (48)

which, by inspection, is maximized for $\hat{y}^{(i)}=1$. Similarly, for
$y^{(i)}=0$, the objective function for a single example is

$\mathcal{L}(0,\hat{y}^{(i)})$ (49)

which reduces to binary cross-entropy loss, and $\hat{y}^{(i)}=0$ minimizes
the objective. Combining the two cases, the minimizer of the M-step objective
when $t^{(i)}=1$ is $\hat{y}^{(i)}=y^{(i)}$ as desired. ∎

###  B.5 Causal identifiability

For completeness, we provide the derivation of the causal identifiability
results, though it follows directly from existing results (Imbens & Rubin,
2015).

######  Proposition.

Suppose that conditional exchangeability, or $\tilde{Y}(t)\perp\\!\\!\\!\perp
T\mid X$, holds. Then $\mathbb{E}[Y\mid X]=\mathbb{E}[\tilde{Y}\mid
X,do(T=1)]$, which is identifiable as $\mathbb{E}[\tilde{Y}\mid X,T=1]$.

###### Proof.

We can write

$\displaystyle\mathbb{E}[Y\mid X]=\mathbb{E}[Y\mid
X,T=1]=\mathbb{E}[\tilde{Y}\mid X,T=1]=\mathbb{E}[\tilde{Y}\mid X,do(T=1)]$
(50)

where the first equality is due to $Y\perp\\!\\!\\!\perp T\mid X$, the second
equality results from $T=1\implies Y=\tilde{Y}$, and the final equality
applies conditional exchangeability. Since $\mathbb{E}[Y\mid
X]=\mathbb{E}[\tilde{Y}\mid X,do(T=1)]=\mathbb{E}[\tilde{Y}\mid X,T=1]$, the
theorem follows. ∎

###  B.6 Lemmas used

Below are the lemmas and proofs referenced in the preceding theorem and
proposition proofs.

######  Lemma 1.

Define $B(Q(y^{(i)}),\hat{t}^{(i)})\triangleq 2Q(y^{(i)})\hat{t}^{(i)}+1$ and
$D(Q(y^{(i)}),\hat{t}^{(i)})\triangleq(-B(Q(y^{(i)}),\hat{t}^{(i)}))^{2}-4Q(y^{(i)})(Q(y^{(i)})\hat{t}^{(i)}+\hat{t}^{(i)})$
on $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$. Then,
$D(Q(y^{(i)}),\hat{t}^{(i)})\geq 0$.

###### Proof.

Choose any $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$. We can write:

$\displaystyle D(Q(y^{(i)},\hat{t}^{(i)})\geq 0$ (51) $\displaystyle\iff$
$\displaystyle B(Q(y^{(i)}),\hat{t}^{(i)})^{2}\geq
4Q(y^{(i)})(Q(y^{(i)})\hat{t}^{(i)}+\hat{t}^{(i)})$ (52) $\displaystyle\iff$
$\displaystyle 4Q(y^{(i)})^{2}\hat{t}^{(i)2}+4Q(y^{(i)})\hat{t}^{(i)}+1\geq
4Q(y^{(i)})(Q(y^{(i)})\hat{t}^{(i)}+\hat{t}^{(i)})$ (53) $\displaystyle\iff$
$\displaystyle
4Q(y^{(i)})^{2}\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1\geq 0.$ (54)

The final LHS is convex (by inspection) in $\hat{t}^{(i)}$, such that

$\displaystyle\underset{\hat{t}^{(i)}}{\min}\;4Q(y^{(i)})^{2}\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1=1-Q(y^{(i)})\geq
0$ (55)

where the minimum is attained at $\hat{t}^{(i)}=\frac{1}{2}$, and concave (by
inspection) in $Q(y^{(i)})$, such that it suffices to evaluate the final LHS
at $Q(y^{(i)})\in\\{0,1\\}$:666Recall that, for a concave function $f$,
$f(\alpha x+(1-\alpha)y)\geq\alpha f(x)+(1-\alpha)f(y)$ for $\alpha\in[0,1]$
with equality for $\alpha=0$ or 1. Thus, via the extreme value theorem, the
minimum of $f$ on $[x,y]$ is achieved at $x$ or $y$.

$\displaystyle\left.4Q(y^{(i)})^{2}\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1\right|_{Q(y^{(i)})=0}$
$\displaystyle=1\geq 0$ (56)
$\displaystyle\left.4Q(y^{(i)})^{2}\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1\right|_{Q(y^{(i)})=1}$
$\displaystyle=4(\hat{t}^{(i)2}-\hat{t}^{(i)})+1\geq
4\cdot\left(-\frac{1}{4}\right)+1=0,$ (57)

such that for all other $Q(y^{(i)})\in(0,1)$,
$4Q(y^{(i)})^{2}\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1\geq 0$ as
needed.

∎

######  Lemma 2.

Define $B(Q(y^{(i)}),\hat{t}^{(i)})$ and $D(Q(y^{(i)},\hat{t}^{(i)})$ as in
Lemma 1. Then, for $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$,

$\frac{B(Q(y^{(i)}),\hat{t}^{(i)})+\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}\geq
1.$ (58)

###### Proof.

Choose any $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$. First, we rewrite

$\displaystyle\frac{B(Q(y^{(i)}),\hat{t}^{(i)})+\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}\geq
1$ (59) $\displaystyle\iff$ $\displaystyle
2Q(y^{(i)})\hat{t}^{(i)}+1+\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\geq
2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})$ (60) $\displaystyle\iff$
$\displaystyle\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\geq 2\hat{t}^{(i)}-1.$ (61)

For $\hat{t}^{(i)}\in(0,\frac{1}{2})$, Lemma 1 yields the desired conclusion.
For $\hat{t}^{(i)}\in[\frac{1}{2},1]$, we can write

$\displaystyle\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\geq 2\hat{t}^{(i)}-1$ (62)
$\displaystyle\iff$ $\displaystyle
4Q(y^{(i)2})\hat{t}^{(i)2}-4Q(y^{(i)}2)\hat{t}^{(i)}+1\geq
4\hat{t}^{(i)2}-4\hat{t}^{(i)}+1$ (63) $\displaystyle\iff$ $\displaystyle
Q(y^{(i)2})(\hat{t}^{(i)}-1)\geq\hat{t}^{(i)}-1$ (64) $\displaystyle\iff$
$\displaystyle Q(y^{(i)2})\leq 1$ (65)

which all $Q(y^{(i)2})\in[0,1]$ satisfy. This completes the proof. ∎

######  Lemma 3.

Define $B(Q(y^{(i)}),\hat{t}^{(i)})$ and $D(Q(y^{(i)},\hat{t}^{(i)})$ as in
Lemma 1. Then, for $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$,

$0\leq\frac{B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})}\leq
1.$ (66)

###### Proof.

Choose any $Q(y^{(i)})\in[0,1]$ and $\hat{t}^{(i)}\in(0,1]$. Equivalently, we
can show

$0\leq B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\leq
2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)}).$ (67)

For the first inequality, note that

$\displaystyle\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}$
$\displaystyle=\sqrt{(-B(Q(y^{(i)}),\hat{t}^{(i)}))^{2}-4Q(y^{(i)})(Q(y^{(i)})\hat{t}^{(i)}+\hat{t}^{(i)})}\leq\sqrt{(-B(Q(y^{(i)}),\hat{t}^{(i)}))^{2}}$
(68)
$\displaystyle=|B(Q(y^{(i)}),\hat{t}^{(i)}))|=B(Q(y^{(i)}),\hat{t}^{(i)}))$
(69)

which rearranges to $0\leq
B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}$ as desired. For
the second inequality, note that

$\displaystyle
B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\leq
2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})$ (70) $\displaystyle\iff$
$\displaystyle 1-\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\leq 2\hat{t}^{(i)}$ (71)
$\displaystyle\iff$ $\displaystyle
1-2\hat{t}^{(i)}\leq\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}.$ (72)

For $\hat{t}^{(i)}\in[\frac{1}{2},1]$, Lemma 1 yields the desired conclusion.
For $\hat{t}^{(i)}\in(0,\frac{1}{2})$, the proof proceeds similarly to Lemma
58:

$\displaystyle\sqrt{D(Q(y^{(i)},\hat{t}^{(i)})}\geq 1-2\hat{t}^{(i)}$ (73)
$\displaystyle\iff$ $\displaystyle
4Q(y^{(i)2})\hat{t}^{(i)2}-4Q(y^{(i)})^{2}\hat{t}^{(i)}+1\geq
4\hat{t}^{(i)2}-4\hat{t}^{(i)}+1$ (74) $\displaystyle\iff$ $\displaystyle
Q(y^{(i)2})(\hat{t}^{(i)}-1)\geq\hat{t}^{(i)}-1$ (75) $\displaystyle\iff$
$\displaystyle Q(y^{(i)2})\leq 1$ (76)

which all $Q(y^{(i)2})\in[0,1]$ satisfy. This completes the proof. ∎

######  Lemma 4.

Define $B(Q(y^{(i)}),\hat{t}^{(i)})$ and $D(Q(y^{(i)},\hat{t}^{(i)})$ as in
Lemma 1, and define $\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$ as
in Definition 88. Then,

$\text{sign}\left(\frac{\partial}{\partial\hat{t}^{(i)})}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})\right)=\text{sign}\left(1-2\hat{t}^{(i)}Q(y^{(i)})^{2}-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})\right)$
(77)

where $\text{sign}(x):\mathbb{R}\to\\{-1,0,1\\}$ is the sign function:

$\text{sign}(x)\triangleq\begin{cases}-1&x<0\\\ 0&x=0\\\ 1&x>0\end{cases}.$
(78)

###### Proof.

The proof is largely algebraic simplification based on sign-preserving
operations. Taking derivatives:

$\frac{\partial}{\partial\hat{t}^{(i)}}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})=\frac{Q(y^{(i)})-\frac{2\hat{t}^{(i)}Q(y^{(i)})^{2}-Q(y^{(i)})^{2}}{\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}}}{\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)}}-\frac{\left(B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})\right)(Q(y^{(i)})+1)}{2(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})^{2}}$
(79)

via the quotient rule of derivatives and cancelling terms. We can apply sign-
preserving operations, namely, positive scalar multiplication, canceling
additive zeroes, and commuting additive terms, as follows:

$\displaystyle\frac{\partial}{\partial\hat{t}^{(i)}}\;\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$
$\displaystyle\propto(\hat{t}^{(i)}+Q(y^{(i)})\hat{t}^{(i)})\left(2Q(y^{(i)})-\frac{4\hat{t}^{(i)}Q(y^{(i)})^{2}-2Q(y^{(i)})^{2}}{\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}}\right)$
$\displaystyle\quad-\left(B(Q(y^{(i)}),\hat{t}^{(i)})-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})\right)(Q(y^{(i)})+1)$
(80)
$\displaystyle\propto\hat{t}^{(i)}\left(2Q(y^{(i)})-\frac{4\hat{t}^{(i)}Q(y^{(i)})^{2}-2Q(y^{(i)})^{2}}{\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}}\right)-B(Q(y^{(i)}),\hat{t}^{(i)})+\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})$
(81)
$\displaystyle=\left(\frac{2\hat{t}^{(i)}Q(y^{(i)})^{2}-4\hat{t}^{(i)2}Q(y^{(i)})^{2}}{\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})}}\right)-1+\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})$
(82) $\displaystyle\propto
2\hat{t}^{(i)}Q(y^{(i)})^{2}-4\hat{t}^{(i)2}-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})+D(Q(y^{(i)}),\hat{t}^{(i)})$
(83)
$\displaystyle=1-2\hat{t}^{(i)}Q(y^{(i)})^{2}-\sqrt{D(Q(y^{(i)}),\hat{t}^{(i)})})$
(84)

which completes the proof. ∎

###  B.7 Definition 3.3: causal regularization strength

We expand on our definition of causal regularization strength here.
Conventionally, regularization strength is operationalized in terms of a
regularization parameter $\lambda\in\mathbb{R}_{+}$, given a loss
$\ell(\theta)$ and a regularizer $r(\theta)$ (_e.g._ ,
$r(\theta)=\lVert\theta\rVert_{2}^{2}$ for some objective $J(\theta)$ of the
form

$J(\theta)\triangleq\ell(\theta)+\lambda r(\theta).$ (85)

Eq. 85 is an instance of _regularized risk minimization_ (Shalev-Shwartz &
Ben-David, 2014). It is also identical to _linear scalarization_ , a technique
for characterizing tradeoffs in multi-objective optimization. The equivalence
between regularized risk minimization and linear scalarization simply reflects
that regularization can impose tradeoffs in optimizing $J(\theta)$ between
minimizing $\ell(\theta)$ versus $r(\theta)$. Regularized risk minimization
treats $r(\theta)$ as a “penalty” term, while linear scalarization treats
$r(\theta)$ as merely another objective. As $\lambda$ increases, the tradeoff
increasingly favors $r(\theta)$, and vice versa.

Now, consider our M-step objective example-wise:

$\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})+Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)}).$
(86)

The M-step objective can similarly be interpreted as variation of a
regularized risk minimziation problem, where
$\ell(\theta)=\mathcal{L}(Q(y^{(i)},\hat{y}^{(i)})$, and
$Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)},\hat{t}^{(i)})=r(\theta),\lambda=1$.
However, $\hat{t}^{(i)}$ is a constant that can affect regularization
strength, but is not a multiplier like $\lambda$. The purpose of our result is
to characterize the impact of $\hat{t}^{(i)}$ on the tradeoff between the two
terms of the M-step objective.

Thus, motivated by the tradeoff/multi-objective perspective of regularization,
we define regularization strength in terms of a tradeoff between optimizing
$\mathcal{L}(Q(y^{(i)},\hat{y}^{(i)})$ and
$Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)})$. We
observe that

$\displaystyle Q(y^{(i)})$
$\displaystyle=\underset{\hat{y}^{(i)}}{\arg\min}\;\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})$
(87)

and define causal regularization strength as the absolute distance between
$Q(y^{(i)})$, the minimizer of $\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})$, and
the optimum of the example-wise M-step objective.

######  Definition B.6 (Causal regularization strength).

Given an example indexed by $i$, and a finite loss function
$\mathcal{L}:[0,1]^{2}\to\mathbb{R}$ convex in $\hat{y}^{(i)}$ on [0, 1] for
all $i$, define

$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})\triangleq\underset{\hat{y}^{(i)}}{\arg\min}\;J^{(i)}(\hat{y}^{(i)},\dots)\triangleq\underset{\hat{y}^{(i)}}{\arg\min}\;\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})+Q(y^{(i)})\mathcal{L}(\tilde{y}^{(i)},\hat{y}^{(i)}\hat{t}^{(i)}).$
(88)

The causal regularization strength of objective $J^{(i)}$ is defined as
$R(J^{(i)})=|Q({y}^{(i)})-\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})|$.

Intuitively, we define causal regularization strength in terms of the absolute
distance between the optimum of each term of the M-step objective, which
captures some notion of a tradeoff between the two terms. Note that this
definition does not relate to convergence to
$\hat{y}^{(i)}_{\text{OPT}}(Q(y^{(i)}),\hat{t}^{(i)})$; we are largely
interested in how much the solution to
$\min\;\mathcal{L}(Q(y^{(i)}),\hat{y}^{(i)})$ shifts after adding the causal
regularization term.

##  Appendix C Additional experimental setup

For both settings, we set random seeds to 42 to facilitate reproducibility.

###  C.1 Additional details for fully synthetic dataset

We choose $s_{Y}$ as follows:

$\displaystyle S_{Y}(x)$ $\displaystyle=(s_{Y1}\circ s_{Y2})(\mathbf{x})$
$\displaystyle s_{Y1}(\mathbf{x})$ $\displaystyle=x_{1}-\frac{1}{4}\sin(8\pi
x_{0}+\psi)\quad s_{Y2}(\mathbf{x})=\mathbf{R}_{\pi/6}\mathbf{x}+0.5$

where $\psi$ is a simulation parameter, and $\mathbf{R}_{\pi/6}$ is a 2D
rotation matrix. Intuitively, $s_{Y}(\mathbf{x})$ rotates and translates
$\mathbf{x}$, then applies a sinewave-based function that yields a similarly
rotated, sinewave-shaped decision boundary.

We choose $s_{T}$ as follows:

$s_{T}(\mathbf{x}^{(i)},a^{(i)})=\mathbf{1}^{\top}\mathbf{x}^{(i)}-\tau_{a^{(i)}}$

where $\tau_{a}$ is a simulation parameter. For demonstration, we set $c_{y}$
such that $P(Y=1)=0.25$ to allow for sufficiently-sized performance gaps
across groups to emerge.777Empirically, at extreme values of $P(Y=1)$, we
found artificially small performance gaps. This is because model errors tend
to concentrate near the true decision boundary, which lies in the tails of the
covariate distributions defining $X\mid A=a$. In those tail regions, the
difference between the densities $X\mid A=a$ across values of $A$ is smaller
in our two-Gaussian simulation design. As a sensitivity analysis, we also
replicate all experiments on fully synthetic data across
$\psi\in[0,\pi/6,\pi/3,\dots,11\pi/6]$, representing the “phase” of the
decision boundary.

#### Computing simulation parameters.

We discuss how we find simulation parameters for each value of $q_{y},q_{t}$,
and $k$. Given $q_{y}$ and $P(A=0)=P(A=1)=0.5$, we have:

$\displaystyle q_{y}$ $\displaystyle=\frac{P(Y=1\mid A=0)}{P(Y=1\mid A=1)}$
$\displaystyle P(Y=1)=\frac{1}{4}$ $\displaystyle=\frac{P(Y=1\mid
A=0)+P(Y=1\mid A=1)}{2}$

which yields, by substitution,

$\displaystyle\frac{q_{y}}{2(q_{y}+1)}$ $\displaystyle=P(Y=1\mid A=0)$
$\displaystyle\frac{1}{2(q_{y}+1)}$ $\displaystyle=P(Y=1\mid A=1),$

from which we use a binary search algorithm (bisection) evaluated using
simulated versions of $X\mid A=a$ with the current estimate of the mean
$\mu_{a}$ to solve for the requisite values of $\mu_{a}$. Given values of
$\mu_{a}$, we can then solve for $\tau_{a}$ using $q_{t}$ and $k$ identically:

$\displaystyle q_{t}$ $\displaystyle=\frac{P(T=1\mid A=0)}{P(T=1\mid A=1)}$
$\displaystyle P(T=1)=kP(Y=1)=\frac{k}{4}$ $\displaystyle=\frac{P(T=1\mid
A=0)+P(T=1\mid A=1)}{2}$

which yields, again by substitution,

$\displaystyle\frac{q_{t}k}{2(q_{t}+1)}$ $\displaystyle=P(T=1\mid A=0)$
$\displaystyle\frac{k}{2(q_{t}+1)}$ $\displaystyle=P(T=1\mid A=1),$

and we can again use binary search to solve for $\tau_{a}$.

###  C.2 Additional details for pseudo-synthetic sepsis risk-stratification
task

Figure 6: Cohort diagram for our Sepsis-3 cohort ($N=5301$). *: Further
details are provided in (Johnson et al., 2018). ^: Our cohort size differs
slightly from that reported in (Johnson et al., 2018) due to an apparent pre-
processing error in defining Sepsis-3 (Singer et al., 2016); our reported
cohort size applies the relevant correction.

#### Cohort description.

Our cohort follows from the MIMIC-III Sepsis-3 cohort (Johnson et al., 2018).
Their cohort exclusion criteria are publicly
available.888https://github.com/alistairewj/sepsis3-mimic We corrected an
apparent Sepsis-3 definition bug that erroneously labeled individuals with
suspicion of infection if they received a blood culture at any time after an
antibiotic before re-running their pipeline. In contrast, the Sepsis-3 (Singer
et al., 2016) definition requires the blood culture to occur _within 24 hours_
of the antibiotic time for suspicion of infection.999There are multiple
“paths” for meeting the criteria for suspicion of infection; for a full
enumeration, see Table 2 of (Singer et al., 2016). In practice, this stricter
condition affects $<1\%$ of rows in their original cohort: their cohort size
is $N=11,791$, while ours is $N=11,705$.

#### Feature extraction.

Following the Risk of Sepsis model (Delahanty et al., 2019), we extract the
following 13 summary statistics over the initial 3-hour observation period:

  1. 1.

Maximum lactic acid measurement,

  2. 2.

first shock index times age (years),

  3. 3.

last shock index times age (years),

  4. 4.

maximum white blood cell count,

  5. 5.

change in lactic acid (last - first),

  6. 6.

maximum neutrophil count,

  7. 7.

maximum blood glucose,

  8. 8.

maximum blood urea nitrogen,

  9. 9.

maximum respiratory rate,

  10. 10.

last albumin measurement,

  11. 11.

minimum systolic blood pressure,

  12. 12.

maximum creatinine, and

  13. 13.

maximum body temperature (Fahrenheit).

The shock index is defined as the ratio of heart rate (beats per minute) and
systolic blood pressure. Missing features are replaced with -9999 following
the original manuscript.

#### Testing decision boundary.

We define $s_{T}$ as

$\beta\cdot\frac{RR_{\max}-22}{RR_{\sigma}}+(1-\beta)\cdot\frac{SBP_{\min}-100}{SBP_{\sigma}}-\tau_{a},$
(89)

where $RR_{\max}$ and $SBP_{\min}$ are maximum respiratory rate and minimum
systolic blood pressure, respectively, and $RR_{\sigma},SBP_{\sigma}$ are
their corresponding standard deviations on the training split
($RR_{\sigma}=9.8,SBP_{\sigma}=21.8$). The parameter $\beta$ allows us to
examine different testing decisions. Thus, we replicate all experiments over
$\beta\in\\{0,0.1,\dots,1\\}$.

##  Appendix D Hyperparameters & additional model details

All hyperparameters were selected using a validation set of examples.
Hyperparameters for the sepsis simulation task were chosen such that all
approaches attained similar performance when using $y$. We reimplement all
existing methods following the original papers, using the code repository as a
reference if applicable. We set random seeds to 42 for all models (used for
initialization), unless otherwise noted.

###  D.1 Default hyperparameters

#### Fully synthetic.

All models use a two-layer neural network with layer sizes $(64,64)$, trained
for 1000 epochs via Adam (Kingma & Ba, 2015) with learning rate $10^{-3}$ and
no weight decay unless specified. EM approaches are trained up to 50
iterations with early stopping on validation loss (patience 3) and warm starts
(initialized with solution from the previous iteration).

#### Sepsis classification.

All predictors are three-layer neural networks with sizes $(128,128,16)$
trained for 10000 epochs using Adam with learning rate $10^{-5}$ and weight
decay $10^{-3}$. The DCEM propensity model ($g_{\zeta}$) is trained for 20000
epochs with learning rate $10^{-5}$ and early stopping with patience 1000, and
the DCEM model ($f_{\theta}$) uses learning rate and weight decay $5\times
10^{-7}$ and $10^{-6}$, respectively.

###  D.2 Simulation study

#### Peer loss & group peer loss:

Both peer loss methods depend on a hyperparameter $\alpha$, for which the
optimal value depends on $y$. To show the peer loss methods in the best light,
we manually calculate the optimal value for usage in training.

#### ITE corrected model (DragonNet):

Our estimand of interest is the conditional average treatment effect (CATE) of
the sensitive attribute $A$ on testing $T$, which is identifiable via

$\text{CATE}_{A\to T}(\mathbf{x})\triangleq\mathbb{E}[T(1)\mid
X=\mathbf{x}]-\mathbb{E}[T(0)\mid X=\mathbf{x}]=\mathbb{E}[T\mid
X=\mathbf{x},A=1]-\mathbb{E}[T\mid X=\mathbf{x},A=0]$ (90)

under assumptions of consistency ($T(a)=T$) and conditional exchangeability
($T(a)\perp\\!\\!\\!\perp A\mid X$). We then apply the CATE as a correction
factor to the default model:

$\hat{y}\triangleq\hat{\tilde{y}}-\text{CATE}_{A\to T}(\mathbf{x});$ (91)

_i.e._ , counterbalancing disparate censorship by “subracting out” the
labeling bias. Note that this is an alternative to the counterbalancing
approach of DCEM. We train and conduct inference with targeted regularization.

#### Truncated LQ:

We searched across $k\in\\{0.1,0.2,\dots,1\\}$ and $q\in\\{0.1,0.2,\dots,1\\}$
(using the notation of the original paper), using $k=0.1,q=0.1$ for the final
results.

#### SELF:

We were unable to obtain convergence with Adam, so we used SGD with learning
rate 0.01, momentum 0.9, noise parameter 0.05 (for input augmentation),
consistency regularization parameter 1, and weight decay $1\times 10^{-6}$ as
used for one of the experiments in the original paper. Weight decay was
selected from $\\{0,10^{-6},10^{-5},10^{-4},10^{-3},10^{-2}\\}$. The
ensembling/mean teacher parameters were chosen from $\\{0.9,0.99,0.999\\}$.
The noise was chosen from $\\{0,0.005,0.01,0.05,0.1\\}$. The regularization
parameter was chosen from $\\{1,5,10,50\\}$. SELF proceeds for a maximum of 50
iterations with patience 1 with respect to validation AUC. We set ensembling
momentum to 0.9 and the mean teacher moving average parameter to 0.9. To show
SELF in the best light, we prevented SELF from filtering tested positive
individuals.

#### DivideMix:

We use 20 warmup epochs, with $\alpha=4$ as the Beta parameter for the
MixMatch step, $T=0.5$, $\lambda_{u}=50$, $\lambda_{r}=1$, and $\tau=0.5$, and
weight decay $5\times 10^{-4}$. We also experimented with preventing DivideMix
from filtering tested positive individuals, but DivideMix was unstable in both
settings. Ultimately, we did not prevent DivideMix from filtering tested
positive individuals.

#### EM-based methods (SAREM, DCEM):

We tested SAREM and DCEM with and without the usage of warm starts in the
M-step.

###  D.3 Sepsis risk-stratification

For all baselines, the setup matches the fully synthetic setting except as
specified below.

#### DCEM:

The learning rates under consideration were $[10^{-7},5\times
10^{-7},10^{-6},5\times 10^{-6},10^{-5},10^{-4},10^{-3}]$. The weight decay
was selected from $[0,10^{-6},2\times 10^{-6}]$.

#### SELF:

For the sepsis classification experiments, weused SGD and set the learning
rate to $10^{-8}$, the highest learning rate tested that did not result in NaN
loss. We tested learning rates of the form $\\{10^{-d},5\times 10^{-d}\\}$ for
$d\in\\{2,3,4,5,6,7,8\\}$.

##  Appendix E Additional empirical results and discussion

###  E.1 Full results

Here, we report empirical results for all baselines and settings. Due to the
large number of empirical settings tested (simulation: 224, sepsis
classification: 45), we include a representative subset of the figures, and
report the raw numbers used for these results _and_ results not shown in the
Appendix via CSV files in the code appendix.

For the simulated task, we show results for $k\in[1/3,3]$, $q_{t}\in[0.5,2]$,
and $q_{y}=0.5$. Empirically, changing $q_{y}$ did not affect the general
trends, but amplified/dampened the scale. Increasing $q_{t}$ beyond the
selected range has similar impacts. For lower values of $k$, all methods
perform poorly. For the sepsis classification task, we show results for
$k\in[1/3,5]$ and $q_{t}=1.5$.

#### Summary of results.

We summarize when our method (DCEM) empirically performed the best, when it
performed similarly to baselines, and when it underperformed baselines.

#### DCEM is best where…

  * •

(Both metrics) The higher-prevalance group is undertested ($q_{y}<1$ but
$q_{t}>1$) _and_

  * •

(Both metrics) testing rates are sufficiently high ($k\geq 0.5$).

#### DCEM is similar to baselines when…

  * •

(Both metrics) Testing rates are moderately low ($1/3\leq k\leq 1/2$), or
sufficiently high that it is easier to extrapolate from labeled data ($k\geq
3$).

#### DCEM underperforms baselines when…

  * •

(ROC gap only) when testing rates are low ($k\leq 1/2$) and

  * •

(ROC gap only) the testing disparity aligns with the prevalence disparity
(_e.g._ , $q_{t}$ and $q_{y}<1$ such that learning to predict $\tilde{y}$
preserves ranking in $y$), _or_

  * •

(both metrics) testing rates are extremely low ($k<1/3$).

The strongest alternatives to DCEM in our experiments were SELF (both
datasets, bias mitigation), DragonNet (sepsis only, both metrics), and the
tested-only model (simulation only, discriminative performance).

#### Index of figures.

We provide here a list of all result figures in the Appendix, indexed by
problem parameters $k$ (overall testing rate multiplier), $q_{t}$ (testing
disparity), and $q_{y}$ (prevalence disparity; simulation only).

#### Fully-synthetic data

  * •

Figure 11: $q_{y}=1/2,k=1/3,q_{t}=1/2$

  * •

Figure 12: $q_{y}=1/2,k=1/3,q_{t}=1$

  * •

Figure 13: $q_{y}=1/2,k=1/3,q_{t}=2$

  * •

Figure 14: $q_{y}=1/2,k=1/2,q_{t}=1/2$

  * •

Figure 15: $q_{y}=1/2,k=1/2,q_{t}=1$

  * •

Figure 16: $q_{y}=1/2,k=1/2,q_{t}=2$

  * •

Figure 17: $q_{y}=1/2,k=1,q_{t}=1/2$

  * •

Figure 18: $q_{y}=1/2,k=1,q_{t}=1$

  * •

Figure 19: $q_{y}=1/2,k=1,q_{t}=2$

  * •

Figure 20: $q_{y}=1/2,k=2,q_{t}=1/2$

  * •

Figure 21: $q_{y}=1/2,k=2,q_{t}=1$

  * •

Figure 22: $q_{y}=1/2,k=2,q_{t}=2$

  * •

Figure 23: $q_{y}=1/2,k=3,q_{t}=1/2$

  * •

Figure 24: $q_{y}=1/2,k=3,q_{t}=1$

  * •

Figure 25: $q_{y}=1/2,k=3,q_{t}=2$

#### Sepsis classification

  * •

Figure 26: $k=1/4,q_{t}=3/2$

  * •

Figure 27: $k=1/3,q_{t}=3/2$

  * •

Figure 28: $k=1/2,q_{t}=3/2$

  * •

Figure 29: $k=1,q_{t}=3/2$

  * •

Figure 30: $k=2,q_{t}=3/2$

  * •

Figure 31: $k=3,q_{t}=3/2$

  * •

Figure 32: $k=4,q_{t}=3/2$

  * •

Figure 33: $k=5,q_{t}=3/2$

###  E.2 DCEM ablation study

To understand how DCEM design choices impact performance, we conduct an
ablation study of repeated iterations and causal regularization:

  * •

Imputation-only: This approach trains a model on the tested-only (labeled)
examples, imputes pseudo-labels for the remaining, then trains a model on both
the pseudo-labeled and labeled data. This is equivalent to a single EM
iteration without causal regularization.

  * •

No causal regularization: This approach runs multiple EM iterations, but
without causal regularization.

The results (Table 1) suggest that both repeated iterations and causal
regularization are essential to the bias mitigation and discriminative
capabilities of DCEM. The imputation-only approach fails due to low overlap
between the tested vs. untested regions. Consequently, the imputed outcomes
could be arbitrarily inaccurate. If we keep imputing and retraining (without
causal regularization), we recover a form of pseudo-labeling (Lee, 2013). The
empirical improvement in performance suggests that repeated supervision from
reliably labeled examples helps improve discriminative performance. However,
this approach does not adjust for labeling bias (e.g., by using $A$), and
indeed the ROC gap does not improve. Incorporating causal regularization
recovers the DCEM M-step. Adding causal regularization guarantees that DCEM
locally maximizes log-likelihood, and allows it to mitigate labeling bias by
incorporating $A$ into a propensity score-like term (causal regularization;
see Theorem Theorem).

Table 1: Sensitivity analysis of DCEM components with respect to AUC and ROC gap (min, max across $s_{T}$ in parentheses) for $q_{y}=0.5,k=1,q_{t}=2$. Maximum (minimum) median AUC (ROC gap) in bold. Method |  $\uparrow$ AUC |  $\downarrow$ ROC gap
---|---|---
Imputation-only | .676 [.644, .715] | .063 [.036, .086]
No causal regularization | .767 [.733, .813] | .056 [.016, .086]
DCEM (ours) | .791 [.763, .820] | .031 [.019, .072]
Table 2: Sensitivity analysis of causal effect estimators for estimating $P(Y\mid X)$ compared to DCEM with respect to AUC and ROC gap (min, max across $s_{T}$ in parentheses) for $q_{y}=0.5,k=1,q_{t}=2$. Maximum (minimum) median AUC (ROC gap) in bold. Method |  $\uparrow$ AUC |  $\downarrow$ ROCGap
---|---|---
Tested-only | .808 [.623, .876] | .052 [.020, .093]
Tested-only + group | .764 [.675, .863] | .078 [.025, .278]
IPW | .829 [.598, .874] | .048 [.020, .104]
DR-Learner | .643 [.558, .769] | .117 [.080, .216]
DCEM (ours) | .791 [.763, .820] | .031 [.019, .072]
Figure 7: Sensitivity analysis of causal effect estimators with respect to AUC
and ROC gap (min, max across $s_{T}$ in parentheses) for
$q_{y}=0.5,k=1,q_{t}=2$ across varying levels of overlap. Causally-motivated
methods are shown in green, while DCEM is shown in magenta. Empirically, DCEM
improves robustness to overlap violations.

###  E.3 Sensitivity analysis of causally-motivated approaches

Here, we conduct a sensitivity analysis of causally-motivated approaches under
disparate censorship. The causally-motivated approaches are theoretically
consistent estimators of $P(Y\mid X)$, which we can interpret at the
conditional average treatment effect of testing ($T$) on the observed outcome
($\tilde{Y}$; see Appendix B.5). We examine the following causally-motivated
estimators:

  * •

Tested-only: training models on tested individuals only, using $X$ as
covariates,

  * •

Tested-only + group: training models on tested individuals only, using $X$ and
$A$ as covariates,

  * •

Inverse propensity weighting (IPW): an IPW-based (Rosenbaum & Rubin, 1983)
version of the tested-only approach, and

  * •

Doubly-robust estimator (DR-Learner): a doubly-robust estimator of $P(Y\mid
X)$ (Kennedy, 2023).

Models are evaluated for $q_{y}=0.5,k=1,q_{t}=2$ (_i.e._ , same setting as
Fig. 2). Under disparate censorship, low overlap is common due to the
“sharpness” of the testing boundary. To validate this hypothesis, we also
evaluate causal effect estimators versus DCEM at varying levels of overlap
($1/4$x, $1/2$x, $1$x, $2$x, and $4$x of the original setting). Overlap is
controlled by the coefficient inside the sigmoid for generating $t^{(i)}$
(_i.e._ , 30 in the original experiments).101010Recall that $t^{(i)}$ is
generated as a Bernoulli random variable with parameters of the form
$\sigma(ax+b)$. For the DR-learner, we trimmed propensity scores (threshold:
0.05) to obtain estimates that were in [0, 1] (the possible values of $P(Y\mid
X)$).

#### DCEM has better bias mitigation capabilities than causal approaches, and
a tighter range of discriminative performance.

Table 2 shows that, empirically, DCEM exhibits lower variance under overlap
violations than causally-motivated approaches. Notably, DCEM achieves the
lowest median ROC gap, and maintains competitive (but not necessarily best)
median AUC. Causally-motivated methods generally have good median
discriminative performance, but poor bias mitigation properties. Furthermore,
the wide performance ranges of causally-motivated approaches may be
unacceptable for safety-critical/high-stakes domains. We note that the DR-
learner may underperform in this setting due if the propensity score trimming
introduces sufficient bias: recall that, although double-robustness only
requires one correctly-specified model, the asymptotic properties may still
depend on the asymptotics of each model (_e.g._ , as shown in (Wager, 2020)).

#### DCEM is empirically more robust to overlap violations.

Figure 7 shows that, empirically, as overlap violations increase, DCEM
degrades more slowly than causally-motivated approaches in terms of both bias
mitigation and discriminative performance. Furthermore, DCEM maintains
similarly tight performance ranges across levels of overlap, while the
performance ranges of causal approaches widens as overlap violations increase.
At low overlap, causally-motivated approaches have similarly tight performance
ranges as DCEM.

###  E.4 Sensitivity analysis of softmax temperature scaling

We can further tune the smoothness of $\hat{t}^{(i)}$ via the softmax
temperature $\tau$ of the binary classifier for $\hat{t}^{(i)}$:

$\hat{t}^{(i)}:=\frac{\exp(z_{t}/\tau)}{\exp(z_{t}/\tau)+\exp(z_{1-t}/\tau)}$
(92)

where $z_{t}$ is the logit outputted by $g_{\zeta}$ for each $t\in\\{0,1\\}$.
Lower values of $\tau$ sharpen $\hat{t}^{(i)}$ towards $\\{0,1\\}$, while
larger values smooth $\hat{t}^{(i)}$ toward $\frac{1}{2}$. Note that $\tau=1$
recovers the standard softmax function. Thus, adjusting $\tau$ allows us to
control the smoothness of the $\tilde{y}^{(i)}=t^{(i)}y^{(i)}$ constraint.

Table 3: Sensitivity analysis of softmax temperature scaling parameter ($T$) with respect to DCEM AUC and ROC gap (min, max across $s_{T}$ in parentheses) for $q_{y}=0.5,k=1,q_{t}=2$. Maximum (minimum) median AUC (ROC gap) in bold. $\tau$ |  $\uparrow$ AUC |  $\downarrow$ ROC gap
---|---|---
0.01 | .778 [.737, .815] | .051 [.020, .104]
0.1 | .791 [.762, .818] | .025 [.014, .057]
1 (default) | .791 [.763, .820] | .031 [.019, .072]
10 | .800 [.730, .858] | .051 [.021, .096]
100 | .762 [.667, .835] | .071 [.032, .097]
Figure 8: Calibration plot of $\hat{t}$ for (from left to right)
$\tau\in\\{0.1,1,10\\}$. While $\hat{t}$ is well-calibrated for $\tau=1$,
changing $\tau$ in either direction ($<1$ vs. $>1$) induces miscalibration
error.

Table 3 shows full results (median AUC and ROC gap, plus minima and maxima
across $s_{Y}$) for DCEM across various values of temperature scaling
parameter $\tau$. Empirically, our results suggest that temperature scaling
does not significantly change the AUC, and may trade off with bias mitigation
since $\hat{t}^{(i)}$ may no longer be calibrated. Furthermore, even though
median AUC improves in one case ($\tau=10$), the range of AUC is much larger
(0.057 vs. 0.128), and $\tau=1$ still yields the maximum empirical worst-case
AUC (0.763).

Values of $\tau$ away from 1 tend to yield larger ROC gaps. We find that
$\hat{t}^{(i)}$ is well-calibrated for $\tau=1$, but not so for values of
$\tau$ (Figure 8). Since $\hat{t}^{(i)}$ is critical to counterbalancing
disparate censorship, miscalibration error in $\hat{t}^{(i)}$ could result in
larger ROC gaps by reducing the effectiveness/correctness of the causal
regularization term. Thus, we opt to maintain $\tau=1$.

###  E.5 Sensitivity analysis of E-step initialization

We compare random initialization to using a tested-only model as
initialization (the final approach). Empirically, Table 4 shows trivial
changes to performance when using a model trained on labeled data to
initialize the E-step. This suggests that DCEM is able to overcome poor
initialization in the settings studied; _i.e._ , the gains from tested-only
initialization may be marginal, if nonzero.

Table 4: Sensitivity analysis of E-step initialization with respect to DCEM AUC and ROC gap (min, max across $s_{Y}$ in parentheses) for $q_{y}=0.5,k=1,q_{t}=2$. Maximum (minimum) median AUC (ROC gap) in bold. Initialization scheme | AUC | ROC gap
---|---|---
random | .787 [.768, .822] | .031 [.011, .060]
tested-only | .791 [.763, .820] | .031 [.019, .072]

###  E.6 Tradeoffs between bias mitigation and discriminative performance:
SELF

Figure 9: Relative frequencies of AUC for DCEM vs. SELF at similar ROC gaps,
pooling models across all $k,q_{y},q_{t}$ tested. Dashed lines = mean AUC by
model. DCEM improves AUC among models with similar ROC gaps when the ROC gap
is below 0.04.

We compare instances of DCEM to SELF, controlling for ROC gap. We find that
DCEM optimizes discriminative performance more effectively than SELF. Fig. 9
shows a histogram of AUC for SELF and DCEM models with similar ROC gaps across
$q_{t},q_{y},k$ and $s_{Y}$, increasing in ROC gap to the right. For models
with ROC gaps $<0.04$ (Fig. 9, 1st and 2nd from left), DCEM improves AUC
compared to instances of SELF with similar ROC gaps. At larger ROC gaps, DCEM
and SELF obtain similar AUCs (Fig. 9, 1st and 2nd from right). Similarly to
the comparison with tested-only models, the results suggest that DCEM is not
simply trading improved bias mitigation for performance, but is also able to
optimize discriminative performance. Since SELF is a filtering approach that
does not account for the causal structure of disparate censorship, its
estimates of label bias are likely skewed. In contrast, DCEM explicitly uses
the causal structure of disparate censorship to counterbalance label bias.

###  E.7 Sepsis classification and robustness to shifts in labeling decisions

Figure 10: ROC gaps (left) and AUC (right) of selected models on sepsis
classification task as weighting of systolic blood pressure (BP) and
respiratory (resp.) rate ($s_{T}$) for testing changes (“0.0”/left: consider
systolic BP only; “1.0”/right: consider resp. rate only) at $q_{t}=1.5,k=4$.
If a feature is “more salient,” it is weighted higher than the other in the
testing decision function $s_{T}$.

Fig. 10 shows the performance of DCEM vs. models with bimodial behavior across
different $s_{T}$, indexed by different feature weightings in $s_{T}$. Our
results suggest that the baselines require specific $s_{T}$ to perform above
random. The baselines catastrophically underperform (AUC below 0.5) otherwise.
Trends are analogous for the ROC gap.

Specifically, baseline performance improves when one feature is more heavily
weighted than the other in the labeling decision ($x$-axis near 0 or 1).
However, when both features feature in labeling decisions ($x$-axis near 0.5),
the baselines catastrophically fail, while DCEM performance stays high. As
seen in Fig. 4, DCEM AUC and ROC gap also exhibit less variation across the
different $s_{T}$.

Determining which $s_{T}$ is appropriate is a clinical problem that requires
domain expertise, and we make no claims as to the clinical appropriateness of
$s_{T}$. Thus, ML practitioners should not assume that their data will be
representative of any particular decision-making pattern. DCEM is an
alternative approach that is more robust than baselines to shifts in $s_{T}$,
and thus warrants consideration when narrow assumptions about labeling biases
are undesirable.

##  Appendix F Computing Infrastructure

#### Hardware.

We parallelize experiments across 4 A6000 GPUs and 256 AMD CPU cores (4x AMD
EPYC 7763 64-Core processors), though the memory requirements of each model
are under 2GB of VRAM.

#### Software.

All experiments are run on a distribution of Ubuntu 20.04.5 LTS (Focal Fossa)
with Python 3.9.16 managed by conda 23.3.1. We use Pytorch 1.13.1 with CUDA
11.6 for all experiments (Paszke et al., 2019), with scikit-learn 1.2.2