# Maintaining fairness across distribution shift: do we have viable solutions
for real-world applications?

Jessica Schrouff  Google ResearchLondonUnited Kingdom<EMAIL_ADDRESS>,
Natalie Harris  Google ResearchLondonUnited Kingdom ,  Oluwasanmi Koyejo
Google ResearchMountain ViewCAUSA ,  Ibrahim Alabdulmohsin  Google
ResearchZurichSwitzerland ,  Eva Schnider  University of BaselBaselSwitzerland
,  Krista Opsahl-Ong  GoogleMountain ViewCAUSA ,  Alex Brown  Google
ResearchLondonUnited Kingdom ,  Subhrajit Roy  Google ResearchLondonUnited
Kingdom ,  Diana Mincu  Google ResearchLondonUnited Kingdom ,  Christina Chen
Google ResearchMountain ViewCAUSA ,  Awa Dieng  Google ResearchMontrealCanada
,  Yuan Liu  Google ResearchMountain ViewCAUSA ,  Vivek Natarajan  Google
ResearchMountain ViewCAUSA ,  Alan Karthikesalingam  Google
ResearchLondonUnited Kingdom ,  Katherine Heller  Google ResearchMountain
ViewCAUSA ,  Silvia Chiappa  DeepMindLondonUnited Kingdom and  Alexander
D’Amour  Google ResearchCambridgeMAUSA

###### Abstract.

Fairness and robustness are often considered as orthogonal dimensions when
evaluating machine learning models. However, recent work has revealed
interactions between fairness and robustness, showing that fairness properties
are not necessarily maintained under distribution shift. In healthcare
settings, this can result in e.g. a model that performs fairly according to a
selected metric in “hospital A” showing unfairness when deployed in “hospital
B”. While a nascent field has emerged to develop provable fair and robust
models, it typically relies on strong assumptions about the shift, limiting
its impact for real-world applications. In this work, we explore the settings
in which recently proposed mitigation strategies are applicable by referring
to a causal framing. Using examples of predictive models in dermatology and
electronic health records, we show that real-world applications are complex
and often invalidate the assumptions of such methods. Our work hence
highlights technical, practical, and engineering gaps that prevent the
development of robustly fair machine learning models for real-world
applications. Finally, we discuss potential remedies at each step of the
machine learning pipeline.

Machine Learning, Fairness, Robustness, Healthcare

##  1\. Introduction

As progress is made to integrate machine learning (ML) systems in real-world
applications, a concern is to reduce disparities between subsets of the
population (e.g. based on age, sex, race) impacted by ML. The ML community is
approaching this issue from different perspectives: an assortment of metrics
to assess potential (un)fairness across subgroups as well as techniques for
mitigating unfairness have been proposed (see (Del Barrio et al., 2020;
Makhlouf et al., 2020; Mitchell et al., 2018; Verma and Rubin, 2018; Barocas
et al., 2019; Oneto and Chiappa, 2020) for overviews).

Fairness metrics and mitigation techniques currently used in real-world
applications are typically evaluated on a single learning task or data
distribution (e.g. (Zhao et al., 2017; Agarwal et al., 2018; Wang et al.,
2020; Pfohl et al., 2021; Cherepanova et al., 2021)). Recent work has however
highlighted that fairness properties that are satisfied on the data used to
develop a model might not always hold if the data distribution changes (Lan
and Huan, 2017; Kallus and Zhou, 2018). In high-risk domains like healthcare,
this can translate to ‘if a model satisfies fairness criteria when trained in
“Hospital A”, it may not satisfy them when tested in “Hospital B”’. Fueled by
this observation, a new field of research is emerging to understand the
relationship between ML fairness and robustness to distribution shift (e.g.
(Schumann et al., 2019; Adragna et al., 2020; Veitch et al., 2021; Creager et
al., 2021; Zhang et al., 2021)) and to provide mitigation techniques for
obtaining robustly fair ML models (e.g. (Singh et al., 2021; Subbaswamy et
al., 2020; Slack et al., 2020; Zhao et al., 2020)).

Importantly, the success of robustly fair machine learning depends strongly on
the nature of the distribution shifts that the system will encounter in
practice. This point has been made elegantly in work that uses causal
structure to provide fairness guarantees under distribution shift (Singh et
al., 2021; Subbaswamy et al., 2020). However, these guarantees are only
practical when distribution shifts satisfy rather strict assumptions on their
causal structure, namely, that the shift only directly affect parts of the
data generating process in isolation.

The key contribution of this work is to assess how realistic these assumptions
are by examining plausible distribution shifts in two applications of machine
learning in dermatology and clinical risk prediction. In these contexts, we
use statistical tests to show that clinically plausible shifts directly affect
many aspects of the data distribution simultaneously. We then demonstrate
empirically that such compound shifts have practical impact on the
transferability of fairness properties in these applications. Our results
highlight fundamental limitations of purely algorithmic approaches to
achieving robustly fair models in practice. Our hope is that this work will
spark additional research into a broader set of remedies across the entire ML
pipeline for applications where models are expected to behave fairly across a
range of real-world settings.

Our contributions can be summarized as follows: (1) we contextualize a number
of recent transferable fairness approaches into a causal framework; (2) we
design statistical tests to characterize plausible distribution shifts in two
medical applications; (3) we show that these shifts induce practically
significant fairness violations in their respective applications; and (4) we
summarize technical, practical and engineering gaps that prevent the
development of robustly fair models and discuss potential remedies that span
the ML pipeline.

##  2\. Setting and Formalization

Let $A,X,Y$ denote random variables, where $A$ represents one or more
sensitive attributes (such as age, gender, or race), $X\in\mathcal{R}^{d}$ a
set of features, and $Y$ a categorical or continuous outcome. We consider a
classification or regression model
$f:\text{dom}(X^{\prime})\rightarrow\text{dom}(Y)$ that outputs a prediction
$f(X^{\prime})$ of $Y$ based on $X^{\prime}=A\cup X$ or $X^{\prime}=X$, and
are interested in studying the fairness properties of $f$ when the
distribution of the data on which $f$ is deployed is different from the one in
which $f$ is trained/validated. More specifically, denoting with $S$ a random
variable representing a specific environment, such as e.g. a geographical
area, a given hospital or a limited period in time, we assume a _source
environment_ ($S=0$) and a _target environment_ ($S=1$), and consider how
fairness properties generalize when $f$ is trained/validated on a dataset of
samples from the distribution $P(A,X,Y\,|\,S=0)$ but deployed on samples from
the distribution $P(A,X,Y\,|\,S=1)$.

Fairness. Our work focuses on statistical group definitions of fairness
(Barocas et al., 2019). In particular, we frame the discussion around
demographic parity in which the model’s output is statistically independent of
the sensitive attribute, i.e.
$f(X^{\prime})\,\mathrel{\perp\mspace{-10.0mu}\perp}\,A$, and equalized odds
in which the independence holds conditionally on the outcome, i.e.
$f(X^{\prime})\,\mathrel{\perp\mspace{-10.0mu}\perp}\,A\,|\,Y$. In the
experiments, these metrics are computed according to Supplement A.1 for binary
tasks. We also compute the gap in subgroup accuracy, as this is amenable to
multiclass tasks. For all fairness metrics, the lower, the better.

Causal framework. We formalize the interplay between fairness and distribution
shift using causal Bayesian networks (Pearl, 2000; Pearl et al., 2016; Spirtes
et al., 2000), which are directed acyclic graphs in which nodes represent
random variables and links express causal relationships between them (see
Supplement A.2 for an introduction). In a causal Bayesian network, a node
$X^{i}$ is a cause of node $X^{j}$ if there exists a causal path
$X^{i}\rightarrow\ldots\rightarrow X^{j}$ from $X^{i}$ to $X^{j}$ such that
intervening on $X^{i}$ affects $X^{j}$, but not conversely. If the path is
direct. i.e. has not intermediate nodes, then $X^{i}$ is called a direct cause
of $X^{j}$. ${\text{pa}}(X^{i})$ denote the _parents_ of $X^{i}$, namely the
nodes with a link into $X^{i}$. We use causal Bayesian networks to express
relationships between $A,X,Y$, and between these variables and the environment
$S$, following the joint causal inference framework proposed in (Mooij et al.,
2020). We then assess when the (conditional) distributions of $A,X$ and/or $Y$
are preserved across the environments and define a ‘robustly fair’ model as a
model whose fairness properties are independent of $S$.

Distribution shift. We categorize distribution shifts into the following four
categories:

  * •

_Demographic shift_ : a shift in the distribution of the demographics $A$ of
the population, $P(A\,|\,S=0)\neq P(A\,|\,S=1)$, that is due to the presence
of a direct path $S\rightarrow A$ in the causal graph, i.e. to $S$ being a
direct cause of $A$. An example of such a shift is a different proportion of
female patients in $S=0$ and in $S=1$.

  * •

_Covariate shift_ : a shift in the distribution of the features $X$,
$P(X\,|\,{\text{pa}}(X),S=0)\neq P(X\,|\,{\text{pa}}(X),S=1)$, that is due to
a direct path $S\rightarrow X$. A covariate shift can be induced e.g. by using
different cameras or views to acquire an image in $S=0$ and $S=1$ (acquisition
shift) (Castro and Glocker, 2020).

  * •

_Label shift_ : a shift in the distribution of the outcome $Y$,
$P(Y\,|\,{\text{pa}}(Y),S=0)\neq P(Y\,|\,{\text{pa}}(Y),S=1)$, that is due to
a direct path $S\rightarrow Y$. A label shift can manifest itself via a
different prevalence of a disease in $S=0$ compared to $S=1$ (prevalence
shift), and/or different strategies to obtain labels, e.g. silver standard vs
gold standard, in $S=0$ and $S=1$ (annotation shift) (Castro and Glocker,
2020).

  * •

_Compound shift_ : a shift that is due to a combination of some of the three
above shifts. Compound shifts are typical in real-world applications. For
instance, it is common for healthcare ML models to be trained on one or
multiple datasets, and then be deployed on datasets recorded in other
locations (Wiens et al., 2014; Zech et al., 2018). Another example is that of
a new device on the market (Finlayson et al., 2021). Such device might be
available to a specific demographic population, which could potentially be
more healthy. This device will also affect the resulting image. Hence
introducing the device simultaneously affects $A,X$ and $Y$. More examples of
shifts observed in clinical settings are discussed in (Finlayson et al.,
2021).

These definitions are independent of the causal graph considered. We define
demographic, covariate and label shifts as mutually exclusive, e.g. the
presence of a direct causal path between $S$ and $A$ prevents direct paths
between $S$ and $Y$ or $S$ and $X$, as any other combination would be referred
to as a ‘compound’ shift. We note that when considering causal graphs of $X,Y$
and $S$ only, we recover the assumptions typically associated with covariate
shift (i.e. $S\rightarrow X\rightarrow Y:P(Y\,|\,X,S=0)=P(Y\,|\,X,S=1)$) and
label shift (i.e. $S\rightarrow Y\rightarrow
X:P(X\,|\,Y,S=0)=P(X\,|\,Y,S=1)$).

##  3\. Related Work

Robustness to distribution shift: has been investigated from many different
angles. Extensive reviews on the topic exist (Pan and Yang, 2010; Weiss et
al., 2016; Farahani et al., 2020; Wang et al., 2021), and generally
distinguish between the type of information available about the shift. We
consider the setting in which the shifts are known and the target
distribution(s) (i.e. $P(A,X,Y\,|\,S=1)$ or $P(A,X\,|\,S=1)$) are available at
training time (we refer to this setting as domain adaptation (Ben-David et
al., 2010), and as causal domain adaptation if shifts are expected on specific
paths of a causal graph), and the setting in which no information is given
about the shift (we refer to this setting as domain generalization). With
regards to the nature of the shifts, the literature typically targets either
covariate shift, i.e. (combinations of) demographic and covariate shifts
according to our definitions, or label shift. The research on these shifts is
imbalanced, with label shift being less studied (Lipton et al., 2018). The
work of (Lipton et al., 2018) proposes an approach to detect a label shift and
correct it using unlabelled target data. More recently, (Garg et al., 2020)
unified multiple techniques. None of the works cited above center on
simultaneous label and covariate shifts, or consider their interaction with
fairness.

Fairness and robustness: Nevertheless, relations between robustness guarantees
and individual and group fairness have been studied in (Yurochkin et al.,
2019; Nanda et al., 2021; Xu et al., 2021; Yeom and Fredrikson, 2020) and in
(Veitch et al., 2021) respectively. (Adragna et al., 2020; Creager et al.,
2021; Zhang et al., 2021) have also investigated the relationship between
fairness and distribution shift experimentally by studying how robustness
mitigation strategies affect fairness metrics. Work explicitly investigating
fairness properties under distribution shift can mostly be divided into
methods that provide a priori guarantees or bounds (e.g. (Schumann et al.,
2019; Singh et al., 2021; Subbaswamy et al., 2020)) and methods that adapt to
the new distribution (e.g. (Coston et al., 2019; Oneto et al., 2019; Slack et
al., 2020; Zhao et al., 2020; Kallus and Zhou, 2018; Du and Wu, 2021; Taskesen
et al., 2020)). Schumann et al. (2019) provide bounds on fairness transfer and
suggest to add adversarial heads on both the in-source attribute $A$ and the
environment $S$. We cast this work into a causal framework in Section 4. Of
particular relevance is the recent work of ‘feature selection’ (Magliacane et
al., 2018) which considers a set $\\{\mathbf{V}\\}$ of variables to include as
inputs to the model such that $\\{\mathbf{V}\\}$ d-separates $Y$ from $S$,
namely $Y\mathrel{\perp\mspace{-10.0mu}\perp}S\,|\,\\{\mathbf{V}\\}$. This
condition alone does not provide fairness guarantees under distribution shift.
Singh et al. (2021) then add the condition that
$\\{\mathbf{V}\\}\mathrel{\perp\mspace{-10.0mu}\perp}S\,|\,A$ for demographic
parity or $\\{\mathbf{V}\\}\mathrel{\perp\mspace{-10.0mu}\perp}S\,|\,Y,A$ for
equalized odds. While the proposed method can capture simultaneous shifts
across different paths of the causal graph by referring to causal domain
adaptation, it explicitly does not take label shift into account.

Another line of work considers adapting transfer learning and meta-learning
algorithms to consider fairness constraints and therefore ensure
generalization to new tasks also in terms of fairness (Coston et al., 2019;
Oneto et al., 2019; Slack et al., 2020; Zhao et al., 2020). Specifically,
Coston et al. (2019) modify the standard approach to transfer learning under
covariate shift by adding a constraint to the weight learning objective that
enforces closeness between all pairs of protected groups. Slack et al. (2020)
introduce a technique to identify mean shifts that could lead to unfairness.
They then propose meta-learning with a regularizer that imposes demographic
parity or equalized odds to improve the transfer of fairness. The method
requires a small set of labelled target data (K-shot learning) and multiple
tasks, and comes with no guarantees. Work on fine-tuning and transfer learning
(Oneto et al., 2019) leverages task similarity to learn fair representations
that provably generalize to unseen tasks, similarly requiring a small set of
labelled target data and generating a model per environment.

We discuss the applicability of these works to compound shifts in real-world
applications in Section 4, with a special interest for the work of Singh et
al. (2021) and Slack et al. (2020).

##  4\. Technical and practical limitations

In this section, we review how various transferable fairness approaches that
have been proposed in the literature interact with different causal structures
of domain shift depicted in Figures 1 and 2. We assume the same sensitive
attributes $A$ in the source and target environments, and consider different
relationships between the environment and other variables. As in (Castro and
Glocker, 2020; Veitch et al., 2021), we split the analysis into causal or
anti-causal prediction tasks (Schölkopf et al., 2012). A prediction task is
causal if the features $X$ are causes of the outcome $Y$, and anti-causal if
the outcome $Y$ is a cause of the features $X$. Many risk prediction problems
are causal, where risk factors may cause an adverse outcome, while many
diagnostic problems are anti-causal, where the underlying disease may cause
symptoms (Castro and Glocker, 2020).

A takeaway from this review is that most methods are tailored to particular
restricted shift structures, and their useful properties can break down under
compound shifts.

###  4.1. Anti-causal prediction tasks

First, we consider anti-causal prediction tasks, in which the outcome $Y$ is a
direct cause of the features $X$, and that the sensitive attribute $A$ is a
direct cause of $X$ and $Y$ (the reality will likely be more complex and
involve other variables in the paths (observed or not)). The assumption that
$A$ is a cause of $X$ and $Y$ is a reasonable approximation in healthcare
settings: e.g. on average, skin images from men are more hairy than those of
women, breasts are visible in chest x-rays, and comorbidities are different
across sexes and age ranges. Sensitive attributes can also be related to the
label, as prevalence can vary across subgroups (e.g. baldness is more
prevalent in older patients) and some conditions might present differently
across attributes (e.g. a heart attack in men vs women).

In this context, we consider strategies for learning robustly fair models
under demographic, covariate, label and compound distribution shift scenarios
represented in the causal graphs of Figure 1:

(a) (b) (c) (d)
$S$$A$$Y$$X$$X\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A$$X\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A,Y$$Y\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A$
$S$$A$$Y$$X$$A\mathrel{\perp\mspace{-10.0mu}\perp}S$$Y\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S$
$S$$A$$Y$$X$$A\mathrel{\perp\mspace{-10.0mu}\perp}S$$X\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;Y$
$S$$A$$Y$$X$

Figure 1. Causal graphs under different distribution shifts when considering
an anti-causal prediction task. (a) Demographic shift where the demographics
$A$ are directly affected by $S$. (b) Covariate shift where the $X$ is
directly affected by $S$. (c) Label shift where $Y$ is directly affected by
$S$. (d) Compound shift given by the combination of demographic, covariate,
and label shifts.

a) Demographic shift. In Figure 1(a), we consider a demographic shift through
the direct path $S\rightarrow A$. In this case, the effects of $S$ on $Y$ and
$X$ are intercepted by $A$. Therefore, training a fair model based on $X$
(either via imposing equalized odds (Singh et al., 2021) or counterfactual
invariance with respect to $A$ (Veitch et al., 2021)) leads to a robustly fair
model.

b) Covariate shift. In Figure 1(b), we consider a covariate shift through the
direct path $S\rightarrow X$. In this case, there are multiple causal arrows
into $X$ that need to be addressed: $A\rightarrow X$ for fairness, and term
$S\rightarrow X$ for robustness to distribution shift. In the case where some
labeled target domain data is available, Schumann et al. (2019) implement
separate independence regularizers to address each path, and demonstrate its
effectiveness in settings that match this causal structure. In the fully
unlabeled domain adaptation case, the shift is harder to account for. For
example, the set of features that satisfy the selection criteria of Singh et
al. (2021) would only include $A$, leading to a predictor that relies solely
on demographics data (Supplement A.3).

c) Label shift. Lipton et al. (2018) propose a technique to detect a label
shift, as well as mitigate it through reweighting, requiring unlabelled target
data. This technique can be used in addition to a fairness mitigation
technique to lead to a robustly fair model.

d) Compound shift. Such a shift leads to multiple factors being affected by
$S$, and current techniques would either be ineffective or degenerate to the
trivial predictor. While Figure 1(d) represents the worst case scenario, even
simpler combinations of shifts would lead to similar results in practice. For
instance, adding a correlation between $S$ and $A$ to Figure 1(b) leads to a
compound shift. In this specific case, mitigation might be possible by taking
all intersections of $A$ and $S$ when using (Schumann et al., 2019). In
practice, if the attribute or the shift have multiple discrete values, this
will quickly become intractable, especially if distribution matching needs to
be applied within each mini-batch. When considering feature selection, (Singh
et al., 2021) would return an empty set for both equalized odds or demographic
parity, leading to a trivial predictor. (Partial) mitigation might be obtained
through the use of an adaptation technique as proposed in (Slack et al., 2020;
Oneto et al., 2019).

###  4.2. Causal prediction tasks

We repeat the analysis above for the simplest causal prediction task in which
$X$ is a direct cause of $Y$, and assuming that $A$ is a direct cause of $X$
and $Y$. We then consider the following distribution shift scenarios
represented in Figure 2:

(a) (b) (c) (d)
$S$$A$$X$$Y$$X\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A$$Y\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A$$Y\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;A,X$
$S$$A$$X$$Y$$A\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S$$Y\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S\;|\;X$
$S$$A$$X$$Y$$A\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S$$X\,\mathrel{\perp\mspace{-10.0mu}\perp}\,S$
$S$$A$$X$$Y$

Figure 2. Causal graphs under different distribution shifts when considering a
causal prediction task. (a) Demographic shift, (b) Covariate Shift, (c) Label
shift, and (d) Compound shift.

a) Demographic shift. As in the anti-causal case, the relationship between $X$
and $Y$ is not affected by $S$ if we impose invariance of $Y$ on $A$. To this
end, training a fair model based on the set {$X,A$} would lead to a robustly
fair model (Singh et al., 2021).

b) Covariate shift. In this setting, note that
$(Y,A)\mathrel{\perp\mspace{-10.0mu}\perp}S\,|\,X$. Thus, fairness and
robustness to distribution shift guarantees can be obtained independently.
This setting corresponds to the classic covariate shift scenario treated in
much of the domain adaptation literature, where potential solutions are to
match the distributions of $X$ across environments by strategies such as
reweighting (e.g. Shimodaira, 2000) or invariant representation learning (e.g.
Ben-David et al., 2010). The advantage of such a techniques is that they only
require unlabelled target data.

c) Label shift. While solutions are investigated for anti-causal prediction
tasks, label shifts on causal prediction tasks are understudied and the
absence of a direct path $S\rightarrow Y$ is often an assumption for
mitigation techniques (Singh et al., 2021; Veitch et al., 2021). To the best
of our knowledge, there is no method proposing formal fairness guarantees
under label shift in causal prediction tasks.

d) Compound shift. As for the anticausal case, a compound shift would lead to
insufficient or trivial predictors.

##  5\. Illustration with real-world healthcare applications

In this section, we illustrate that the shifts encountered in real-world
applications are typically compound by performing a series of statistical
tests to assess the causal structure of distribution shifts encountered in
real-world applications. We focus on the healthcare domain and select an anti-
causal prediction task in dermatology, and a causal prediction task in
Electronic Health Records (EHR). Given a sample of labelled target data, we
test for various invariances based on simplified causal graphs of the
applications, highlighting that distribution shifts directly affect multiple
aspects of the data distribution simultaneously. In both the dermatology and
EHR cases, we show that fairness properties do not transfer across these
compound shifts.

###  5.1. Assessing the causal structure of a shift

For each variable $X^{d}$ in the graph, we are assessing whether there is a
direct effect of the environment $S$ on $X^{d}$. This operation is often non-
trivial when $X^{d}$ has causal ancestors that could also be affected by the
shift; in this case, the distribution of $X^{d}$ would differ between
environments, but there would be no direct effect of $S$ on $X^{d}$. To
isolate the direct effect of $S$, we derive the set of variables
$\\{\mathbf{V}\\}$ that block all indirect paths from $S$ to $X^{d}$ and
compute weights that balance these background variable across environments. We
then test the hypothesis that the reweighted distributions of $X^{d}$ match
across the environments $S$. Depending on the dimensionality of $X^{d}$,
different strategies are used for the testing (Rabanser et al., 2019). See
Supplement A.4 for the details of our testing procedure.

Compared to the test proposed in Slack et al. (2020), our strategy, while
requiring a causal graph of the application, allows to estimate the joint
causal inference framework (Mooij et al., 2020) and hence to identify which
mitigation strategies might be applicable. In addition, the test in (Slack et
al., 2020) requires features to be semantic, and thus is not directly
applicable in the context of computer vision.

###  5.2. Predicting skin conditions in dermatology

In this task, we are interested in predicting common skin conditions (26
conditions plus an additional ‘other’ category to capture the long tail),
based on one or multiple images of the pathology of interest and additional
metadata (Liu et al., 2020). As is relatively common in healthcare settings,
the model is trained on source data, and considered for application on target
data that has been collected using different protocols in a different clinical
environment. The data used as source is a subset of the data described in (Liu
et al., 2020). Briefly, it consists of de-identified retrospective adult cases
from a teledermatology service serving 17 sites from 2 states in the United
States. Each case contained 1-6 clinical photographs of the affected skin
areas as well as patient demographic information and medical history (for a
complete list, see Table S1 of (Liu et al., 2020)). We split the source data
such that it comprises 12,024 cases for training, 1,925 for validation and
hyper-parameter tuning and 1,924 for testing. The target data, unobserved at
model development time, comes from skin cancer clinics in Australia and New
Zealand and was partially labelled, leading to 21,661 cases used for model
evaluation. We consider a second target dataset of 1,843 labelled
teledermatology cases from Colombia in Supplement B.3111The dermatology data
is not available to the public.

Ethics approval: the images and metadata were de-identified according to
Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor prior
to transfer to study investigators. The protocol was reviewed by [Anonymous]
IRB, which determined that it was exempt from further review under 45 CFR 46.

####  5.2.1. A compound shift

We first show that the shift between the source and the target data affects
all relationships in the causal graph. To this end, we draw a simplified
causal graph of the dermatology task at hand, similar in spirit to the anti-
causal prediction task discussed in Section 4, but with added metadata. In the
source data, we can divide the metadata in multiple variables: demographics
$A$, co-morbidities/medical history $M$ and symptoms $X_{s}$. Each of these
plays a different role in the causal graph (see Figure 3(a)), with the graph
itself becoming more complex. We however note that we omit unobservable
variables such as environmental factors that could affect the skin or co-
morbidities and medical history that are not recorded in the data.

(a)

$S$$X$$A$$M$$Y$$X_{s}$

(b)

(c)

(d)

Figure 3. A compound shift in dermatology. (a) Simplified causal structure of
the application. $A$ refers to demographic metadata, $M$ to the patient’s
medical history and comorbidities, $Y$ to the skin condition as labelled by
clinicians, $X$ represents the set of pictures of the skin pathology and
$X_{s}$ represents other symptoms as shared by the patient (e.g. whether the
lesion is ‘burning’ or ‘itchy’). In our setup, the environment $S$ directly
affects all variables in the graph. (b) Distribution in the source train and
in the target dataset of the sensitive attributes, computed in terms of
percentage of pre-defined subgroups. (c) Prevalence of each condition in cases
where the patients are female and over 65 years of age. (d) Distribution of
the number of images in cases labelled as ‘SK/ISK’ in older females. The
distributions in the source data are represented on the left, and their
corresponding distributions in the target data on their immediate right with a
hashed pattern.

We first note that the metadata available in the source data is more
comprehensive than the data available in the target data, with only age and
sex222Sex is mainly recorded by clinicians, with a small set of the source
data containing self-reported sex. being common to both. These features have
different distributions across the two datasets (see Figure 3(b)), with the
population in the source data being typically younger (median age: 40 years
old, 25$\%$ quantile: 27, 75$\%$ quantile: 54) than in the target data (median
age: 58 years old, 25$\%$ quantile: 44, 75$\%$ quantile: 70). The source
distribution also includes more female patients than the target data (62$\%$,
compared to 47$\%$). Other demographics and its proxies (e.g. pregnancy
status, ethnicity, skin type), as well as $M$ and $X_{s}$ are absent from the
target data. We therefore see direct effects of $S$ on $M,X_{s}$ and $A$, and
add these relationships to the causal graph (Figure 3a, in purple).

We then assess whether $S$ directly affects the labels. By looking at the
graph, we see that a shift on $P(A)$ could lead to a shift on the marginal
$P(Y)$. However, conditioning on $A$ and $M$ could lead to an invariance of
$Y$ on $S$. As $M$ is absent from the target data, we condition on $A$ only.
Using overlap weights to represent the conditioning (Li et al., 2019), we
obtain significant differences between $P(Y\,|\,A,S=0)$ and $P(Y\,|\,A,S=1)$
for 24 conditions ($p<0.05$, Bonferroni corrected). Figure 3(c) illustrates
the label shift in a specific age and sex subgroup (here females aged 65+, 409
cases in the source, 3,198 in the target data). We see that the source data
includes more cases of ‘eczema’ and ‘psoriasis’ and that cancerous conditions
such as ‘basal cell carcinoma’ and ‘melanoma’ are more prevalent in the target
data. While this analysis is limited due to the absence of $M$ and of
sensitive attributes in the target data, our results suggest that the
environment also directly affects the labels (indicated by the orange link in
Figure 3(a)). We note that this analysis is limited to the distribution of
labels and cannot assess differences in their quality: for instance, the
labels for the source data were sourced from multiple clinicians, while some
target labels were verified by biopsies. This is an added difference between
the two environments (known as annotation shift, (Castro and Glocker, 2020))
that only in-depth dataset knowledge can bring.

Finally, we consider whether the features of the images themselves (designated
by $X$) are directly affected by $S$. If the environment does not directly
affect the images, then $P(X\,|\,Y,A,S=0)=P(X\,|\,Y,A,S=1)$. However, our
weighted t-tests suggest a significant difference between these two
distributions ($p<0.05$ corrected on 22 dimensions). Figure 3(d) illustrates
this difference by computing the number of images per case in the group of
older females considered above with cases labelled as ‘SK/ISK’ ( source
median: 3, $n=61$, target median: 2, $n=341$). This result suggests the
existence of the direct path $S\rightarrow X$ and we add this relationship to
the graph (green link in Figure 3(a)).

Based on these different analyses, we conclude that the environment is
affecting all the nodes in our simplified causal graph, which contrasts with
previous works investigating the transferability of fairness. Similar results
are obtained for the second target dataset considered, although 17 conditions
present invariances to $S$ after controlling for demographics (see Supplement
B.3).

####  5.2.2. Fairness metrics are affected by the environment

We train a deep learning model to predict skin conditions from the images $X$,
age and sex. Our research model follows the overall approach described in (Liu
et al., 2020; Roy et al., 2021b) with some minor architectural changes. We use
the train and validation sets of the source data to train 10 replicates of the
model from different random seeds. Each model is then evaluated based on its
top-1 and top-3 accuracy across all conditions on the test set of the source
data and on the target data. We assess fairness by estimating the difference
in top-k model performance between subgroups. When multiple subgroups refer to
the same sensitive attribute, we compute the maximum difference between any
pair of subgroups.

Figure 4. Model performance in dermatology, as estimated via Top-3 accuracy
(in %). The plot displays the total performance, as well as performance
stratified by sex and by age on the source (circles with plain line) and
target (diamonds with dashed line) data. Each marker represents one replicate
of the model.

Overall, our model performs in line with (Liu et al., 2020; Roy et al., 2021b)
on the source data333Notice that we are only using a subset of the data, and
different splits, leading to an apparent decrease in performance (93% in (Liu
et al., 2020) compared to 88% here)., obtaining a top-3 accuracy of $88.52\pm
0.68\%$. When stratifying per sex, the model performs slightly better for
females ($88.95\pm 0.93\%$) than for males ($87.78\pm 0.52\%$). We also
observe a $1.05\%$ maximum difference across age subgroups (based on binary
comparisons), with the 65+ group under-performing ($87.84\pm 1.70\%$). We
however note that these numbers are close to one another, and that there is no
significant difference when considering multiple training seeds. Therefore,
not performing any fairness mitigation might seem like a reasonable option.

Nevertheless, the gaps between the groups increase when the environment
changes (Figure 4), with the difference between females and males doubling
(from 1.17 to 2.33%) and the maximum age gap reaching 21.32%. Similar results
are obtained when considering top-1 accuracy (Supplement B.2.2). When
considering conditions separately, the patterns can vary, but often differ
between the source and the target data (Supplement B.2.3). We therefore
observe that the fairness metrics are affected by the environment. These
results are replicated for the other target dataset considered, leading to
similar conclusions. Interestingly, considering only those 17 conditions
displaying invariance to $S$ leads to a noticeable reduction in fairness
metric disparities between environments (Supplement B.3.2).

####  5.2.3. Current mitigation techniques for fairness violations under
distribution shift are not applicable

As we observed, the shift is complex and does not satisfy the assumptions in
(Singh et al., 2021), or the independence between $A$ and $S$ required for
(Schumann et al., 2019). In this case, the meta-learning strategy proposed in
(Slack et al., 2020) cannot be directly applied as this is a single multiclass
problem which does not refer to multiple tasks. In the present case, we also
notice that (fairness) mitigation strategies for multiclass applications are
scarce (e.g. (Yang et al., 2020; Denis et al., 2021)).

In addition, the features are pixels and non-semantic, meaning that feature
selection techniques that directly test for invariances based on the features
are not applicable (Magliacane et al., 2018; Singh et al., 2021). On the other
hand, masking has been used previously to constrain the learning towards
specific areas of the image (Ross et al., 2017; Simpson et al., 2019). In this
case, each training image is accompanied by a segmentation of the lesion. By
‘zooming in’ on the lesion, we hope that the model will more easily isolate
the signal of interest compared to confounding factors, without discarding
(too much) useful signal. While this technique has shown some promise with
chest x-rays (Viviano et al., 2019), it is impractical due to the additional
requirement of a mask for all training images. In addition, it increases the
ratio of signal of interest compared to undesirable signal, but does not
exclude such signals. Therefore, in cases where the undesirable signal remains
strong in the masked area, the technique comes with no guarantee.

###  5.3. Clinical risk prediction from Electronic Health Records

In this application, we predict the clinical outcomes of patients based on
EHRs. EHRs record the time series of interactions between a patient and the
clinical system (e.g. medication, labs, vitals, …). In the context of EHR
data, evidence has previously shown heterogeneity across centers (Wilson et
al., 2021). However, important shifts can also be observed within a single
system (Nestor et al., 2019). As an example, over half of hospitals in the US
with intensive care units (ICUs) only have a single unit where all critically
ill patients (medical, surgical, cardiac, …) are admitted (noa, [n. d.]). Risk
scores, such as the Acute Physiology and Chronic Health Evaluation II (APACHE
II) score are used to evaluate all critically ill patients, but do not perform
equally across clinical subgroups, e.g. underperforming on patients with
cardiac disease (Pierpont and Parenti, 1999). We illustrate this shift by
considering a model developed on data from generalist ICUs and deployed on
data from specialized ICUs. The dataset used in this study is the open access,
de-identified Medical Information Mart for Intensive Care III (MIMIC-III)
dataset (Johnson et al., 2016)444The de-identified EHR data is available based
on a user agreement at Physionet (Goldberger et al., 2000)., which consists of
data from admissions to ICUs at the Beth Israel Deaconess Medical Center
between 2001 and 2012. Based on clinical input, we consider the Medical ICU
(MICU), Surgical ICU (SICU) and Trauma Surgical ICU (TSICU) to be generalist
ICUs; whereas the Cardiac Surgery Recovery Unit (CSRU) and Coronary Care Unit
(CCU) are specialized ICUs. This split leads to 17,641 patients included in
the source dataset and 10,442 patients in the target dataset after cohort
selection. Our goal is to predict prolonged ICU stay (i.e. length of stay $>3$
days, as in (Wang et al., 2019; Pfohl et al., 2021)) using data from the first
24 hours of first ICU admission. Please see Supplement C.1 for additional
information.

####  5.3.1. A compound shift

We frame this task by referring to the causal graph described in (Singh et
al., 2021). In their work, the authors used a restricted set of variables
related to demographics $A$, comorbidities $M$, observed features $X$ and the
label $Y$. Similarly, we define the demographics $A$ as age and self-reported
sex, comorbidities $M$ as previous medical history defined by ICD-9 codes
(Charlson et al., 1987; Deyo et al., 1992; Halfon et al., 2002; Li et al.,
2008), observed features $X$ as all labs and vitals, and $Y$ as our prolonged
length of stay label. Given that a more comprehensive feature set leads to
better predictive performance (Roy et al., 2021a), we use all 59,351 features
at our disposal and add ‘treatments’ $T$ (e.g. medications) to the graph
(Figure 5(a)). In our graph, $S$ represents the ICU unit type that a patient
is admitted to and is therefore correlated with the ‘reason for the ICU visit’
$R$. We note that variables like labs and vitals might correlate with $R$ but
that the model does not have a direct access to $R$ within the first 24 hours
of admission for which the predictions are issued.

As performed in Section 5.2, we investigate potential (conditional)
independencies between the environment and the different variables. We first
estimate the distribution of age and sex in each dataset (Figure 5(b)). We
observe that the population in the source is typically younger (median: 62
years old, 25% quantile: 49, 75% quantile: 76) than the population in the
target (median: 68 years old, 25% quantile: 58, 75% quantile: 77, t-test:
$p<0.0001$). In addition, the proportion of males in the source and target
differ (source: 50%, target: 65%). Therefore,
$A\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}S$.

(a)

$S$$X$$A$$M$$Y$$T$

(b)

(c)

(d)

Figure 5. A compound shift in EHR. (a) Causal graph for this application (see
text for variable description). (b) Distribution in each dataset of the
sensitive attributes, computed in terms of percentage of pre-defined
subgroups. (c) Prevalence of comorbidities for male patients over 65 years of
age. (d) Prevalence of the label for male patients over 65 with a solid tumor
comorbidity and under vasopressors/inotropes. The distributions in the source
data (train split) are represented on the left, and their corresponding
distributions in the target data on their immediate right with a hashed
pattern.

To assess comorbidities, we refer to Elixhauser and Vanwalraven scores for
comorbidities (Elixhauser et al., 1998; van Walraven et al., 2009) and compute
the scores on the source and target data (Johnson et al., 2018)555Code
publicly available at https://doi.org/10.5281/zenodo.821872. Figure 5(c)
displays the distribution of comorbities in a specific slice of the data (here
older males, $n=3,951$ in source and $n=3,588$ in target). As expected,
patients in the target data have a higher prevalence of comorbidities related
to the peripheral vascular system than in the source data and
$M\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}S\,|\,A$ for 8 comorbidities
(weighted t-test: $p<0.05$, Bonferroni corrected).

We repeat the analysis to assess the direct effect of $S$ on the treatments
($P(T\,|\,A,M,S=0)=P(T\,|\,A,M,S=1):p<0.0001$), on the labs and vitals
($P(X\,|\,A,M,T,S=0)=P(X\,|\,A,M,T,S=1):p<0.05$) and on the label
($P(Y|A,M,T,S=0)=P(Y\,|\,A,M,T,S=1):p<0.05$) in Supplement C.2. Figure 5(d)
displays the prevalence of prolonged length of stay for the specific subgroup
of older males with a history of a solid tumor comorbidity and receiving
vasopressors/inotropes ($n=55$ in source and $n=16$ in target). Our results
suggest that the shift is compound, i.e. the ICU unit a patient is admitted to
will directly affect all variables in our causal graph. We note that this
conclusion is not surprising clinically, given that the environment $S$ is
correlated with the reason for their ICU admission $R$, and that this reason
for admission (unobserved) is a main driver for all variables.

####  5.3.2. Fairness metrics are affected by the shift

As performed in (Roy et al., 2021a; Tomašev et al., 2019; Tomašev et al.,
2021), each patient’s medical history is converted to a time series of one-
hour aggregates including the different structured data elements (medication,
labs, vitals, …) represented by numerical and binary variables. Based on the
first 24 hourly aggregates, we predict the binary label for prolonged length
of stay using a recurrent network architecture (Tomašev et al., 2019; Tomašev
et al., 2021)666We take inspiration from the code made publicly by the authors
of (Tomašev et al., 2021) and available at https://github.com/google/ehr-
predictions. We then estimate model performance in terms of accuracy across
patients and per sex and age subgroups. In addition to per-group metrics, we
compute demographic parity and equalized odds.

Figure 6(a) displays model accuracy across subgroups on the source and target
data. On the source data, the model performs to $78.6\pm 0.7\%$ accuracy, with
$\sim 1\%$ performance gap between sex groups (Female: $77.8\pm 0.8\%$, Male:
$79.2\pm 0.5\%$), and a gap of $7.4\%$ between age subgroups (18-29: $84.0\pm
1.6\%$, 30-44: $82.8\pm 0.9\%$, 45-64: $78.8\pm 0.5\%$, 65+: $76.6\pm 0.7\%$).
Performance increases slightly between the two environments, with the model
reaching $79.7\pm 0.9\%$. The gap between sexes increases to $2.7\%$ on the
target data (Female: $78.0\pm 0.6\%$, Male: $80.7\pm 0.5\%$). For ages, the
maximum gap remains similar (7% on average across seeds, 18-29: $82.5\pm
0.9\%$, 30-44: $84.2\pm 0.9\%$, 45-64: $83.2\pm 0.6\%$, 65+: $77.2\pm 0.6\%$).

(a)

(b)

Figure 6. Model performance in EHR, as estimated via accuracy. The plot
displays the total performance, as well as performance stratified by sex and
by age on the source (circles with plain line) and target (diamonds with
dashed line) data. Each marker represents one replicate of the model. (a) Un-
mitigated. (b) After mitigating for equalized odds across age groups on the
source data.

In terms of fairness metrics, demographic parity is $0.002\pm 0.002$ for sex
on the source, and $0.016\pm 0.003$ on the target. It is $0.05\pm 0.006$ for
age on the source, and $0.066\pm 0.010$ on the target. For both attributes, we
hence observe an increase in demographic parity between the source and the
target. Equalized odds do not display a significant difference between the two
environments, for age or sex. On the other hand, we observe significant
differences in model accuracy for specific subgroups (e.g. 45-64 years old)
between the two environments, suggesting that investigating the predictions
for specific subgroups is needed, in addition to computing overall scores like
demographic parity, equalized odds or maximum performance gaps.

####  5.3.3. Fairness properties after mitigation do not transfer

As the gap between age groups in the source is considerable, we perform post-
processing of the predictions to enforce either demographic parity or
equalized odds across age subgroups. We use the method in Alabdulmohsin and
Lučić (2021), fitting the debiasing model on the tune split of the source, and
applying it to the test set of the source and to the target set.

This results in demographic parity to decrease to value $10^{-5}$ in the
source, but to (slightly) increase to value $0.078\pm 0.025$ in the target
data. Similarly, equalized odds decrease to value $10^{-5}$ and the gap in
model accuracy between groups is reduced from 7% to 0.8% (Figure 6(b)) in the
source, but increases from $0.05\pm 0.008$ to $0.16\pm 0.04$ on the target
data, and the gap between age groups increases to 11.4%.

We therefore observe that mitigating for either demographic parity or
equalized odds on the source leads to no improvement, or even worsening of the
fairness properties on the target. Noteworthy, the variance across seeds also
increases on the target, which might highlight a manifestation of
underspecification (D’Amour et al., 2020).

####  5.3.4. Current mitigation for fairness violations under distribution
shift are not applicable

Singh et al. (2021) clearly mention which shifts can be accounted for by their
feature selection strategy: shifts that are mediated by $A$ (i.e.
$S\rightarrow A\rightarrow X^{d}$), or shifts that affect variables $X$
correlated with the outcome $Y$. For the latter, affected variables are
considered as ‘unstable’ and need to be removed from the input feature set. In
the present case, the graph is more complex, as is the shift, and the
covariate shift assumption is not valid (i.e. here
$Y\,|\,A,M,T\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}S$). Therefore,
the method would lead to an empty feature set. We also note that shifts
affecting single variables in $X$ are unlikely due to the presence of
correlations among variables (e.g. labs sampled as part of ‘panels’ and not in
isolation). In terms of experimentally-driven techniques, we observe the same
limitation as for dermatology, whereby we only have a single task and the work
of Slack et al. (2020) is not directly applicable. Engineering-wise, it is
also unclear how provable transfer techniques could apply to the recurrent,
multidimensional, sparse problem that represents EHR modeling.

##  6\. Discussion

In this work, we formalized the interaction between population-level fairness
and distribution shift using a causal framework. We used this formalization to
discuss previous work and to investigate two healthcare applications. Our
analysis highlighted the following gaps for the application of current
unfairness mitigation methods in real-world settings:

  * •

Technical gap. In the case of complex causal graph and shift, provable
mitigation strategies for fairness violations under distribution shift have
limited applicability in real-world settings. Therefore, currently available
methods often lead to a solution with no guarantee or to a trivial predictor.

  * •

Practical gap. Even experimentally-driven techniques (Adragna et al., 2020;
Schumann et al., 2019; Slack et al., 2020) would need further work before they
could be considered for the applications discussed in this work (e.g. single
task, multivariate sparse time series).

  * •

Engineering gap. Given a small sample of labelled target data, fine-tuned
models that are provably fair (Zhao et al., 2020) could be considered. These
however come at the cost of maintaining one tuned model per environment, which
is not sustainable for all real-world deployments of ML. Techniques relying on
regularization (e.g. (Schumann et al., 2019; Veitch et al., 2021)) also
require that there be enough samples across the different subgroups to match
the distributions of interest. Given that regularization is applied per batch,
this necessitates batches that are of a minimal size (limiting for e.g. long
time series in EHR) and diverse in terms of representation. This limitation
can be compounded in low data settings, which are typical in healthcare.

Given that each task leads to different complexities in terms of causal graph,
expected shifts, and fairness requirements, we can envisage remedies at every
step of the ML pipeline (Chen et al., 2021):

  1. (1)

Problem selection. Consider focusing on tasks with lower anticipated harms or
clinical/policy safeguards against ML unfairness.

  2. (2)

Data collection. Consider early data collection from anticipated deployment
environments in which distribution shifts are suspected. We indeed observed
that statistical tests based on a (labelled) subset of the target data can
provide evidence of how the environment affects the data and the task. In one
case (second target dataset in dermatology), it allowed pinpointing the main
drivers of the fairness gap between the source and the target to 10 conditions
out of 27. In addition, Section 4 showed that if some invariances to the
environment are observed, it is possible that an unlabelled or labelled sample
from the target data could be used for mitigation or that feature selection
(Singh et al., 2021) methods are applicable.

  3. (3)

Outcome definition. Consider intermediate outcomes for which unfairness under
distribution shift might be less consequential (e.g. image segmentation
compared to diagnosis).

  4. (4)

Algorithm development. Based on (2), some strategies can be used if their
assumptions are satisfied. In addition, in the present work we used a causal
framework. However, techniques that are not easily cast into this framework
exist. For instance, manual feature engineering has been used to improve the
robustness of EHR models to distribution shift (Nestor et al., 2019). While
this work has not evaluated the fairness of the model in the source and target
distributions, we believe that feature engineering might be a promising avenue
to enforce specific inductive biases during learning while reducing the risks
of distribution shifts. Compared to previous works on the topic (e.g. (Castro
and Glocker, 2020)), this would mean having an observable proxy for a latent
variable $Z$ that represents the patient’s health. As per (Castro and Glocker,
2020), having access to $Z$ would lead to the modelling task being invariant
to some (not all) combinations of shift and attributes, hence producing more
robustly fair models.

  5. (5)

Post-deployment. Further safeguards may be envisaged to safely deploy ML
systems, for example, commencing such deployments with a prospective
observational (non-interventional) integration (Finlayson et al., 2021).
Transfer learning or retraining with site-specific updates might then be
considered as necessary prior to interventional deployment. Finally, those
deployments may be accompanied with robust post-market surveillance to
continuously monitor not only the performance but also the fairness of the
systems.

We note that, while our analysis focused on two healthcare applications, our
observations are not limited to clinical settings and are likely relevant to
many real-world applications.

Limitations and future work. In this work, we recast recent work on the
transferability of fairness leveraging causal graphs, grouping prediction
tasks as causal or anti-causal, inspired by (Schölkopf et al., 2012; Lipton et
al., 2018; Castro and Glocker, 2020; Veitch et al., 2021). This grouping is,
however, limiting since, as we observed, in real-world applications causal
graphs often contain input features that are both the cause and the effect of
the outcome (e.g. $M$ and $A$ are causes of $Y$ in dermatology, while $X$ and
$X_{s}$ are effects, and the set ${A,X}$ is used for predicting $Y$).
Similarly, the prediction task can be causal with silver standard labels ($Y$
determined from images by dermatologists), but anti-causal with gold standard
labels ($Y$ obtained from biopsy), and label type can be mixed even within a
single dataset due to sampling effects (Castro and Glocker, 2020). We also
note that we ignored the temporal setting of EHR.

While our work covers different types of shift, and their interaction with
sensitive attributes, future work should investigate further sources of shift
and biases, such as sampling biases or missingness (especially relevant for
EHR) (Castro and Glocker, 2020; Zhang et al., 2021). We refer the reader to
Castro and Glocker (2020) for a causal framing of shifts in the context of
medical imaging, noting that the discussion does not include interactions with
fairness or sensitive attributes.

By specifying causal graphs for each problem, we make assumptions about the
relationships between variables. These assumptions might not represent the
true underlying data generation process. For instance, in Section 4, we made
assumptions about the relationship between the demographics, the covariates,
and the labels. While most of our conclusions hold under slightly different
relationships, assuming that $A$ and $X$ are correlated rather than directly
related would mean that the technique in (Singh et al., 2021) leads to an
empty set {$\mathbf{V}$} for a covariate shift on the causal graph in Figure
2(b). Therefore, this exercise might need to be replicated if different
assumptions are considered. In addition, the effects of $A$ could potentially
be divided into ‘fair’ and ‘unfair’ effects, as proposed in (Chiappa, 2019;
Mhasawade et al., 2020). For real-world applications, further assumptions were
made, e.g. considering the silver standard label in dermatology as a good
approximation for the gold standard label (hence an anti-causal task) (Castro
and Glocker, 2020). The value of these graphs is purely illustrative and
should not be considered for medical applications without validation.

Furthermore, the data at hand usually represents a set of proxies for the
underlying variables. Therefore, even if a shift seems to satisfy some
invariance assumptions, there are no sharp guarantees (Finkelstein et al.,
2021). This is especially true for demographic factors: features such as
ethnicity, age or sex only approximate sensitive attributes. We are also
limited by which factors are observed, across all environments. For instance,
skin type is only available in the source data and in one target environment
(Supplement B.3.3). In addition, the uneven distribution of the data across
skin types leads to under-represented groups and comparisons/conclusions on
such small numbers might be irrelevant and/or misleading. Future work will
hence involve acquiring more demographic data to ensure that all sensitive
attributes can be investigated with respect to fairness and its
transferability.

In terms of metrics, we have focused mostly on demographic parity and
equalized odds, due to their recent causal grounding (Veitch et al., 2021).
Different metrics might, however, be affected differently, and it is possible
that some metrics are more or less ‘sensitive’ to distribution shift.
Similarly, different fairness mitigation strategies might be affected
differently by distribution shift, although none provides guarantees under
distribution shift. We also note that ‘fairness’ in the healthcare domain is
an active discussion, and that equalized odds or demographic parity metrics do
not consider factors such as socio-determinants of health or health equity. As
discussed in (Pfohl et al., 2021), considering human elicited metrics might be
an avenue forward. Similarly, ‘fairness’ might change depending on the context
(Sambasivan et al., 2021) and different metrics might be relevant in different
environments.

## Declarations

### Conflict of interest

The authors declare that they have no conflict of interest.

### Funding

This work was funded by Google.

### Data and method availability

The dermatology data is not available to the public. The de-identified EHR
data is available based on a user agreement at Physionet (Goldberger et al.,
2000). The code for extracting Elixhauser and van Walren comorbidity scores
from MIMIC-III is available at https://doi.org/10.5281/zenodo.821872 (Johnson
et al., 2018). We use publicly available code for EHR modelling at
https://github.com/google/ehr-predictions (Tomašev et al., 2019; Tomašev et
al., 2021).

## Acknowledgements

The authors would like to acknowledge and thank Lucas Dixon, Noah Broestl,
Sara Mahdavi, Nenad Tomasev, Cameron Chen, Stephen Pfohl, Matt Kusner, Victor
Veitch, Shannon Sequeira, Abhijit Guha Roy, Jan Freyberg, Aaron Loh, Martin
Seneviratne, Patricia MacWilliams, Yun Liu, Christopher Semturs, Dale Webster,
Greg Corrado and Marian Croak for their contributions to this effort.

## References

  * (1)
  * noa ([n. d.]) [n. d.].  Critical Care Statistics.  https://www.sccm.org/Communications/Critical-Care-Statistics.
  * Adragna et al. (2020) Robert Adragna, Elliot Creager, David Madras, and Richard Zemel. 2020.  Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification.  (Nov. 2020).  arXiv:2011.06485 [cs.LG]
  * Agarwal et al. (2018) Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. 2018\.  A Reductions Approach to Fair Classification. In _Proceedings of the 35th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer Dy and Andreas Krause (Eds.). PMLR, 60–69.  https://proceedings.mlr.press/v80/agarwal18a.html
  * Alabdulmohsin and Lučić (2021) Ibrahim Alabdulmohsin and Mario Lučić. 2021.  A Near-Optimal Recipe for Debiasing Trained Machine Learning Models. In _35th Conference on Neural Information Processing Systems (to appear)_.  https://arxiv.org/abs/2106.12887
  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019\.  _Fairness and machine learning_.  fairmlbook.org.
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010.  A theory of learning from different domains.  _Mach. Learn._ 79, 1 (May 2010), 151–175.
  * Castro and Glocker (2020) Walker Ian Castro, Daniel C. and Ben Glocker. 2020.  Causality matters in medical imaging.  _Nature Communications_ 11 (2020).  Issue 1.
  * Charlson et al. (1987) M E Charlson, P Pompei, K L Ales, and C R MacKenzie. 1987\.  A new method of classifying prognostic comorbidity in longitudinal studies: development and validation.  _J. Chronic Dis._ 40, 5 (1987), 373–383.
  * Chen et al. (2021) Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. 2021\.  Ethical Machine Learning in Healthcare.  _Annu Rev Biomed Data Sci_ 4 (July 2021), 123–144.
  * Cherepanova et al. (2021) Valeriia Cherepanova, Vedant Nanda, Micah Goldblum, John P Dickerson, and Tom Goldstein. 2021\.  Technical Challenges for Training Fair Neural Networks.  (Feb. 2021).  arXiv:2102.06764 [cs.LG]
  * Chiappa (2019) Silvia Chiappa. 2019\.  Path-Specific Counterfactual Fairness.  _AAAI_ 33, 01 (July 2019), 7801–7808.
  * Coston et al. (2019) Amanda Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R. Varshney, Skyler Speakman, Zairah Mustahsan, and Supriyo Chakraborty. 2019.  Fair Transfer Learning with Missing Protected Attributes. In _AAAI/ACM Conference on AI, Ethics, and Society_. 91–98.
  * Cowell et al. (2007) Robert G. Cowell, A. Philip Dawid, Steffen Lauritzen, and David J. Spiegelhalter. 2007.  _Probabilistic Networks and Expert Systems, Exact Computational Methods for Bayesian Networks_.  Springer-Verlag.
  * Creager et al. (2021) Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. 2021.  Environment Inference for Invariant Learning. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 2189–2200.
  * D’Amour et al. (2020) Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D Sculley. 2020.  Underspecification Presents Challenges for Credibility in Modern Machine Learning.  (Nov. 2020).  arXiv:2011.03395 [cs.LG]
  * Del Barrio et al. (2020) E. Del Barrio, P. Gordaliza, and J.-M. Loubes. 2020\.  Review of Mathematical Frameworks for Fairness in Machine Learning.  (2020).  arXiv:2005.13755
  * Denis et al. (2021) Christophe Denis, Romuald Elie, Mohamed Hebiri, and François Hu. 2021.  Fairness guarantee in multi-class classification.  (Sept. 2021).  arXiv:2109.13642 [math.ST]
  * Deyo et al. (1992) R A Deyo, D C Cherkin, and M A Ciol. 1992.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.  _J. Clin. Epidemiol._ 45, 6 (June 1992), 613–619.
  * Du and Wu (2021) Wei Du and Xintao Wu. 2021\.  Fair and Robust Classification Under Sample Selection Bias. In _ACM International Conference on Information & Knowledge Management_. 2999–3003.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through awareness. In _Innovations in Theoretical Computer Science_.
  * Elixhauser et al. (1998) A Elixhauser, C Steiner, D R Harris, and R M Coffey. 1998\.  Comorbidity measures for use with administrative data.  _Med. Care_ 36, 1 (Jan. 1998), 8–27.
  * Farahani et al. (2020) Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. 2020.  A Brief Review of Domain Adaptation.  (Oct. 2020).  arXiv:2010.03978 [cs.LG]
  * Finkelstein et al. (2021) Noam Finkelstein, Roy Adams, Suchi Saria, and Ilya Shpitser. 2021.  Partial Identifiability in Discrete Data With Measurement Error. In _Proceedings of 37th Conference on Uncertainty in Artificial Intelligence_.
  * Finlayson et al. (2021) Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane, and Suchi Saria. 2021.  The Clinician and Dataset Shift in Artificial Intelligence.  _N. Engl. J. Med._ 385, 3 (July 2021), 283–286.
  * Garg et al. (2020) Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. 2020.  A Unified View of Label Shift Estimation. In _Advances in Neural Information Processing Systems_ , H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (Eds.), Vol. 33. Curran Associates, Inc., 3290–3300.
  * Goldberger et al. (2000) A L Goldberger, L A Amaral, L Glass, J M Hausdorff, P C Ivanov, R G Mark, J E Mietus, G B Moody, C K Peng, and H E Stanley. 2000\.  PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals.  _Circulation_ 101, 23 (June 2000), E215–20.
  * Halfon et al. (2002) Patricia Halfon, Yves Eggli, Guy van Melle, Julia Chevalier, Jean Blaise Wasserfallen, and Bernard Burnand. 2002.  Measuring potentially avoidable hospital readmissions.  _J. Clin. Epidemiol._ 55, 6 (June 2002), 573–587.
  * Johnson et al. (2018) Alistair Ew Johnson, David J Stone, Leo A Celi, and Tom J Pollard. 2018.  The MIMIC Code Repository: enabling reproducibility in critical care research.  _J. Am. Med. Inform. Assoc._ 25, 1 (Jan. 2018), 32–39.
  * Johnson et al. (2016) Alistair E W Johnson, Tom J Pollard, Lu Shen, Li-Wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Sci Data_ 3 (May 2016), 160035.
  * Kallus and Zhou (2018) Nathan Kallus and Angela Zhou. 2018.  Residual Unfairness in Fair Machine Learning from Prejudiced Data. In _International Conference on Machine Learning_. 2439–2448.
  * Kolesnikov et al. (2020) Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. 2020\.  Big Transfer (BiT): General Visual Representation Learning. In _ECCV_.
  * Koller and Friedman (2009) Daphne Koller and Nir Friedman. 2009.  _Probabilistic Graphical Models: Principles and Techniques_.  MIT Press.
  * Lan and Huan (2017) Chao Lan and Jun Huan. 2017\.  Discriminatory Transfer.  (July 2017).  arXiv:1707.00780 [cs.CY]
  * Li et al. (2008) Bing Li, Dewey Evans, Peter Faris, Stafford Dean, and Hude Quan. 2008.  Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases.  _BMC Health Serv. Res._ 8 (Jan. 2008), 12.
  * Li et al. (2019) Fan Li, Laine E Thomas, and Fan Li. 2019.  Addressing Extreme Propensity Scores via the Overlap Weights.  _Am. J. Epidemiol._ 188, 1 (Jan. 2019), 250–257.
  * Lipton et al. (2018) Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018\.  Detecting and Correcting for Label Shift with Black Box Predictors. In _Proceedings of the 35th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer Dy and Andreas Krause (Eds.). PMLR, 3122–3130.
  * Liu et al. (2020) Yuan Liu, Ayush Jain, Clara Eng, David H Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, Vishakha Gupta, Nalini Singh, Vivek Natarajan, Rainer Hofmann-Wellenhof, Greg S Corrado, Lily H Peng, Dale R Webster, Dennis Ai, Susan J Huang, Yun Liu, R Carter Dunn, and David Coz. 2020\.  A deep learning system for differential diagnosis of skin diseases.  _Nat. Med._ 26, 6 (June 2020), 900–908.
  * Magliacane et al. (2018) Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. 2018\.  Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions. In _Advances in Neural Information Processing Systems_ , S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett (Eds.), Vol. 31. Curran Associates, Inc.
  * Makhlouf et al. (2020) Karima Makhlouf, Sami Zhioua, and Catuscia Palamidessi. 2020\.  Survey on Causal-based Machine Learning Fairness Notions.  arXiv:2010.09553
  * Mehrabi et al. (2019) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019\.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ (2019).
  * Mhasawade et al. (2020) Vishwali Mhasawade, Nabeel Abdur Rehman, and Rumi Chunara. 2020\.  Population-aware hierarchical bayesian domain adaptation via multi-component invariant learning. In _Proceedings of the ACM Conference on Health, Inference, and Learning_ (Toronto, Ontario, Canada) _(CHIL ’20)_. Association for Computing Machinery, New York, NY, USA, 182–192.
  * Mitchell et al. (2018) Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2018\.  Prediction-based decisions and fairness: A catalogue of choices, assumptions, and definitions.  arXiv:1811.07867
  * Mooij et al. (2020) Joris M Mooij, Sara Magliacane, and Tom Claassen. 2020\.  Joint Causal Inference from Multiple Contexts.  _J. Mach. Learn. Res._ 21, 99 (2020), 1–108.
  * Nanda et al. (2021) Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. 2021.  Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 466–477.
  * Nestor et al. (2019) Bret Nestor, Matthew B A McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C Hughes, Anna Goldenberg, and Marzyeh Ghassemi. 2019\.  Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks. In _Proceedings of the 4th Machine Learning for Healthcare Conference_ _(Proceedings of Machine Learning Research, Vol. 106)_ , Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.). PMLR, Ann Arbor, Michigan, 381–405.
  * Oneto and Chiappa (2020) Luca Oneto and Silvia Chiappa. 2020.  Fairness in Machine Learning. In _Recent Trends in Learning From Data. Studies in Computational Intelligence_ , L. Oneto, N. Navarin, A. Sperduti, and D. Anguita (Eds.), Vol. 896. Springer, Cham.
  * Oneto et al. (2019) Luca Oneto, Michele Donini, A. Maurer, and Massimiliano Pontil. 2019.  Learning Fair and Transferable Representations.  arXiv:1906.10673
  * Pan and Yang (2010) Sinno Jialin Pan and Qiang Yang. 2010.  A Survey on Transfer Learning.  _IEEE Trans. Knowl. Data Eng._ 22, 10 (Oct. 2010), 1345–1359.
  * Pearl (1988) Judea Pearl. 1988\.  _Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference_.  Morgan Kaufmann Publishers Inc.
  * Pearl (2000) Judea Pearl. 2000\.  _Causality: Models, Reasoning, and Inference_.  Cambridge University Press.
  * Pearl et al. (2016) Judea Pearl, M. Glymour, and Nicholas P. Jewell. 2016.  _Causal Inference in Statistics: A Primer_.  Wiley.
  * Pfohl et al. (2021) Stephen R Pfohl, Agata Foryciarz, and Nigam H Shah. 2021\.  An empirical characterization of fair machine learning for clinical risk prediction.  _J. Biomed. Inform._ 113 (Jan. 2021), 103621.
  * Pierpont and Parenti (1999) G L Pierpont and C M Parenti. 1999.  Physician risk assessment and APACHE scores in cardiac care units.  _Clin. Cardiol._ 22, 5 (May 1999), 366–368.
  * Rabanser et al. (2019) Stephan Rabanser, Stephan Günnemann, and Zachary C Lipton. 2019\.  Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In _33rd Conference on Neural Information Processing Systems_ (Vancouver, Canada).
  * Rajkomar et al. (2018) Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Yi Zhang, Gerardo Flores, Gavin E Duggan, Jamie Irvine, Quoc Le, Kurt Litsch, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H Shah, Atul J Butte, Michael D Howell, Claire Cui, Greg S Corrado, and Jeffrey Dean. 2018.  Scalable and accurate deep learning with electronic health records.  _NPJ Digit Med_ 1 (May 2018), 18.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017.  Right for the right reasons: Training differentiable models by constraining their explanations. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence_ (Melbourne, Australia). International Joint Conferences on Artificial Intelligence Organization, California, 2662–2670.
  * Roy et al. (2021b) Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, Nam Vo, Peggy Bui, Samantha Winter, Patricia MacWilliams, Greg S Corrado, Umesh Telang, Yun Liu, Taylan Cemgil, Alan Karthikesalingam, Balaji Lakshminarayanan, and Jim Winkens. 2021b.  Does Your Dermatology Classifier Know What It Doesn’t Know? Detecting the Long-Tail of Unseen Conditions.  (April 2021).  arXiv:2104.03829 [cs.CV]
  * Roy et al. (2021a) Subhrajit Roy, Diana Mincu, Eric Loreaux, Anne Mottram, Ivan Protsyuk, Natalie Harris, Yuan Xue, Jessica Schrouff, Hugh Montgomery, Alistair Connell, Nenad Tomasev, Alan Karthikesalingam, and Martin Seneviratne. 2021a.  Multitask prediction of organ dysfunction in the intensive care unit using sequential subnetwork routing.  _J. Am. Med. Inform. Assoc._ (June 2021).
  * Sambasivan et al. (2021) Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021\.  Re-imagining Algorithmic Fairness in India and Beyond.  (Jan. 2021).  arXiv:2101.09995 [cs.CY]
  * Schölkopf et al. (2012) Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. 2012.  On Causal and Anticausal Learning. In _International Conference on Machine Learning_. 459–466.
  * Schumann et al. (2019) Candice Schumann, Xuezhi Wang, Alex Beutel, Jilin Chen, Hai Qian, and Ed H Chi. 2019\.  Transfer of Machine Learning Fairness across Domains.  (June 2019).  arXiv:1906.09688 [cs.LG]
  * Shimodaira (2000) Hidetoshi Shimodaira. 2000\.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of statistical planning and inference_ 90, 2 (2000), 227–244.
  * Simpson et al. (2019) Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. 2019.  GradMask: Reduce Overfitting by Regularizing Saliency.  (April 2019).  arXiv:1904.07478 [cs.CV]
  * Singh et al. (2021) Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. 2021.  Fairness Violations and Mitigation under Covariate Shift.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. Association for Computing Machinery, New York, NY, USA, 3–13.
  * Slack et al. (2020) Dylan Slack, Sorelle A Friedler, and Emile Givental. 2020\.  Fairness warnings and fair-MAML: learning fairly with minimal data. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 200–209.
  * Spirtes et al. (2000) Peter Spirtes, Clark Glymour, and Richard Scheines. 2000\.  _Causation, Prediction, and Search_.  MIT Press.
  * Subbaswamy et al. (2020) Adarsh Subbaswamy, Roy Adams, and Suchi Saria. 2020\.  Evaluating Model Robustness and Stability to Dataset Shift.  (Oct. 2020).  arXiv:2010.15100 [cs.LG]
  * Taskesen et al. (2020) Bahar Taskesen, Viet Anh Nguyen, Daniel Kuhn, and José H. Blanchet. 2020.  A Distributionally Robust Approach to Fair Classification.  arXiv:2007.09530
  * Tomašev et al. (2019) Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, Alistair Connell, Cían O Hughes, Alan Karthikesalingam, Julien Cornebise, Hugh Montgomery, Geraint Rees, Chris Laing, Clifton R Baker, Kelly Peterson, Ruth Reeves, Demis Hassabis, Dominic King, Mustafa Suleyman, Trevor Back, Christopher Nielson, Joseph R Ledsam, and Shakir Mohamed. 2019.  A clinically applicable approach to continuous prediction of future acute kidney injury.  _Nature_ 572, 7767 (Aug. 2019), 116–119.
  * Tomašev et al. (2021) Nenad Tomašev, Natalie Harris, Sebastien Baur, Anne Mottram, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Valerio Magliulo, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, Alistair Connell, Cían O Hughes, Alan Karthikesalingam, Julien Cornebise, Hugh Montgomery, Geraint Rees, Chris Laing, Clifton R Baker, Thomas F Osborne, Ruth Reeves, Demis Hassabis, Dominic King, Mustafa Suleyman, Trevor Back, Christopher Nielson, Martin G Seneviratne, Joseph R Ledsam, and Shakir Mohamed. 2021.  Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records.  _Nat. Protoc._ 16, 6 (June 2021), 2765–2787.
  * van Walraven et al. (2009) Carl van Walraven, Peter C Austin, Alison Jennings, Hude Quan, and Alan J Forster. 2009.  A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.  _Med. Care_ 47, 6 (June 2009), 626–633.
  * Veitch et al. (2021) Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021.  Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests.  (May 2021).  arXiv:2106.00545 [cs.LG]
  * Verma and Rubin (2018) Sahil Verma and Julia Rubin. 2018.  Fairness definitions explained. In _IEEE/ACM International Workshop on Software Fairness_.
  * Viviano et al. (2019) Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. 2019.  Saliency is a possible red herring when diagnosing poor generalization.  (Oct. 2019).  arXiv:1910.00199 [cs.CV]
  * Wang et al. (2021) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. 2021\.  Generalizing to Unseen Domains: A Survey on Domain Generalization.  (March 2021).  arXiv:2103.03097 [cs.LG]
  * Wang et al. (2019) Shirly Wang, Matthew B A McDermott, Geeticka Chauhan, Michael C Hughes, Tristan Naumann, and Marzyeh Ghassemi. 2019.  MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III.  (July 2019).  arXiv:1907.08322 [cs.LG]
  * Wang et al. (2020) Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. 2020.  Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 8916–8925.
  * Weiss et al. (2016) Karl Weiss, Taghi M Khoshgoftaar, and Dingding Wang. 2016\.  A survey of transfer learning.  _Journal of Big Data_ 3, 1 (May 2016), 1–40.
  * Wiens et al. (2014) Jenna Wiens, John Guttag, and Eric Horvitz. 2014.  A study in transfer learning: leveraging data from multiple hospitals to enhance hospital-specific predictions.  _J. Am. Med. Inform. Assoc._ 21, 4 (July 2014), 699–706.
  * Wilson et al. (2021) F Perry Wilson, Melissa Martin, Yu Yamamoto, Caitlin Partridge, Erica Moreira, Tanima Arora, Aditya Biswas, Harold Feldman, Amit X Garg, Jason H Greenberg, Monique Hinchcliff, Stephen Latham, Fan Li, Haiqun Lin, Sherry G Mansour, Dennis G Moledina, Paul M Palevsky, Chirag R Parikh, Michael Simonov, Jeffrey Testani, and Ugochukwu Ugwuowo. 2021.  Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial.  _BMJ_ 372 (Jan. 2021), m4786.
  * Xu et al. (2021) Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021.  To be Robust or to be Fair: Towards Fairness in Adversarial Training. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 11492–11501.
  * Yang et al. (2020) Forest Yang, Moustapha Cisse, and Sanmi Koyejo. 2020\.  Fairness with Overlapping Groups.  (June 2020).  arXiv:2006.13485 [cs.LG]
  * Yeom and Fredrikson (2020) Samuel Yeom and Matt Fredrikson. 2020.  Individual fairness revisited: Transferring techniques from adversarial robustness. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ (Yokohama, Japan). International Joint Conferences on Artificial Intelligence Organization, California.
  * Yurochkin et al. (2019) Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. 2019.  Training individually fair ML models with Sensitive Subspace Robustness.  (June 2019).  arXiv:1907.00020 [stat.ML]
  * Zafar et al. (2017) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In _International Conference on World Wide Web_.
  * Zech et al. (2018) John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. 2018.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  _PLoS Med._ 15, 11 (Nov. 2018), e1002683.
  * Zhang et al. (2021) Haoran Zhang, Natalie Dullerud, Laleh Seyyed-Kalantari, Quaid Morris, Shalmali Joshi, and Marzyeh Ghassemi. 2021.  An empirical framework for domain generalization in clinical settings. In _Proceedings of the Conference on Health, Inference, and Learning_ (Virtual Event, USA) _(CHIL ’21)_. Association for Computing Machinery, New York, NY, USA, 279–290.
  * Zhao et al. (2020) Chen Zhao, Changbin Li, Jincheng Li, and Feng Chen. 2020\.  Fair Meta-Learning For Few-Shot Classification. In _IEEE International Conference on Knowledge Graph_. 275–282.
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017.  Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, 2979–2989.

## Appendix

##  Appendix A Related works and Methods

###  A.1. Fairness metrics

We define demographic parity as (Dwork et al., 2012; Zafar et al., 2017;
Mehrabi et al., 2019; Alabdulmohsin and Lučić, 2021):

(1)
$\max_{a\in\mathcal{A}}\,\mathbb{E}_{\mathbf{x}}[f(\mathbf{x})\,|\,A=a]\,-\,\min_{a\in\mathcal{A}}\,\mathbb{E}_{\mathbf{x}}[f(\mathbf{x})\,|\,A=a]$

Equalized odds is computed in a similar fashion by conditioning on the
positive and negative classes, taking the average of the discrepancies across
classes.

Subgroup performance is computed as the accuracy (top-1 or top-3 in the case
of dermatology) within each subgroup.

###  A.2. Causal framework

A _Bayesian network_ (Cowell et al., 2007; Koller and Friedman, 2009; Pearl,
1988, 2000) is a _directed acyclic graph_ (DAG) ${\mathcal{G}}$ whose nodes
$X^{1},\ldots,X^{D}$ represent random variables and links express statistical
dependencies among them. Each node $X^{d}$ is associated with a _conditional
probability distribution_ (CPD) $p(X^{d}\,|\,{\text{pa}}(X^{d}))$, where
${\text{pa}}(X^{d})$ denote the _parents_ of $X^{d}$, namely the nodes with a
link into $X^{d}$. The joint distribution of all nodes is given by the product
of all CPDs, i.e.
$p(X^{1},\dots,X^{D}\,|\,{\mathcal{G}})=\prod_{d=1}^{D}p(X^{d}\,|\,{\text{pa}}(X^{d}))$.
This function is assumed to be invariant across distributions. A _path_ from
$X_{i}$ to $X_{j}$ is a sequence of linked nodes starting at $X_{i}$ and
ending at $X_{j}$. A path is called _directed_ if the links point from
preceding towards following nodes in the sequence. A node $X_{i}$ is an
_ancestor_ of a node $X_{j}$ if there exists a directed path from $X_{i}$ to
$X_{j}$. In this case, $X_{j}$ is a _descendant_ of $X_{i}$. We say that a set
of nodes {$X^{i},\ldots,X^{j}$} _d-separates_ two random variables $U$ and $W$
if $U\mathrel{\perp\mspace{-10.0mu}\perp}W\,|\,{\\{X^{i},\ldots,X^{j}\\}}$. A
causal Bayesian network is a Bayesian network in which a link expresses causal
influence rather than statistical dependence. In causal Bayesian networks,
directed paths are called _causal paths_.

###  A.3. Feature selection mitigation strategies

We summarize the results for feature selection, and in particular the method
proposed in (Singh et al., 2021) in Table 1. We observe that covariate and
label shifts cannot be mitigated without excluding the signal, leading to an
empty set $\\{\mathbf{V}\\}$ in some of our simplified cases.

Table 1. Review of the coverage of (Singh et al., 2021) across different prediction tasks including both $A$ and $S$. We refer to equalized odds for the anti-causal predictive case, and demographic parity for the causal predictive case, as in (Veitch et al., 2021). Here, $\\{\mathbf{V}\\}$ is the set of variables to include as inputs to the model. Predictive task |  Shift |  $\\{\mathbf{V}\\}$
---|---|---
Anti-causal |  demographic |  $\\{X\\}$
|  covariate |  $\\{A\\}$
|  label |  $\emptyset$
|  complex |  $\emptyset$
Causal |  demographic |  $\\{X,A\\}$
|  covariate |  $\emptyset$
|  label |  $\emptyset$
|  complex |  $\emptyset$

###  A.4. Assessing the causal structure of shifts

####  A.4.1. General Strategy

To assess the causal structure of a shift, we examine whether there is a
direct effect of the shift $S$ on a focal variable $X^{d}$. This requires
controlling for all other pathways by which $X^{d}$ may depend on $S$. To do
this, we select a set of variables $\mathbf{V}$ such that {$\mathbf{V}$}
blocks all indirect paths from $S$ to $X^{d}$. We then test the following
equality in conditional distributions:
$P(X^{d}\mid\mathbf{V},S=0)=P(X^{d}\mid\mathbf{V},S=1)$ for almost all values
of $\mathbf{V}$. 777“Almost all” specific terminology from probability theory.
In most practical cases, it can be understood informally to mean “all”.

To test the equality of conditional distributions, we reduce the problem to
testing whether two marginal distributions of $X^{d}$ are equal when the
distributions of the background variables $\mathbf{V}$ are adjusted to follow
the same distribution. Using observed data from environments $S=0$ and $S=1$,
we can perform such an apples-to-apples comparison using importance weighting.
In particular, for any distribution $\pi(\mathbf{V})$,

(2) $\displaystyle P(X^{d}|\mathbf{V},S=0)$
$\displaystyle=P(X^{d}|\mathbf{V},S=1)\quad\text{for almost all
}\mathbf{V}\quad\implies$ (3) $\displaystyle\int E[X^{d}\mid\mathbf{V}\mid
S=0]\pi(\mathbf{V})d\mathbf{V}$ $\displaystyle=\int E[X^{d}\mid\mathbf{V}\mid
S=1]\pi(\mathbf{V})d\mathbf{V}\quad\implies$ (4) $\displaystyle
E\left[\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid S=0)}E[X^{d}\mid\mathbf{V}\mid
S=0]\mid S=0\right]$
$\displaystyle=E\left[\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid
S=1)}E[X^{d}\mid\mathbf{V}\mid S=1]\mid S=1\right]\implies$ (5) $\displaystyle
E\left[\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid S=0)}X^{d}\mid S=0\right]$
$\displaystyle=E\left[\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid S=1)}X^{d}\mid
S=1\right]$

Define the weights $w_{0}(\mathbf{V}):=\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid
S=0)}$, and $w_{1}(\mathbf{V}):=\frac{\pi(\mathbf{V})}{P(\mathbf{V}\mid
S=1)}$. This result shows that we can test the equality of the conditional
distributions by testing the implication that the mean of weighted outcomes is
the same.

This leaves a degree of freedom for choosing the test distribution
$\pi(\mathbf{V})$ on which the conditional distributions will be compared.
This distribution then determines the weighting scheme that will be used. We
consider two different weighting schemes. In the first, we define
$\pi(\mathbf{V})$ to be a constant, i.e., uniform. Then the weights are

$w_{0}(\mathbf{V})\propto P(\mathbf{V}\mid S=0)^{-1}\quad\text{and}\quad
w_{1}(\mathbf{V})\propto P(\mathbf{V}\mid S=1)^{-1}.$

This weighting scheme is known as inverse probability weighting or IPW, and is
a popular choice.

A drawback of IPW is that they can be highly variable, and inject high
variance into test statistics, reducing the power of a test. An alternative
weighting scheme employs so-called overlap weights (OW) (Li et al., 2019),
which correspond to the test distribution $\pi(\mathbf{V})\propto
P(\mathbf{V}\mid S=0)P(\mathbf{V}\mid S=1)$. This smoothly downweights values
of $\mathbf{V}$ that are rare under either $S=0$ or $S=1$ and focuses the
comparison on values of $\mathbf{V}$ where there is substantial overlap
between the two distributions. In this case, the weights are

$w_{0}(\mathbf{V})\propto P(\mathbf{V}\mid S=1)\quad\textrm{and}\quad
w_{1}(\mathbf{V})\propto P(\mathbf{V}\mid S=0).$

Note that this scheme weights each datapoint by the probability that it would
appear in the opposite environment.

We estimate both OW and IPW, with a preference for OW given its favorable
variance properties. We present the results with OW for compactness, and
explicitly mention where OW and IPW lead to different results. It is important
to note that each weighting scheme tests different implications of the
equality of conditional distributions, and that falsifying either implication
would indicate that the conditional distributions differ.

In practice, the likelihoods $P(\mathbf{V}\mid S=s)$ used to define the
weights need to be learned from data. Usefully, because of how the weights are
normalized, the likelihoods can be replaced with classification scores that
estimate $P(S=s\mid\mathbf{V})$. We estimate these scores with logistic
regression, and plug these estimates into the expressions for our chosen
weights, to obtain estimated weights $\hat{w}_{0}$ and $\hat{w}_{1}$.

We then use a standard t-test with unequal variances to test the null
hypothesis

$H_{0}:E[\hat{w}_{0}X^{d}\mid S=0]=E[\hat{w}_{1}X^{d}\mid S=1].$

####  A.4.2. Testing with High-Dimensional $X^{d}$

If $X^{d}$ has low dimensionality, Rabanser et al. (2019) show that multiple
one-dimensional tests can be used with correction for multiple comparisons
(here Bonferroni). On the other hand, if $X^{d}$ has higher dimensionality
(e.g. an image), summaries of $X^{d}$ can be constructed and the test can be
performed on these lower-dimensional summaries. The validity of this approach
follows from the fact that if the conditional distributions of $X^{d}$ are the
same, then so will the conditional distributions of any summary $f(X^{d})$.
The trade-off is that the test loses all power to detect distributional
differences that are compressed out by the summary $f(X^{d})$, or to highlight
specific dimensions in $X^{d}$ that are more or less affected by $S$. To
define a summary that is relevant to the problem, Lipton et al. (2018) suggest
defining $f(X^{d})$ to be the output of a model the predicts some variable of
interest (say the outcome $Y$) using $X^{d}$. As reported in (Rabanser et al.,
2019), other summarizing techniques could be considered.

##  Appendix B Dermatology

###  B.1. Assessing the shift as compound

####  B.1.1. The environment $S$ affects the conditions $Y$

We aim at assessing the direct effect of $S$ on $Y$, under the hypothesis that
$P(Y|A,S=0)=P(Y|A,S=1)$, hence no label shift. To this end, we need to account
for the effect of $S$ on $A$ (see causal graph in main text). Therefore, we
perform overlap weighting (Li et al., 2019) by subsampling 10,000 random
samples from the source and from the target, selecting age and sex as our
variables, and using the environment as our label for prediction. We train a
logistic regression using 80% of the samples and assess its accuracy on the
remaining 20% samples. We estimate weights for each sample in the source and
in the target, normalizing the weights across samples.

For this logistic classifier, model accuracy on the left-out test sample is
$66\%$. We note that, given the absence of ground truth, we are not able to
directly assess the quality of our derived weights. Using these weights in a
t-test, we obtain significant differences between the label distributions in
the source and in the target for 24 conditions out of 27 ($p<0.05$, Bonferroni
corrected for multiple comparisons).

####  B.1.2. The environment $S$ affects the images $X$

To assess the direct effect of $S$ on the images $X$, we need to control for
the set $\\{A,Y\\}$. In this case, our hypothesis is that
$P(X|A,Y,S=0)=P(X|A,Y,S=1)$. Using a similar methodology as above, we obtain a
classifier that discriminates between environments with an accuracy of
$89.3\%$.

As $X$ is highly multi-dimensional, we build a ‘summary’ by training a
classifier that predicts the label $Y$ from the images, following (Lipton et
al., 2018). We note that we did have to train another model (without metadata
compared to the model reported in the main text) to provide a summary of $X$,
contrarily to what was reported as an advantage of the method in (Lipton et
al., 2018).

Based on this process, we find that 22 out of the 27 conditions (as predicted
from $X$) lead to significant differences between the environments ($p<0.05$,
Bonferroni corrected for multiple comparisons).

###  B.2. Model and performance

####  B.2.1. Model architecture

To train the model, we consider all images pertaining to a case (min 1, max
6). Each image is encoded with a wide ResNet-101$\times$3 feature extractor
initialised using BiT-L pretraining checkpoints (Kolesnikov et al., 2020). The
embeddings are then averaged across images and concatenated with the metadata
(here age and sex) before passing through a fully connected layer which is
followed by classification heads predicting the 26 + 1 conditions (Liu et al.,
2020; Roy et al., 2021b). An additional classification head covering a more
fine-grained set of 419 conditions, defined on the same examples, is used at
train time only.

####  B.2.2. Overall model performance

Detailed top-3 and top-1 model performance can be found in Tables 2 and 3,
respectively. For top-1 accuracy, we observe similar results as for top-3
accuracy: model performance is relatively similar between groups on the source
data, but differences become apparent on the target data (Figure 7).

Table 2. Top-3 model accuracy (in %) in the source and target data, on average across model runs. Group | Source | Target
---|---|---
Total | $88.52\pm 0.68$ | $70.87\pm 0.85$
Female |  $88.95\pm 0.93$ (n=1,221) |  $72.11\pm 0.98$ (n=10,195)
Male |  $87.78\pm 0.52$ (n=703) |  $69.77\pm 0.80$ (n=11,466)
$[18,30)$ |  $88.51\pm 1.11$ (n=563) |  $87.52\pm 1.15$ (n=1,434)
$[30,45)$ |  $88.38\pm 1.02$ (n=548) |  $77.64\pm 0.79$ (n=4,365)
$[45,65)$ |  $88.89\pm 0.70$ (n=619) |  $68.39\pm 0.97$ (n=8,355)
$[65,90)$ |  $87.84\pm 1.70$ (n=194) |  $66.20\pm 1.44$ (n=7,401)
Table 3. Top-1 model accuracy (in %) in the source and target data, on average across model runs. Group | Source | Target
---|---|---
Total | $53.62\pm 0.88$ | $32.14\pm 0.99$
Female | $53.62\pm 0.88$ | $32.34\pm 1.14$
Male | $55.09\pm 1.05$ | $31.96\pm 0.91$
$[18,30)$ | $54.49\pm 1.73$ | $45.80\pm 2.14$
$[30,45)$ | $53.12\pm 1.14$ | $36.63\pm 1.45$
$[45,65)$ | $52.68\pm 0.92$ | $29.32\pm 1.01$
$[65,90)$ | $55.52\pm 2.00$ | $29.72\pm 1.05$
Figure 7. Model performance in dermatology, as estimated via Top-1 accuracy
(in %). The plot displays the total performance, as well as performance
stratified by sex and by age on the source (circles with plain line) and
target (diamonds with dashed line) data. Each marker represents one replicate
of the model.

####  B.2.3. Per condition model performance

We further compute model performance per condition when enough data samples
are available both in the source and target data. We observe that the gaps
between groups vary based on the condition, with the long tail of conditions
(represented as a single ‘other’ condition in our task) showing a similar
pattern as observed overall (Figure 8(a)). Some conditions like Melanocytic
Nevus present disparities across age groups in both datasets (with the caveat
of few test samples in the source data), with performance decreasing with
increasing age (max gap in source: 24.47%, in target: 39.00%, Figure 8(b)). On
the other hand, the SK/ISK condition displays similar performance across age
groups in the source, but increasing performance with increasing age in the
target (max gap in source: 4.00%, max gap in target: 31.78%, Figure 8(c)).
Where the sample size allowed for comparisons, we therefore observe that the
gaps across groups are not reproducible between the source and target data.

(a)

(b)

(c)

Figure 8. Per condition model performance in dermatology, as estimated by
top-3 accuracy (in %). (a) ’Other’. (b) Melanocytic Nevus. (c) SK/ISK.

###  B.3. Second target dataset: teledermatology cases in Colombia

In this section, we repeat the analyses performed in the main text for another
target dataset.

####  B.3.1. A compound shift

As previously, we first estimate the demographic distribution of the target
population (age and sex), compared to the source distribution (Figure 9(a)).
We observe that the sex888Sex is self-reported in the target. distributions
are well matched across the source and target (binomial test, p=0.22).
However, the target data includes a significant number of pediatric cases, and
hence the age distribution is significantly affected by the shift (t-test,
p$<0.001$). We note that removing the pediatric cases still leads to a
significant shift (t-test, p$<0.001$), with the target population being on
average older (median age: 44 years old, 25$\%$ quantile: 30, 75$\%$ quantile:
59) than the source population (median age: 40 years old, 25$\%$ quantile: 27,
75$\%$ quantile: 54).

(a)

(c)

(b)

Figure 9. Another compound shift in dermatology. (a) Distribution in each
dataset of the sensitive attributes, computed in terms of percentage of pre-
defined subgroups. (b) Prevalence of each condition in cases where the
patients are female and over 65 years of age. (c) Distribution of the number
of images in cases labelled as ‘SK/ISK’. The distributions in the source data
are represented on the left, and their corresponding distributions in the
target data on their immediate right with a hashed pattern.

When selecting a specific age and sex subgroup (here elder women for
illustration, $n=409$ cases in the source, $n=128$ in the target data), we
note differences in the distribution of the conditions $Y\,|\,A$ with Eczema
and Psoriasis being more prevalent in the source, while conditions like
Actinic Keratosis and Seborrheic Dermatitis are more represented in the target
(Figure 9(b)). We note that the shift is significant for 8 conditions (i.e.
$P(Y\,|\,A,S=0)\neq P(Y\,|\,A,S=1)$, weighted t-test, $p<0.05$ Bonferroni
corrected), but is nonetheless not as stark as for the target considered in
the main text.

Finally, we investigate how the images themselves are affected by $S$. We
illustrate this difference via the number of images per case (Figure 9(c)),
which displays that on average, cases from elder female patients with the
‘SK/ISK’ condition include more images in the target (median=5, $n=18$) than
in the source (median=3, $n=61$). When assessing the hypothesis
$P(X\,|\,Y,A,S=0)=P(X\,|\,Y,A,S=1)$ as per our weighted t-test, we observe
significant differences between the source and target data for 10 conditions
out of 27 ($p<0.05$ Bonferroni corrected).

Based on these results, we observe that all aspects of the data are affected
by the environment, although some conditions might display invariance (17
conditions out of 27) once controlled for demographics.

####  B.3.2. Fairness properties are affected by the environment

As in the main text, we then compute top-3 model performance per subgroup in
the source and target distribution. We observe a similar pattern as the one
depicted in the main text: while the model performs similarly across age and
sex subgroups in the source dataset, performance correlates with age in the
target dataset (Figure 10).

Figure 10. Model performance in dermatology, as estimated via Top-3 accuracy
(in %). The plot displays the total performance, as well as performance
stratified by sex and by age on the source (circles with plain line) and
target (diamonds with dashed line) data. Each marker represents one replicate
of the model.

We repeat the analyses for the same conditions depicted in Section B.2.3. In
this case, the tail of other conditions (‘other’, Figure 11(a)) shows more
similarity with the source, and even improved performance. As previously, the
fairness patterns for Melanocytic Nevus seem to be similar, decreasing with
age, while SK/ISK shows an increase in performance with age in the target, but
not in the source. We note that the analyses in this section include fewer
samples per subgroup than in Section B.2.3 as the target dataset is smaller
(1,299 adult cases in this second target compared to 21,661 cases in the
target depicted in the main text).

(a)

(b)

(c)

Figure 11. Per condition model performance in dermatology, as estimated by
top-3 accuracy (in %). (a) ’Other’. (b) Melanocytic Nevus. (c) SK/ISK.

Interestingly, using the model to predict only the 17 conditions for which we
have not identified significant differences in prevalence or presentation (as
per Section B.3.1) reduces the fairness gap between source and target. Figure
12 displays the model performance across those 17 conditions, where we observe
a performance gap of 4.1 % in the source ($n=412$) and 8.6 % in the target
($n=577$). The remaining fairness gap between the source and the target could
be due to interactions between age and sex, noise or unobserved variables. In
comparison, using the 10 conditions for which we do have evidence of a
significant shift leads to a gap of 1.1 % on the source and 10.0 % on the
target. Therefore, our tests have identified potential strong drivers of the
fairness gap between the source and the target environment.

Figure 12. Model performance in dermatology, as estimated via Top-3 accuracy
(in %) for 17 conditions with
$P(Y\,|\,A)\mathrel{\perp\mspace{-10.0mu}\perp}S$ and
$P(X\,|\,Y,A)\mathrel{\perp\mspace{-10.0mu}\perp}S$. The plot displays the
total performance, as well as performance stratified by sex and by age on the
source (circles with plain line) and target (diamonds with dashed line) data.
Each marker represents one replicate of the model.

####  B.3.3. Fitzpatrick’s skin type

In this second target dataset, skin type is available as an attribute. We
therefore perform similar analyses as performed for age and sex. We first
observe that the proportions of cases across the different skin types are
different across the source and target datasets (Figure 13(a), t-test:
$p<0.001$).

(a)

(b)

Figure 13. (a) Prevalence of the six Fitzpatrick’s skin types in each dataset
(left: source, right: target). (b) Top-3 model performance for each subgroup
in the source and in the target.

Due to a low number of examples for skin types 1 and 6 (in both datasets), we
group skin types according to ‘unknown’, ‘1/2’, ‘3/4’ and ‘5/6’ before
assessing top-3 performance per subgroup. We observe that model performance is
relatively similar across the different subgroups, with the maximum pairwise
difference being 6% on the source and 5.5% on the target (Figure 13(b)). We
however note that, despite the groupings, the number of examples in group 5/6
remains low (source: $n=53$, target: $n=131$). Performance results in this
group are hence to take with a grain of salt (as displayed by large
variability across seeds), and single conditions cannot be investigated.

##  Appendix C Electronic health records

###  C.1. Dataset

Cohort selection: Our cohort includes patients with age $>$ 18 at time of
admission and ICU length of stay (LOS) $>$ 1 day. We restrict the patient
sequences to the first ICU admission, as in the MIMIC-Extract project (Wang et
al., 2019), and additionally restrict to only those admissions with a
FIRST_CAREUNIT defined in the MIMIC-III icustays table. This procedure results
in a cohort of 28,083 patients.

Feature and target definition: The full feature set from (Roy et al., 2021a)
is used, leading to 31,303 binary features and 28,048 numeric features. The
target is defined as a binary outcome, i.e. remaining ICU LOS $>$ 3 days,
following the approach used in (Wang et al., 2019; Pfohl et al., 2021).

Patient demographics are used as metadata for fairness mitigation and
evaluation. For illustration, we focus on sex and age (bucketed as in the
dermatology application).

Data splits: The FIRST_CAREUNIT defined in the MIMIC-III icustays table is
used to split the dataset into a source dataset with 17,641 patients (MICU,
SICU and TSICU) and target set with 10,442 patients (CCU, CSRU). The source
dataset is then randomly split into training (80%), validation (5%),
calibration (5%) and test (10%) sets.

###  C.2. A compound shift

####  C.2.1. The ICU unit $S$ affects the comorbidities $M$

We define comorbidities according to (Elixhauser et al., 1998; van Walraven et
al., 2009) using code in (Johnson et al., 2018), obtaining a set of 30
comorbidities associated with each patient (multi-label). In this work, we
refer to comorbidities as a ‘summary’ of ICD codes that represent the
patient’s medical history prior to the current admission.

Statistical testing: We define OW weights based on a logistic regression that
predicts the environment based on 10,000 admissions from the source and 10,000
admissions from the target data. The classifier performs with an accuracy of
59.55% on a left-out test set comprising 20% of the data. For each
comorbidity, we assess whether its prevalence is significantly affected by the
environment using weighted t-tests. We observe that 8 comorbidities lead to
significant results (after Bonferroni correction for multiple comparisons):
valvular disease, peripheral vascular, pulmonary circulation, chronic
pulmonary, diabetes complicated, liver disease, solid tumor and metastatic
cancer. We however caveat this analysis by the low number of patients whom
have recorded comorbidities prior to admission.

####  C.2.2. The ICU unit $S$ affects the treatments $T$

Definition of treatments: We identify treatments based on the
‘MedicationRequest’ field in the FHIR representation (Rajkomar et al., 2018)
of MIMIC. For each of those treatments, we construct a summary by counting the
number of hourly requests in the first 24 hours after admission. This summary
is used for illustrating specific medications (beta-blockers,
vasopressors/inotropes and ACE inhibitors) and to control for treatments in
section C.2.3. Given that this representation includes 5,873 dimensions, we
build a further 1-dimensional summary (Lipton et al., 2018) by training a
model that predicts LOS from the time series of the medication requests on the
source training split. This model reaches an accuracy of 76.9% (AUPRC: 0.54,
AUROC: 0.75) on the source test split.

Statistical testing: We assess whether treatments are directly affected by $S$
by computing weights to balance $\\{A,M\\}$. We note that this analysis is
limited as comorbidities are not reported for all patients. Nevertheless, a
binary model discriminating between environments based on $\\{A,M\\}$ leads to
62% accuracy. Given the dimensionality of treatments, we perform a weighted
t-test on the predicted probability of the recurrent network trained to
predict LOS from treatments only. Considering all medication, we observe that
patients receive different treatments in different units (weighted t-test,
$p<0.0001$), hence
$T\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}S|\\{A,M\\}$.

As an illustration, we estimate what proportion of patients receives
treatments such as beta-blockers, vasopressors/inotropes and/or ACE inhibitors
(Figure 14) for patients with (a) vascular comorbidities, and (b) tumor
history.

a

b

Figure 14. Per comorbity prevalence of treatment (in %). (a) Peripheral
vascular comorbidities. (b) Solid tumor.

####  C.2.3. The ICU unit $S$ affects the length of stay $Y$

We refer to the summary of treatments of dimension 5,873 built above to define
$T$ and build the feature set $\\{A,M,T\\}$ to estimate overlap weights such
that we can test whether $P(Y|A,M,T,S=0)=P(Y|A,M,T,S=1)$. The treatment
features with non-null values are normalized to be in the range [0, 1] before
we build a logistic regression that distinguishes between the two
environments. The logistic model achieves $87.2\%$ accuracy in predicting the
environment. We derive the weights for each data point in the two environments
and assess with a weighted t-test whether the prevalence in the two
environments is similar. We obtain evidence that the environment directly
affects the prevalence of the length of stay ($p=0.026$ with OW, $p=6.7e-6$
with IPW).

Figure 15 displays the prevalence of prolonged length of stay when
conditioning on specific comorbidities and treatments for older males: (a)
peripheral vascular comorbidities, (b) solid tumor.

(a)

(b)

Figure 15. Per comorbity and treatment prevalence of prolonged length of stay
(in %). (a) Peripheral vascular comorbidities. (b) Solid tumor.

####  C.2.4. The ICU unit $S$ affects the labs and vitals $X$

Based on the causal graph, we use the same OW and IPW weights as in Section
C.2.3. We build a summary of $X$ by predicting $Y$ from all features that are
not comorbidities, demographics or treatments. This recurrent model reaches
77.84% accuracy (AUPRC: 0.62, AUROC: 0.80) on the source test split. Based on
this summary, we observe that labs and vitals display some evidence of being
affected by the environment ($p=0.011$ for OW, $p=0.081$ for IPW). We note a
discrepancy between OW and IPW results in this case.