# Transferring Fairness under Distribution Shifts
via Fair Consistency Regularization

Bang An
Department of Computer Science
University of Maryland, College Park
<EMAIL_ADDRESS>
&Zora Che
Department of Computer Science
Boston University
<EMAIL_ADDRESS>
&Mucong Ding
Department of Computer Science
University of Maryland, College Park
<EMAIL_ADDRESS>
&Furong Huang
Department of Computer Science
University of Maryland, College Park
<EMAIL_ADDRESS>
###### Abstract

The increasing reliance on ML models in high-stakes tasks has raised a major
concern about fairness violations. Although there has been a surge of work
that improves algorithmic fairness, most are under the assumption of an
identical training and test distribution. In many real-world applications,
however, such an assumption is often violated as previously trained fair
models are often deployed in a different environment, and the fairness of such
models has been observed to collapse. In this paper, we study how to transfer
model fairness under distribution shifts, a widespread issue in practice. We
conduct a fine-grained analysis of how the fair model is affected under
different types of distribution shifts and find that domain shifts are more
challenging than subpopulation shifts. Inspired by the success of self-
training in transferring accuracy under domain shifts, we derive a sufficient
condition for transferring group fairness. Guided by it, we propose a
practical algorithm with fair consistency regularization as the key component.
A synthetic dataset benchmark, which covers diverse types of distribution
shifts, is deployed for experimental verification of the theoretical findings.
Experiments on synthetic and real datasets, including image and tabular data,
demonstrate that our approach effectively transfers fairness and accuracy
under various types of distribution shifts111Code is available at
https://github.com/umd-huang-lab/transfer-fairness..

##  1 Introduction

| Source | Target
---|---|---
Sampled from | UTKFace zhifei2017cvpr  | FairFace karkkainen2019fairface
Class | Gender
Sensitive attribute | Race
Group | Gender $+$ Race
Shared | Underlying data generation process
Differ | Capturing-bias, e.g., diff
angles, facial expressions

Figure 1: Illustration of intra-group expansion assumption in the input space.
An example of gender classification task with the sensitive attribute being
race. Intra-group expansion assumes that different groups are separated but
every group is self-connected under certain transformations. If a model has
consistent predictions under those transformations, we can propagate labels
within each group. Under this assumption, we propose to obtain fairness and
accuracy in both domains by a self-training algorithm with fair consistency
regularization.

Machine learning’s social impact has broadened as it is widely used to aid
decision-making in real-world applications, such as hiring, loan approval,
facial recognition, and criminal justice. To avoid discrimination against a
subset of the population (e.g., w.r.t race or gender), many efforts on
algorithmic fairness have been carried out chouldechova2016fair ;
friedler2016impossibility ; pmlr-v54-zafar17a ; survey ; rajkomar2018ensuring
; corbettdavies2018measure ; caton2020fairness . Although existing work has
achieved remarkable success in ensuring fairness, most of them assume the
distribution of data at test time is identical to that in the training set.
However, recent studies show that the fairness of a model is likely to
collapse when encountering a distribution shift. For example, ding2021retiring
observes that a fair income predictor trained with data from one state might
not be fair when used in other states. schrouff2022maintaining  tries to
maintain fairness in healthcare settings, but a model that performs fairly
according to the metric evaluated in “Hospital A” shows unfairness when
applied to “Hospital B”. Such observations motivate us to find the reason
behind the collapse of fairness and investigate how to transfer fairness under
distribution shifts. Specifically, when we have labeled data in the source
domain and unlabeled data in the target domain, we investigate how to adapt
the fair source model to a target domain with the goal of achieving both
accuracy and fairness in both domains.

Intuitively, the fairness of a model in the target domain strongly depends on
the nature of distribution shifts. In this paper, we only consider cases where
the oracle model is the same in two domains. We characterize distribution
shifts by assuming two domains share the same underlying data generation
process where data is generated from a set of latent factors with a fixed
generative model, and the shift is caused by the shift of the marginal
distribution of some factors. We categorize distribution shifts into three
types koh2021wilds : 1) Domain shift where source and target distributions
comprise data from related but distinct domains (e.g., train a model in
hospital A but test it in hospital B). 2) Subpopulation shift where two
domains overlap, but relative proportions of subpopulations differ (e.g., the
proportion of female candidates increases at test time). 3) Hybrid shift where
domain shift and subpopulation shift happen at the same time. We find domain
shift more challenging for transferring fairness since the model’s performance
is unpredictable in unseen domains. Such a finding is supported empirically on
a synthetic dataset that is developed to simulate diverse types of
distribution shifts. While recent work explores methods to transfer fairness
singh_violate ; rezaei_robust ; giguere2022fairness , most considered settings
fall into subpopulation shifts. In this paper, we consider all three types of
distribution shifts. Our analysis suggests we encourage consistent fairness
under different factor values.

We draw inspiration from recent progress on self-training in transferring
accuracy under domain shifts colin2021 ; tianle2021 ; zhang2021semisupervised
; berthelot2021adamatch ; Sagawa2021ExtendingTW ; sohn2020fixmatch . The
success of self-training is due to an expansion assumption and a consistency
regularization algorithm. The expansion assumption also assumes two domains
share one underlying generative model and the support of the distribution on
each class is a connected compact set under data transformations (i.e., has a
good continuity). Under the expansion assumption, colin2021  and tianle2021
prove that self-training, which enforces consistent predictions for the same
input under different transformations (i.e., under shifts of nuisance
factors), can propagate labels from the source to the target domain. This
approach exhibits superior performance in transferring accuracy
zhang2021semisupervised ; Sagawa2021ExtendingTW , but does not consider
fairness.

Taking demography into consideration, we relax the expansion assumption to a
more realistic intra-group expansion assumption, as shown in Figure 1, which
only requires continuity of the underlying distribution within every group
(i.e., data points with the same class and sensitive attribute) rather than
the entire class. Based on the intra-group expansion assumption, we derive a
sufficient condition that guarantees fairness in both source and target
domains. This sufficient condition suggests that ensuring the trained model
gains the same consistency across groups under a fair teacher classifier
guarantees fairness in both domains. However, such a teacher classifier is not
available in practice, and we need a practical treatment.

Guided by the theoretical algorithm, we propose a practical self-training
algorithm to minimize and balance consistency loss across groups. Our
algorithm builds upon Laftr david_learning_fair , an adversarial learning
method for fairness, and FixMatch sohn2020fixmatch , a self-training
framework. To encourage similar consistency in different groups, we propose a
novel fair consistency regularization. By reweighting the consistency loss of
each group dynamically according to the model’s performance, the algorithm
encourages the model to pay more attention to the high-error group while
training. Our method results in a model that is fair in source and has similar
consistency across groups. As indicated by our theory, it would have similar
accuracy across groups in the target domain so that we can transfer fairness.
We evaluate our method under different types of distribution shifts with the
synthetic and real datasets. Experiments show that our approach achieves high
accuracy and fairness in the target domain without sacrificing performance in
the source domain. To the best of our knowledge, this is the first work using
self-training to transfer fairness under distribution shifts.

Summary of contributions: (1) We provide a fine-grained analysis of fairness
under distribution shifts and develop a synthetic dataset to study model
fairness under different types of distribution shifts. (2) Theoretically, we
derive a sufficient condition for transferring fairness under distribution
shifts. (3) Algorithmically, we propose a theory-guided algorithm for
transferring fairness with a fair consistency regularization as the key
component. (4) Experimentally, we evaluate our method on synthetic data, real
image data, and real tabular data. All results show the effectiveness of our
approach in transferring fairness.

##  2 Preliminaries and Notations

Transfer Fairness. Let $X,A,Y$ and $\mathcal{X},\mathcal{A},\mathcal{Y}$
denote random variables and sample space of input features, sensitive
attribute, and label. For simplicity, we assume binary sensitive attribute and
binary classification, while our method can easily extend to multi-sensitive
attributes and multi-class cases (see Appendix E). We aim to learn a
classifier $g:\mathcal{X}\rightarrow\mathcal{Y}$ and are interested in its
fairness under distribution shifts. Specifically, with $S$ and $T$ denoting
source and target domains, we study how to transfer fairness and accuracy when
$\mathbb{P}_{S}(X,A,Y)\neq\mathbb{P}_{T}(X,A,Y)$, with the access to $X,A,Y$
in the source domain, but only $X,A$ in the target domain. In the self-
training algorithm, we use $g_{tc}$ to denote a teacher classifier, and
$g^{*}$ to denote the oracle classifier. We use the word “group” to denote the
set of data that has the same label and sensitive attribute.

Fairness Metric. Since we consider classification problems in this paper, we
expect the fairness metrics could encourage models to achieve similar
classification performance across groups. We use two metrics in this paper,
equalized odds and variance of group accuracy. Equalized odds
hardt2016equality  is a widely used unfairness metric in classification
problems that requires the true positive rate and the true negative rate to be
the same among groups. It is defined as
$\Delta_{odds}=\frac{1}{2}\sum_{y=0}^{1}\big{|}\mathbb{P}(\hat{Y}=y|A=0,Y=y)-\mathbb{P}(\hat{Y}=y|A=1,Y=y)\big{|}$,
where $\hat{Y}=g(X)$ is the prediction. Additionally, we also evaluate the
variance of group accuracy which is defined as
$V_{acc}=Var(\\{\mathbb{P}(\hat{Y}=y|A=a,Y=y),\forall a,y\\})$. Smaller
$V_{acc}$ indicates the model is fairer since it performs similarly across
groups. Note that the variance of group accuracy can help avoid trivial
fairness where a model with constant output has $\Delta_{odds}=0$, but such
fairness is meaningless.

##  3 Fairness under Distribution Shifts

In this section, we provide a fine-grained analysis of fairness under various
types of distribution shifts based on a unified framework of distribution
shift characterization.

A Unified Framework to Characterize Distribution Shifts. Following wiles2022a
, we characterize distribution shifts by assuming a unified latent variable
model for the underlying data generation process. We denote the underlying
factors as $Y^{1},Y^{2},...,Y^{K}$, and data point as $X$. Two of the factors
are label $Y^{l}$ (i.e. $Y$) and sensitive attribute $Y^{a}$ (i.e. $A$). We
call other factors nuisance factors since they are irrelevant to the
classification task.

######  Assumption 1.

(Underlying data generation process) We assume the data is generated from a
latent generative model as $\bm{y}^{1:K}\sim\mathbb{P}(Y^{1:K})$ and
$\bm{x}\sim\mathbb{P}(X|Y^{1:K}=\bm{y}^{1:K})$. The generative model is fixed
$\mathbb{P}_{S}(X|Y^{1:K}=\bm{y}^{1:K})=\mathbb{P}_{T}(X|Y^{1:K}=\bm{y}^{1:K})$
but the marginal distribution of factors varies in two domains
$\mathbb{P}_{S}(Y^{1:K})\neq\mathbb{P}_{T}(Y^{1:K})$, causing the distribution
shift $\mathbb{P}_{S}(Y^{1:K},X)\neq\mathbb{P}_{T}(Y^{1:K},X)$.

It is realistic to assume two domains share the same data generation process.
For example, the underlying physical process of cell imaging is fixed, while
the distribution of underlying factors (e.g. gender, age or equipment) may
vary in two hospitals (i.e. two domains), resulting in the distribution shift
of the observed tissue images. Based on the unified framework, we consider two
major types of distribution shifts, namely subpopulation shift and domain
shift, which are widely considered in many practical applications koh2021wilds
.

######  Definition 3.1.

(Subpopulation shift) We say it is a subpopulation shift, if for any factor
$Y^{i}$, the sample space of it remains the same in two domains (i.e.,
$\mathcal{Y}^{i}_{S}=\mathcal{Y}^{i}_{T}$), but the marginal distribution of
at least one factor changes (e.g.,
$\mathbb{P}_{S}(Y^{j})\neq\mathbb{P}_{T}(Y^{j})$), resulting in
$\mathbb{P}_{S}(Y^{1:K})\neq\mathbb{P}_{T}(Y^{1:K})$ and
$\mathbb{P}_{S}(Y^{1:K},X)\neq\mathbb{P}_{T}(Y^{1:K},X)$.

######  Definition 3.2.

(Domain shift) We say it is a domain shift, if at least one nuisance factor
$Y^{i},i\neq l,i\neq a$, has different sample space in two domains, $\exists
y^{i}\in\mathcal{Y}^{i}_{T}$, but $y^{i}\notin\mathcal{Y}^{i}_{S}$, resulting
in $\mathbb{P}_{S}(Y^{1:K})\neq\mathbb{P}_{T}(Y^{1:K})$ and
$\mathbb{P}_{S}(Y^{1:K},X)\neq\mathbb{P}_{T}(Y^{1:K},X)$.

Intuitively, under subpopulation shift, the sample space overlaps, and only
the marginal distributions of factors vary in the two domains. For example,
the proportion of females versus males in training and deployment time
differs. In contrast, under domain shift, the source model has never seen the
data with factor values that only exist in the target domain. For instance,
the source model is unaware of the equipment used for cell imaging at
deployment time.

Why do distribution shifts cause unfairness? Suppose the marginal
distributions of a binary nuisance factor $Y^{i}$ differ in two domains with
$\mathbb{P}_{S}(Y^{i})\neq\mathbb{P}_{T}(Y^{i})$. The unfairness in two
domains are

$\displaystyle\Delta_{odds}^{S}$
$\displaystyle=\mathbb{P}_{S}(Y^{i}=0)\times\Delta_{odds}^{S}|_{Y^{i}=0}+\mathbb{P}_{S}(Y^{i}=1)\times\Delta_{odds}^{S}|_{Y^{i}=1}$
(1) $\displaystyle\Delta_{odds}^{T}$
$\displaystyle=\mathbb{P}_{T}(Y^{i}=0)\times\Delta_{odds}^{T}|_{Y^{i}=0}+\mathbb{P}_{T}(Y^{i}=1)\times\Delta_{odds}^{T}|_{Y^{i}=1}.$

Due to the same generation process where
$\mathbb{P}_{S}(X|Y^{i}=y^{i})=\mathbb{P}_{T}(X|Y^{i}=y^{i})$, we have
$\Delta_{odds}^{S}|_{Y^{i}=y^{i}}=\Delta_{odds}^{T}|_{Y^{i}=y^{i}}$, $\forall
y^{i}\in\\{0,1\\}$. Under subpopulation shift, $Y^{i}$ has the same sample
space in two domains but with different proportions (e.g.,
${\mathbb{P}_{S}(Y^{i}=0)=0.9},{\mathbb{P}_{S}(Y^{i}=1)=0.1},\mathbb{P}_{T}(Y^{i}=0)=0.1,{\mathbb{P}_{T}(Y^{i}=1)=}0.9$),
while under domain shift the sample space differs (e.g.
$\mathbb{P}_{S}(Y^{i}=0)=1,\mathbb{P}_{T}(Y^{i}=1)=1$). It is easy to see from
(1) that if a model is perfectly fair on data with $Y^{i}=0$ but unfair on
data with $Y^{i}=1$, then the model is highly fair in the source domain but
highly unfair in the target domain under both cases. Therefore, if the model
has inconsistent performance on data generated from different nuisance factor
values, then the shifted marginal distribution of those factors may cause
fairness collapse.

How to transfer fairness under distribution shifts? Based on the above
analysis, one way is to train the model to be fair under any values of
factors. It is possible under subpopulation shift as stated in the following
proposition (see proof and discussion in Appendix B).

######  Proposition 3.1.

(Transfer fairness under subpopulation shift) Consider the subpopulation shift
that is caused by the shifted marginal distribution of nuisance factor $Y^{i}$
(i.e., $\mathbb{P}_{S}(Y^{i})\neq\mathbb{P}_{T}(Y^{i})$), while
$\mathcal{Y}_{S}^{i}=\mathcal{Y}_{T}^{i}=\mathcal{Y}^{i}$. If model $f$ is
strictly fair in source domain under any value of factor $Y^{i}$ satisfying
$\mathbb{P}_{S}(g(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i})=\mathbb{P}_{S}(g(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}),\forall
y^{i}\in\mathcal{Y}^{i},y^{l}\in\\{0,1\\}$, then model $g$ is also fair in
target domain with $\Delta_{odds}=0$.

Our empirical results (Figure 3) also support this finding. However, domain
shift is more challenging. The source model’s performance on target data is
unpredictable due to the distinct sample space. One promising way to tackle
domain shift is to enforce the model’s invariance to nuisance factors so that
the source model would have the same behavior on target data. Note that this
solution also works for subpopulation shift since it leads to the case in
Proposition 3.1 directly. The above analysis motivates us to transfer fairness
by encouraging consistent fairness under different nuisance factor values.

##  4 Transfer Fairness via Fair Consistency Regularization

###  4.1 Theoretical Analysis: A Sufficient Condition for Transferring
Fairness

In reality, distribution shifts are usually hybrid, and we may not know all
the underlying factor values. In this section, we consider a general case
where we only have access to input $X$, label $Y$, and sensitive attribute
$A$. We use data transformations to simulate the shift of nuisance factors.
Our theory is based on colin2021  and tianle2021  which prove that encouraging
consistency under transformations can propagate labels so that to transfer
accuracy. In this section, we find that in order to transfer fairness, we need
a fair label propagation process that requires the model to have similar
consistency across groups. We introduce assumptions and our findings as
follows.

######  Assumption 2 (Separability of the input).

Let $S_{a}^{y}$ and $T_{a}^{y}$ denote the sample space of $X|_{A=a,Y=y}$ in
source and target domains. The ground truth class and sensitive attribute for
$\bm{x}\in S_{a}^{y}\cup T_{a}^{y}$ are consistent, which are $y\in\\{0,1\\}$
and $a\in\\{0,1\\}$. We assume the sample spaces of $X$ in two domains are
$S=\cup_{y}\cup_{a}S_{a}^{y}$ and $T=\cup_{y}\cup_{a}T_{a}^{y}$, where groups
are separated with  1) $S_{a}^{y}\cap S_{a^{\prime}}^{y}=T_{a}^{y}\cap
T_{a^{\prime}}^{y}=S_{a}^{y}\cap T_{a^{\prime}}^{y}=\emptyset,\forall y,a\neq
a^{\prime}$, and  2) $S_{a}^{y}\cap S_{a^{\prime}}^{y^{\prime}}=T_{a}^{y}\cap
T_{a^{\prime}}^{y^{\prime}}=S_{a}^{y}\cap
T_{a^{\prime}}^{y^{\prime}}=\emptyset,\forall a,a^{\prime},y\neq y^{\prime}$.

This is a realistic assumption as illustrated in Figure 1 where the data from
two domains are from the same underlying conditional distribution $X|_{Y,A}$,
and groups are separated by label and sensitive attribute. We define
$U_{a}^{y}=\frac{1}{2}(S_{a}^{y}+T_{a}^{y})$ as the group distribution, and
$U$ as the population distribution on the entire data. Next, we characterize
the good continuity of group distributions with the definition of neighbor and
intra-group expansion assumption.

######  Definition 4.1 (Neighbor).

Let $\mathcal{T}$ denote a set of input transformations and define the
transformation set of $\bm{x}$ as
$\mathcal{B}(\bm{x})\triangleq\\{\bm{x}^{\prime}|\exists
t\in\mathcal{T},\textnormal{s.t. }\|\bm{x}^{\prime}-t(\bm{x})\|\leq r\\}$. For
any $\bm{x}\in S_{a}^{y}\cup T_{a}^{y}$, we define the neighbor of $\bm{x}$ as
$\mathcal{N}(\bm{x}):=(S_{a}^{y}\cup
T_{a}^{y})\cap\\{\bm{x}^{\prime}|\mathcal{B}(\bm{x})\cap\mathcal{B}(\bm{x}^{\prime})\neq\emptyset\\}$
and define the neighbor of a set $V\in\mathcal{X}$ as
$\mathcal{N}(V):=\cup_{\bm{x}\in V\cap(\cup_{y}\cup_{a}S_{a}^{y}\cup
T_{a}^{y})}\mathcal{N}(x)$.

Intuitively, two examples are neighbors if they are near each other after
applying some transformations. Note that we only consider neighbors that have
the same class and sensitive attribute (i.e., from the same group). Based on
this definition, we characterize the continuity of group distribution with
intra-group expansion assumption where any small set has a large neighbor in
its group.

######  Assumption 3 (Intra-group expansion).

We say that $U_{a}^{y}$ satisfies $(\alpha,c)$-multiplicative expansion for
some constant $\alpha\in(0,1)$ and $c>1$, if for all $V\subset U_{a}^{y}$ with
$\mathbb{P}_{U_{a}^{y}}(V)\leq\alpha$, the following holds:

$\displaystyle\mathbb{P}_{U_{a}^{y}}(\mathcal{N}(V))\geq\min\\{c\mathbb{P}_{U_{a}^{y}}(V),1\\}.$

Different from the expansion assumption proposed in colin2021  which considers
the class continuity, intra-group expansion assumes group continuity. As shown
in Figure 1, this is more realistic since groups are separated by both label
and sensitive attribute. We can also interpret it as the transformations that
change the value of nuisance factors will generate neighbors within the same
group.

This assumption allows us to propagate labels within the group from one domain
to another by encouraging consistency under transformations. We use
$R_{U_{a}^{y}}(g)\triangleq\mathbb{P}_{U_{a}^{y}}[\exists\bm{x}^{\prime}\in\mathcal{B}(\bm{x}),\textnormal{s.t.
}g(\bm{x})\neq g(\bm{x}^{\prime})]$ to denote the consistency loss of
classifier $g$ on the group distribution $U_{a}^{y}$, which is the fraction of
examples where $g$ is not robust to input transformations. Since we only have
partial supervision (i.e., no labels in the target domain), we use a self-
training framework to obtain a model that is accurate and fair in both domains
(i.e., on $U_{a}^{y}$). Based on the theory of self-training in colin2021 , we
derive a sufficient condition in Theorem 4.1 that bounds the unfairness and
error on the population distribution. We use 0-1 loss to evaluate the error of
$g$ as
$\varepsilon_{U_{a}^{y}}(g)\triangleq\mathbb{P}_{U_{a}^{y}}[g(\bm{x})\neq
g^{*}(\bm{x}^{\prime})]$, and the disagreement between $g$ and a teacher
classifier $g_{tc}$ as
$L_{U_{a}^{y}}(g,g_{tc})\triangleq\mathbb{P}_{U_{a}^{y}}[g(\bm{x})\neq
g_{tc}(\bm{x}^{\prime})]$.

######  Theorem 4.1 (Guarantee fairness).

Suppose we have a teacher classifier $g_{tc}$ with bounded unfairness such
that
$|\varepsilon_{U_{a}^{y}}(g_{tc})-\varepsilon_{U_{a^{\prime}}^{y^{\prime}}}(g_{tc})|\leq\gamma,\forall
a,a^{\prime}\in\mathcal{A}$ and $y,y^{\prime}\in\mathcal{Y}$. We assume intra-
group expansion where $U_{a}^{y}$ satisfies
$(\bar{\alpha},\bar{c})$-multiplicative expansion and
$\varepsilon_{U_{a}^{y}}(g_{tc})\leq\bar{\alpha}<1/3$ and $\bar{c}>3,\forall
a,y$. We define $c\triangleq\min\\{1/\bar{\alpha},\bar{c}\\}$, and set
$\mu\leq\varepsilon_{U_{a}^{y}}(g_{tc}),\forall a,y$. If we train our
classifier with the algorithm

$\displaystyle\min_{g\in
G}\max_{a,y}R_{U_{a}^{y}}(g),\quad\quad\textnormal{s.t. }\quad
L_{U_{a}^{y}}(g,g_{tc})\leq\mu\quad\forall a,y$

then the error and unfairness of the optimal solution $\hat{g}$ on the
distribution $U$ are bounded with

$\displaystyle\varepsilon(\hat{g})$
$\displaystyle\leq\frac{2}{c-1}\varepsilon_{U}(g_{tc})+\frac{2c}{c-1}R_{U}(\hat{g}),$
(2) $\displaystyle\Delta_{odds}(\hat{g})$
$\displaystyle\leq\frac{2}{c-1}(\gamma+\mu+c\max_{a,y}R_{U_{a}^{y}}(\hat{g}))$
(3)

Remark. This sufficient condition suggests we fit a teacher classifier which
is fair on the population distribution and minimize the consistency loss in
every group. The unfairness of the resulting model is bounded by the quality
(unfairness and error) of the teacher classifier and the worst-group
consistency loss. Intuitively, we can understand the consistency loss as the
model invariance to the nuisance factors. With a group-balanced consistency
loss, the model would have similar invariance to the nuisance factors
resulting in similar group performance on the unseen data so that to transfer
accuracy and fairness. We also bound the variance of group accuracy with the
variance of consistency loss (Appendix C). Both bounds suggest we balance and
minimize the consistency loss across groups.

###  4.2 Practical Algorithm: Fair Consistency Regularization

There are two challenges in realizing the theoretical algorithm in Theorem
4.1. First, we need a high-quality teacher model, but the model trained with
labeled source data is only fair and accurate in the source domain. Second,
existing consistency regularization methods do not consider fairness. We
tackle the first problem by leveraging the iterative self-training paradigm
that updates the teacher model with the student model while training, thus
making it fairer and fairer. We tackle the second problem by proposing a novel
fair consistency regularization.

Figure 2: Training diagram.

Algorithm. Figure 2 shows the overall training diagram. There are three major
components:
(1) In every training epoch, we use the student model obtained in the last
epoch as the teacher model and automatically fit the teacher model by
initializing the student model to be the same as the teacher model. In other
words, only one model is training itself iteratively.
(2) To ensure the accuracy and fairness in the source domain, we adopt Laftr
david_learning_fair , an adversarial learning method consisting of a
classification loss $L_{cls}$ and a fairness loss $L_{fair}$.
(3) To transfer fairness and accuracy, we do consistency training on all
unlabeled data (including source and target data). Following FixMatch
sohn2020fixmatch , we use the pseudo-labels generated by the teacher model as
supervision for consistency training where the model should have consistent
predictions under transformations. Different from FixMatch, we propose a fair
consistency regularization with a balanced group consistency loss
$L_{fconsis}$.
We train the model with the weighted summation of these three losses as shown
in Figure 2. We defer the detailed loss functions of $L_{cls}$ and $L_{fair}$
with a detailed algorithm description to Appendix D.

Fair Consistency Regularization. To tighten the upper bound of the unfairness
in Theorem 4.1, we need to minimize and balance consistency loss across
groups. However, the consistency regularization in FixMatch sohn2020fixmatch
does not distinguish groups and might amplify the bias as observed in
zhu2022the  and our experiments. Instead, we propose to use a fair consistency
regularization that evaluates the consistency loss per group and minimizes the
balanced consistency loss $L_{fconsis}$ defined as below.

$\displaystyle
L_{fconsis}(g)=\sum_{y=0}^{1}\sum_{a=0}^{1}\lambda_{a}^{y}L_{a}^{y}(g)$ (4)
where $\displaystyle
L_{a}^{y}(g)=\frac{1}{\sum_{\bm{x}_{a}^{y}}\mathds{1}}\sum_{\bm{x}_{a}^{y}}\mathds{1}(\max(g_{tc}(\bm{x}_{a}^{y}))\geq\tau)H(\mathop{\mathrm{argmax}}(g_{tc}(\bm{x}_{a}^{y})),g(t(\bm{x}_{a}^{y})))$
(5)

where $\bm{x}_{a}^{y}$ denotes an input with sensitive attribute $A=a$ and
class $Y=y$. $L_{a}^{y}(g)$ is model $g$’s consistency in the group of
$\\{\bm{x}_{a}^{y}\\}$, and $\lambda_{a}^{y}$ is the corresponding weight of
the group consistency loss. Here, we abuse the usage of $g(\bm{x})$ to denote
the output logits of model $g$ on input $\bm{x}$ and thus,
$\mathop{\mathrm{argmax}}(g_{tc}(\bm{x}_{a}^{i}))$ is the pseudolabel
generated by teacher classifier. $t(\bm{x}_{a}^{y})$ is the transformed input
as defined in Definition 4.1. We use a cross-entropy loss $H(\cdot)$ to
encourage the consistency under transformation $t(\cdot)$ and only consider
examples that the teacher model has high confidence in with a confidence
threshold $\tau$. Note that data is classified into groups according to the
true sensitive attribute and pseudolabels. To balance the group consistency
loss, we propose to weigh each group inversely with the number of confident
pseudolabels, and set $\lambda_{a}^{y}$ as

$\displaystyle\hat{\lambda}_{a}^{y}=\frac{1}{\sum_{\bm{x}_{a}^{y}}\mathds{1}(\max(g_{tc}(\bm{x}_{a}^{y}))\geq\tau)},\quad\lambda_{a}^{y}=\hat{\lambda}_{a}^{y}/\sum_{a,y}\hat{\lambda}_{a}^{y}.$
(6)

The weights will dynamically change while training. Heuristically, if the
teacher model is only confident in a few examples in a group, the model’s
consistency in this group is more likely to be low. With the proposed weights,
a larger penalty will be applied to such groups. Therefore, the proposed fair
consistency regularization will enforce the model to pay more attention to
high-error groups. By doing so, the trained model would enjoy similar
consistency loss across groups. Together with the self-training algorithm, it
would have similar accuracy across groups in the target domain.

##  5 Related Work

This section features related work for transferring fairness. Another
discussion of related work in fair machine learning, domain adaptation, and
self-training is deferred to Appendix A. Out-of-distribution fairness remains
an under-explored area. We categorize prior works into five classes.  1)
Group-wise distribution matching. candice_transfer  derives an upper bound for
fairness in the target domain which suggests training a fair model in the
source domain and matching the distributions of relevant groups from two
domains in feature space at the same time. yoon_wasser  also applies group-
wise distribution matching but with Wasserstein distance. Such methods are
hard to achieve if we do not have supervision in the target domain and it also
shares the drawback of distribution matching methods.  2) Reweighting. When
the proportions of groups differ in two domains, reweighting the examples in
the source domain can approximate the target distribution. coston_missing
uses reweighting to deal with fairness problems under covariate shift and
giguere2022fairness  uses reweighting together with a fairness test to
guarantee fairness under demographic shift. Reweighting methods strongly rely
on the support cover assumption which is not satisfied under domain shift.  3)
Distributionally robust optimization (DRO). This line of work considers
unknown target data that can be any arbitrary weighted combinations of the
source dataset and train a fair model that is robust to the worst-case shift
rezaei_robust ; mandal2020ensuring . These methods also assume subpopulation
shift instead of domain shift.  4) Causal inference. singh_violate  conducts
causal domain adaptation and DRO based on a well-characterized causal graph
that describes the data construction and distribution shift. Causal methods
highly rely on the correct causal graph which is hard to obtain in reality.
For example, schrouff2022maintaining  finds that the causal graph in real
applications (e.g. predicting the skin condition in dermatology) is far more
complicated which violates normal assumptions, thus making those approaches
inapplicable.  5) Others. chen2022fairness  derives bound for fairness
violation under distribution shifts. There are also studies that aim to
maintain fairness under distribution shifts through online learning
zhang2021farf , and loss curvature matching wang2022equalized .  To the best
of our knowledge, this is the first work that uses self-training to transfer
fairness. Some work also studies self-supervised learning and fairness, yet
they use unlabeled data and self-training to improve the in-distribution
fairness chzhen2019leveraging ; 9117188 ; chakraborty2021can  which is
different from our goal.

##  6 Experiments

Figure 3: Accuracy and unfairness (error bar denotes the standard deviation)
in two domains under subpopulation shifts (Sshift 1, Sshift 2), domain shift
(Dshift), and hybrid shift (Hshif). (S) and (T) denotes the evaluation in the
source and target domains respectively. Results show that domain shift is more
challenging than subpopulation shift, and our method can effectively transfer
accuracy and fairness under all the distribution shifts considered.

###  6.1 Evaluation under Different Types of Distribution Shifts with a
Synthetic Dataset

In order to study the fairness under distribution shifts and verify our
theoretical findings, we develop a synthetic dataset to simulate different
types of distribution shifts.

Synthetic dataset. The synthetic dataset is adapted from the 3dshapes dataset
pmlr-v80-kim18b  which contains images of 3D objects generated from six
independent latent factors (shape, object hue, scale, orientation, floor hue,
wall hue). This dataset satisfies our assumption on the shared underlying data
generation process. We simulate different types of distribution shifts by
varying the marginal distributions of the latent factors and sample the data
accordingly (see Appendix D.1 for details).

Distribution shifts. We set the image as input $X$, and select three latent
factors to be class ($Y=\textit{shape}$), sensitive attribute
($A=\textit{object hue}$), and a nuisance factor that might shift
($D=\textit{scale}$). We consider four widely observed distribution shifts in
reality ( $\mathbb{P}_{S}(X,Y,A,D)\neq\mathbb{P}_{T}(X,Y,A,D)$):
(1) Sshift 1: Subpopulation shift where only the nuisance factor shift (i.e.
more small objects in source but more large objects in target),
$\mathbb{P}_{S}(Y,A)=\mathbb{P}_{T}(Y,A)$,
$\mathbb{P}_{S}(D)\neq\mathbb{P}_{T}(D)$.
(2) Sshift 2: Subpopulation shift where $A$ and $Y$ have different
correlations in two domains (i.e. most red objects are cubes in source but are
capsules in target), $\mathbb{P}_{S}(Y,A)\neq\mathbb{P}_{T}(Y,A)$,
$\mathbb{P}_{S}(D)=\mathbb{P}_{T}(D)$.
(3) Dshift: Domain shift where the nuisance factor has different sample spaces
(i.e. only small objects in source but only large objects in target),
$\mathbb{P}_{S}(Y,A)=\mathbb{P}_{T}(Y,A)$,
$\mathbb{P}_{S}(D)\neq\mathbb{P}_{T}(D)$,
$\mathcal{Y}^{d}_{S}\neq\mathcal{Y}^{d}_{T}$.
(4) Hshift: Hybrid shift of (2) and (3).

Baselines. We do shape classification task with an MLP model and compare our
method with four baselines: Base (standard ERM); Laftr; Laftr+DANN (a
combination of Laftr and a domain adaptation method ganin2016domain );
Laftr+FixMatch. In our method, we also use Laftr and FixMatch but with the
proposed fair consistency regularization. Since the shifted nuisance factor is
scale, we use random padding and cropping as transformations in our method and
Laftr+FixMatch. We train Base and Laftr with labeled source data and train
others with unlabeled target data as well.

Domain shift is more challenging than subpopulation shift. Figure 3 shows that
under subpopulation shifts, the fair source model trained with Laftr also has
high accuracy and fairness in the target domain although it has not seen any
target date. This is because the sample space is shared (e.g. small and large
objects both exist in the source data), and the model has similar performance
under all factor values. Thus, good performance remains even if the proportion
of data changes, verified Proposition 3.1. In contrast, under domain shift and
hybrid shift, the fair source model performs poorly in the target domain where
data is sampled from a different sample space, suggesting the difficulty of
domain shift.

Our method can transfer fairness and accuracy under various types of
distribution shifts. Under domain shift, the domain adaptation method DANN
does not help in transferring fairness or accuracy. Consistency regularization
forces the model to behave consistently under cropping and padding, resulting
in a model that has similar predictions regardless of the object’s scale and
thus transfers accuracy. However, it may cause bias as shown in the results of
Laftr+FixMatch. With the proposed fair consistency regularization, the model
gains similar consistency across groups, resulting in a similar accuracy in
all groups in the target domain and thus transfers fairness. Therefore, our
method achieves high accuracy and fairness in two domains under all the
considered distribution shifts.

Table 1: Transfer fairness and accuracy from UTKFace to FairFace

| Source |  | Target
---|---|---|---
| Acc | Unfairness |  | Acc | Unfairness
Method |  | $V_{acc}$ | $\Delta_{odds}$ |  |  | $V_{acc}$ | $\Delta_{odds}$
Base |  92.85±0.49 |  2.30±0.97 |  4.81±0.69 |  |  74.49±0.83 |  5.79±3.49 |  9.90±1.27
Laftr |  93.24±0.41 |  1.19±0.46 |  2.44±0.51 |  |  74.35±1.46 |  6.92±0.72 |  9.79±1.54
CFair |  92.51±0.22 |  1.76±0.53 |  4.75±0.85 |  |  73.53±0.89 |  7.51±0.73 |  7.26±1.95
Laftr+DANN |  91.33±0.08 |  2.12±1.72 |  2.70±0.67 |  |  74.28±1.63 |  6.25±2.59 |  8.27±2.11
CFair+DANN |  90.89±0.76 |  2.01±0.70 |  4.43±1.36 |  |  74.62±1.06 |  6.23±0.90 |  5.26±2.07
Laftr+FixMatch |  96.62±0.06 |  0.77±0.21 |  2.23±0.44 |  |  83.87±0.48 |  8.21±0.67 |  9.32±1.01
CFair+FixMatch |  96.13±0.53 |  1.28±0.53 |  2.78±0.74 |  |  83.11±0.49 |  7.87±1.86 |  7.89±0.40
Ours (w/ Laftr) |  96.08±0.07 |  0.96±0.39 |  2.59±0.35 |  |  85.52±0.40 |  2.82±0.87 |  5.70±0.52
Ours (w/ CFair) |  95.65±0.22 |  1.56±0.37 |  3.85±0.97 |  |  84.48±0.42 |  2.88±0.99 |  5.43±0.65

(a) Pareto frontiers of ours (w/ Laftr) and baselines in the target domain.

(b) Pareto frontiers of ours (w/ Cfair) and baselines in the target domain.

Figure 4: Comparison of Pareto frontiers. Upper left is preferred. Our method
outperforms baseline methods in achieving accuracy and fairness at the same
time.

###  6.2 Evaluation on Real Datasets

Evaluation on images. We use UTKFace zhifei2017cvpr  as the source data and
FairFace karkkainen2019fairface  as the target data. Although both are facial
images, there is a distribution shift between them due to different image
sources. We consider a gender classification task with race as the sensitive
attribute. We use VGG16 simonyan2014very  as the model and RandAugment
cubuk2020randaugment  (excluding transformations that may change the group) as
the transformation function. Additional to previous baselines, we also use
CFair conditionalzhao  as the method for in-distribution fairness. As shown in
Table 1, there is indeed a distribution shift as the source model trained with
Laftr or CFair is no longer accurate or fair in the target domain. The domain
adaptation method has a limited effect on transferring accuracy and fairness.
As expected, self-training (Laftr+Fixmatch and CFair+Fixmatch) significantly
improves the accuracy in the target domain, but the unfairness is high. With
the proposed fair consistency regularization, our method outperforms it
remarkably on fairness with a 70% decrease in the variance of group accuracy
and a 30% decrease in the equalized odds. We further sweep the weights of
losses and draw Pareto frontiers. As shown in Figure 4, our method
significantly outperforms others in achieving accuracy and fairness at the
same time.

(a) Laftr

(b) Ours

(c) Decrease of unfairness

Figure 5: Unfairness and accuracy tested on NewAdult. CA as the source domain
(red star) and other states as the target domain (blue dots). Red lines
indicate the average of unfairness. The relative decrease is calculated by
comparing with Laftr.

Evaluation on tabular data. We further evaluate our method on the NewAdult
dataset ding2021retiring  which contains census data from all states of the
United States. We consider gender as the sensitive attribute and do income
classification with an MLP as the model. We set CA as the source domain and
all the other states as the target domain. We use random perturbation on
tabular data (see details in Appendix D) as the transformations. Results are
shown in Figure 5. When applied to other states, the fair model trained on CA
becomes unfair (Figure 5(a)). Our method improves the fairness in most states
with a slight improvement in accuracy (Figure 5(b)). Compared with the one
without fair consistency regularization, our method achieves better fairness
with a decrease in unfairness in most states (Figure 5(c)).

###  6.3 Ablation Study

The role of transformation. We design transformation functions based on our
domain knowledge of latent factors. To investigate the importance of
transformations, we test a weaker set of transformations, which includes only
cropping and flipping, on the UTKFace-FairFace experiment and report the
performance in Table 2. Compared with RandAugment in Table 1, consistency
under weak transformations leads to a less effective transfer of accuracy
since the neighbor generated transformations is much smaller. The limited
transformations also restrict the performance of our method on tabular data
(see Appendix E). Though the ability to transfer accuracy is limited by weak
transformations, our method can still make the transfer process fair as
there’s a significant decrease in unfairness, as shown in Table 2.

Figure 6: Per-group accuracy and consistency. Compared with the standard
consistency regularization (SCR), the model trained with fair consistency
regularization (FCR) has more balanced consistency and accuracy.

Fair consistency is essential in transferring fairness. To see whether
enhanced consistency improves accuracy and whether unbalanced consistency
leads to unfairness as suggested by Theorem 4.1, we evaluate the accuracy and
consistency of each group in the UTKFace-FairFace experiment on the target
data. The consistency is measured by testing the model’s agreement on the
outputs under two random transformations. As shown in Figure 6, groups that
obtain higher consistency have higher accuracy, which validates the ability of
consistency regularization for transferring accuracy. The training methods
that use standard consistency regularization (e.g. Laftr+FixMatch) have been
observed to be unfair in the target domain. Figure 6 shows that it is because
the model has imbalanced consistency across groups. With our fair consistency
regularization, the model gains similar consistency for all groups, resulting
in similar group accuracy.

| Source |  | Target
---|---|---|---
| Acc | Unfairness |  | Acc | Unfairness
Method |  | $V_{acc}$ | $\Delta_{odds}$ |  |  | $V_{acc}$ | $\Delta_{odds}$
Laftr+FixMatch |  94.08±0.70 |  1.64±0.46 |  3.51±1.46 |  |  77.05±0.26 |  12.23±3.83 |  6.55±1.54
CFair+FixMatch |  94.09±0.33 |  0.97±0.36 |  2.16±0.97 |  |  77.25±0.21 |  12.93±2.66 |  9.77±0.95
Ours (w/ Laftr) |  94.25±0.22 |  1.06±0.46 |  2.09±0.55 |  |  77.32±0.21 |  2.35±1.67 |  4.27±1.41
Ours (w/ CFair) |  94.24±0.26 |  1.67±0.38 |  4.43±0.63 |  |  77.96±0.38 |  3.34±1.08 |  5.70±1.14

Table 2: Transfer fairness and accuracy from UTKFace to FairFace with weak
transformations

| Acc | Unfairness
---|---|---
Method |  | $V_{acc}$ | $\Delta_{odds}$
Ours |  85.52±0.40 |  2.82±0.87 |  5.70±0.52
w/o consistency in target |  82.43±1.05 |  6.80±1.30 |  5.85±0.40
w/o consistency in source |  82.5±1.58 |  6.63±0.71 |  8.18±1.27
w/o dynamic weights |  84.34±0.19 |  6.86±0.50 |  7.68±0.81
w/o updating $g_{tc}$ |  79.13±0.52 |  3.49±0.63 |  6.65±1.31

Table 3: Ablation study on UTKFace-FairFace task

The role of components in fair consistency regularization. Table 3 shows the
ablation study. We can see that the consistency in both domains matters.
Giving every group the same weight instead of using dynamic weights leads to
increased unfairness. Fixing the teacher classifier to be the fair source
model, we observe a significant decrease in the accuracy, suggesting the
important role of iterative self-training in our algorithm.

##  7 Conclusion

In this paper, we explore how to transfer fairness under distribution shifts.
We derive a sufficient condition and present a theory-guided self-training
algorithm based on an intra-group expansion assumption. The key component of
our algorithm is fair consistency regularization. We simulate different types
of distribution shifts with a synthetic dataset and examine our theoretical
findings with it. Abundant experiments with synthetic data and real data have
shown that our method has superior performance in transferring fairness and
accuracy. Like other self-training methods, one limitation of our method is
the reliance on a well-defined data transformation set. Future work will relax
this limitation for application to more real-world problems.

## Acknowledgements

This work is supported by National Science Foundation NSF-IIS-FAI program,
DOD-ONR-Office of Naval Research, DOD-DARPA-Defense Advanced Research Projects
Agency Guaranteeing AI Robustness against Deception (GARD), and Adobe, Capital
One and JP Morgan faculty fellowships.

## References

  * (1) Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna M. Wallach.  A reductions approach to fair classification.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 60–69. PMLR, 2018.
  * (2) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Mach. Learn., 79(1-2):151–175, 2010.
  * (3) David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin.  Adamatch: A unified approach to semi-supervised learning and domain adaptation, 2021.
  * (4) Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H. Chi.  Data decisions and theoretical implications when adversarially learning fair representations, 2017.
  * (5) Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei.  A theory of label propagation for subpopulation shift.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1170–1182. PMLR, 18–24 Jul 2021.
  * (6) Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy.  Building classifiers with independency constraints.  In 2009 IEEE International Conference on Data Mining Workshops, pages 13–18, 2009.
  * (7) Simon Caton and Christian Haas.  Fairness in machine learning: A survey, 2020.
  * (8) L. Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K. Vishnoi.  Classification with fairness constraints: A meta-algorithm with provable guarantees.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 319–328, New York, NY, USA, 2019. Association for Computing Machinery.
  * (9) Joymallya Chakraborty, Huy Tu, Suvodeep Majumder, and Tim Menzies.  Can we achieve fairness using semi-supervised learning?  arXiv preprint arXiv:2111.02038, 2021.
  * (10) Yatong Chen, Reilly Raab, Jialu Wang, and Yang Liu.  Fairness transferability subject to bounded distribution shift.  arXiv preprint arXiv:2206.00129, 2022.
  * (11) Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma.  Self-training avoids using spurious features under domain shift.  Advances in Neural Information Processing Systems, 33:21061–21071, 2020.
  * (12) Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, 2016.
  * (13) Ching-Yao Chuang and Youssef Mroueh.  Fair mixup: Fairness via interpolation.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * (14) Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil.  Leveraging labeled and unlabeled data for consistent fair binary classification.  Advances in Neural Information Processing Systems, 32, 2019.
  * (15) Sam Corbett-Davies and Sharad Goel.  The measure and mismeasure of fairness: A critical review of fair machine learning, 2018.
  * (16) Amanda Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R. Varshney, Skyler Speakman, Zairah Mustahsan, and Supriyo Chakraborty.  Fair transfer learning with missing protected attributes.  In Vincent Conitzer, Gillian K. Hadfield, and Shannon Vallor, editors, Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, January 27-28, 2019, pages 91–98. ACM, 2019.
  * (17) Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A. Weis, Kevin Swersky, Toniann Pitassi, and Richard S. Zemel.  Flexibly fair representation learning by disentanglement.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1436–1445. PMLR, 2019.
  * (18) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * (19) Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt.  Retiring adult: New datasets for fair machine learning.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * (20) Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil.  Empirical risk minimization under fairness constraints.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2796–2806, 2018.
  * (21) Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian.  On the (im)possibility of fairness, 2016.
  * (22) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * (23) Stephen Giguere, Blossom Metevier, Yuriy Brun, Philip S. Thomas, Scott Niekum, and Bruno Castro da Silva.  Fairness guarantees under demographic shift.  In International Conference on Learning Representations, 2022.
  * (24) Bruce Glymour and Jonathan Herington.  Measuring the biases that matter: The ethical and casual foundations for measures of fairness in algorithms.  In danah boyd and Jamie H. Morgenstern, editors, Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pages 269–278. ACM, 2019.
  * (25) Karan Goel, Albert Gu, Yixuan Li, and Christopher Re.  Model patching: Closing the subgroup performance gap with data augmentation.  In International Conference on Learning Representations, 2021.
  * (26) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * (27) Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  Advances in neural information processing systems, 29, 2016.
  * (28) Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1989–1998. PMLR, 10–15 Jul 2018\.
  * (29) Taotao Jing, Bingrong Xu, and Zhengming Ding.  Towards fair knowledge transfer for imbalanced domain adaptation.  IEEE Transactions on Image Processing, 30:8200–8211, 2021.
  * (30) Kimmo Kärkkäinen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age.  arXiv preprint arXiv:1908.04913, 2019.
  * (31) Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR, 10–15 Jul 2018\.
  * (32) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021.
  * (33) Ananya Kumar, Tengyu Ma, and Percy Liang.  Understanding self-training for gradual domain adaptation.  In International Conference on Machine Learning, pages 5468–5479. PMLR, 2020.
  * (34) Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4066–4076, 2017.
  * (35) Chao Lan and Jun Huan.  Discriminatory transfer.  arXiv preprint arXiv:1707.00780, 2017.
  * (36) Bo Li, Yezhen Wang, Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer.  Rethinking distributional matching based domain adaptation.  arXiv preprint arXiv:2006.13352, 2020.
  * (37) Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma.  Self-supervised learning is more robust to dataset imbalance.  In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  * (38) Hong Liu, Jianmin Wang, and Mingsheng Long.  Cycle self-training for domain adaptation, 2021.
  * (39) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In International conference on machine learning, pages 97–105. PMLR, 2015.
  * (40) Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  Advances in neural information processing systems, 31, 2018.
  * (41) Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard S. Zemel.  The variational fair autoencoder.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * (42) David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel.  Learning adversarially fair and transferable representations.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3381–3390. PMLR, 2018\.
  * (43) Debmalya Mandal, Samuel Deng, Suman Jana, Jeannette Wing, and Daniel J Hsu.  Ensuring fairness beyond the training data.  Advances in neural information processing systems, 33:18445–18456, 2020.
  * (44) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  CoRR, abs/1908.09635, 2019.
  * (45) Luca Oneto and Silvia Chiappa.  Fairness in machine learning.  Studies in Computational Intelligence, page 155–196, 2020.
  * (46) Alvin Rajkomar, Michaela Hardt, Michael D Howell, Greg Corrado, and Marshall H Chin.  Ensuring fairness in machine learning to advance health equity.  Annals of internal medicine, 169(12):866–872, 2018.
  * (47) Charan Reddy, Deepak Sharma, Soroush Mehri, Adriana Romero-Soriano, Samira Shabanian, and Sina Honari.  Benchmarking bias mitigation algorithms in representation learning through fairness metrics.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * (48) Ashkan Rezaei, Anqi Liu, Omid Memarrast, and Brian D. Ziebart.  Robust fairness under covariate shift.  CoRR, abs/2010.05166, 2020.
  * (49) Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori B. Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang.  Extending the wilds benchmark for unsupervised adaptation.  ArXiv, abs/2112.05090, 2021.
  * (50) Jessica Schrouff, Natalie Harris, Oluwasanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alex Brown, Subhrajit Roy, Diana Mincu, Christina Chen, Awa Dieng, Yuan Liu, Vivek Natarajan, Alan Karthikesalingam, Katherine Heller, Silvia Chiappa, and Alexander D’Amour.  Maintaining fairness across distribution shift: do we have viable solutions for real-world applications?, 2022.
  * (51) Candice Schumann, Xuezhi Wang, Alex Beutel, Jilin Chen, Hai Qian, and Ed H. Chi.  Transfer of machine learning fairness across domains.  CoRR, abs/1906.09688, 2019.
  * (52) Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon.  A dirt-t approach to unsupervised domain adaptation.  arXiv preprint arXiv:1802.08735, 2018.
  * (53) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * (54) Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara.  Fairness violations and mitigation under covariate shift.  In Madeleine Clare Elish, William Isaac, and Richard S. Zemel, editors, FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 3–13. ACM, 2021.
  * (55) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  arXiv preprint arXiv:2001.07685, 2020.
  * (56) Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon.  Learning controllable fair representations.  In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 2164–2173. PMLR, 2019.
  * (57) Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon.  Domain adaptation with conditional distribution matching and generalized label shift.  Advances in Neural Information Processing Systems, 33:19276–19289, 2020.
  * (58) Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176, 2017.
  * (59) Haotao Wang, Junyuan Hong, Jiayu Zhou, and Zhangyang Wang.  Equalized robustness: Towards sustainable fairness under distributional shifts, 2022.
  * (60) Tongxin Wang, Zhengming Ding, Wei Shao, Haixu Tang, and Kun Huang.  Towards fair cross-domain adaptation via generative learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 454–463, 2021.
  * (61) Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma.  Theoretical analysis of self-training with deep networks on unlabeled data.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * (62) Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil.  A fine-grained analysis on distribution shift.  In International Conference on Learning Representations, 2022.
  * (63) Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton.  Domain adaptation with asymmetrically-relaxed distribution alignment.  In International Conference on Machine Learning, pages 6872–6881. PMLR, 2019.
  * (64) Tae-Ho Yoon, Jaewook Lee, and Woojin Lee.  Joint transfer of model knowledge and fairness over domains using wasserstein distance.  IEEE Access, 8:123783–123798, 2020.
  * (65) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P. Gummadi.  Fairness Constraints: Mechanisms for Fair Classification.  In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 962–970. PMLR, 20–22 Apr 2017.
  * (66) Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.  Learning fair representations.  In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 325–333, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
  * (67) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell.  Mitigating unwanted biases with adversarial learning.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340, 2018.
  * (68) Tao Zhang, tianqing zhu, Jing Li, Mengde Han, Wanlei Zhou, and Philip Yu.  Fairness in semi-supervised learning: Unlabeled data help to reduce discrimination.  IEEE Transactions on Knowledge and Data Engineering, pages 1–1, 2020.
  * (69) Wenbin Zhang, Albert Bifet, Xiangliang Zhang, Jeremy C Weiss, and Wolfgang Nejdl.  Farf: A fair and adaptive random forests classifier.  In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 245–256. Springer, 2021.
  * (70) Yabin Zhang, Haojian Zhang, Bin Deng, Shuai Li, Kui Jia, and Lei Zhang.  Semi-supervised models are strong unsupervised domain adaptation learners, 2021.
  * (71) Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
  * (72) Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon.  Conditional learning of fair representations.  In International Conference on Learning Representations, 2019.
  * (73) Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J. Gordon.  Conditional learning of fair representations.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * (74) Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon.  On learning invariant representations for domain adaptation.  In International Conference on Machine Learning, pages 7523–7532. PMLR, 2019.
  * (75) Sicheng Zhu, Bang An, and Furong Huang.  Understanding the generalization benefit of model invariance from a data perspective.  Advances in Neural Information Processing Systems, 34, 2021.
  * (76) Zhaowei Zhu, Tianyi Luo, and Yang Liu.  The rich get richer: Disparate impact of semi-supervised learning.  In International Conference on Learning Representations, 2022.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] See Section F

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes]

    2. (b)

Did you include complete proofs of all theoretical results? [Yes]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] see Section D

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Section D

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

Supplementary Material

##  Appendix A More on Related Work

Fair machine learning. Generally, fair machine learning methods fall into
three categories: pre-processing, in-processing, and post-processing [44, 7].
In this paper, we focus on in-processing methods that modify learning
algorithms to remove discrimination during the training process. As for fair
classification, several approaches have been proposed including fair
representation learning [66, 41, 4, 67, 42, 56, 17, 73], fairness-constrained
optimization [20, 1], causal methods [34, 24, 45], and many other approaches
with different techniques [8, 13, 25]. All of those works are for in-
distribution fairness, and we investigate out-of-distribution fairness in this
paper. We use LAFTR [42], an adversarial learning method that shows advanced
performance on fairness [47], to learn a fair model in the source domain and
adapt it to the target domain. We also test CFair[72] in our experiments. Many
metrics of fairness have been proposed [15] including demographic parity [6],
equalized opportunity, and equalized odds [27] which are most widely adopted.
In this paper, we use equalized odds to measure unfairness in both domains.

Distribution shifts. In many real-world applications, distribution shifts are
unavoidable. The goal of existing work addressing distribution shifts is
simply to transfer accuracy. [32] propose a benchmark of in-the-wild datasets
to study the real distribution shifts. We follow their category of
distribution shifts, including subpopulation shifts and domain shifts. Their
empirical results on many state-of-the-art methods show that self-training
outperforms others on image datasets significantly while having limited
performance due to the limited data augmentation on non-image modalities [49].
This finding aligns with our experimental results. [62] conduct a fine-grained
analysis of various distribution shifts based on an underlying data generation
assumption similar to ours. They also use 3dshapes dataset to simulate
different types of distribution shifts. Additional to accuracy, we aim to
transfer fairness at the same time in this paper.

Domain adaptation and self-training. Inspired by the theoretical work [2],
numerous distribution matching approaches have emerged for domain adaption
over the past decade. Domain-adversarial training [22] and many of its
variants [58, 40, 28, 57] that aim at matching the distribution of two domains
in the feature space have shown encouraging results in many applications.
However, recent studies [63, 74, 36] show that such methods may fail in many
cases since they only optimize part of the theoretical bound. We test DANN
[22], and MMD [39], two distribution matching methods in our experiments, and
also find them less effective in transferring accuracy and fairness. Recently,
another line of work that uses self-training draws increasing attention [70,
3]. Those methods enjoy guarantees [61, 5] and demonstrate superior empirical
results with desirable properties such as robustness to spurious features [33,
11, 38] and robustness to dataset imbalance [37]. However, all of those work
on domain adaptation only aims at transferring accuracy. Although there is
work that studies fairness issues in current domain adaptation methods [35]
and proposes to alleviate it by balancing the data [29, 60, 76], fair domain
adaptation is still under-explored. Based on the findings that the model’s
consistency to input transformations is important to generalization [75] and
is a core component of self-training [52, 55, 26], we improve the consistency
regularization in [55] to achieve fair transferring.

##  Appendix B Proof and More Discussion of Fairness under Distribution Shifts

######  Lemma B.1.

Under Assumption 1, for a subpopulation shift that is caused by the shift of
the marginal distribution of factor $Y^{i}$, we have
$\mathbb{P}_{S}(X|Y^{i}=y^{i})=\mathbb{P}_{T}(X|Y^{i}=y^{i}),\forall
y^{i}\in\mathcal{Y}^{i}$.

###### Proof.

Under Assumption 1,
$\mathbb{P}_{S}(X|Y^{1:K}=\bm{y}^{1:K})=\mathbb{P}_{T}(X|Y^{1:K}=\bm{y}^{1:K})$.
Since the shift only happens on the factor $Y^{i}$, the marginal distribution
of other factors remains the same in the two domains,
$\mathbb{P}_{S}(Y^{\\{1:K\\}\setminus i})=\mathbb{P}_{T}(Y^{\\{1:K\\}\setminus
i})$ where we use $\\{1:K\\}\setminus i$ to denote $1,..,i-1,i+1,...,K$. Then

$\displaystyle\mathbb{P}_{S}(X|Y^{i}=y^{i})$
$\displaystyle=\sum_{\bm{y}^{\\{1:K\\}\setminus
i}}\mathbb{P}_{S}(X,Y^{\\{1:K\\}\setminus i}=\bm{y}^{\\{1:K\\}\setminus
i}|Y^{i}=y^{i})$ $\displaystyle=\sum_{\bm{y}^{\\{1:K\\}\setminus
i}}\mathbb{P}_{S}(Y^{\\{1:K\\}\setminus i}=\bm{y}^{\\{1:K\\}\setminus
i})\mathbb{P}_{S}(X|Y^{i}=y^{i},Y^{\\{1:K\\}\setminus
i}=\bm{y}^{\\{1:K\\}\setminus i})$
$\displaystyle=\sum_{\bm{y}^{\\{1:K\\}\setminus
i}}\mathbb{P}_{T}(Y^{\\{1:K\\}\setminus i}=\bm{y}^{\\{1:K\\}\setminus
i})\mathbb{P}_{T}(X|Y^{i}=y^{i},Y^{\\{1:K\\}\setminus
i}=\bm{y}^{\\{1:K\\}\setminus i})$
$\displaystyle=\mathbb{P}_{T}(X|Y^{i}=y^{i})$

where the second line holds because of the independence of the latent factors
$Y^{1},...,Y^{K}$. ∎

Now, we restate Proposition 3.1 and provide the proof.

######  Proposition B.1 (Transfer of fairness under subpopulation shift).

Consider the subpopulation shift that is caused by the shifted marginal
distribution of a nuisance factor $Y^{i}$ (i.e.,
$\mathbb{P}_{S}(Y^{i})\neq\mathbb{P}_{T}(Y^{i})$), while
$\mathcal{Y}_{S}^{i}=\mathcal{Y}_{T}^{i}=\mathcal{Y}^{i}$. If model $f$ is
strictly fair in the source domain under any value of factor $Y^{i}$
satisfying
$\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i})=\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}),\forall
y^{i}\in\mathcal{Y}^{i},y^{l}\in\\{0,1\\}$, then $f$ is also fair in target
domain with $\Delta_{odds}=0$.

###### Proof.

In the target domain, the equalized odds (unfairness) is defined as

$\displaystyle\Delta_{odds}=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\mathbb{P}_{T}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})-\mathbb{P}_{T}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l})\big{|}.$

Since all latent factors are independent, we have

$\displaystyle\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l}\right)$
$\displaystyle=\sum_{y^{i}\in\mathcal{Y}^{i}}\mathbb{P}_{T}(Y^{i}=y^{i})\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)$

and

$\displaystyle\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l}\right)$
$\displaystyle=\sum_{y^{i}\in\mathcal{Y}^{i}}\mathbb{P}_{T}(Y^{i}=y^{i})\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right).$

Therefore, the $\Delta_{odds}$ in the target domain can be decomposed into

$\displaystyle\Delta_{odds}=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\sum_{y^{i}\in\mathcal{Y}^{i}}\mathbb{P}_{T}(Y^{i}=y^{i})$
$\displaystyle\big{(}\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)$
$\displaystyle-\mathbb{P}_{T}\left(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right)\big{)}\big{|}.$

Since two domains share the same underlying data generative model, and the
distribution shift is caused by the shift of the marginal distribution of
factor $Y^{i}$, from Lemma B.1, we have

$\displaystyle\mathbb{P}_{T}\left(X|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)=\mathbb{P}_{S}\left(X|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)$

and

$\displaystyle\mathbb{P}_{T}\left(X|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right)=\mathbb{P}_{S}\left(X|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right).$

Thus the conditional distribution of the model’s prediction also remains, as

$\displaystyle\mathbb{P}_{T}\left(f(X)|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)=\mathbb{P}_{S}\left(f(X)|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)$

and

$\displaystyle\mathbb{P}_{T}\left(f(X)|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right)=\mathbb{P}_{S}\left(f(X)|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right).$

In this case, if the source model is strictly fair that $\forall
y^{i}\in\mathcal{Y}^{i},y^{l}\in\\{0,1\\}$ the following holds

$\displaystyle\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i})=\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}),$

then it is also fair in the target domain with

$\displaystyle\Delta_{odds}=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\sum_{y^{i}\in\mathcal{Y}^{i}}\mathbb{P}_{T}(Y^{i}=y^{i})$
$\displaystyle\big{(}\mathbb{P}_{S}\left(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l},Y^{i}=y^{i}\right)$
$\displaystyle-\mathbb{P}_{S}\left(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l},Y^{i}=y^{i}\right)\big{)}\big{|}=0.$

∎

This proposition explains why the fair source model is also fair in the target
domain under Sshift 1 in our experiments (see Section 6.1). In addition to
shifts of nuisance factors, the subpopulation shifts can also be caused by the
marginal distribution shift of the label and sensitive attribute. The
following proposition argues that the fair model is also in the target domain
under such distribution shifts.

######  Proposition B.2 (Transfer of fairness under subpopulation shift of
sensitive attribute).

Consider the subpopulation shift that is caused by the shifted marginal
distribution of sensitive attribute $Y^{a}$ (i.e.,
$\mathbb{P}_{S}(Y^{a})\neq\mathbb{P}_{T}(Y^{a})$), while
$\mathcal{Y}_{S}^{a}=\mathcal{Y}_{T}^{a}=\mathcal{Y}^{a}=\\{0,1\\}$. If model
$f$ is fair in the source domain with $\Delta^{S}_{odds}=0$, then it is also
fair in the target domain with $\Delta^{T}_{odds}=0$.

###### Proof.

The proof is similar to the previous one. Since

$\displaystyle\Delta^{S}_{odds}=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})-\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l})\big{|},$

and from Lemma B.1 we know that

$\displaystyle\mathbb{P}_{S}(X|Y^{a}=0,Y^{l}=y^{l})$
$\displaystyle=\mathbb{P}_{T}(X|Y^{a}=0,Y^{l}=y^{l})$
$\displaystyle\mathbb{P}_{S}(X|Y^{a}=1,Y^{l}=y^{l})$
$\displaystyle=\mathbb{P}_{T}(X|Y^{a}=1,Y^{l}=y^{l}),$

thus,

$\displaystyle\Delta^{T}_{odds}$
$\displaystyle=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\mathbb{P}_{T}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})-\mathbb{P}_{T}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l})\big{|}$
$\displaystyle=\frac{1}{2}\sum_{y^{l}=0}^{1}\big{|}\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})-\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=1,Y^{l}=y^{l})\big{|}$
$\displaystyle=\Delta^{S}_{odds}=0.$

∎

Such a result also holds for subpopulation shifts caused by the shift of label
$Y^{l}$. This proposition explains why the fair source model is also fair in
the target domain under Sshift 2 in our experiments (see Section 6.1). It
suggests that encouraging fairness is able to alleviate spurious correlation.
We leave more studies on this interesting finding to future work.

Remark. All the above analyses are based on the population distribution where
${\mathbb{P}_{S}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})=\mathbb{E}_{\mathbb{P}_{S}(X,Y^{1:K})}(f(X)=y^{l}|Y^{a}=0,Y^{l}=y^{l})}$.
In practice, it is estimated by finite samples. Insufficient samples would
cause estimation errors in fairness and bring another challenge for
transferring fairness. In this paper, we only consider the fairness measured
by population distribution. Future work will investigate the impact of
estimation error on transferring fairness and the way to resolve it.

##  Appendix C Proof of the Sufficient Condition for Transferring Fairness

Our proof is based on the theory in [61].

######  Theorem C.1.

(Restatement of Lemma A.8 in [61]) We assume that $U_{a}^{y}$ satisfies
$(\bar{\alpha},\bar{c})$-multiplicative expansion for
$\varepsilon_{U_{a}^{y}}(g_{tc})\leq\bar{\alpha}<1/3$ and $\bar{c}>3$. We
define $c\triangleq\min\\{1/\bar{\alpha},\bar{c}\\}$. Then for any classifier
$g:\mathcal{X}\rightarrow\mathcal{Y}$, the error of it on the group
$U_{a}^{y}$ is upper bounded as:

$\displaystyle\varepsilon_{U_{a}^{y}}(g)\leq\frac{c+1}{c-1}L_{U_{a}^{y}}(g,g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(g)-\varepsilon_{U_{a}^{y}}(g_{tc})$

######  Theorem C.2.

(A restricted version of the above theorem) We assume that $U_{a}^{y}$
satisfies $(\bar{\alpha},\bar{c})$-multiplicative expansion for
$\varepsilon_{U_{a}^{y}}(g_{tc})\leq\bar{\alpha}<1/3$ and $\bar{c}>3$. We
define $c\triangleq\min\\{1/\bar{\alpha},\bar{c}\\}$. Then for any classifier
$g:\mathcal{X}\rightarrow\mathcal{Y}$ satisfies
$L_{U_{a}^{y}}(g,g_{tc})\leq\varepsilon_{U_{a}^{y}}(g_{tc})$, the error of it
on the group $U_{a}^{y}$ is upper bounded as:

$\displaystyle\varepsilon_{U_{a}^{y}}(g)\leq\frac{2}{c-1}\varepsilon_{U_{a}^{y}}(g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(g)$

###### Proof.

$\displaystyle\varepsilon_{U_{a}^{y}}(g)$
$\displaystyle\leq\frac{c+1}{c-1}L_{U_{a}^{y}}(g,g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(g)-\varepsilon_{U_{a}^{y}}(g_{tc})$
$\displaystyle\varepsilon_{U_{a}^{y}}(g)$
$\displaystyle\leq\frac{2}{c-1}\varepsilon_{U_{a}^{y}}(g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(g)$
(because $L_{U_{a}^{y}}(g,g_{tc})\leq\varepsilon_{U_{a}^{y}}(g_{tc})$)

∎

######  Theorem C.3.

If $L_{U_{a}^{y}}(g,g_{tc})\leq\varepsilon_{U_{a}^{y}}(g_{tc})$, we have

$\displaystyle\varepsilon_{U_{a}^{y}}(g)\geq\varepsilon_{U_{a}^{y}}(g_{tc})-L_{U_{a}^{y}}(g,g_{tc})$

###### Proof.

By triangle inequality. ∎

Now, we restate Theorem 4.1 and provide the proof.

######  Theorem C.4.

Suppose we have a teacher classifier $g_{tc}$ with bounded unfairness such
that
$|\varepsilon_{U_{a}^{y}}(g_{tc})-\varepsilon_{U_{a^{\prime}}^{y^{\prime}}}(g_{tc})|\leq\gamma,\forall
a,a^{\prime}\in\mathcal{A}$ and $y,y^{\prime}\in\mathcal{Y}$. We assume intra-
group expansion where $U_{a}^{y}$ satisfies
$(\bar{\alpha},\bar{c})$-multiplicative expansion and
$\varepsilon_{U_{a}^{y}}(g_{tc})\leq\bar{\alpha}<1/3$ and $\bar{c}>3,\forall
a,y$. We define $c\triangleq\min\\{1/\bar{\alpha},\bar{c}\\}$, and set
$\mu\leq\varepsilon_{U_{a}^{y}}(g_{tc}),\forall a,y$. If we train our
classifier with the algorithm

$\displaystyle\min_{g\in G}\max_{a,y}R_{U_{a}^{y}}(g)$ (7) s.t.
$\displaystyle\quad L_{U_{a}^{y}}(g,g_{tc})\leq\mu\quad\forall a,y$

then the error and unfairness of the optimal solution $\hat{g}$ on the
distribution $U$ are bounded with

$\displaystyle\varepsilon(\hat{g})$
$\displaystyle\leq\frac{2}{c-1}\varepsilon_{U}(g_{tc})+\frac{2c}{c-1}R_{U}(\hat{g}),$
$\displaystyle\Delta_{odds}(\hat{g})$
$\displaystyle\leq\frac{2}{c-1}(\gamma+\mu+c\max_{a,y}R_{U_{a}^{y}}(\hat{g})).$

###### Proof.

The upper bound of error is derived from Theorem C.2. For the unfairness, by
definition

$\displaystyle\Delta_{odds}(\hat{g})=\frac{1}{2}\left(\left|\varepsilon_{U_{0}^{0}}(\hat{g})-\varepsilon_{U_{1}^{0}}(\hat{g})\right|+\left|\varepsilon_{U_{0}^{1}}(\hat{g})-\varepsilon_{U_{1}^{1}}(\hat{g})\right|\right).$

Based on the upper bound of group error from Theorem C.2, and the lower bound
of it from Theorem C.3, we have

$\displaystyle\left|\varepsilon_{U_{0}^{0}}(\hat{g})-\varepsilon_{U_{1}^{0}}(\hat{g})\right|$
$\displaystyle\leq\max\Big{\\{}\frac{2}{c-1}\gamma+\frac{2}{c-1}L_{U_{1}^{0}}(\hat{g},g_{tc})+\frac{2c}{c-1}R_{U_{0}^{0}}(\hat{g}),$
$\displaystyle\quad\quad\quad\quad\frac{2}{c-1}\gamma+\frac{2}{c-1}L_{U_{0}^{0}}(\hat{g},g_{tc})+\frac{2c}{c-1}R_{U_{1}^{0}}(\hat{g})\Big{\\}}$
(because $c>3$)
$\displaystyle=\frac{2}{c-1}\gamma+\frac{2}{c-1}\max\left\\{L_{U_{1}^{0}}(\hat{g},g_{tc})+cR_{U_{0}^{0}}(\hat{g}),L_{U_{0}^{0}}(\hat{g},g_{tc})+cR_{U_{1}^{0}}(\hat{g})\right\\}$
$\displaystyle\leq\frac{2}{c-1}(\gamma+\mu+c\max_{a}\
R_{U_{a}^{0}}(\hat{g})).$

Therefore,

$\displaystyle\Delta_{odds}(\hat{g})$
$\displaystyle\leq\frac{2}{c-1}\left(\gamma+\mu+\frac{c}{2}\left(\max_{a}\
R_{U_{a}^{0}}(\hat{g})+\max_{a}\ R_{U_{a}^{1}}(\hat{g})\right)\right)$
$\displaystyle\leq\frac{2}{c-1}(\gamma+\mu+c\max_{a,y}\
R_{U_{a}^{y}}(\hat{g})).$

∎

Upper bound of $V_{acc}$. From Theorem C.2 we know that the group accuracy is
upper bounded by
$\varepsilon_{U_{a}^{y}}(\hat{g})\leq\frac{2}{c-1}\varepsilon_{U_{a}^{y}}(g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(\hat{g})$.
The variance of group accuracy is defined as

$\displaystyle V_{acc}(\hat{g})$
$\displaystyle=Var(\\{\mathbb{P}(\hat{Y}=y|A=a,Y=y),\forall a,y\\})$
$\displaystyle=Var(\\{\varepsilon_{U_{a}^{y}}(\hat{g}),\forall a,y\\})$

If we assume the same estimation error for all the groups when we use the
upper bound to estimate the group accuracy with
$\varepsilon_{U_{a}^{y}}(\hat{g})=\frac{2}{c-1}\varepsilon_{U_{a}^{y}}(g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(\hat{g})$,
then

$\displaystyle V_{acc}(\hat{g})$
$\displaystyle=Var\left(\left\\{\frac{2}{c-1}\varepsilon_{U_{a}^{y}}(g_{tc})+\frac{2c}{c-1}R_{U_{a}^{y}}(\hat{g}),\forall
a,y\right\\}\right)$

When the teacher classifier has bounded unfairness with
$|\varepsilon_{U_{a}^{y}}(g_{tc})-\varepsilon_{U_{a^{\prime}}^{y^{\prime}}}(g_{tc})|\leq\gamma,\forall
a,a^{\prime},y,y^{\prime}$, the variance of group accuracy would be mainly
affected by the variance of group consistency loss
$Var(\\{R_{U_{a}^{y}}(\hat{g}),\forall a,y\\})$. Therefore, this upper bound
also suggests us to balance the consistency loss while minimizing it.

Multi-sensitive attribute and multi-class cases. It is obvious that the
Theorem 4.1 still holds for the binary-sensitive attribute and multi-class
case where $\mathcal{Y}=\\{1,2,..,M\\}$. For the multi-sensitive attribute
case, the key problem is how to define the unfairness. If we define the
equalized odds in general cases to be the following one, then it is easy to
see that Theorem 4.1 still holds.

$\displaystyle\Delta_{odds}(\hat{g})=\frac{1}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}\max_{a,a^{\prime}\in\mathcal{A}}\left|\varepsilon_{U_{a}^{y}}(\hat{g})-\varepsilon_{U_{a^{\prime}}^{y}}(\hat{g})\right|$

##  Appendix D Details of Experiments

###  D.1 Synthetic Dataset

The 3dshapes dataset 222https://github.com/deepmind/3d-shapes [31] contains
480000 RGB images (the size is $64\times 64\times 3$) of 3D objects. Every
image is generated by six latent factors (shape, object hue, scale,
orientation, floor hue, wall hue) which are annotated along with images. The
sample spaces of these factors are shown in Table 4.

Factor | Sample space
---|---
shape | 4 values in [0, 1, 2, 3]
object hue | 10 values linearly spaced in [0, 1]
scale | 8 values linearly spaced in [0, 1]
orientation | 15 values linearly spaced in [-30, 30]
floor hue | 10 values linearly spaced in [0, 1]
wall hue | 10 values linearly spaced in [0, 1]
Table 4: Latent factors in 3dshapes dataset.

How to simulate different types of distribution shift? By varying the marginal
distribution of latent factors and then sample images according to the
distribution of latent factors, we can simulate different distribution shifts.
In this paper, we set the image as input $X$, and set class
$Y=\textit{shape}$, sensitive attribute $A=\textit{object hue}$, and a
nuisance factor that might shift to be $D=\textit{scale}$. We consider a
binary case, and restrict the shape to be in $\\{0,1\\}$ (i.e.
$\\{\text{cube},\text{cylinder}\\}$) and object hue to be in $\\{0,1\\}$ (i.e.
$\\{\text{red},\text{yellow}\\}$). In our experiments, we simulate four types
of distribution shift. Their specific settings are shown in Table 5. We show
examples from two domains under different shifts in Figure 7(d).

Figure 7: Randomly sampled examples from two domains under different
distribution shifts.

Source

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

Target

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

(a) Sshift 1

Source

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

Target

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

(b) Sshift 2

Source

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

Target

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

(c) Dshift

Source

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

Target

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

ct-.2in

(d) Hshift | Factor | Source | Target
---|---|---|---
Sshift 1 | $\mathbb{P}(Y,A)$ | $[0.1,0.4,0.4,0.1]$ | same
$\mathbb{P}(D)$ | $[\frac{4}{16},\frac{4}{16},\frac{3}{16},\frac{1}{16},\frac{1}{16},\frac{1}{16},\frac{1}{16},\frac{1}{16}]$ | $[\frac{1}{16},\frac{1}{16},\frac{1}{16},\frac{1}{16},\frac{1}{16},\frac{3}{16},\frac{4}{16},\frac{4}{16}]$
Sshift 2 | $\mathbb{P}(Y,A)$ | $[0.1,0.4,0.4,0.1]$ | $[0.4,0.1,0.1,0.4]$
$\mathbb{P}(D)$ | $[\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8}]$ | same
Dshift | $\mathbb{P}(Y,A)$ | $[0.1,0.4,0.4,0.1]$ | same
$\mathbb{P}(D)$ | $[\frac{1}{2},\frac{1}{2},0,0,0,0,0,0]$ | $[\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8}]$
Hshift | $\mathbb{P}(Y,A)$ | $[0.1,0.4,0.4,0.1]$ | $[0.4,0.1,0.1,0.4]$
$\mathbb{P}(D)$ | $[\frac{1}{2},\frac{1}{2},0,0,0,0,0,0]$ | $[\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8},\frac{1}{8}]$
|  |  |
Table 5: Simulate different distribution shifts. $\mathbb{P}(Y,A)$ is
represented by the proportions of four groups as
$[\mathbb{P}(Y=0,A=0),\mathbb{P}(Y=0,A=1),\mathbb{P}(Y=1,A=0),\mathbb{P}(Y=1,A=1)]$.
$\mathbb{P}(D)$ is represented by the proportions of eight possible values of
scale. Other factors have uniform marginal distributions. Images in two
domains are sampled according to the marginal distributions of six latent
factors.

###  D.2 Real Datasets

UTKFace333https://susanqq.github.io/UTKFace/ [71] is a face dataset with
images annotated with age, gender, and race. The data is collected from MORPH,
CACD and Web. In our experiments, we use the aligned and cropped face images
with ages larger than 10. We do gender classification which is a binary
classification task, and set sensitive attribute to be the race. We consider
binary-sensitive attribute case in our experiment by setting race to be white
or non-white. The statistics of this dataset are shown in Table 6.

$(Y,A)$ | (Male, White) | (Male, Black) | (Female, White) | (Female, Black)
---|---|---|---|---
UTK (S) | train | 3127 | 1508 | 2480 | 1450
test | 1377 | 651 | 1027 | 617
FairFace (T) | train | 7796 | 4650 | 6946 | 5160
test | 984 | 620 | 839 | 635
|  |  |  |  |
Table 6: Statistics of UTKFace and FairFace datasets.

FairFace444https://github.com/joojs/fairface [30] is another large-scale face
dataset with images annotated with age, gender, and race as well. Different
from UTKFace, the data in FairFace is collected from Flickr, Twitter,
Newspapers, and the Web. We also use images with ages larger than 10 in our
experiments. We set the label to be the gender and the sensitive attribute to
be the race. See statistics in Table 6. All face images in UTK and FairFace
are resized to $96\times 96\times 3$ in our experiments.

NewAdult555https://github.com/zykls/folktables [19] is a suite of datasets
derived from US Census surveys. The data spans multiple years and all states
of the United States which is a good fit for studying distribution shifts. In
our experiments, we use 2018 data that span all states and do income
classification with a threshold of 50,000 dollars. We set gender to be the
sensitive attribute. We consider a problem that we train a fair classifier in
California (source domain) and deploy it in other states (target domain). The
statistics are shown in Table 7. The input contains 10 features (see Appendix
B.1 in [19]) which are preprocessed to one-hot embeddings in our experiments.

$(Y,A)$ | (High, Male) | (High, Female) | (Low, Male) | (Low, Female)
---|---|---|---|---
CA (S) | train | 33258 | 22314 | 39224 | 42169
test | 14839 | 9924 | 15990 | 17947
Other states (T) | train | 232162 | 140876 | 296826 | 351970
test | 101934 | 57798 | 127654 | 150544
|  |  |  |  |
Table 7: Statistics of NewAdult dataset.

###  D.3 Experimental Settings

####  D.3.1 Experiments on 3dshapes

Model. We use a two-layer MLP with 512 hidden units as the encoder and one
linear layer as the classifier. The adversaries used in Laftr, CFair, and DANN
are also two-layer MLP with 512 hidden units. ReLU is used as the activation
function.

Transformations. We use random center cropping and padding as the
transformation functions in consistency regularization. Such transformations
can perfectly change the scale of the objects to propagate labels from the
source domain to the target domain.

Setup. We use SGD as the optimizer. We train every model with 200 epochs and
select the best model according to the model’s performance on the validation
set. Base and Laftr only have access to the source data, and the model
selection is based on the source validation set. For other methods that can
access the unlabeled target data, the model selection is based on the labeled
target validation set. Since accuracy and fairness are both important metrics,
we select the best model according to the value of accuracy minus unfairness
which is $Acc-\Delta_{odds}$. The coefficients of the fairness loss and
consistency loss are both set to be 1. We run every method five times and
report the mean and the standard deviation.

####  D.3.2 Experiments on UTK-FairFace

Model. We use VGG16 and ResNet18 as the model in our experiments. The last
linear layer is the classifier, and all the previous layers construct the
encoder. When we use VGG16 as the model, we set every adversary used in Laftr,
CFair, and DANN to be a two-layer MLP with 1024 hidden neurons. When ResNet18
is the model, the adversary has 512 hidden neurons.

Transformations. We use RandAugment [18] as the transformation function which
contains data augmentations that are the best for the CIFAR-10 dataset. To
restrict the transformations to be group-preserving, we exclude augmentations
that may change the color (so to change the race). The transformation function
used in our experiments contains AutoContrast, Brightness, Equalize, Identity,
Posterize, Rotate, Sharpness, ShearX, ShearY, TranslateX, and TranslateY. In
this experiment, we use a weak augmented (with random cropping and flipping)
image as the original input $\bm{x}$ and a strong augmented (with RandAugment)
image as the transformed input $t(\bm{x})$.

Setup. We use SGD as the optimizer. We train every model with 200 epochs and
use the way introduced in the 3dshapes experiment to select the best model.
The coefficients of the fairness loss and consistency loss are both set to be
1. We run every method five times and report the mean and the standard
deviation.

####  D.3.3 Experiments on NewAdult

Model. We use a 3-layer MLP with hidden sizes of (256, 512, 256) as the
encoder and a 2-layer MLP with a hidden size of 128 as the classifier. Every
adversary is a two-layer MLP with 128 hidden neurons.

Transformations. Studies on data augmentations for tabular data are very
limited. In this paper, we use random corruptions on the input features as the
transformation function. There are ten features in the input, and every time
we only corrupt half of them. Additionally, for important factors that are
highly correlated with the label, including the OCCP (occupation), COW (class
of worker), we do not do any corruption. For factor SEX (gender), we do not do
any corruption to preserve the group. For continuous factors including AGEP
(age), SCHL (educational attainment), and WKHP (work hours), we do
perturbations within a range. For other factors, we do uniformly sampling from
their value spaces as corruptions. We do such transformations based on our
assumption that they do not change the label. For example, two individuals
that have five years of age gap but have the same other features should have
similar income, and two individuals that only differ in the place of birth
should have similar income. We admit that such transformations may not be the
best ones. We need better domain knowledge on income prediction to design more
powerful transformations. We leave the improvement of transformations for
tabular data to future work.

Setup. We use SGD as the optimizer. We train every model with 200 epochs and
use the metric introduced in the 3dshapes experiment to select the best model.
The coefficients of the fairness loss and consistency loss are both set to be
1. We run every method five times and report the mean and the standard
deviation.

###  D.4 Baselines

Laftr is an adversarial learning method for algorithmic fairness. The
adversary aims at accurately predicting the sensitive attribute based on the
representation, while the encoder aims at making it hard. By adversarial
learning, the representation will not contain information on sensitive
attributes, so the prediction based on it will be fair. We denote the data in
each group to be $\mathcal{D}_{a}^{y}=\\{\bm{x}\in\mathcal{D}|A=a,Y=y\\}$. The
fairness loss is designed to be

$\displaystyle
L_{fair}=\sum_{(a,y)\in\\{0,1\\}^{2}}\frac{1}{|\mathcal{D}_{a}^{y}|}\sum_{\bm{x}\in\mathcal{D}_{a}^{y}}|h(f(\bm{x}))-a|.$

where $f$ is the encoder, $h$ is the adversary. [42] prove that this loss is
an upper bound of the equalized odds. The adversary minimizes this loss, while
the encoder maximizes this loss with a gradient reversal layer.

CFair is similar to Laftr but uses two adversaries $h^{\prime}$, and
$h^{\prime\prime}$ for two classes with a balanced error rate (BER) defined as
follows. We denote the data from one class to be
$\mathcal{D}^{y}=\\{\bm{x}\in\mathcal{D}|Y=y\\}$.

$\displaystyle
L_{fair}=\text{BER}_{\mathcal{D}^{0}}(h^{\prime}(f(\bm{x}))\parallel
A)+\text{BER}_{\mathcal{D}^{1}}(h^{\prime\prime}(f(\bm{x}))\parallel A)$

where $\text{BER}_{\mathcal{D}^{0}}(h^{\prime}(f(\bm{x}))\parallel
A)=\frac{1}{2}\mathbb{P}_{\mathcal{D}^{0}}(h^{\prime}(f(\bm{x}))\neq
A|A=0)+\frac{1}{2}\mathbb{P}_{\mathcal{D}^{0}}(h^{\prime}(f(\bm{x}))\neq
A|A=1)$. In practice, the balanced error rate is estimated by the following
cost-sensitive cross-entropy loss.

$\displaystyle\mathbb{P}_{\mathcal{D}^{0}}(h^{\prime}(f(\bm{x}))\neq
A|A=0)\leq\frac{\text{CE}_{\mathcal{D}_{0}^{0}}(h^{\prime}(f(\bm{x}))\parallel
A)}{\mathbb{P}_{\mathcal{D}^{0}}(A=0)}$

Laftr+FixMatch uses the same framework as our method but with a standard
consistency regularization that does not care about group performance. The
consistency loss is defined as

$\displaystyle
L_{consis}(g)=\frac{1}{|\mathcal{D}|}\sum_{\bm{x}\in\mathcal{D}}\mathds{1}(\max(g_{tc}(\bm{x}))\geq\tau)H(\mathop{\mathrm{argmax}}(g_{tc}(\bm{x})),g(t(\bm{x})))$

where $\mathcal{D}$ denotes the entire dataset.

###  D.5 Time and Space Complexity

Compared with Base, Laftr, and CFair which only uses labeled source data, our
method needs more training time and memory since we use unlabeled target data
as well. Compared with other baselines that also use target data, such as
Laftr+DANN, the time complexity of our method is comparable to theirs.
Nevertheless, our method needs much fewer parameters than Laftr+DANN since it
requires an adversary to do domain classification while we do not need it. Our
method has the same model parameters as that in Laftr but with an additional
consistency loss.

##  Appendix E More Experimental Results

###  E.1 Additional Results on UTKFace-FairFace with a Different Data Setting

To evaluate our method in extreme circumstances, we conduct the UTKFace-
FairFace experiment with less labeled source data and more unlabeled target
data (see Table LABEL:tab:dataset_face_app). We also consider the race "white"
and "non-white". Are shown in Table 9, we get consistent results that our
method outperforms all baselines and can effectively transfer accuracy as well
as fairness.

Table 8: Statistics of UTK and FairFace datasets used in Table 9.

$(Y,A)$ | (Male, White) | (Male, Non-white) | (Female, White) | (Female, Non-white)
---|---|---|---|---
UTK (S) | train | 1373 | 750 | 1650 | 1227
test | 565 | 285 | 614 | 370
FairFace (T) | train | 11429 | 16574 | 8024 | 16838
test | 1712 | 2453 | 1176 | 2518

Table 9: Transfer fairness and accuracy from UTKFace to FairFace with less
source data.

| Source |  | Target
---|---|---|---
| Acc | Unfairness |  | Acc | Unfairness
Method |  | $V_{acc}$ | $\Delta_{odds}$ |  |  | $V_{acc}$ | $\Delta_{odds}$
Base |  89.93±0.43 |  2.79±0.74 |  4.65±0.44 |  |  73.48±0.56 |  7.49±3.50 |  6.09±1.07
Laftr |  90.61±0.33 |  1.28±0.43 |  3.62±1.17 |  |  73.29±0.70 |  5.42±1.33 |  7.78±1.77
CFair |  90.68±0.35 |  1.20±0.59 |  3.61±0.93 |  |  73.82±0.81 |  5.71±1.54 |  7.37±1.40
Laftr+DANN |  90.53±0.98 |  1.59±0.97 |  4.62±1.24 |  |  74.44±1.38 |  6.94±1.53 |  10.26±1.85
CFair+DANN |  90.23±0.88 |  1.82±0.97 |  4.96±1.15 |  |  74.53±1.46 |  9.27±2.16 |  9.96±1.49
Laftr+FixMatch |  95.01±0.10 |  1.37±0.44 |  4.65±1.00 |  |  83.77±0.45 |  11.58±1.16 |  6.56±1.74
CFair+FixMatch |  95.37±0.24 |  1.13±0.21 |  3.58±0.90 |  |  83.62±0.51 |  11.96±1.05 |  5.29±1.76
Ours (w/ Laftr) |  94.77±0.33 |  1.35±0.70 |  3.28±0.79 |  |  84.65±1.13 |  2.92±0.72 |  6.99±0.41
Ours (w/ CFair) |  94.92±0.43 |  1.09±0.30 |  3.00±1.09 |  |  84.71±1.10 |  3.57±0.60 |  7.34±0.91

###  E.2 Additional Results on UTKFace-FairFace with Different Transformations

Table 10: Results by using different transformations in our method. Average
results of three trials.

| Source |  | Target
---|---|---|---
| Acc | Unfairness |  | Acc | Unfairness
Transformation |  | $V_{acc}$ | $\Delta_{odds}$ |  |  | $V_{acc}$ | $\Delta_{odds}$
None | 93.24 | 1.19 | 2.44 |  | 74.35 | 6.92 | 9.79
All | 96.08 | 0.96 | 2.59 |  | 85.52 | 2.82 | 5.70
AutoContrast | 94.82 | 1.12 | 2.66 |  | 79.69 | 5.55 | 7.48
Brightness | 95.61 | 0.95 | 1.48 |  | 82.16 | 4.89 | 6.39
Color | 95.53 | 1.07 | 1.28 |  | 81.32 | 6.66 | 8.22
Contrast | 94.93 | 1.31 | 2.29 |  | 79.35 | 6.37 | 8.39
Equalize | 95.15 | 1.47 | 2.33 |  | 79.17 | 5.88 | 6.91
Identity | 96.21 | 1.03 | 1.31 |  | 81.58 | 3.44 | 7.29
Posterize | 94.92 | 1.77 | 3.06 |  | 79.63 | 5.26 | 6.01
Rotate | 96.13 | 0.72 | 1.83 |  | 84.33 | 3.80 | 6.34
Sharpness | 95.73 | 1.03 | 2.64 |  | 81.26 | 5.33 | 7.09
ShearX | 95.45 | 1.70 | 0.99 |  | 82.47 | 3.30 | 3.72
ShearY | 96.25 | 0.54 | 1.75 |  | 84.26 | 3.96 | 6.07
Solarize | 95.89 | 0.98 | 2.67 |  | 80.38 | 7.37 | 8.79
TranslateX | 96.11 | 0.89 | 1.79 |  | 83.49 | 2.31 | 6.13
TranslateY | 95.53 | 0.97 | 2.83 |  | 83.04 | 7.16 | 6.17

To investigate the effect of different transformations in our method, we
evaluate 14 transformations in RandAugment and report the results in Table 9.
All the transformations can improve the accuracy in both domains. The effect
on fairness varies. We find that Solarize, Color, and TranslateX increase the
unfairness in the source domain the most, and Contrast, Color and Solarize
have the highest unfairness in the target domain. Note that, it does not mean
that these augmentations always lead to unfairness but that they are not
suitable for our method. Recall that our theory and algorithm are built upon
the intra-group expansion assumption. Transformations like Contrast, Color,
and Solarize may change the sensitive attribute "race" and break this
assumption. Thus, in our experiments (Table 1) we use all the transformations
excluding Contrast, Color, and Solarize.

###  E.3 A Byproduct: Alleviate the Disparate Impact of Semi-supervised
Learning

Figure 8: With fair consistency regularization, our method alleviates the
disparate impact of FixMatch.

[76] find that semi-supervised learning methods may have a disparate impact.
The classes that have high accuracy on labeled data are likely to benefit more
from semi-supervised learning on unlabeled data. We test this argument on
CIFAR-10 with FixMatch as the semi-supervised learning method. We use ResNet18
as the model. We randomly sample 500 images to be labeled data and treat
others as unlabeled data. We use the benefit ratio proposed in [76] as the
metric for the benefit of semi-supervised learning, defined as

$\displaystyle
BR(\mathcal{D})=\frac{a_{semi}(\mathcal{D})-a_{baseline}(\mathcal{D})}{a_{ideal}(\mathcal{D})-a_{baseline}(\mathcal{D})}.$
(8)

where $\mathcal{D}$ denotes the data from one class. $a_{semi}(\mathcal{D})$
is the model’s test accuracy after semi-supervised learning,
$a_{baseline}(\mathcal{D})$ is the test accuracy of the base model that is
trained on labeled data, and $a_{ideal}(\mathcal{D})$ is the test accuracy of
the ideal model where all data are labeled. We evaluate the benefit ratio of
FixMatch on ten classes. As the blue line in Figure 8 shows, the rich gets
richer, and the poor gets poorer after semi-supervised learning. Our method
(without using Laftr) can directly apply to this task. By using the proposed
fair consistency regularization (red line in Figure 8), we can significantly
improve the benefit ratio of the poor classes. Therefore, fair consistency
regularization is a byproduct of this paper which is able to alleviate the
disparate impact of semi-supervised learning.

##  Appendix F Impact and Limitations

The fairness of machine learning is a critical problem in today’s real-world
applications. When distribution shifts happen, the collapse of fair systems
will cause unexpected discrimination, resulting in severe negative social
impacts. The fairness that is robust to distribution shifts is essential but
is less explored. In this paper, the theoretical analysis of how fairness
changes under different distribution shifts sheds light on the deep reasons
for the collapse of fairness. The theory-guided self-training algorithm
proposed in this paper explores a promising way to tackle distribution shifts.
We hope our work will inspire more algorithms for this important and practical
task.

The major limitation of our method is that it strongly relies on pre-defined
transformations as all the other self-training methods. The transformations
are designed to be group-preserving based on our prior knowledge. Our
experiments show that self-training with less powerful transformations has
limited ability in propagating labels from source to target (i.e. transfer
accuracy). Valid transformation functions on image data are thoroughly studied
in existing work, while transformations on non-image data such as tabular data
are much less explored. Our method with more powerful transformations on
tabular data is expected to have significant improvement. Future work is
encouraged to relax this limitation, such as by using a generative model as
the transformation function.