# FairerCLIP: Debiasing CLIP’s Zero-Shot Predictions using Functions in RKHSs

Sepehr Dehdashtian* Lan Wang Vishnu Naresh Boddeti
Michigan State University
{sepehr, wanglan3<EMAIL_ADDRESS>Equal Contribution.

###### Abstract

Large pre-trained vision-language models such as CLIP provide compact and
general-purpose representations of text and images that are demonstrably
effective across multiple downstream zero-shot prediction tasks. However,
owing to the nature of their training process, these models have the potential
to 1) propagate or amplify societal biases in the training data and 2) learn
to rely on spurious features. This paper proposes FairerCLIP, a general
approach for making zero-shot predictions of CLIP more fair and robust to
spurious correlations. We formulate the problem of jointly debiasing CLIP’s
image and text representations in reproducing kernel Hilbert spaces (RKHSs),
which affords multiple benefits: 1) _Flexibility:_ Unlike existing approaches,
which are specialized to either learn with or without ground-truth labels,
FairerCLIP is adaptable to learning in both scenarios. 2) _Ease of
Optimization:_ FairerCLIP lends itself to an iterative optimization involving
closed-form solvers, which leads to $4\times$-$10\times$ faster training than
the existing methods. 3) _Sample Efficiency:_ Under sample-limited conditions,
FairerCLIP significantly outperforms baselines when they fail entirely. And,
4) _Performance:_ Empirically, FairerCLIP achieves appreciable accuracy gains
on benchmark fairness and spurious correlation datasets over their respective
baselines.

##  1 Introduction

Vision-Language Models such as CLIP (Radford et al., 2021) are trained on
large-scale datasets of image-text pairs to learn representations that have
high similarity for related image-text pairs. While these models have gained
significant attention in recent years due to their remarkable zero-shot
classification capabilities, they are not flawless. There is growing evidence
that such models suffer from biases w.r.t. demographic (e.g., sex or skin
tone) attributes (Agarwal et al., 2021; Wang et al., 2021; Birhane et al.,
2023a; b) and even non-demographic(e.g., image background or illumination)
attributes (Du et al., 2022; Zhang & Ré, 2022).

[width=0.9]figs/causal-graphs

Figure 1: Dependence graphs for debiasing.

The above-mentioned biases can be viewed based on dependencies between the
data attributes. We show these dependencies in Fig. 1: $X$ is the data (e.g.,
face images) that depends on some attributes, including $Y$, the target
attribute we wish to predict, and $S$, the attribute that leads to bias. The
goal of bias mitigation is to ensure that the prediction $\hat{Y}$ is
independent of $S$. We group the biases into those arising from two scenarios:
(1) $Y$ and $S$ are dependent (Fig. 1 a): for example, _high cheekbones_ as
$Y$ and _sex_ as $S$ since males typically have higher cheekbones than
females. We refer to this type of correlation as intrinsic dependence. (2) $Y$
and $S$ are independent (Fig. 1 b): for example _hair color_ as $Y$ and _sex_
as $S$ since the hair color of a person does not depend on their sex. In this
case, we refer to any observed correlation as a spurious correlation.

Several efforts (Zhang & Ré, 2022; Gao et al., 2021; Kumar et al., 2022;
Kirichenko et al., 2022; Chuang et al., 2023; Wortsman et al., 2022; An et
al., 2023; Adila et al., 2023), have been made to debias zero-shot predictions
from CLIP models. However, they are limited in either one or more respects:
(1) Type of Bias: Existing CLIP debiasing methods only consider spurious
correlations in the data (Fig. 1 b) and do not seek to address bias induced by
pairs of attributes with intrinsic dependencies (Fig. 1 a), (2) Labels for
training: All existing approaches are tailored to train/fine-tune either with
(supervised) or without (unsupervised) ground-truth labels and, as such,
cannot be employed in both scenarios, (3) Efficiency: Some approaches adopt
iterative methods to debias the features. However, they are computationally
expensive to _train_ , i.e., slow to converge, leading to high training
latency and many parameters in the debiasing modules, increasing model sizes
even further.

[width=]figs/overview

Figure 2: Overview of the train and inference phases of FairerCLIP. (a) Shows
the label prediction step. When labels are not available for training,
FairerCLIP uses cosine similarity between the $X_{T}$ and $X_{I}$, and
$X_{TS}$ and $X_{I}$ to predict the target attributes and sensitive
attributes, respectively. (b) Shows the inputs and outputs for FairerCLIP in
its training stage. FairerCLIP uses representation of images and the
corresponding text prompts that are constructed by target attribute ($Y$)
along with the predicted labels to find the image and text encoders, i.e.,
$\bm{f}^{*}_{I}(.;\bm{\Theta}_{I})$ and $\bm{f}^{*}_{T}(.;\bm{\Theta}_{T})$.
(c) Shows the inference phase of FairerCLIP in which we use the trained image
and text encoders to generate debiased representations from the ones generated
by CLIP.

We propose FairerCLIP to address the aforementioned limitations of existing
debiasing approaches. FairerCLIP affords sufficient flexibility to mitigate
bias arising from both spurious correlations and intrinsic dependencies and,
in both scenarios, learn with or without ground-truth labels. FairerCLIP
utilizes a non-parametric measure of statistical dependence that accounts for
all linear and non-linear relations between the debiased representation and
the sensitive attribute of interest. Our formulation lends itself to
alternating optimization, with each update having a closed-form solution and,
in comparison to baselines, enjoying fast training convergence and requiring
fewer parameters to train. An overview of FairerCLIP in its train and
inference phases along with how we integrate this transformation over the
underlying CLIP model is shown in Fig. 2.

Summary of Contributions: (1) We demonstrate that a single general method can
debias the image and text features from frozen CLIP backbones under different
scenarios more effectively than those specialized for each scenario. The
scenarios include accounting for both spurious correlations and intrinsic
dependencies (Sec. 4.2), learning with and without ground-truth labels (Sec.
4.2), and learning from small and medium-sized datasets (Sec. A.5). (2) We
demonstrate that kernel methods are particularly effective compared to shallow
MLPs when operating on features and optimizing possibly competing objectives,
as is the case for debiasing CLIP representations. They enjoy closed-form
solutions that allow for significantly faster training, can scale to medium-
sized datasets, and are more effective under limited training data (Sec. 4.3,
Sec. A.4, and Tab. 3 (left)).

##  2 The Debiasing CLIP Representations Problem

Notation: Scalars are denoted by regular lower case letters, e.g. $r$, $\tau$.
Deterministic vectors are denoted by boldface lowercase letters, e.g.,
$\bm{x}$, $\bm{s}$. We denote both scalar-valued and multi-dimensional Random
Variables (RVs) by regular uppercase letters, e.g. $X$, $S$. Deterministic
matrices are denoted by boldface uppercase letters, e.g. $\bm{H}$,
$\bm{\Theta}$, and the entry at $i^{th}$ row, $j^{th}$ column of matrix
$\bm{M}$ is denoted by $\left(\bm{M}\right)_{ij}$ or $m_{ij}$. $\bm{I}_{n}$ or
simply $\bm{I}$ denotes an $n\times n$ identity matrix, $\bm{1}_{n}$ or
$\bm{1}$ and $\bm{0}_{n}$ or $\bm{0}$ are $n\times 1$ vector of ones and
zeros, respectively. We denote the trace of any square matrix $\bm{K}$ by
$\text{Tr}[\bm{K}]$. Finite or infinite sets are denoted by calligraphy
letters, e.g., $\mathcal{H}$, $\mathcal{A}$.

Problem Setup: We assume that the joint RV $(X_{I},X_{T},Y,S)$ contains the
pre-trained image features $X_{I}\in\mathbb{R}^{d}$, pre-trained text features
of target attribute $X_{T}\in\mathbb{R}^{d}$, target attribute
$Y\in\mathbb{R}^{d_{Y}}$, and sensitive attribute $S\in\mathbb{R}^{d_{S}}$.
Their joint distribution will be $\bm{p}_{X_{I},X_{T},Y,S}$. Furthermore, $Y$
and $S$ can also belong to any finite set, such as a categorical set.

Our aim is to debias $X_{I}$ and $X_{T}$ by generating representations,
$Z_{I}=\bm{f}_{I}(X_{I})$ and $Z_{T}=\bm{f}_{T}(X_{T})$, with no or reduced
statistical dependence on $S$. To measure this dependency, we need to employ a
metric capable of capturing both linear and non-linear statistical
dependencies.

Choice of Dependence Measure: We will adopt a simplified definition of the
Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005)
introduced by Sadeghi et al. (2022), defined as,

$\text{Dep}(Z,S):=\sum_{j=1}^{r}\sum_{\beta\in\mathcal{U}_{S}}\mathbb{C}\text{ov}^{2}\left(Z_{j},\beta(S)\right),$
(1)

where $\mathcal{U}_{S}$ is a countable orthonormal basis set for the separable
universal RKHS $\mathcal{H}_{S}$, and $Z_{j}=f_{j}(X)$ for
$f_{j}\in\mathcal{H}_{X}\forall j=1,...,r$. $\text{Dep}(Z,S)$ can be estimated
(see Lemma 1 in Sadeghi et al. (2022)) empirically as,

$\text{Dep}(Z,S):=\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{S}\right\|^{2}_{F},$
(2)

where $n$ is the number of data samples, $\bm{K}_{X}\in\mathbb{R}^{n\times n}$
is the Gram matrix corresponding to $\mathcal{H}_{X}$, $\bm{\Theta}$ is the
encoder parameter in
$\bm{f}(X)=\bm{\Theta}[k_{X_{1}},k_{X_{2}},\cdots,k_{X_{n}}]^{T}$,
$\bm{H}=\bm{I}_{n}-\frac{1}{n}\bm{1}_{n}\bm{1}_{n}^{T}$ is the centering
matrix, and $\bm{L}_{S}$ is a full column-rank matrix corresponding to the
Cholesky factorization of $K_{S}$, i.e.,
$\bm{L}_{S}\bm{L}_{S}^{T}=\bm{K}_{S}$. While HSIC and related measures like
Maximum Mean Discrepancy (MMD) Gretton et al. (2012) have been employed by
prior fairness approaches (Bahng et al., 2020; Quadrianto et al., 2019; Jung
et al., 2021), the HSIC variation we use in Eq. 2 has several attractive
properties (Sadeghi et al., 2022). This includes a convergence rate of
$\mathcal{O}(n^{-1/2})$111In scenarios where only a single or few samples are
available, to an extent, heavy data augmentation can compensate for the lack
of sufficient samples to accurately estimate Dep. However, this is beyond the
scope of this paper, and all our experiments are performed without data
augmentation., a practical ability to capture all non-linear modes of
dependencies when projecting from a high-dimensional representation to a low-
dimensional representation, and, as we demonstrate next, analytical
tractability.

In addition to the above-mentioned dependence metric, we also need another
metric that can mimic the cosine similarity-based classification employed by
CLIP. This metric will be used to make the representations of images and their
corresponding text prompts align with each other to improve the accuracy of
the predictions. As a result, we modify the definition of Dep metric in Eq. 1
and use a linear kernel as $\beta$ in Lemma 1.

######  Lemma 1.

Let $\bm{K}_{X_{I}},\bm{K}_{X_{T}}\in\mathbb{R}^{n\times n}$ be the Gram
matrices corresponding to $\mathcal{H}_{X_{I}}$ and $\mathcal{H}_{X_{T}}$,
respectively, i.e.,
$\left(\bm{K}_{X_{I}}\right)_{ij}=k_{X_{I}}(\bm{x}_{I_{i}},\bm{x}_{I_{j}})$
and
$\left(\bm{K}_{X_{T}}\right)_{ij}=k_{X_{T}}(\bm{x}_{T_{i}},\bm{x}_{T_{j}})$,
where covariance is empirically estimated as

$\displaystyle\mathbb{C}\text{ov}\left(f_{j}(X_{I}),g_{m}(X_{T})\right)\approx\frac{1}{n}\sum_{i=1}^{n}f_{j}(\bm{x}_{I_{i}})g_{m}(\bm{x}_{T_{i}})-\frac{1}{n^{2}}\sum_{p=1}^{n}f_{j}(\bm{x}_{I_{p}})\sum_{k=1}^{n}g_{m}(\bm{x}_{T_{k}}).$

It follows that, the corresponding empirical estimator for
$\text{Dep}\left(Z_{I},Z_{T}\right)$ is

$\displaystyle\text{Dep}\left(Z_{I},Z_{T}\right)$ $\displaystyle=$
$\displaystyle\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\Theta}^{T}_{T}\right\|^{2}_{F},$
(3)

where $\bm{\Theta}_{I}$ and $\bm{\Theta}_{T}$ are the parameters of the image
and text encoders, respectively, and $\bm{K}_{X_{I}}$ and $\bm{K}_{X_{T}}$ are
the kernel matrices for the image and text features, respectively.

###### Proof.

The main idea for proving equality equation 3 is to employ the representer
theorem to express $f_{j}$ and $g_{m}$. The complete proof is available in the
supplementary material. ∎

Objective Function: After choosing the appropriate dependence measure, we now
define our objective function. Our goal is to mitigate bias in CLIP’s zero-
shot predictions by debiasing the underlying representations. This can be
achieved by (1) reducing the information related to the sensitive attribute
while (2) preserving information about the target attribute as much as
possible in the pair of image-text representations and (3) keeping the image
and corresponding text representations aligned with each other.

We formulate the above-mentioned learning objective through the following
optimization problem.

######  Definition 1.

$\displaystyle\sup_{\bm{f}_{I}\in\mathcal{A}^{I}_{r},\bm{f}_{T}\in\mathcal{A}^{T}_{r}}\big{\\{}J\left(\bm{f}_{I},\bm{f}_{T},\tau_{I},\tau_{T},\tau_{z}\right)$
$\displaystyle:=\text{Dep}\left(Z_{I},Y\right)-\tau_{I}\text{Dep}\left(Z_{I},S\right)$
(4)
$\displaystyle+\text{Dep}\left(Z_{T},Y\right)-\tau_{T}\text{Dep}\left(Z_{T},S\right)$
$\displaystyle+\tau_{z}\text{Dep}\left(Z_{I},Z_{T}\right)\big{\\}}$

where $\mathrm{Dep}(\cdot,\cdot)\geq 0$ is the statistical dependence measure
defined in Eq. 2. $\mathrm{Dep}(Q,U)=0$ implies $Q$ is independent of $U$
(i.e., $Q\perp\\!\\!\\!\perp U$), and $\mathrm{Dep}(Q,U)>0$ implies $Q$ is
dependent on $U$ (i.e., $Q\not\perp\\!\\!\\!\perp U$), with larger values
indicating greater degrees of dependence. $\tau_{I}$ and $\tau_{T}$ control
the contribution of the corresponding debiasing terms and $\tau_{z}$ controls
the alignment of the debiased image and text features
$Z_{I}=\bm{f}_{I}(X_{I})$ and $Z_{T}=\bm{f}_{T}(X_{Y})$, respectively.

In the above definition, the terms $\text{Dep}(Z_{I},Y)$ and
$\text{Dep}(Z_{T},Y)$ contribute to maximizing the statistical dependence
between the representations and the target label $Y$, the terms
$-\tau_{I}\text{Dep}(Z_{I},S)$ and $-\tau_{T}\text{Dep}(Z_{T},S)$ seek to make
the representations independent of $S$, and the term
$\tau_{z}\text{Dep}(Z_{I},Z_{T})$ ensures that the text and image features are
still aligned with each other after debiasing.

Choice of Encoder: We construct the mappings through a set of $r$ functions
from $\mathbb{R}^{d_{X}}\rightarrow\mathbb{R}$ in a reproducing kernel Hilbert
space (RKHS) $\left(\mathcal{H}_{X},\,k_{X}(\cdot,\cdot)\right)$, such as the
RBF Gaussian kernel. Hence, the representation $Z$ can be expressed as,

$\displaystyle
Z=\bm{f}(X):=\left[Z_{1},\cdots,Z_{r}\right]^{T}\in\mathbb{R}^{r},\quad
Z_{j}=f_{j}(X),f_{j}\in\mathcal{H}_{X}\ \forall{j=1,\dots,r},$ (5)

where $r$ is the dimensionality of the transformed representation.

Our choice of RKHS is motivated by several reasons. As we observe in Eq. 4,
debiasing is inherently an optimization problem with multiple competing
objectives. In such cases, optimization is the primary bottleneck rather than
model expressivity. This was also observed in Sadeghi et al. (2022). The
closed-form solution afforded by our approach mitigates the optimization
challenges (Sec. 4.3 and Sec. A.6). RKHS has nice universal approximation
properties and has performance comparable to shallow MLPs while being more
computationally efficient for training (Sec. 4.3) and is performant under
limited data scenarios (Sec. 4.2).

##  3 A Solution to The Debiasing CLIP Representations Problem

Given the choice of dependence measure in Eq. 2, the optimization problem in
Eq. 4 can be expressed as,

$\displaystyle\max_{\bm{f}_{I}\in A_{r},\bm{f}_{T}\in
A_{r}}\big{\\{}J\left(\bm{f}_{I},\bm{f}_{T},\tau_{I},\tau_{T},\tau_{z},\bm{X}_{I},\bm{X}_{T},\bm{Y},\bm{S}\right)$
$\displaystyle:=\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{L}_{Y}\right\|^{2}_{F}-\tau_{I}\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{L}_{S}\right\|^{2}_{F}$
(6)
$\displaystyle\frac{1}{n^{2}}\left\|\bm{\Theta}_{T}\bm{K}_{X_{T}}\bm{H}\bm{L}_{Y}\right\|^{2}_{F}-\tau_{T}\frac{1}{n^{2}}\left\|\bm{\Theta}_{T}\bm{K}_{X_{T}}\bm{H}\bm{L}_{S}\right\|^{2}_{F}$
$\displaystyle+\tau_{z}\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\Theta}^{T}_{T}\right\|^{2}_{F}\big{\\}}$

[width=]figs/FairVLM-v5

Figure 3: FairerCLIP acts on representations extracted from a frozen CLIP
model. It has two mapping functions, $\bm{f}_{I}$ and $\bm{f}_{T}$, for the
image and text representations. These functions are learned through an
alternating optimization algorithm with two closed-form solvers. When ground-
truth labels are unavailable for training, FairerCLIP learns from pseudo-
labels $\hat{y}$, which are initialized from CLIP’s zero-shot predictions and
refined iteratively. The bold words in the input text prompts are the
information of the target task label included in the text prompts.

Our solution to the constrained optimization problem in Eq. 6 is based on the
observation that it has a closed-form solution when either $\bm{f}_{I}$ or
$\bm{f}_{T}$ are fixed. Let $\bm{Z}_{O}$ be the feature corresponding to the
fixed parameter and $\bm{f}$ the optimization parameter of the other feature
of interest. Then Eq. 6 reduces to two optimization problems of the following
general form,

$\max_{\bm{f}\in
A_{r}}\\{J\left(\bm{f},\tau,\tau_{z},\bm{X},\bm{Y},\bm{S},\bm{Z}_{O}\right):=\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{Y}\right\|^{2}_{F}-\tau\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{S}\right\|^{2}_{F}+\tau_{z}\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{Z}_{O}\right\|^{2}_{F}\\}$
(7)

This is easy to see since fixing either of the parameters in Eq. 6 renders the
terms involving them to a constant w.r.t. the variable of interest, and hence
can be ignored during optimization.

######  Theorem 2.

Let the Cholesky factorization of $\bm{K}_{X}$ be
$\bm{K}_{X}=\bm{L}_{X}\bm{L}_{X}^{T}$, where $\bm{L}_{X}\in\mathbb{R}^{n\times
d}$ ($d\leq n$) is a full column-rank matrix. Let $r\leq d$, then a solution
to Eq. 7 is

$\displaystyle\bm{f}^{\text{opt}}(X)=\bm{\Theta}^{\text{opt}}\left[k_{X}(\bm{x}_{1},X),\cdots,k_{X}(\bm{x}_{n},X)\right]^{T},$

where $\bm{\Theta}^{\text{opt}}=\bm{U}^{T}\bm{L}_{X}^{\dagger}$ and the
columns of $\bm{U}$ are eigenvectors corresponding to the $r$ largest
eigenvalues of the following generalized eigenvalue problem.

$\displaystyle\bm{L}_{X}^{T}\left(\bm{H}\bm{K}_{Y}\bm{H}-\tau\bm{H}\bm{K}_{S}\bm{H}+\tau_{z}\bm{H}\bm{Z}_{O}\bm{Z}_{O}^{T}\bm{H}\right)\bm{L}_{X}\bm{u}=\lambda\left(\frac{1}{n}\,\bm{L}_{X}^{T}\bm{H}\bm{L}_{X}+\gamma\bm{I}\right)\bm{u}.$
(8)

Furthermore, the objective value of (7) is equal to
$\sum_{j=1}^{r}\lambda_{j}$, where $\\{\lambda_{1},\cdots,\lambda_{r}\\}$ are
$r$ largest eigenvalues of Eq. 8.

###### Proof.

The objective in Eq. 7 can be expressed as a trace optimization problem, which
reduces to a generalized eigenvalue problem (Kokiopoulou et al., 2011). See
the supplementary material for detailed proof. ∎

Building upon the above closed-form solution, we adopt alternating
optimization to solve Eq. 6, by fixing $\bm{f}_{I}$ and solving for
$\bm{f}_{T}$ and vice-versa (Fig. 3). The formulation in Eq. 4 requires labels
of the downstream target task $Y$ and the sensitive labels $S$ to learn
FairerCLIP’s parameters. While such labels are readily available for
supervised learning and partially available for semi-supervised learning (Jung
et al., 2022; Chen et al., 2023), this is not the case for unsupervised
learning. Therefore, in this case, we initialize the labels $Y$ and $S$ by the
original zero-shot predictions $\hat{Y}$ and $\hat{S}$ from CLIP (Fig. 2 a).
Then we refine $\hat{Y}$ by predicting it after every iterative update of
$\bm{f}_{I}$ and $\bm{f}_{T}$. However, note that we do not update
$\hat{\bm{S}}$ in the same way since our initial prediction of $\hat{S}$ has
the most information about the label $S$, but as we debias the representations
in the subsequent iterations, we remove the information of $S$. Therefore,
updated values of $\hat{S}$ will lead to inaccurate estimates of
$\text{Dep}(Z,S)$ and affect the overall optimization. This procedure is
detailed in Algorithm 1 of Sec. A.1.

###  3.1 A Geometric Illustration of FairerCLIP

[width=]figs/intuitive

Figure 4: A geometric illustration of FairerCLIP training steps. The encoder
utilizes the implicit mapping functions $\phi_{I}(X)$ and $\phi_{T}(X)$ of the
RBF kernel to map image and text features into an infinite-dimensional space,
facilitating linear separability of samples with different target attributes.
The optimization process seeks a direction that aligns with labels $Y$,
statistically independent of $S$, and aligned with the other representation.

A geometric illustration of the steps that FairerCLIP takes to debias the
representations is shown in Fig. 4. In theory, the RBF kernels used in our
encoder ($\phi_{I}(X)$ and $\phi_{T}(X)$) map the image and text features into
an infinite-dimensional space, where the samples corresponding to different
target attributes are linearly separable. In the infinite-dimensional space,
the encoder that optimizes Eq. 4 for $\bm{\Theta}_{I}$ and $\bm{\Theta}_{T}$
by alternating between closed-form solvers and seeks a direction for mapping
the image and text features that have low angular distance w.r.t. the
direction of (1) $Y$ labels (small $\alpha_{IY}$ and $\alpha_{TY}$), (2)
$S_{\perp}$ (small $\alpha_{IS_{\perp}}$ and $\alpha_{TS_{\perp}}$), and (3)
the other representation (small $\alpha_{IT}$).

##  4 Experimental Evaluation

We evaluate FairerCLIP on datasets with spurious correlation and intrinsic
dependence and compare it to several existing baselines. In summary, the
experimental results indicate that the baseline methods are effective in
mitigating spurious correlations, but they are not as effective at mitigating
the bias caused by the intrinsic dependencies. In contrast, FairerCLIP
effectively and efficiently mitigates both spurious correlations and intrinsic
dependencies in CLIP’s zero-shot predictions. In all our experiments, to
overcome the $\mathcal{O}(n^{3})$ computational and $\mathcal{O}(n^{2})$
memory complexity of the kernel matrices $\bm{K}$, we use random Fourier
features (RFF) (Rahimi & Recht, 2007). All the implementation details are
provided in Sec. A.3

###  4.1 Datasets

We evaluate FairerCLIP on an assortment of classification tasks across many
datasets. This includes Waterbirds (Sagawa et al., 2019), which contains
spurious correlations between the types of birds and background of the images,
different settings of CelebA (Liu et al., 2015) that contains more than
200,000 face images of the celebrities in the wild annotated with 40 binary
attributes and contains both spurious correlations and intrinsic dependencies
among its attributes, FairFace dataset (Karkkainen & Joo, 2021) which contains
more than 108,000 face images from 7 different race groups (White, Black,
Indian, East Asian, Southeast Asian, Middle Eastern, and Latino) collected
from the YFCC-100M Flickr dataset and labeled with race, sex, and age groups,
and Chicago Face Database (CFD) (Ma et al., 2015) which includes face images
with different annotations such as facial attributes, ethnicity, age, and sex.

###  4.2 Empirical Results

We report the results of FairerCLIP and compare them with the performance of
related baselines on various datasets and settings. Following the experimental
settings of prior work (Zhang & Ré, 2022; Koh et al., 2021; Chuang et al.,
2023), we do not presume sensitive attributes ($S$) during the training
process but assume them in the validation dataset for hyperparameter tuning
and model selection, as proposed in Koh et al. (2021). Thus, following prior
work (Zhang & Ré, 2022), for FairerCLIP and other baselines that need label
$S$, we use the zero-shot predictions of $S$ ($\hat{S}$) from CLIP as the
sensitive attribute.

####  4.2.1 Mitigating Intrinsic Dependency

To evaluate the ability of FairerCLIP to mitigate intrinsic dependency, we
conduct numerical evaluations on the CelebA dataset with _high cheekbones_ as
the target attribute and _sex_ as the sensitive attribute. As discussed in
Sec. 1, these two attributes are intrinsically dependent. To measure the
fairness of predictions, we employ the Equal Opportunity Difference (EOD)
(Hardt et al., 2016) metric, defined as,
$\text{EOD}:=\left|P(\hat{Y}=1|Y=1,S=1)-P(\hat{Y}=1|Y=1,S=0)\right|$, where
$S$ is the sensitive attribute, and $\hat{Y}$ and $Y$ are the predicted and
the ground-truth target labels, respectively. Our choice of EOD is justified
since other fairness definitions, like Demographic Parity Violation (DPV), are
not well suited for many practical scenarios (Hardt et al., 2016;
Chouldechova, 2017).

Table 1: Fairness on the CelebA dataset with intrinsic dependency. All values
are in %.

Method | CLIP ResNet-50 |  | CLIP ViT-L/14
---|---|---|---
Avg | EOD |  | Avg | EOD
Zero-shot (Radford et al., 2021) | 50.5 | 5.8 |  | 48.8 | 2.8
ERM Linear Probe (Kumar et al., 2022) | 84.8 | 19.0 |  | 84.8 | 14.0
ERM Adapter (Gao et al., 2021) | 85.3 | 11.0 |  | 84.6 | 14.0
DFR (Subsample) (Kirichenko et al., 2022) | 83.2 | 4.2 |  | 84.1 | 7.4
DFR (Upsample) (Kirichenko et al., 2022) | 83.6 | 4.1 |  | 84.1 | 6.6
Contrastive Adapter (Zhang & Ré, 2022) | 84.2 | 1.0 |  | 83.6 | 6.3
FairerCLIP (ours) | 83.4 | 0.02 |  | 83.8 | 0.005

Table 1 compares the performance of FairerCLIP and the baselines on the CelebA
dataset with intrinsic dependency. For this experiment, we train all methods
except the zero-shot baseline, which is included to demonstrate the level of
unfairness in the CLIP features with the ground-truth labels. We observe that
among all baselines, Contrastive Adapter (Zhang & Ré, 2022) performs well and
achieves appreciable EOD for the CLIP ResNet-50 model. However, most other
methods seem to even amplify the bias in the original CLIP features while
improving average accuracy. FairerCLIP performs the best in terms of
debiasing, achieving an EOD of 0.002% and 0.005% for CLIP ResNet-50 and CLIP
ViT-L/14, respectively. Overall, FairerCLIP is very effective at mitigating
unfairness to a significant extent, achieving an EOD value close to zero while
maintaining a high classification accuracy.

####  4.2.2 Mitigating Spurious Correlation

To evaluate FairerCLIP’s effectiveness in mitigating spurious correlation, we
perform numerical experiments on spurious correlation benchmarks, Waterbirds
and CelebA, following the settings in Zhang & Ré (2022). Since FairerCLIP can
be learned with or without ground-truth labels, we compare it against methods
from these two categories. For performance evaluation, we use three metrics:
1) Average accuracy (Avg.), 2) Worst-Group accuracy (WG), i.e., the lowest
accuracy of all subgroups, and 3) Gap, which is the difference between average
and worst-group accuracy.

Table 2: Comparison of prior methods and FairerCLIP in terms of worst group
accuracy (WG), average accuracy (Avg), and their gap on the WaterBirds and
CelebA datasets. For the latter, the target and sensitive attributes are
blonde hair and sex for two different variants of CLIP, CLIP ResNet-50 and
CLIP ViT-L/14, in two different settings–w/ and w/o labels. 1st / 2nd best
results are in bold / underlined.

| Method / Acc. | CLIP ViT-L/14 |  | CLIP ResNet-50
---|---|---|---|---
| Waterbirds |  | CelebA |  | Waterbirds |  | CelebA
|  | WG ($\uparrow$) | Avg ($\uparrow$) | Gap ($\downarrow$) |  | WG ($\uparrow$) | Avg ($\uparrow$) | Gap ($\downarrow$) |  | WG ($\uparrow$) | Avg ($\uparrow$) | Gap ($\downarrow$) |  | WG ($\uparrow$) | Avg$\uparrow$ | Gap ($\downarrow$)
w/ labels | ERM Linear Probe (Kumar et al., 2022) | 65.4$\pm$0.5 | 97.7$\pm$0.1 | 32.3$\pm$0.5 |  | 30.4$\pm$1.5 | 94.6$\pm$0.1 | 64.2$\pm$1.5 |  | 13.2$\pm$0.7 | 94.6$\pm$0.1 | 81.4$\pm$0.7 |  | 13.1$\pm$0.9 | 94.8$\pm$0.0 | 81.6$\pm$0.8
ERM Adapter(Gao et al., 2021) | 76.1$\pm$1.8 | 97.8$\pm$0.1 | 21.7$\pm$1.7 |  | 40.0$\pm$5.6 | 94.3$\pm$0.3 | 54.3$\pm$5.6 |  | 63.0$\pm$0.4 | 96.0$\pm$1.1 | 32.9$\pm$0.8 |  | 41.9$\pm$4.5 | 94.7$\pm$0.4 | 52.8$\pm$4.1
DFR (Subsample)(Kirichenko et al., 2022) | 58.8$\pm$0.8 | 95.9$\pm$0.2 | 37.1$\pm$0.8 |  | 78.7$\pm$3.6 | 91.8$\pm$0.2 | 13.1$\pm$3.6 |  | 66.1$\pm$5.5 | 92.9$\pm$2.2 | 26.9$\pm$6.5 |  | 80.9$\pm$0.6 | 91.7$\pm$0.5 | 10.8$\pm$3.2
DFR (Upsample)(Kirichenko et al., 2022) | 66.5$\pm$0.8 | 96.4$\pm$0.9 | 29.8$\pm$1.5 |  | 83.9$\pm$2.3 | 91.2$\pm$0.8 | 7.2$\pm$3.1 |  | 54.2$\pm$6.2 | 90.3$\pm$2.0 | 36.1$\pm$7.9 |  | 89.9$\pm$0.2 | 91.3$\pm$0.3 | 1.4$\pm$0.5
Contrastive Adapter (Zhang & Ré, 2022) | 85.3$\pm$2.3 | 94.5$\pm$2.4 | 9.3$\pm$1.1 |  | 83.9$\pm$1.1 | 90.4$\pm$0.2 | 6.4$\pm$1.1 |  | 82.5$\pm$0.9 | 88.2$\pm$2.6 | 5.7$\pm$3.1 |  | 88.4$\pm$1.7 | 90.8$\pm$1.2 | 2.5$\pm$1.5
FairerCLIP (ours) | 86.0$\pm$1.8 | 92.2$\pm$0.8 | 6.1$\pm$1.9 |  | 85.2$\pm$2.3 | 87.8$\pm$1.7 | 2.5$\pm$0.9 |  | 75.4$\pm$1.9 | 84.3$\pm$2.2 | 8.9$\pm$3.1 |  | 81.5$\pm$0.7 | 85$\pm$0.9 | 3.5$\pm$0.3
w/o labels | Zero-shot(Radford et al., 2021) | 45.3$\pm$0.0 | 84.4$\pm$0.0 | 39.1$\pm$0.0 |  | 72.8$\pm$0.0 | 87.6$\pm$0.0 | 14.9$\pm$0.0 |  | 39.6$\pm$0.0 | 77.3$\pm$0.0 | 37.7$\pm$0.0 |  | 75.9$\pm$0.0 | 82.3$\pm$0.0 | 6.4$\pm$0.0
Orth-Cali(Chuang et al., 2023) | 68.8 $\pm$ 0.0 | 84.5 $\pm$ 0.0 | 15.7$\pm$0.0 |  | 76.1$\pm$0.0 | 86.2$\pm$0.0 | 10.1$\pm$0.0 |  | 74.0$\pm$0.0 | 78.7$\pm$0.0 | 4.7$\pm$0.0 |  | 82.2$\pm$0.0 | 84.4$\pm$0.0 | 2.2$\pm$0.0
FairerCLIP (ours) | 78.1$\pm$1.4 | 85.1$\pm$1.1 | 7.1$\pm$2.4 |  | 86.1$\pm$0.8 | 88.0$\pm$1.0 | 1.9$\pm$0.6 |  | 74.8$\pm$1.7 | 81.4$\pm$0.9 | 6.6$\pm$2.5 |  | 80.4$\pm$1.0 | 84.7$\pm$0.7 | 4.3$\pm$0.4

Tab. 2 shows the results of our empirical evaluation. We make the following
observations: (i) On CLIP ViT-L/14, FairerCLIP has the lowest Gap and highest
WG accuracy. (ii) For the CLIP ResNet-50, FairerCLIP outperforms the baselines
in the w/o labels setting but not in the w/ label setting. The discrepancy
between the performance FairerCLIP with CLIP ViT-L/14 and CLIP ResNet-50 can
be attributed to the fact that CLIP ResNet-50 features contain less
information about target attributes than CLIP ViT-L/14 features, as shown in
Sec. A.7. Overall, the results of Tab. 2 indicate that FairerCLIP effectively
improves the worst group’s accuracy and reduces the Gap. Notably, our approach
can be applied to and is effective in both scenarios, with and without ground-
truth labels. At the same time, the baselines are specialized to operate in
one or the other scenario only.

In Tab. 3, we evaluate FairerCLIP on the FairFace dataset. Here, we consider
sex and race as the sensitive attributes and follow the experimental setup in
Chuang et al. (2023). We use five target attributes and ten text prompts (2
prompts per attribute) unrelated to the samples’ facial or sensitive
attributes; we do not have access to ground-truth labels. As an example, the
text prompt can be “A photo of a criminal person" or “A photo of a friendly
person". All the ten specific prompts are in Sec. A.3. To evaluate the models,
we calculate MaxSkew@1000 (Geyik et al., 2019), which assesses the maximum
imbalance in certain sensitive attributes within a dataset. As is shown in
Tab. 3 FairerCLIP outperforms

Table 3: Comparison of FairerCLIP with baselines on FairFace dataset.

Method / MaxSkew@1000 | CLIP ViT-B/32 |  | CLIP ViT-L/14
---|---|---|---
Sex | Race |  | Sex | Race
Zero-Shot (Radford et al., 2021) | 0.206 | 0.743 |  | 0.206 | 0.768
Orth-Proj (Chuang et al., 2023) | 0.146 | 0.755 |  | 0.349 | 0.605
Orth-Cali (Chuang et al., 2023) | 0.102 | 0.638 |  | 0.200 | 0.461
FairerCLIP (ours) | 0.097 | 0.408 |  | 0.099 | 0.428

the other baselines for both sensitive attributes across two different CLIP
backbones.

In summary, the results in Tab. 2, Tab. 3, and Tab. 1 suggest that FairerCLIP
can effectively mitigate the demographic bias caused by spurious correlation
and intrinsic dependency in the data in both w/ and w/o the ground-truth
labels settings.

$0$$10$$20$$30$$40$$45$$50$$55$$60$$65$WG (%)Avg (%)Zero-Shot (Radford et al.,
2021)ERM Adapter (Gao et al., 2021)Contrastive Adapter (Zhang & Ré,
2022)FairerCLIP (ours)

Figure 5: Results of FairerCLIP and baselines on CFD

Next, we consider a more challenging task to evaluate the data-efficiency of
FairerCLIP. We use CFD images with _attractive_ and _sex_ as the target and
sensitive group attributes. The former is a continuous label, which we
binarize by using the mean value of all samples as a threshold. Moreover, the
sex attribute is a binary label. This task presents challenges in two aspects.
First, the number of samples in this dataset is very small (597 samples),
which may not be sufficient for training some of the baselines. Second, the
performance of the zero-shot classifier for this case shows that the features
generated by the CLIP model are not well separated, rendering it difficult to
correctly predict $\hat{S}$ (see Section A.8). Fig. 5 shows the results of
this experiment. We first observe that all the baselines almost completely
fail at mitigating the bias for the worst group. In contrast, FairerCLIP’s
performance is satisfyingly better, both in terms of the worst group (WG) and
average (Avg) accuracy. Furthermore, the Gap is significantly lower (21.8% vs
53.7% for (Zhang & Ré, 2022)).

###  4.3 Computational Efficiency of Training

Table 4: Training time comparison (in seconds).

Method | Waterbirds |  | CelebA
---|---|---|---
Contrastive Adapter(Zhang & Ré, 2022) | 1202 |  | 20602
ERM Linear Probe(Kumar et al., 2022) | 157 |  | 2437
ERM Adapter(Gao et al., 2021) | 161 |  | 1924
DFR (Subsample)(Kirichenko et al., 2022) | 128 |  | 1878
DFR (Upsample)(Kirichenko et al., 2022) | 176 |  | 2662
FairerCLIP (ours) | 32 |  | 222

To show the computational efficiency of FairerCLIP we report and compare the
training time of FairerCLIP and other baselines in Tab. 4. The results show
that FairerCLIP is an order of magnitude faster than most baselines and almost
two orders faster than Contrastive Adapter (Zhang & Ré, 2022). The underlying
model for this experiment is CLIP ViT-L/14, and all the numbers are measured
on the same machine.

##  5 Ablation Studies

We conduct systematic ablation studies in different settings to investigate
the effectiveness of individual components of our approach. The settings
include spurious correlation, intrinsic dependency experiments, and scenarios
where ground-truth labels are unavailable. The results are shown in Tab. 5,
where Tab. 5 (left) shows results of training w/ labels and Tab. 5 (right)
shows training w/o labels. In the following, we describe each of these
studies. For more ablation studies please refer to Sec. A.6.

Effect of $\text{Dep}{(Z,Y)}$ term: Here we study the effect of
$\text{Dep}{(Z,Y)}$ by only retaining $\text{Dep}{(Z,Y)}$ in the objective. In
this case, the worst group accuracy dropped by 14.5% for the group robustness
experiment, and the EO increased to 0.195% for the fairness experiment.
Although the new features were still able to maintain good separation w.r.t.
$Y$, they lost their debiasing ability to a large extent.

Effect of $\text{Dep}(Z_{I},Z_{T})$ term: We remove $\text{Dep}(Z_{I},Z_{T})$
to investigate its effect on the alignment between the image and text
embeddings. Results show that while maintaining a similar worst group
accuracy, there is a decrease in average accuracy for both under the w/ and
w/o ground-truth setups. This result demonstrates the contribution of this
component in improving the predictions. Similarly, the results for the
fairness experiment show that $\text{Dep}(Z_{I},Z_{T})$ also aids in enhancing
the debiasing ability of FairerCLIP.

Effect of updating $\hat{Y}$: In this experiment, we predict $\hat{Y}$ once
and fix it through the training process. Updating it during the training
iterations has a considerable impact on worst group accuracy. The initial
zero-shot accuracy for this group was $72.8\%$. Using the same initial
$\hat{Y}$ during training improves the accuracy to $81.1\%$ while updating
$\hat{Y}$ while training improves it further to $86.1\%$.

Table 5: Ablation study w/ (left) and w/o (right) ground-truth labels on the
CelebA dataset for different target attributes with sex as the sensitive
attribute. We compare the effect of different components and parameters of
FairerCLIP on its performance. Both $\text{Dep}(Z_{I},Z_{T})$ and
$\text{Dep}(Z,Y)$ prove to be necessary and effective in maximizing the
metrics. All values are in %.

Method | Blonde Hair |  | High cheekbones
---|---|---|---
WG | Avg | Gap |  | Avg | EOD
$\text{Dep}(Z,Y)$ only ($\tau=0$) | 72.2 | 89.8 | 17.6 |  | 83.8 | 6.4
w/o $\text{Dep}(Z_{I},Z_{T})$ ($\tau_{z}=0$) | 87.0 | 88.7 | 1.7 |  | 83.8 | 0.2
FairerCLIP | 86.7 | 89.3 | 2.6 |  | 83.8 | 0.005

Method | Blonde Hair
---|---
WG | Avg | Gap
$\text{Dep}(Z,Y)$ only ($\tau=0$) | 75.0 | 87.7 | 12.7
w/o $\text{Dep}(Z_{I},Z_{T})$ ($\tau_{z}=0$) | 81.8 | 86.1 | 4.3
w/o updating $\hat{y}$ | 81.1 | 87.3 | 6.2
FairerCLIP | 86.1 | 88.8 | 2.7

##  6 Related Work

CLIP and Bias: Recent advancements in CLIP like models utilize multimodal data
to learn representations that demonstrably generalize well to many downstream
tasks and associated datasets (Radford et al., 2021; Desai & Johnson, 2021;
Singh et al., 2022; Zellers et al., 2021; Zhang et al., 2021; Alayrac et al.,
2022). Radford et al. (2021) demonstrated that utilizing a simple pretraining
task with massive amounts of image-text pairs collected from the Internet,
lead to models with strong transferability on different downstream tasks.
FLAVA (Singh et al., 2022) learns representations by jointly pretraining on
both unimodal and multimodal data. Flamingo Alayrac et al. (2022) demonstrated
excellent generalization performance in few-shot and zero-shot scenarios.
However, growing evidence (Birhane et al., 2023a; b) shows these models suffer
from spurious correlations and bias towards certain demographic groups. For
instance, Chuang et al. (2023) showed that textual prompt embeddings capture
spurious correlations. In addition, Agarwal et al. (2021) discovered that
zero-shot prediction from CLIP representations showed a high misclassification
rate for certain races. Similarly, Wolfe et al. (2022) observed that CLIP
embeddings exhibit stereotypes about sex and race. Complementing these
observations, we observe that CLIP exhibits high levels of demographic bias on
the CFD and CelebA datasets.

Debiasing CLIP: Several approaches have been proposed to debias CLIP
embeddings. Wang et al. (2021) addressed bias in image search by combining
balanced sampling and pruning spuriously correlated embeddings. Wang et al.
(2022) proposed a two-stage method that used learnable word vector prefixes
and a re-representation Matrix for debiasing image retrieval problems. Berg et
al. (2022) jointly trained an adversarial classifier and image-text
contrastive loss, effectively reducing different bias measures. Zhang & Ré
(2022) employed a contrastive adapter training strategy to enhance group
robustness. Following the group robustness evaluation, Chuang et al. (2023)
proposed to remove bias from text embeddings by projecting out the biased
direction with text data only. Seth et al. (2023) adapted an additive residual
learner module that separates the protected attribute information from the
image representation generated by the visual encoder of CLIP.

##  7 Concluding Remarks

This paper proposed FairerCLIP to mitigate bias in zero-shot predictions from
CLIP. It is versatile enough to mitigate bias caused by both spurious
correlations and intrinsic dependencies in data and can be trained with or
without ground-truth labels. Our key idea was to model the CLIP debiasing
problem in reproducing kernel Hilbert spaces and employ a non-parametric
statistical dependence measure that considers all linear and non-linear
relations between the representation and the attribute of interest. Our
solution in the form of an alternating optimization algorithm is effective
across a diverse set of datasets, including Waterbirds, CelebA, FairFace, and
the Chicago Face Database, spanning a variety of intrinsic dependencies and
spurious correlations among attributes. Lastly, kernel-based approaches are
underrepresented in current learning solutions, and FairerCLIP shows their
strong potential for the type of task considered in this paper due to its
flexibility, ease of optimization, and promising performance.

Acknowledgements: This work was supported in part by the National Science
Foundation (award #2147116) and the Office of Naval Research (award
#N00014-23-1-2417).

## References

  * Adila et al. (2023) Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala.  Zero-shot robustification of zero-shot models with foundation models.  _arXiv preprint arXiv:2309.04344_ , 2023.
  * Agarwal et al. (2021) Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage.  Evaluating clip: towards characterization of broader capabilities and downstream implications.  _arXiv preprint arXiv:2108.02818_ , 2021.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * An et al. (2023) Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang.  More context, less distraction: Visual classification by inferring and conditioning on contextual attributes.  _arXiv preprint arXiv:2308.01313_ , 2023.
  * Bach & Jordan (2002) Francis R Bach and Michael I Jordan.  Kernel independent component analysis.  _Journal of machine learning research_ , 3(Jul):1–48, 2002.
  * Bahng et al. (2020) Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh.  Learning de-biased representations with biased representations.  In _International Conference on Machine Learning_. PMLR, 2020.
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(8):1798–1828, 2013.
  * Berg et al. (2022) Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain.  A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.  _arXiv preprint arXiv:2203.11933_ , 2022.
  * Birhane et al. (2023a) Abeba Birhane, Vinay Prabhu, Sang Han, and Vishnu Naresh Boddeti.  On hate scaling laws for data-swamps.  _arXiv preprint arXiv:2306.13141_ , 2023a.
  * Birhane et al. (2023b) Abeba Birhane, Vinay Uday Prabhu, Sanghyun Han, Vishnu Naresh Boddeti, and Sasha Luccioni.  Into the laion’s den: Investigating hate in multimodal datasets.  _Advances in Neural Information Processing Systems_ , 2023b.
  * Chen et al. (2023) Annie S Chen, Yoonho Lee, Amrith Setlur, Sergey Levine, and Chelsea Finn.  Project and probe: Sample-efficient domain adaptation by interpolating orthogonal features.  _arXiv preprint arXiv:2302.05441_ , 2023.
  * Chouldechova (2017) Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. big data 5, 2 (2017), 153–163.  _arXiv preprint arXiv:1610.07524_ , 2017.
  * Chuang et al. (2023) Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka.  Debiasing vision-language models via biased prompts.  _arXiv preprint arXiv:2302.00070_ , 2023.
  * Desai & Johnson (2021) Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 11162–11173, 2021.
  * Du et al. (2022) Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Fukumizu et al. (2007) Kenji Fukumizu, Francis R Bach, and Arthur Gretton.  Statistical consistency of kernel canonical correlation analysis.  _Journal of Machine Learning Research_ , 8(2), 2007.
  * Gao et al. (2021) Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_ , 2021.
  * Geyik et al. (2019) Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi.  Fairness-aware ranking in search & recommendation systems with application to linkedin talent search.  In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 2221–2231, 2019.
  * Gretton et al. (2005) Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, and Bernhard Schölkopf.  Kernel methods for measuring independence.  _Journal of Machine Learning Research_ , 6(12):2075–2129, 2005.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Jung et al. (2021) Sangwon Jung, Donggyu Lee, Taeeon Park, and Taesup Moon.  Fair feature distillation for visual recognition.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Jung et al. (2022) Sangwon Jung, Sanghyuk Chun, and Taesup Moon.  Learning fair classifiers with partially annotated group labels.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Karkkainen & Joo (2021) Kimmo Karkkainen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1548–1558, 2021.
  * Kirichenko et al. (2022) Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  _arXiv preprint arXiv:2204.02937_ , 2022.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pp. 5637–5664. PMLR, 2021.
  * Kokiopoulou et al. (2011) Effrosini Kokiopoulou, Jie Chen, and Yousef Saad.  Trace optimization and eigenproblems in dimension reduction methods.  _Numerical Linear Algebra with Applications_ , 18(3):565–602, 2011.
  * Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ , 2022.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3730–3738, 2015.
  * Ma et al. (2015) Debbie S Ma, Joshua Correll, and Bernd Wittenbrink.  The chicago face database: A free stimulus set of faces and norming data.  _Behavior research methods_ , 47:1122–1135, 2015.
  * Quadrianto et al. (2019) Novi Quadrianto, Viktoriia Sharmanska, and Oliver Thomas.  Discovering fair representations in the data domain.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 2021.
  * Rahimi & Recht (2007) Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  _Advances in neural information processing systems_ , 20, 2007.
  * Sadeghi et al. (2022) Bashir Sadeghi, Sepehr Dehdashtian, and Vishnu Boddeti.  On characterizing the trade-off in invariant representation learning.  _Transactions on Machine Learning Research_ , 2022.  ISSN 2835-8856.  URL https://openreview.net/forum?id=3gfpBR1ncr.  Featured Certification.
  * Sagawa et al. (2019) Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _arXiv preprint arXiv:1911.08731_ , 2019.
  * Seth et al. (2023) Ashish Seth, Mayur Hemani, and Chirag Agarwal.  Dear: Debiasing vision-language models with additive residuals.  _arXiv preprint arXiv:2303.10431_ , 2023.
  * Singh et al. (2022) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15638–15650, 2022.
  * Wang et al. (2021) Jialu Wang, Yang Liu, and Xin Eric Wang.  Are gender-neutral queries really gender-neutral? mitigating gender bias in image search.  _arXiv preprint arXiv:2109.05433_ , 2021.
  * Wang et al. (2022) Junyang Wang, Yi Zhang, and Jitao Sang.  Fairclip: Social bias elimination based on attribute prototype learning and representation neutralization.  _arXiv preprint arXiv:2210.14562_ , 2022.
  * Wolfe et al. (2022) Robert Wolfe, Mahzarin R Banaji, and Aylin Caliskan.  Evidence for hypodescent in visual semantic ai.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 1293–1304, 2022.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7959–7971, 2022.
  * Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  _Advances in Neural Information Processing Systems_ , 34:23634–23651, 2021.
  * Zhang & Ré (2022) Michael Zhang and Christopher Ré.  Contrastive adapters for foundation model group robustness.  _arXiv preprint arXiv:2207.07180_ , 2022.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5579–5588, 2021.

##  Appendix A Appendix

In our main paper, we proposed FairerCLIP to debias the text and image
features from pre-trained vision-language models. Here, we provide some
additional analysis to support our main results. The supplementary material is
structured as follows:

  1. 1.

Representation Disentanglement and Training Algorithm in Sec. A.1

  2. 2.

Proofs Lemmas and Theorems in Sec. A.2

  3. 3.

Implementation details in Sec. A.3

  4. 4.

Analysis of analytical computational complexity and memory complexity in Sec.
A.4

  5. 5.

Effect of data size on the performance of FairerCLIP in Sec. A.5

  6. 6.

More ablation studies in Sec. A.6

  7. 7.

Comparison of CLIP ViT-L/14 and CLIP ResNet-50 in Sec. A.7

  8. 8.

Comparison of more than 100 Zero-Shot CLIP models on CFD in Sec. A.8

###  A.1 Techniqual Details

Representation Disentanglement: To ensure that the mapping functions avoid
learning representations with redundant information where different dimensions
are highly correlated with each other, we seek a compact (Bengio et al., 2013)
debiased embedding space. Therefore, we impose additional constraints on the
representation. Specifically, we constrain the search space of the mapping
functions $\bm{f}(\cdot)$ to learn a disentangled representation (Bengio et
al., 2013) as follows

$\displaystyle\mathcal{A}_{r}:=\left\\{\left(f_{1},\cdots,f_{r}\right)\,\big{|}\,f_{i},f_{j}\in\mathcal{H}_{X},\,\mathbb{C}\text{ov}\left(f_{i}(X),f_{j}(X)\right)+\gamma\,\langle
f_{i},f_{j}\rangle_{\mathcal{H}_{X}}=\delta_{i,j}\right\\}.$ (9)

In the above set, $\mathbb{C}\text{ov}\left(f_{i}(X),f_{j}(X)\right)$ part
enforces the covariance matrix of $Z=\bm{f}(X)$ to be an identity matrix and
encourages the variance of each entry of $Z$ to be one and different entries
of $Z$ to be uncorrelated with each other. The regularization part,
$\gamma\,\langle f_{i},f_{j}\rangle_{\mathcal{H}_{X}}$ encourages the
components of the mapping functions to be of unit norm and as orthogonal as
possible to each other. These constraints also aid with numerical stability
during empirical estimation (Fukumizu et al., 2007).

Training Algorithm: Details of training FairerCLIP are presented in Algorithm
1. FairerCLIP uses representation of images, representation of texts
corresponding to target attribute labels ($X_{T}$), and representation of text
corresponding to sensitive attribute labels ($X_{TS}$) as its inputs.
FairerCLIPś goal is to find the image encoder ($\bm{f}^{*}_{I}$) and text
encoder ($\bm{f}^{*}_{T}$) that can map the biased features generated by the
CLIP to a debiased representation space. The training algorithm starts with
initializing the label predictions. Since this algorithm is used for scenarios
where we do not have access to the ground-truth labels of target attributes
and sensitive attributes, we need to predict them by zero-shot classification
from CLIP features. However, in scenarios where we have access to the true
labels of the target attribute, we can skip the pseudo $Y$ initialization step
and use the ground truth $Y$ instead. In the last step of initialization, we
need to initialize the representation of images since we are using an
alternating method to optimize the parameters of both the image encoder and
text encoder. After the initialization step, we start to train both models
alternatingly. After each optimization iteration, we update our prediction of
$Y$ labels. After reaching the stop condition, the training process is
complete.

Input: $\bm{X}_{I}\in\mathbb{R}^{n\times d}$,
$\bm{X}_{T}\in\mathbb{R}^{|Y|\times d}$, $\bm{X}_{TS}\in\mathbb{R}^{|S|\times
d}$, $m\in\mathbb{N}$

Output: $\bm{f}^{*}_{I}$, $\bm{f}^{*}_{T}$

Initialize:

$\hat{\bm{Y}}^{(0)}\leftarrow\Big{\\{}\forall\bm{x}_{I}\in\bm{X}_{I},\bm{x}_{T}\in\bm{X}_{T}\Big{|}\underset{\bm{x}_{T}}{\text{argmax}}\frac{\bm{x}_{I}^{T}\bm{x}_{T}}{\|\bm{x}_{I}\|\|\bm{x}_{T}\|}\Big{\\}}$;

/* initialize pseudo Y */

$\hat{\bm{S}}\leftarrow\Big{\\{}\forall\bm{x}_{I}\in\bm{X}_{I},\bm{x}_{TS}\in\bm{X}_{TS}\Big{|}\underset{\bm{x}_{TS}}{\text{argmax}}\frac{\bm{x}_{I}^{T}\bm{x}_{TS}}{\|\bm{x}_{I}\|\|\bm{x}_{TS}\|}\Big{\\}}$;

/* initialize pseudo S */

$\bm{Z}_{I}^{(0)}\leftarrow\\{\bm{f}^{*}_{I}(\bm{X}_{I})|\bm{f}^{*}_{I}\leftarrow\underset{\bm{f}_{I}}{\text{argmax}}\mbox{
}J(\bm{f}_{I},\tau_{I},0,\bm{X}_{I},\hat{\bm{Y}}^{(0)},\hat{\bm{S}},\bm{0})\\}$;

/* Eq. 7 */

$i\leftarrow 0$;

while  _$i <m$_ do

$\bm{Z}^{(i+1)}_{T}\leftarrow\\{\bm{f}^{*}_{T}(\bm{X}_{T})|\bm{f}^{*}_{T}\leftarrow\underset{\bm{f}_{T}}{\text{argmax}}\mbox{
}J(\bm{f}_{T},\tau_{T},\tau_{z},\bm{X}_{T},\hat{\bm{Y}}^{(i)},\hat{\bm{S}},\bm{Z}_{I}^{(i)})\\}$;

/* solve Eq. 7 */

$\bm{Z}_{I}^{(i+1)}\leftarrow\\{\bm{f}^{*}_{I}(\bm{X}_{I})|\bm{f}^{*}_{I}\leftarrow\underset{\bm{f}_{I}}{\text{argmax}}\mbox{
}J(\bm{f}_{I},\tau_{I},\tau_{z},\bm{X}_{I},\hat{\bm{Y}}^{(i)},\hat{\bm{S}},\bm{Z}_{T}^{(i)})\\}$;

/* solve Eq. 7 */

$\hat{\bm{Y}}^{(i+1)}\leftarrow\Big{\\{}\forall\bm{z}_{I}\in\bm{Z}^{(i+1)}_{I},\bm{z}_{T}\in\bm{Z}^{(i+1)}_{T}\Big{|}\underset{\bm{z}_{T}}{\text{argmax}}\frac{\bm{z}_{I}^{T}\bm{z}_{T}}{\|\bm{z}_{I}\|\|\bm{z}_{T}\|}\Big{\\}}$;

/* refine pseudo Y */

$i\leftarrow i+1$

end while

Algorithm 1 FairerCLIP Training Without Labels

###  A.2 Proofs

######  Lemma 1.

Let $\bm{K}_{X_{I}},\bm{K}_{X_{T}}\in\mathbb{R}^{n\times n}$ be the Gram
matrices corresponding to $\mathcal{H}_{X_{I}}$ and $\mathcal{H}_{X_{T}}$,
respectively, i.e.,
$\left(\bm{K}_{X_{I}}\right)_{ij}=k_{X_{I}}(\bm{x}_{Ii},\bm{x}_{Ij})$ and
$\left(\bm{K}_{X_{T}}\right)_{ij}=k_{S}(\bm{x}_{Ti},\bm{x}_{Tj})$, where
covariance is empirically estimated as

$\displaystyle\mathbb{C}\text{ov}\left(f_{j}(X_{I}),g_{m}(X_{T})\right)\approx\frac{1}{n}\sum_{i=1}^{n}f_{j}(\bm{x}_{I_{i}})g_{m}(\bm{x}_{T_{i}})-\frac{1}{n^{2}}\sum_{p=1}^{n}f_{j}(\bm{x}_{I_{p}})\sum_{k=1}^{n}g_{m}(\bm{x}_{T_{k}}).$

It follows that, the corresponding empirical estimator for
$\text{Dep}\left(Z_{I},Z_{T}\right)$ is

$\displaystyle\text{Dep}\left(Z_{I},Z_{T}\right)$ $\displaystyle=$
$\displaystyle\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\Theta}^{T}_{T}\right\|^{2}_{F},$
(10)

###### Proof.

$\displaystyle\text{Dep}(Z_{I},Z_{T})$ $\displaystyle:=$
$\displaystyle\sum_{m=1}^{r}\sum_{j=1}^{r}\left\\{\frac{1}{n}\sum_{i=1}^{n}f_{j}(\bm{x}_{I_{i}})g_{m}(\bm{x}_{T_{i}})-\frac{1}{n^{2}}\sum_{p=1}^{n}f_{j}(\bm{x}_{I_{p}})\sum_{k=1}^{n}g_{m}(\bm{x}_{T_{k}})\right\\}^{2}$
(11) $\displaystyle=$
$\displaystyle\sum_{m=1}^{r}\sum_{j=1}^{r}\Big{\\{}\frac{1}{n}\bm{\theta}_{I_{j}}^{T}\bm{K}_{X_{I}}\bm{K}_{X_{T}}\bm{\theta}_{T_{m}}-\frac{1}{n^{2}}\bm{\theta}_{I_{j}}^{T}\bm{K}_{X_{I}}\bm{1}_{n}\bm{1}_{n}^{T}\bm{K}_{X_{T}}\bm{\theta}_{T_{m}}\Big{\\}}^{2}$
$\displaystyle=$
$\displaystyle\sum_{m=1}^{r}\sum_{j=1}^{r}\left\\{\frac{1}{n}\bm{\theta}_{I_{j}}^{T}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\theta}_{T_{m}}\right\\}^{2}$
$\displaystyle=$
$\displaystyle\sum_{m=1}^{r}\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\theta}_{T_{m}}\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{n^{2}}\left\|\bm{\Theta}_{I}\bm{K}_{X_{I}}\bm{H}\bm{K}_{X_{T}}\bm{\Theta}_{T}^{T}\right\|^{2}_{F}$

∎

######  Theorem 2.

Let the Cholesky factorization of $\bm{K}_{X}$ be
$\bm{K}_{X}=\bm{L}_{X}\bm{L}_{X}^{T}$, where $\bm{L}_{X}\in\mathbb{R}^{n\times
d}$ ($d\leq n$) is a full column-rank matrix. Let $r\leq d$, then a solution
to

$\max_{\bm{f}\in
A_{r}}\\{J\left(\bm{f},\tau,\tau_{z},\bm{X},\bm{Y},\bm{S},\bm{Z}_{O}\right):=\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{Y}\right\|^{2}_{F}-\tau\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{S}\right\|^{2}_{F}+\tau_{z}\frac{1}{n^{2}}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{Z}_{O}\right\|^{2}_{F}\\}$
(12)

is

$\displaystyle\bm{f}^{\text{opt}}(X)=\bm{\Theta}^{\text{opt}}\left[k_{X}(\bm{x}_{1},X),\cdots,k_{X}(\bm{x}_{n},X)\right]^{T}$

where $\bm{\Theta}^{\text{opt}}=\bm{U}^{T}\bm{L}_{X}^{\dagger}$ and the
columns of $\bm{U}$ are eigenvectors corresponding to the $r$ largest
eigenvalues of the following generalized eigenvalue problem.

$\displaystyle\bm{L}_{X}^{T}\left(\bm{H}\bm{K}_{Y}\bm{H}-\tau\bm{H}\bm{K}_{S}\bm{H}+\tau_{z}\bm{H}\bm{Z}_{O}\bm{Z}_{O}^{T}\bm{H}\right)\bm{L}_{X}\bm{u}=\lambda\left(\frac{1}{n}\,\bm{L}_{X}^{T}\bm{H}\bm{L}_{X}+\gamma\bm{I}\right)\bm{u}.$
(13)

Furthermore, the supremum value of the objective function is equal to
$\sum_{j=1}^{r}\lambda_{j}$, where $\\{\lambda_{1},\cdots,\lambda_{r}\\}$ are
$r$ largest eigenvalues of equation 13.

###### Proof.

Using the representer theorem, the disentanglement property in

$\displaystyle\mathcal{A}_{r}:=\left\\{\left(f_{1},\cdots,f_{r}\right)\,\big{|}\,f_{i},f_{j}\in\mathcal{H}_{X},\,\mathbb{C}\text{ov}\left(f_{i}(X),f_{j}(X)\right)+\gamma\,\langle
f_{i},f_{j}\rangle_{\mathcal{H}_{X}}=\delta_{i,j}\right\\}.$ (14)

can be expressed as

$\displaystyle\mathbb{C}\text{ov}\left(f_{i}(X),\,f_{j}(X)\right)+\gamma\,\langle
f_{i},f_{j}\rangle_{\mathcal{H}_{X}}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{k=1}^{n}f_{i}(\bm{x}_{k})f_{j}(\bm{x}_{k})-\frac{1}{n^{2}}\sum_{k=1}^{n}f_{i}(\bm{x}_{k})\sum_{m=1}^{n}f_{j}(\bm{x}_{m})+\gamma\,\langle
f_{i},f_{j}\rangle_{\mathcal{H}_{X}}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{k=1}^{n}\sum_{t=1}^{n}\bm{K}_{X}(\bm{x}_{k},\bm{x}_{t})\theta_{it}\sum_{m=1}^{n}\bm{K}_{X}(\bm{x}_{k},\bm{x}_{m})\theta_{jm}-\frac{1}{n^{2}}\bm{\theta}_{i}^{T}\bm{K}_{X}\bm{1}_{n}\bm{1}_{n}^{T}\bm{K}_{X}\bm{\theta}_{j}+\gamma\,\langle
f_{i},f_{j}\rangle_{\mathcal{H}_{X}}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\left(\bm{K}_{X}\bm{\theta}_{i}\right)^{T}\left(\bm{K}_{X}\bm{\theta}_{j}\right)-\frac{1}{n^{2}}\bm{\theta}_{i}^{T}\bm{K}_{X}\bm{1}_{n}\bm{1}_{n}^{T}\bm{K}_{X}\bm{\theta}_{j}+\gamma\,\left\langle\sum_{k=1}^{n}\theta_{ik}k_{X}(\cdot,\bm{x}_{k}),\sum_{t=1}^{n}\theta_{it}k_{X}(\cdot,\bm{x}_{t})\right\rangle_{\mathcal{H}_{X}}$
$\displaystyle=$
$\displaystyle\frac{1}{n}\bm{\theta}_{i}^{T}\bm{K}_{X}\bm{H}\bm{K}_{X}\bm{\theta}_{j}+\gamma\,\bm{\theta}^{T}_{i}\bm{K}_{X}\bm{\theta}_{j}$
$\displaystyle=$
$\displaystyle\frac{1}{n}\bm{\theta}_{i}^{T}\bm{L}_{X}\left(\bm{L}^{T}_{X}\bm{H}\bm{L}_{X}+n\gamma\,\bm{I}\right)\bm{L}^{T}_{X}\bm{\theta}_{j}$
$\displaystyle=$ $\displaystyle\delta_{i,j}.$

As a result, $\bm{f}\in\mathcal{A}_{r}$ is equivalent to

$\displaystyle\bm{\Theta}\bm{L}_{X}\underbrace{\Big{(}\frac{1}{n}\bm{L}^{T}_{X}\bm{H}\bm{L}_{X}+\gamma\bm{I}\Big{)}}_{:=\bm{C}}\bm{L}^{T}_{X}\bm{\Theta}^{T}=\bm{I}_{r},$

where
$\bm{\Theta}:=\big{[}\bm{\theta}_{1},\cdots,\bm{\theta}_{r}\big{]}^{T}\in\mathbb{R}^{r\times
n}$.

Let $\bm{V}=\bm{L}_{X}^{T}\bm{\Theta}^{T}$ and consider the optimization
problem in equation 7:

$\displaystyle\sup_{\bm{f}\in\mathcal{A}_{r}}\frac{1}{n^{2}}\left\\{\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{Y}\right\|^{2}_{F}-\tau\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{L}_{S}\right\|^{2}_{F}+\tau_{z}\left\|\bm{\Theta}\bm{K}_{X}\bm{H}\bm{Z}_{O}\right\|^{2}_{F}\right\\}$
(15) $\displaystyle=$
$\displaystyle\sup_{\bm{f}\in\mathcal{A}_{r}}\frac{1}{n^{2}}\left\\{\text{Tr}\left\\{\bm{\Theta}\bm{K}_{X}\bm{H}\bm{K}_{Y}\bm{H}\bm{K}_{X}\bm{\Theta}^{T}\right\\}-\tau\text{Tr}\left\\{\bm{\Theta}\bm{K}_{X}\bm{H}\bm{K}_{S}\bm{H}\bm{K}_{X}\bm{\Theta}^{T}\right\\}+\tau_{z}\text{Tr}\left\\{\bm{\Theta}\bm{K}_{X}\bm{H}\bm{Z}_{O}\bm{Z}_{O}^{T}\bm{H}\bm{K}_{X}\bm{\Theta}^{T}\right\\}\right\\}$
$\displaystyle=$
$\displaystyle\max_{\bm{V}^{T}\bm{C}\bm{V}=\bm{I}_{r}}\frac{1}{n^{2}}\text{Tr}\left\\{\bm{\Theta}\bm{L}_{X}\bm{B}\bm{L}_{X}^{T}\bm{\Theta}^{T}\right\\}$
$\displaystyle=$
$\displaystyle\max_{\bm{V}^{T}\bm{C}\bm{V}=\bm{I}_{r}}\frac{1}{n^{2}}\text{Tr}\left\\{\bm{V}^{T}\bm{B}\bm{V}\right\\}$

where

$\displaystyle\bm{B}$ $\displaystyle:=$
$\displaystyle\bm{L}^{T}_{X}\left(\bm{H}\bm{K}_{Y}\bm{H}-\tau\bm{H}\bm{K}_{S}\bm{H}+\tau_{z}\bm{H}\bm{Z}_{O}\bm{Z}_{O}^{T}\bm{H}\right)\bm{L}_{X}$

It is shown in Kokiopoulou et al. (2011) that an222Optimal $\bm{V}$ is not
unique. optimizer of (15) is any matrix $\bm{U}$ whose columns are
eigenvectors corresponding to $r$ largest eigenvalues of generalized problem

$\displaystyle\bm{B}\bm{u}=\tau\,\bm{C}\bm{u}$ (16)

and the maximum value is the summation of $r$ largest eigenvalues. Once
$\bm{U}$ is determined, then, any $\bm{\Theta}$ in which
$\bm{L}_{X}^{T}\bm{\Theta}^{T}=\bm{U}$ is optimal $\bm{\Theta}$ (denoted by
$\bm{\Theta}^{\text{opt}}$). Note that $\bm{\Theta}^{\text{opt}}$ is not
unique and has a general form of

$\displaystyle\bm{\Theta}^{T}=\left(\bm{L}_{X}^{T}\right)^{\dagger}\bm{U}+\bm{\Lambda}_{0},\quad\mathcal{R}(\bm{\Lambda}_{0})\subseteq\mathcal{N}\left(\bm{L}^{T}_{X}\right).$

However, setting $\bm{\Lambda}_{0}$ to zero would lead to minimum norm for
$\bm{\Theta}$. Therefore, we opt
$\bm{\Theta}^{\text{opt}}=\bm{U}^{T}\bm{L}_{X}^{\dagger}$. ∎

###  A.3 Implementation Details

We conducted experiments on CelebA, Waterbirds, FairFace, and the Chicago Face
Dataset (CFD). For CelebA and Waterbirds, we follow their official
train/val/test splits and only use ground truth labels from the val split for
hyperparameter tuning. For CFD, since there is no official dataset split, we
randomly split it with a ratio of 0.5/0.1/0.4 for train/val/test. Following
the standard setting Zhang & Ré (2022), we use val split to decide the optimal
$\tau$, $\tau_{z}$, and dimensionality of the random Fourier features (RFF).
For CelebA, the optimal $\tau$, $\tau_{z}$, and RFF dimensions are 0.8, 0.5,
and 8000. For Waterbirds, the optimal $\tau$, $\tau_{z}$, and RFF dimensions
are 0.7, 0.7, and 3000. And for CFD, the optimal $\tau$, $\tau_{z}$, and RFF
dimensions are 0.6, 0.3, and 1000. In the scenario where the group labels are
not available, we follow the same setup as the scenario where the group labels
of the val split are available. For the CelebA dataset, we also conduct a pre-
sampling process on the training split to balance the number of each class
from the predicted $\hat{y}$.

For the FairFace dataset, we use 10 text prompts that are unrelated to the
facial attributes or the sensitive attributes of the samples. In this setting,
the sensitive attribute is gender or race, and the text prompts are
constructed as _“This is a photo of a [attribute] person"_ where _[attribute]_
can be one of the elements of the {good, evil, smart, dumb, attractive,
unattractive, lawful, criminal, friendly, unfriendly} set.

For all the above-mentioned experiments under different settings, we set the
representation dimensionality $r$ to $c-1$ where $c$ is the number of classes
of the downstream target task.

For clarity, we summarized all the above-mentioned implementation details in
Tab. 6

Table 6: Implementation details of FairerCLIP for each dataset. Dataset | RFF Dim. | $r$ | $\tau$ | $\tau_{Z}$ | Train/Val/Test | Training samples
---|---|---|---|---|---|---
CelebA | 8000 | 1 | 0.8 | 0.5 | Official Splits | 162,770
Waterbirds | 3000 | 1 | 0.7 | 0.7 | Official Splits | 4,795
FairFace | 3000 | 1 | 0.8 | 0.8 | Official Splits | 86,744
CFD | 1000 | 1 | 0.6 | 0.3 | 0.5/0.1/0.4 | 298

###  A.4 Numerical Complexity

Computational Complexity: If $\bm{L}_{X}$ in equation 13 is provided in the
training dataset, then the computational complexity of obtaining the optimal
encoder is $\mathcal{O}(l^{3})$, where $l\leq n$ is the numerical rank of the
Gram matrix $\bm{K}_{X}$. However, the dominating part of the computational
complexity is due to the Cholesky factorization,
$\bm{K}_{X}=\bm{L}_{X}\bm{L}_{X}^{T}$, which is $\mathcal{O}(n^{3})$. Using
random Fourier features (RFF) (Rahimi & Recht, 2007),
$k_{X}(\bm{x},\bm{x}^{\prime})$ can be approximated by
$\bm{r}_{X}(\bm{x})^{T}\,\bm{r}_{X}(\bm{x}^{\prime})$, where
$\bm{r}_{X}(\bm{x})\in\mathbb{R}^{d}$. In this situation, the Cholesky
factorization can be directly calculated as

$\bm{L}_{X}=\begin{bmatrix}\bm{r}_{X}(\bm{x}_{1})^{T}\\\ \vdots\\\
\bm{r}_{X}(\bm{x}_{n})^{T}\end{bmatrix}\in\mathbb{R}^{n\times d}.$ (17)

As a result, the computational complexity of obtaining the optimal encoder
becomes $\mathcal{O}(d^{3})$, where the RFF dimension, $d$, can be
significantly less than the sample size $n$ with negligible error on the
approximation
$k_{X}(\bm{x},\bm{x}^{\prime})\approx\bm{r}_{X}(\bm{x})^{T}\,\bm{r}_{X}(\bm{x}^{\prime})$.

Memory Complexity: The memory complexity of equation 13, if calculated
naively, is $\mathcal{O}(n^{2})$ since $\bm{K}_{Y}$, $\bm{K}_{S}$, and
$\bm{Z}_{O}\bm{Z}_{O}^{T}$ are $n$ by $n$ matrices. However, using RFF
together with Cholesky factorization $\bm{K}_{Y}=\bm{L}_{Y}\bm{L}_{Y}^{T}$,
$\bm{K}_{S}=\bm{L}_{S}\bm{L}_{S}^{T}$, the left-hand side of equation 13 can
be re-arranged as

$\left(\bm{L}^{T}_{X}\tilde{\bm{L}}_{Y}\right)\left(\tilde{\bm{L}}^{T}_{Y}\bm{L}_{X}\right)-\tau\left(\bm{L}^{T}_{X}\tilde{\bm{L}}_{S}\right)\left(\tilde{\bm{L}}^{T}_{S}\bm{L}_{X}\right)+\tau_{z}\left(\bm{L}^{T}_{X}\tilde{\bm{Z}}_{O}\right)\left(\tilde{\bm{Z}}^{T}_{O}\bm{L}_{X}\right),$
(18)

where
$\tilde{\bm{Z}}^{T}_{O}=\bm{H}\bm{Z}_{O}=\bm{Z}_{O}-\frac{1}{n}\bm{1}_{n}(\bm{1}_{n}^{T}\bm{Z}_{O})$
and
$\tilde{\bm{L}}^{T}_{Y}=\bm{H}\bm{L}_{Y}=\bm{L}_{Y}-\frac{1}{n}\bm{1}_{n}(\bm{1}_{n}^{T}\bm{L}_{Y})$;
therefore, the required memory complexity is $\mathcal{O}(nd)$. Note that
$\tilde{\bm{L}}^{T}_{S}$ and $\bm{H}\bm{L}_{X}$ can be calculated similarly.

###  A.5 Effect of Data Size on the performance of FairerCLIP

To evaluate the effectiveness of the FairerCLIP under limited data samples
condition we report the performance of FairerCLIP as the size of the training
dataset is varied. In this experiment, we randomly sampled 5, 25, 50, 75, and
100 percent of the training data as our training set. Then the Avg., WG, and
Gap are evaluated. Table 7 shows the results of the evaluation for the
Waterbird and CelebA datasets. The results indicate that FairerCLIP is able to
perform sufficiently well when a small sub-sample of the dataset is used for
its training. More specifically, in the CelebA dataset, FairerCLIP is only
losing $2.9\%$ of its WG accuracy when only $25\%$ of the original training
data is employed in its training phase. Moreover, on Waterbirds, a relatively
smaller dataset, it loses less than $6\%$ of WG accuracy when only $50\%$ of
training data is used. This experiment shows the effectiveness of FairerCLIP
under a limited number of training samples.

Table 7: Effect of training data size on the performance of FairerCLIP # Samples | CelebA |  | Waterbird
---|---|---|---
Avg. ($\uparrow$) | WG ($\uparrow$) | Gap ($\downarrow$) |  | Avg. ($\uparrow$) | WG ($\uparrow$) | Gap ($\downarrow$)
$5\%$ | 84.21 | 73.88 | 10.32 |  | 86.31 | 65.26 | 21.05
$25\%$ | 88.26 | 83.88 | 4.37 |  | 86.54 | 78.66 | 7.88
$50\%$ | 86.65 | 84.29 | 2.36 |  | 89.11 | 81.31 | 7.80
$75\%$ | 90.44 | 85.56 | 4.88 |  | 87.04 | 84.11 | 2.92
$100\%$ | 89.3 | 86.7 | 2.6 |  | 92.30 | 87.70 | 4.60

###  A.6 More Ablation Studies

In 5, we studied the effect of different components of FairerCLIP such as
$\text{Dep}(Z,Y)$, $\text{Dep}(Z_{I},Z_{T})$, and updating the prediction of
the target labels in each iteration. Here, we add another ablation study on
the effect of our control hyper-parameters, $\tau$ and $\tau_{z}$ on the
performance of the method.

Effect of $\tau$ and $\tau_{z}$: We illustrate the performance of average
accuracy and worst group’s accuracy for varying values of $\tau$ and
$\tau_{z}$ in Figure 6. First, as $\tau$ and $\tau_{z}$ vary, there is a
smooth change in the average and worst group accuracy, which demonstrates the
stability of FairerCLIP. Second, as $\tau$ increases, the worst group’s
accuracy also improves, which alludes to the effectiveness of $\tau$ as a
control for the degree of debiasing. However, when $\tau$ reaches a certain
value, a further increase in its value leads to a degradation in the worst
group’s accuracy. Similarly, $\tau_{z}$ also plays a gradual but noticeable
effect on improving both the average and worst group accuracy for a given
value of $\tau$.

Figure 6: Effect of $\tau$ and $\tau_{z}$ on the average accuracy (left) and
worst group accuracy (right). For every fixed $\tau_{z}$, the worst group
accuracy increases with $\tau$ and then starts to decrease beyond a point.
Additionally, the variation of the metrics is smooth over the range of
hyperparameters, indicating lower sensitivity towards them.

###  A.7 Comparing Features of CLIP ViT-L/14 and CLIP ResNet-50

Table 8: Comparison between CLIP ViT-L/14 and CLIP ResNet-50 in terms of the amount of the information from $Y$ and $S$ that each of them can embed into their feature space on the Waterbirds. | ($X,Y$) |  | ($X,S$)
---|---|---|---
HSIC | KCC |  | HSIC | KCC
CLIP ViT-L/14 | 0.1849 | 0.8267 |  | 0.2392 | 0.8661
CLIP ResNet-50 | 0.1423 | 0.7556 |  | 0.3823 | 0.8861

To compare the features of CLIP ViT-L/14 and CLIP ResNet-50, we measure the
amount of information from the target attribute and sensitive attribute
contained in their generated representations. The embedded information is
measured in terms of statistical dependency between the features and their
ground-truth $Y$ and $S$ labels. To calculate these dependencies, Hilbert-
Schmidt independence criterion (HSIC) (Gretton et al., 2005) and Kernel
Canonical Covariance (KCC) Bach & Jordan (2002) are used. Tab. 8 compares
these two CLIP models. From the table, we can observe that CLIP ViT-L/14
embeds more information about the target $Y$ while containing less information
about $S$ which indicates that the former provides better features for the
Waterbirds dataset.

###  A.8 Comparing more than 100 Zero-Shot CLIP models on CFD

As we mentioned in Section 4.2.2, zero-shot classification on CFD is a
difficult task for the OpenAI CLIP model. In Figure 7, we show that a majority
of other publicly available CLIP models suffer similarly on CFD. In fact,
several models achieve only near-zero WG accuracy, irrespective of their
parameter count and the training dataset.

Figure 7: Comparison of more than 100 publicly available CLIP models zero-shot
performance on CFD dataset. Colors show the pre-trained dataset and sizes show
the number of parameters of each model.