# Alignment Calibration: Machine Unlearning for Contrastive Learning under
Auditing

Yihan Wang 111Contributed equally. Academy of Mathematics and Systems Science,
Chinese Academy of Sciences  University of Chinese Academy of Sciences  Yiwei
Lu 11footnotemark: 1 University of Waterloo, Vector Institute  Guojun Zhang
Huawei Noah’s Ark Lab  Franziska Boenisch  CISPA  Adam Dziedzic  CISPA
Yaoliang Yu  University of Waterloo, Vector Institute  Xiao-Shan Gao  Academy
of Mathematics and Systems Science, Chinese Academy of Sciences  University of
Chinese Academy of Sciences

###### Abstract

Machine unlearning provides viable solutions to revoke the effect of certain
training data on pre-trained model parameters. Existing approaches provide
unlearning recipes for classification and generative models. However, a
category of important machine learning models, i.e., contrastive learning (CL)
methods, is overlooked. In this paper, we fill this gap by first proposing the
framework of Machine Unlearning for Contrastive learning (MUC) and adapting
existing methods. Furthermore, we observe that several methods are mediocre
unlearners and existing auditing tools may not be sufficient for data owners
to validate the unlearning effects in contrastive learning. We thus propose a
novel method called _Alignment Calibration (AC)_ by explicitly considering the
properties of contrastive learning and optimizing towards novel auditing
metrics to easily verify unlearning. We empirically compare _AC_ with baseline
methods on SimCLR, MoCo and CLIP. We observe that _AC_ addresses drawbacks of
existing methods: (1) achieving state-of-the-art performance and approximating
exact unlearning (retraining); (2) allowing data owners to clearly visualize
the effect caused by unlearning through _black-box auditing_.

##  1 Introduction

The success of modern machine learning models largely relies on training with
a large corpus of data. However, carefully annotated data are expensive and
difficult to obtain, thus urging the utilization of the vast amount of
unlabelled data in the wild. The recent self-supervised learning methods,
especially contrastive learning methods [6, 8, 17], provide viable solutions
to learning general representations for various downstream tasks. For example,
unimodal contrastive learning models employ the InfoNCE loss to maximize the
feature similarity between positive pairs (e.g., different data augmentations
of the same image) while minimizing that between the negative ones (e.g.,
different data samples). This training scheme also applies to multi-modal
training (e.g., CLIP [32]), and the learned encoders are widely applied in
various tasks, e.g., GPT-based models [1] and latent diffusion models [33].

To collect large-scale datasets for training contrastive learning models,
practitioners usually extract the desired data by crawling on the internet
(e.g., web pages using Common Crawl222https://commoncrawl.org/). Such data
collection procedures may disregard data owners’ privacy concerns and retrieve
their data unwillingly. Moreover, part of the acquired training data may be
copyrighted or even contain inappropriate content such as sexual abuse (e.g.,
in recent reports333https://purl.stanford.edu/kh752sm9123?ref=404media.co
against contents in LAION-5B [35]). In such scenarios, data owners or even the
authorities are entitled to send a removal request upon such misused training
data (i.e., unlearning dataset)444Following the policies of European Union’s
General Data Protection Regulation (GDPR), the California Consumer Privacy Act
(CCPA), and Canada’s proposed Consumer Privacy Protection Act (CPPA)., which
consequently affects the trained model parameters. A straightforward solution
is to retrain the model entirely from scratch without the unlearning dataset,
but the computational cost can be tremendous for large models and datasets.

To eliminate the effect of the unlearning dataset on the model with minimum
effort, machine unlearning methods [3, 2, 12, 15, 30, 42, 36, 20, 5, 48, 10,
37] provide recipes for supervised learning methods on group removal and for
generative models on sample or concept removal. However, the study of an
efficient solution for contrastive learning models is underexplored. In this
paper, we establish the foundation of Machine Unlearning for Contrastive
learning models (MUC). MUC adapts various existing methods to contrastive
learning, and introduces the notion of data owners who request unlearning and
model owners who execute unlearning. Given candidate unlearning algorithms,
the model owners first perform _white-box evaluation_ to select the best
method and generate the optimal unlearned model. The data owners then perform
_black-box auditing_ to validate the effect of the unlearning procedure. We
argue that unlearning success is achieved only if the unlearned model meets
the criteria on both sides.

Unfortunately, direct adaptions of existing unlearning approaches are
unsatisfactory on both considerations. Firstly, from the model owners’
perspective, such algorithms are mediocre approximations of exact unlearning
(training from scratch) under different white-box evaluations and there lack
of a good candidate method. Secondly, from the data owner’s perspective, even
given the strongest unlearned model returned by exact unlearning, it is
difficult to discern the unlearning effect under existing _black-box auditing_
tools, rendering it hard to determine the success of unlearning.

Motivated by the above state of affairs, we introduce a novel unlearning
method called _Alignment Calibration (AC)_ that is specifically tailored for
contrastive learning. To approximate exact unlearning, our approach takes the
properties of the InfoNCE loss in contrastive learning and the goals of
unlearning into consideration and introduces an additional positive alignment
calibration term during optimization. Moreover, we propose negative alignment
calibration to provide extra visual auditing tools for data owners to validate
the effect of unlearning. Figure 1 visualize the advantages of _AC_ : enabling
easy auditing without sacrificing the goals of unlearning.

Finally, we empirically compare baseline methods with our _Alignment
Calibration_ algorithms on unlearning models pre-trained on SimCLR [6] MoCo
[17], and CLIP [32]. Under various unlearning settings (e.g., the fraction of
the unlearning dataset) and evaluation metrics, _AC_ consistently outperforms
the baseline methods, especially under unlearn auditing, validating the
benefits of our method. In summary, we make the following contributions:

Figure 1: An overview of the unlearning effect of our _Alignment Calibration_
algorithm under unlearn auditing. Upon receiving an unlearning request, our
method strategically and efficiently modifies the pre-trained encoder $g$ to
an unlearned encoder $\mathbf{g}$. Our method enjoys three properties: (1)
preserving model utility for downstream tasks on the retain dataset; (2)
enlarging the forgetting score FS of unlearned set in the feature space; (3)
enabling easy visual auditing with the Alignment Matrix.

  * •

We propose the MUC framework that considers existing methods and various
evaluation tools in contrastive learning, including _white-box evaluation_ and
_black-box auditing_.

  * •

Motivated by insufficiencies of existing unlearning algorithms and auditing
tools, we propose the novel _Alignment Calibration_ method that satisfies both
model owners and data owners.

  * •

Our experiments initiate the evaluation of existing machine unlearning methods
for contrastive learning and confirm the superiority of our new methods.

##  2 Background and Related Work

We first provide background and related work on contrastive learning and
machine unlearning.

#### Contrastive Learning and Self-supervised Learning

Contrastive learning learns general representations by contrasting sample
pairs (usually without labels), which analytically benefits the downstream
applications [34, 40]. Popular contrastive learning methods such as
Contrastive Predictive Coding (CPC) [31], SimCLR [6], and MoCo [17] employ the
InfoNCE loss to enforce the contrast between positive and negative pairs.
Other variants of the InfoNCE-based loss are also widely applied, e.g.,
$f$-MICL [27], Alignment and Uniformity [43], and Pearson $\chi^{2}$
divergence [41]. This contrastive training diagram is also applied to the
context of multimodal learning, where images and texts are formed as pairs,
e.g., in CLIP [32]. There exist other self-supervised learning methods that
also learn representations [14, 7, 16, 4]. In this paper, we mainly focus on
developing unlearning recipes for contrastive learning methods, especially
SimCLR, MoCo, and CLIP.

Specifically, contrastive learning usually applies the InfoNCE loss to learn a
representation $g$. Given a probability measure $p$, we define the density of
_positive pairs_ sampled from $p$ as $p^{+}$, _i.e._ , two samples with
similar feature embeddings joint distribution; and the density of _negative
pairs_ as $p^{\times}$. Specifically, one minimizes the loss below as the
objective:

$\displaystyle{\mathcal{L}}_{\texttt{InfoNCE}}=-\mathop{\mathbb{E}}_{(x,y)\sim
p^{+}}s(x^{g},y^{g})+\mathop{\mathbb{E}}_{x\sim
p}\log\mathop{\mathbb{E}}_{y\sim p}\exp(s(x^{g},y^{g})),$ (1)

where $s$ is the cosine similarity after normalization with a temperature
parameter, and $x^{g},y^{g}$ are the features extracted by a given encoder
$g$, respectively. The above contrastive learning (pre-training) scheme learns
a general encoder $g$ (image and text encoders for CLIP). Such a (fixed) $g$
can be utilized with an additional linear head or shallow models for
downstream tasks. In this paper, we mainly consider linear probing, where $g$
is used for the classification of the same dataset with pretraining. Notably,
we consider unlearning during the pretraining phase only.

#### Machine Unlearning

_For Supervised Learning_ : Machine unlearning (MU) [3] requires an algorithm
to revert to the state that specific data points are never trained on. While
exact unlearning (e.g., retraining the model entirely on the retaining
dataset) provides a reliable solution, the additional computation requirement
is also tremendous. [2] propose a sharding-based algorithm in a weaker
unlearning model, while other methods explore a probabilistic notion of
unlearning [12, 15, 30, 42, 36, 20, 5, 48, 10, 37].

_For Generative Models:_ MU methods are applied to diffusion models to avoid
copyright infringement and inappropriate image generation [11, 47, 18, 23].
For large language models, MU is applied as a model-editing [45] tool to
enable forgetting on certain training texts [29, 28, 21, 9, 46, 19]. In this
paper, we focus on MU on self-supervised learning, specifically, contrastive
learning methods, which differs from the above two cases in both unlearning
settings and frameworks, which we specify in the following section.

##  3 Machine Unlearning for Contrastive Learning (MUC)

In this section, we specify the problem setting of machine unlearning for
contrastive learning, introduce direct adaptations of existing methods, and
propose evaluation metrics.

###  3.1 Problem Settings

#### Setting and notation:

Suppose a contrastive learning model $g$ with parameter $\mathbf{w}$ is
obtained by training on a training set ${\mathcal{D}}_{\texttt{train}}$. After
$g$ is deployed, unlearning requests may be made to remove part of the
training set, which we denote as ${\mathcal{D}}_{\texttt{unlearn}}$. The goal
here is to acquire a new set of parameters $\mathbf{w}_{u}$, which amounts to
being trained on a retained dataset
${\mathcal{D}}_{\texttt{retain}}={\mathcal{D}}_{\texttt{train}}\backslash{\mathcal{D}}_{\texttt{unlearn}}$
($\backslash$ denotes removal here). Aside from the straightforward and
expensive “retraining from scratch” recipe, unlearning methods aim to achieve
the same goal approximately with improved efficiency. Additionally, we
introduce the notion of the model owner who receives the unlearning request,
and the data owners who wish to remove data as two parties involved in
unlearning. In practical scenarios, consists of a group of individuals
{i}${}_{i=1}^{N}$, who may not know the existence of each other, but
participate in unlearning at the same time.

#### Goal of unlearning:

❶ From the owner ’s perspective, there are two major considerations:
preserving the model performance on ${\mathcal{D}}_{\texttt{retain}}$ while
eliminating the effects of training on ${\mathcal{D}}_{\texttt{unlearn}}$. We
discuss how to measure such effects and provide evidence that these two
criteria may not constitute the success of unlearning in contrastive learning
in Section 3.3; ❷ In contrast, an individual data owner i only has limited
access to his/her own unlearning subset
${\mathcal{D}}_{\texttt{unlearn}}^{i}$, the output of the encoder before/after
unlearning. i wishes to observe different and desired behaviors of
${\mathcal{D}}_{\texttt{unlearn}}^{i}$ on $\mathbf{w}_{u}$ compared with that
of $\mathbf{w}$, which we specify in Section 3.3.

###  3.2 Adapting existing methods to MUC

To acquire $\mathbf{w}_{u}$ from the pre-trained weights $\mathbf{w}$
efficiently, we first adapt some existing unlearning methods designed for
supervised unlearning to contrastive unlearning. Due to the lack of labels in
contrastive learning pre-training, some approaches cannot be directly applied.
For example, random labeling [13, 10] relies on flipping the labels of the
unlearn data; boundary unlearning [5] expands or shrinks the decision
boundary, which does not exist in our context. In contrast, some other
unlearning methods can be tailored to our settings. Specifically, we adapt the
following methods:

  * •

_Retraining_ performs exact unlearning by training the retain dataset from
scratch via minimizing Equation 1;

  * •

_Fine-tuning_ [13] updates the pre-trained model for several epochs on the
retain dataset only;

  * •

_Gradient Ascent_ [13, 30, 39] reversely maximize the Equation 1 on the
unlearn dataset;

  * •

_NegGrad_ [24] jointly minimize and maximize Equation 1 on the retain data and
unlearn data;

  * •

_$l_{1}$ -Sparsity_ [22] propose to regularize the $l_{1}$-norm of model
parameters based on fine-tuning.

The above methods manifest straightforward adaptations of unlearning from
supervised learning to contrastive learning by changing the cross entropy loss
to the InfoNCE loss in Equation 1. In Section 5, we show these adaptations are
mediocre approximations of exact unlearning, which motivates us to design new
unlearning methods specifically for contrastive learning.

###  3.3 Evaluating an unlearned model

Suppose the model owner has obtained the unlearned model $\mathbf{w}_{u}$,
next, we discuss the evaluation metrics on assessing its performance.

#### For the model owner :

To explicitly measure whether the effect of
${\mathcal{D}}_{\texttt{unlearn}}^{i}$ has been removed by $\mathbf{w}_{u}$,
previous works set retraining from scratch as the exact unlearning baseline
for evaluating probabilistic unlearning methods. Note that the comparison with
retraining exhibits _white-box evaluation_ to choose the optimal unlearning
method from a pool of algorithms. Suppose a retrained (and converged) model
has parameter $\mathbf{w}_{r}$, evaluation metrics can be established by
comparing various behaviors of $\mathbf{w}_{u}$ and $\mathbf{w}_{r}$ with
respect to:

  * •

_Encoder-level metrics._ ❶ Forgetting Score: we propose a Forgetting Score
(FS) by directly adapting the memorization score in evaluating data
attribution in [44] to unlearning. FS measures the quantity of forgetting
${\mathcal{D}}_{\texttt{unlearn}}$ by comparing the alignment loss through the
features returned by model parameter before and after unlearning:

$\displaystyle\texttt{FS}\coloneqq\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{u}}^{+}}s(x^{g},y^{g})-\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{u}}{}^{+}}s(x^{\mathbf{g}},y^{\mathbf{g}}),$ (2)

where $p_{\texttt{u}}$ is the density of ${\mathcal{D}}_{\texttt{unlearn}}$,
$g$ and $\mathbf{g}$ are models before/after unlearning, and $s$ is the cosine
similarity.
❷ Membership Inference Attacks: EncoderMI [26] proposed an alignment-based
membership inference attack for self-supervised encoders. It extracts
membership information from the embedded features to distinguish whether input
data is included in the encoder training set. Following the implementation of
[22, 10], we evaluate the attack success rate (ASR) on the unlearn dataset
${\mathcal{D}}_{\texttt{unlearn}}$ and denote it by encoder membership
inference attack (EMIA) efficacy. We compare EMIA on
${\mathcal{D}}_{\texttt{unlearn}}$ before and after unlearning. See Section
A.3 for details of EMIA.

  * •

_Downstream-level metrics._ To quantify the model performance before/after
unlearning, we perform linear probing, _i.e._ , image classification on the
same (labeled) dataset for unimodal contrastive learning. Given the encoder
$g$ with unlearned parameters $\mathbf{w}_{u}$, we train an additional linear
head on top of the fixed $g$ to obtain a classifier. Next we evaluate: ❶
Accuracies: we evaluate retain accuracy (RA) on
${\mathcal{D}}_{\texttt{retain}}$, test accuracy on
${\mathcal{D}}_{\texttt{test}}$ (TA), and unlearn accuracy on
${\mathcal{D}}_{\texttt{unlearn}}$ (UA). For a good unlearning algorithm, the
above three measurements should be close to that of the retrained model
$\mathbf{w}_{r}$, with a common pattern of
$\text{{UA}}\approx\text{{TA}}<\text{{RA}}$; ❷ Membership Inference Attacks:
similarly to EMIA, we implement a confidence-based membership inference attack
[22, 10, 38] on the entire network (encoder and linear head) and report
classifier membership inference attack efficacy (CMIA). We compare CMIA before
and after unlearning. See Section A.3 for details.

After choosing the optimal unlearning algorithm, the model owner generates the
unlearned model $\mathbf{g}$ as a response to the unlearning request made by
data owners. However, it is impossible for the data owners to perform the same
_white-box evaluations_ , thus requiring the following verifications:

#### For the data owners (Unlearn Auditing) :

To validate the unlearning effect from the data owners’ perspective, we
further propose the notion of _unlearn auditing_ in the context of contrastive
learning. Note that this process is _black-box auditing_ as an individual data
owner i can only observe the input ${\mathcal{D}}_{\texttt{unlearn}}^{i}$ and
the output of the encoder before/after unlearning. Specifically, i cannot
train shadow models with ${\mathcal{D}}_{\texttt{unlearn}}^{i}$ alone to
perform MIAs; and cannot obtain TA or RA to quantify performance.
Additionally, due to the lack of the retrain baseline $\mathbf{w}_{r}$, it is
impossible to compare the forgetting score. To this end, the only auditing
tool is the forgetting score FS on ${\mathcal{D}}_{\texttt{unlearn}}$, which
can be calculated with Equation 2. However, we argue this auditing is _neither
sufficient nor reliable_ and we use a simple empirical example to validate
this claim:

_Exact unlearning on MoCo_ [17]: We perform exact unlearning (i.e.,
retraining) to forget 4500 training images of CIFAR-10 (randomly chosen) on
MoCo (ResNet-18). We calculate the forgetting score FS for every unlearn
sample and calculate the mean $\mu$ and the standard derivation $\sigma$
across the 4500 unlearning images. We obtain $\mu=0.025,\sigma=0.081$.

Here we observe the large standard derivation $\sigma=0.081=3.24\times\mu$
makes the current auditing largely unreliable. For individual data owners, if
the unlearn subset size $|{\mathcal{D}}_{\texttt{unlearn}}^{i}|$ is small, its
corresponding sample-wise FS is likely to be biased and the average could
fluctuate around 0 and suggesting little forgetting. This could lead to the
belief in “haven’t performed unlearning” from the data owner’s side, thus
rejecting the exact unlearned model!

This example reveals the insufficiency of both the existing unlearning methods
(even exact unlearning) and unlearn auditing tools in contrastive learning.
Motivated by this state of affairs, we are urged to design new unlearning
algorithms for the model owners and advanced auditing tools for data owners in
contrastive learning in the next section, which would benefit both parties in
engaging the unlearning procedure.

##  4 Alignment Calibration

In Section 3, we introduce the notion of MUC, adapt existing unlearning
methods, and propose evaluation metrics. However, we observe insufficiencies
in the current approaches in two aspects: model owners _do not possess an
effective unlearner_ ; data owners _lack reliable auditing tools._ In this
section, we propose a novel method called Alignment Calibration (AC),
providing a powerful unlearner for contrastive learning and a practical visual
auditing mechanism. _AC_ addresses concerns from both sides and we specify in
a two-step manner below.

###  4.1 Tailored objective for MUC

We first introduce a more effective unlearner for model owners . Recall the ’s
goal of unlearning: preserving the model utility on
${\mathcal{D}}_{\texttt{retain}}$ while revoking the effects of training on
${\mathcal{D}}_{\texttt{unlearn}}$. For the retain dataset
${\mathcal{D}}_{\texttt{retain}}$, we minimize the InfoNCE loss in Equation 1
to achieve reasonable downstream performance after unlearning:

$\displaystyle{\mathcal{L}}_{\texttt{retain}}=$
$\displaystyle-\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{r}}^{+}}s(x^{g},y^{g})+\mathop{\mathbb{E}}_{x\sim
p_{\texttt{r}}}\log\mathop{\mathbb{E}}_{y\sim
p_{\texttt{d}}}\exp(s(x^{g},y^{g})),$ (3)

where $p_{\texttt{r}}$ is the density of ${\mathcal{D}}_{\texttt{retain}}$ and
$p_{\texttt{d}}$ is the density of ${\mathcal{D}}_{\texttt{train}}$.

For the unlearn dataset ${\mathcal{D}}_{\texttt{unlearn}}$, revoking the
effects of training amounts to achieving the following goals upon evaluation
in Section 3.3:

  * •

(_Encoder-level_) Enlarging forgetting on ${\mathcal{D}}_{\texttt{unlearn}}$ :
recall that in Equation 2 the forgetting score FS is measured by the
difference between feature similarity on ${\mathcal{D}}_{\texttt{unlearn}}$
before/after unlearning with pre-trained model $g$ and unlearned model
$\mathbf{g}$. As the first term is fixed during unlearning, increasing FS is
equal to minimizing the second positive alignment term. For this purpose, we
explicitly perform such minimization in our objective function and call it
_positive alignment calibration_.

  * •

(_Downstream-level_) $\text{{UA}}\approx\text{{TA}}<\text{{RA}}$: enlarging FS
alone may also hurt the overall downstream performance on the unlearned model
$\mathbf{g}$. To obtain reasonable UA and TA, we find it beneficial to
maintain the term for negative pairs in contrastive learning, such that for
${\mathcal{D}}_{\texttt{unlearn}}$, we minimize:

$\displaystyle{\mathcal{L}}_{\texttt{unlearn}}=$
$\displaystyle\underbrace{\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{u}}^{+}}s(x^{g},y^{g})}_{\textit{positive alignment
calibration}}+\underbrace{\mathop{\mathbb{E}}_{x\sim
p_{\texttt{u}}}\log\mathop{\mathbb{E}}_{y\sim
p_{\texttt{d}}}\exp(s(x^{g},y^{g}))}_{\textit{performance preserving}},$ (4)

where $p_{\texttt{u}}$ is the density of ${\mathcal{D}}_{\texttt{unlearn}}$.

###  4.2 Calibration under unlearn auditing

#### Auditing beyond FS:

Recall that in Section 3.3, we show that the forgetting score FS is not a
sufficient nor reliable evaluation for unlearning success. Here we introduce
an additional auditing tool: given ${\mathcal{D}}_{\texttt{unlearn}}$ and the
models before unlearning $g$, data owners can easily obtain the feature
vectors with two different data augmentations:
$\mathbf{x}^{g}=\\{x_{i}^{g}\\}_{i=1}^{|{\mathcal{D}}_{\texttt{unlearn}}|}$
and
$\mathbf{y}^{g}=\\{y_{j}^{g}\\}_{j=1}^{|{\mathcal{D}}_{\texttt{unlearn}}|}$.
Then an $\texttt{Alignment Matrix}:$
$\texttt{AM}(\mathbf{x}^{g},\mathbf{y}^{g})$ can be easily acquired by
calculating the pairwise similarity between the two vectors. See Figure 3(a)
for some visualizations of AM in the format of heatmaps. Similarly, can obtain
$\texttt{AM}(\mathbf{x}^{\mathbf{g}},\mathbf{y}^{\mathbf{g}})$ after
unlearning and an additional $\texttt{Alignment Gap Matrix}:$
$\texttt{AGM}=\texttt{AM}(\mathbf{x}^{g},\mathbf{y}^{g})-\texttt{AM}(\mathbf{x}^{\mathbf{g}},\mathbf{y}^{\mathbf{g}})$
. The heatmaps of AM and AGM provide auditing tools beyond FS and allow to
visualize the model change through unlearning by looking at the temperature of
the graphs. Notably, the elements on the diagonal of AGM also visualize
sample-wise forgetting scores.

#### Taking auditing into account for unlearning:

The additional auditing tools enable the model owners to design an algorithm
that allows data owners to clearly visualize the effect caused by unlearning
(_i.e._ , through AM or AGM) without sacrificing the goal of unlearning.

We provide a simple solution to improve existing unlearning methods. As we
have explicitly calibrated the alignment of positive pairs of
${\mathcal{L}}_{\texttt{unlearn}}$ in Equation 4, it suffices to adjust that
of negative pairs (within ${\mathcal{D}}_{\texttt{unlearn}}$) to a larger
value to enlarge the model differences in AM. Specifically, we update the
unlearn loss in Equation 4 with _negative alignment calibration_ :

$\displaystyle{\mathcal{L}}_{\texttt{unlearn}}=$
$\displaystyle-\alpha\cdot\underbrace{\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{u}}^{\times}}s(x^{g},y^{g})}_{\textit{negative alignment
calibration}}+\beta\cdot\underbrace{\mathop{\mathbb{E}}_{(x,y)\sim
p_{\texttt{u}}^{+}}s(x^{g},y^{g})}_{\textit{positive alignment
calibration}}+\gamma\cdot\underbrace{\mathop{\mathbb{E}}_{x\sim
p_{\texttt{u}}}\log\mathop{\mathbb{E}}_{y\sim
p_{\texttt{d}}}\exp(s(x^{g},y^{g}))}_{\textit{performance preserving}},$

where $\alpha,\beta,\gamma$ are tunable parameters to adjust the strength of
each component. In summary, we write the complete objective of _Alignment
Calibration (AC)_ :

$\displaystyle{\mathcal{L}}_{\texttt{retain}}+\varepsilon\cdot{\mathcal{L}}_{\texttt{unlearn}},$
(5)

where
$\varepsilon=|{\mathcal{D}}_{\texttt{unlearn}}|/|{\mathcal{D}}_{\texttt{retain}}|$
varies by the size of the unlearn set. In the next section, we will show _AC_
not only achieves state-of-the-art performance upon model owners’ evaluations
but can also easily pass data owners’ visual auditing on unlearning.

##  5 Experiment

Recall that we made several claims in Section 3 and Section 4: ❶ Existing
methods are ordinary unlearners under _white-box evaluations_ and our _AC_
algorithm approaches exact unlearning in this regard; ❷ Under MUC, the
forgetting score FS is a weak auditing tool, whereas _AC_ introduces alignment
matrices AM for visual auditing and exhibits clear evidence for unlearning. In
this section, we evaluate baseline methods and _AC_ following the above steps
and present _white-box evaluation_ by model owners and _black-box auditing_ by
data owners .

###  5.1 Experimental Setup

#### Data and models.

For unimodal contrastive unlearning, we perform experiments on CIFAR-10 and
SimCLR/MoCo algorithms with the ResNet-18 backbone. We randomly forget 10/50%
training data from a pre-trained encoder. For multimodal contrastive
unlearning, we evaluate CLIP [32] on an Image-Text paired dataset called MS-
COCO [25], which contains $\sim$120K images and $\sim$600K captions. We
perform unlearning on 10% randomly select image-text pairs.

#### White-box Evaluation:

Following Section 3.3, we use FS and EMIA for encoder-level evaluation, and
use CMIA, RA, TA, and UA for downstream-level evaluation after performing
linear probing for SimCLR/MoCo experiments. For the evaluation of CLIP, we
measure the image-text cosine similarity of the retain dataset and unlearn
dataset due to the lack of suitable downstream tasks. Across all experiments,
we compare each unlearning method with the exact unlearning (retraining)
baseline and report the differences across all metrics. We also report the
running time efficacy (RTE) of unlearning methods to evaluate efficiency.

#### Black-box Auditing:

For data owners, calculating FS on their own unlearning subset is also
possible as basic auditing. Moreover, we use the Alignment Matrix (AM) and
Alignment Gap Matrix (AGM) introduced in Section 4 for visual auditing on MoCo
and CLIP.

#### Unlearning Algorithms:

We evaluate Retrain, Fine-Tune, Gradient Ascent, NegGrad, and $l_{1}$-Sparsity
as baselines for MUC. Our _Alignment Calibration_ method updates the pre-
trained encoder for the same number of epochs as FineTune, NegGrad, and
$l_{1}$-Sparsity. For simplicity, we set $\alpha=\gamma=1$ if not otherwise
stated and we tune $\beta$ for the best performance. Implementation details of
the above methods are described in Section A.2.

Table 1: Unlearning performance of different methods on randomly forgetting
10% of CIFAR-10 training data under various metrics. The performance gaps
between retraining and other methods are shown in the parenthesis. We report
the average gap (Avg. Gap) over these 5 metrics. The results are obtained by
averaging over 5 random trials.

Methods | EMIA | RA | TA | UA | CMIA |  Avg. Gap (%) |  RTE (mins)
---|---|---|---|---|---|---|---
MoCo
Retrain | 49.72 | 89.54 | 87.76 | 88.42 | 34.38 | - | 109.47
Fine-Tune | 50.15 (0.43) | 88.34 (1.20) | 86.46 (1.30) | 87.59 (0.83) | 29.42 (4.96) | 1.74 | 1.42
Grad. Ascent | 44.95 (4.77) | 89.92 (0.38) | 88.28 (0.52) | 89.76 (1.34) | 28.53 (5.85) | 2.57 | 0.17
NegGrad | 48.43 (1.29) | 89.25 (0.29) | 87.35 (0.40) | 88.58 (0.16) | 28.89 (5.49) | 1.53 | 1.70
$l_{1}$-Sparsity | 49.38 (0.34) | 88.56 (0.98) | 86.91 (0.84) | 88.12 (0.30) | 29.91 (4.47) | 1.39 | 1.43
Ours | 50.28 (0.56) | 89.14 (0.40) | 87.24 (0.52) | 88.20 (0.22) | 31.50 (2.88) | 0.92 | 1.87
SimCLR
Retrain | 48.11 | 90.87 | 88.94 | 89.68 | 38.87 | - | 151.77
Fine-Tune | 47.72 (0.39) | 89.38 (1.49) | 87.26 (1.68) | 88.93 (0.75) | 30.71 (8.16) | 2.49 | 1.93
Grad. Ascent | 41.48 (6.63) | 91.26 (0.40) | 89.55 (0.61) | 91.11 (1.43) | 29.36 (9.50) | 3.71 | 0.19
NegGrad | 49.56 (1.46) | 89.10 (1.77) | 87.23 (1.71) | 89.07 (0.61) | 29.97 (8.89) | 2.89 | 2.34
$l_{1}$-Sparsity | 48.44 (0.33) | 90.59 (0.28) | 88.56 (0.38) | 90.44 (0.75) | 30.61 (8.26) | 2.00 | 1.96
Ours | 48.64 (0.53) | 90.24 (0.63) | 88.06 (0.88) | 89.24 (0.44) | 33.12 (5.75) | 1.65 | 3.00

Table 2: Unlearning performance of various methods on randomly forgetting 50%
of CIFAR-10 training data. The results are averaged over 5 random trials.

Methods | EMIA | RA | TA | UA | CMIA | Avg. Gap | RTE
---|---|---|---|---|---|---|---
MoCo
Retrain | 55.95 | 85.98 | 83.55 | 83.98 | 46.66 | - | 66.71
Fine-Tune | 47.72 (5.97) | 89.38 (1.90) | 87.26 (2.12) | 88.93 (2.67) | 30.71 (14.31) | 5.39 | 0.86
Grad. Ascent | 41.48 (13.06) | 91.26 (3.53) | 89.55 (4.25) | 91.11 (5.01) | 29.36 (14.82) | 8.13 | 0.45
NegGrad | 49.56 (1.45) | 89.10 (2.94) | 87.23 (3.40) | 89.07 (3.39) | 29.97 (6.02) | 3.44 | 1.66
$l_{1}$-Sparsity | 48.44 (3.76) | 90.59 (4.38) | 88.56 (3.36) | 90.44 (3.20) | 30.61 (8.24) | 4.59 | 0.87
Ours | 48.64 (0.93) | 90.24 (0.30) | 88.06 (0.26) | 89.24 (0.26) | 33.12 (8.27) | 2.00 | 1.84
SimCLR
Retrain | 53.37 | 87.23 | 85.16 | 85.69 | 49.30 | - | 89.74
Fine-Tune | 45.90 (7.47) | 87.88 (0.65) | 85.5 (0.34) | 87.23 (1.54) | 35.44 (13.85) | 4.77 | 1.17
Grad. Ascent | 42.23 (11.13) | 90.52 (3.28) | 88.61 (3.45) | 90.45 (4.77) | 33.28 (16.02) | 7.73 | 0.59
NegGrad | 55.70 (2.33) | 83.98 (3.25) | 82.15 (3.01) | 83.80 (1.89) | 33.67 (15.62) | 5.22 | 2.32
$l_{1}$-Sparsity | 46.51 (6.86) | 89.84 (2.60) | 87.75 (2.59) | 89.48 (3.80) | 35.38 (13.91) | 5.95 | 1.19
Ours | 47.12 (6.25) | 86.11 (1.13) | 83.92 (1.24) | 85.24 (0.45) | 37.57 (11.72) | 4.16 | 3.07

###  5.2 Unlearning Performance under White-box Evaluation

We first provide empirical evidence for model owners on choosing a suitable
unlearning method with superb efficiency and effectiveness. ❶ Unimodal
contrastive learning: we present our evaluation under EMIA, RA, TA, UA and
CMIA in Table 1 and Table 2 for unlearning 10/50% of CIFAR-10 training set and
FS (CIFAR-10, MoCo) for 10/50% separately in Table 3 due to different scales.
For both SimCLR and MoCo, our proposed _Alignment Calibration (AC)_ method
achieves the lowest average performance gap over EMIA, RA, TA, UA, and CMIA.
In terms of unlearning efficiency, our methods only introduce a slight
overhead. Additionally, our method achieves the lowest FS gap compared to
retraining. In Appendix B, we also report the results on CIFAR-100, in which
our methods consistently achieve the best performance. ❷ Multi-modal
contrastive learning: in Table 4, we again observe that our method is the best
approximator of exact unlearning by evaluating the image-text cosine
similarity.

Table 3: Forgetting score (FS) of methods for CIFAR-10 and MoCo. FS gaps are
computed between Retrain and other methods.

Methods | 10% | 50%
---|---|---
FS | Gap | FS | Gap
Retrain | 0.0266 | - | 0.0604 | -
Fine-Tune | 0.0393 | 0.0127 | 0.0423 | 0.0180
Grad.Ascent | 0.0005 | 0.0262 | 0.0007 | 0.0596
NegGrad | 0.0205 | 0.0061 | 0.1002 | 0.0398
$l_{1}$-Sparsity | 0.0216 | 0.0050 | 0.0408 | 0.0195
Ours | 0.0259 | 0.0007 | 0.0672 | 0.0068

Figure 2: Negative alignment of 4500 unlearn samples (10%) and MoCo and
CIFAR-10. The error bar is the standard deviation.

Table 4: Performance of methods on randomly forgetting 10% of MS-COCO data
from a pre-trained CLIP. We report the image-text cosine similarity of the
retain dataset and unlearn dataset respectively, as well as the average
absolute gap from Retrain. The results are averaged over 3 random trials.

Dataset | Pre-train | Retrain | Fine-Tune | Grad. Ascent | NegGrad |  $l_{1}$-Sparsity | Ours
---|---|---|---|---|---|---|---
Retrain | 62.09 | 62.47 (0) | 58.67 (3.80) | 61.96 (0.51) | 58.57 (3.90) | 57.70 (4.76) | 60.75 (1.72)
Unlearn | 62.08 | 49.84 (0) | 54.75 (4.91) | 62.03 (12.19) | 49.19 (0.65) | 53.54 (3.70) | 51.23 (1.39)
Avg. Gap | - | 0 | 4.35 | 6.35 | 2.28 | 4.23 | 1.56

###  5.3 Black-box Auditing

Figure 3: Alignment Matrices and Alignment Gap Matrix on 8 random images in
unlearn dataset of CIFAR-10 (MoCo). The task is forgetting 10% of training
data.

Motivated by the insufficiency of auditing with the FS score (positive
alignment), we propose to apply the Alignment Matrix (AM) and Alignment Gap
Matrix (AGM) in Section 4. AM and AGM naturally introduce additional
quantification of negative alignment. In Figure 2, we report the negative
alignment value (mean and stand deviation of pairwise similarity on negative
samples in AGM) of 4500 unlearn samples and observe our method _AC_ exhibits a
more significant unlearning effect under such auditing. For individual data
owners i, the size of his/her subset $|{\mathcal{D}}_{\texttt{unlearn}}^{i}|$
may be small. Thus we provide additional qualitative results for visual
auditing: we randomly select 8 samples from ${\mathcal{D}}_{\texttt{unlearn}}$
to simulate the budget of i. We construct the AM (before/after unlearning with
_AC_) and AGM for this small set and plot their heatmaps in Figure 3 and
observe the apparent effect of unlearning. We provide additional AGM for other
methods and CLIP unlearning in Figure 5 and Figure 6 in Section B.2, where
_AC_ consistently exhibits the best performance under visual auditing.

###  5.4 Ablation Study

Figure 4: The effect of $\alpha$ and $\beta$ on the forgetting score ratio
between Retrain and our _Alignment Calibration_ , _i.e._ , FS(RT):FS(AC). The
unlearning task is to forget 10% of CIFAR-10 training data from a MoCo
encoder.

#### Influence of negative alignment calibration:

In Equation 5, the coefficient $\alpha$ controls the intensity of maximizing
the negative alignment on unlearn data. To explore the effect of negative
alignment calibration in the unlearning task, we fix $\beta$ and adjust the
$\alpha$ while keeping $\gamma=\alpha$ for simplicity. Figure 4 (orange bars)
reports the ratio between the forgetting score FS of Retrain and _AC_. When
$\alpha$ increases, the ratio decreases, indicating that the resulting model
forgets more information about unlearn data. The ratio equaling 1 denotes FS
of _AC_ equals to that of Retrain. Furthermore, we consider a more extreme
case of $\alpha=0$, representing no negative calibration. In Table 5, without
negative calibration, the average gap over metrics is larger than the standard
_AC_ by 0.33/2.12% (comparing columns “w/o+w/” with “w/+w/”) for 10/50%
forgetting tasks, suggesting this additional term not only benefits the data
owner for unlearn auditing but also improves the unlearn performance for the
model owner.

#### Influence of positive alignment calibration.

In Equation 5, the coefficient $\beta$ controls the intensity of minimizing
the positive alignment on the unlearn data. In Figure 4 (blue columns), we fix
$\alpha=\gamma=1$ and vary $\beta$ from 0 to 16. The forgetting score ratio
decreases by increasing $\beta$ and approximately reaches 1.0 in the range of
[12,16]. In Table 5, the positive alignment calibration term enhances the
unlearning performance from 1.95/3.75% to 0.92/2% (comparing columns “w/+w/o”
with “w/+w/”) for the 10/50% forgetting task regarding the average gap.

Table 5: Ablation study on positive and negative calibration in Equation 5 regarding the average gap over metrics on CIFAR-10 and MoCo with forgetting ratio 10/50%. For example, “w/o + w/” means _AC_ without negative calibration but with positive calibration.  Ratio | w/ + w/ | w/o + w/ | w/ + w/o | w/o + w/o
---|---|---|---|---
10% | 0.92 | 1.25 | 1.95 | 2.70
50% | 2.00 | 4.12 | 3.75 | 6.45

##  6 Conclusion

In this paper, we study the problem of machine unlearning for contrastive
learning pre-training (MUC). We establish the foundations on this line of
study by adapting existing unlearning methods and setting up baseline
evaluation metrics, including _white-box evaluation_ for model owners to
choose the optimal unlearning strategy, and _black-box auditing_ for data
owners to examine the effect of unlearning. Spotting the suboptimality of
existing unlearning methods and the insufficiency of current auditing tools,
we further propose our novel method called _Alignment Calibration_. Our
approach introduces a novel unlearning objective function to strategically
optimize towards the unlearning goal and enable straightforward visual
auditing. Empirically, our method achieves state-of-the-art performance on
unlearning tasks for both unimodal and multimodal contrastive learning. Our
paper initializes the study of machine unlearning in self-supervised learning
but only considers contrastive learning. We plan to extend our exploration of
unlearning towards other SSL methods in the future.

## Acknowledgement

YY gratefully acknowledges funding support from NSERC, the Ontario early
researcher program and the Canada CIFAR AI Chairs program. Resources used in
preparing this research were provided, in part, by the Province of Ontario,
the Government of Canada through CIFAR, and companies sponsoring the Vector
Institute.

## References

  * [1] Josh Achiam et al.  “Gpt-4 technical report” arXiv preprint arXiv:2303.08774, 2023
  * [2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie and Nicolas Papernot  “Machine unlearning”  In _2021 IEEE Symposium on Security and Privacy (SP)_ , 2021, pp. 141–159  IEEE  URL: https://ieeexplore.ieee.org/iel7/9519381/9519382/09519428.pdf
  * [3] Yinzhi Cao and Junfeng Yang  “Towards making systems forget with machine unlearning”  In _2015 IEEE symposium on security and privacy_ , 2015, pp. 463–480  IEEE
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski and Armand Joulin  “Emerging properties in self-supervised vision transformers”  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 9650–9660
  * [5] Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng and Chen Wang  “Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7766–7775  URL: http://openaccess.thecvf.com/content/CVPR2023/papers/Chen_Boundary_Unlearning_Rapid_Forgetting_of_Deep_Networks_via_Shifting_the_CVPR_2023_paper.pdf
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton  “A Simple Framework for Contrastive Learning of Visual Representations”  In _ICML_ , 2020  URL: http://proceedings.mlr.press/v119/chen20j.html
  * [7] Xinlei Chen and Kaiming He  “Exploring simple siamese representation learning”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15750–15758
  * [8] Xinlei Chen, Saining Xie and Kaiming He  “An empirical study of training self-supervised vision transformers”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9640–9649
  * [9] Ronen Eldan and Mark Russinovich  “Who’s Harry Potter? Approximate Unlearning in LLMs”  In _arXiv preprint arXiv:2310.02238_ , 2023  URL: https://arxiv.org/abs/2310.02238
  * [10] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei and Sijia Liu  “SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=gn0mIhQGNM
  * [11] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman and David Bau  “Erasing concepts from diffusion models”  In _arXiv preprint arXiv:2303.07345_ , 2023
  * [12] Antonio Ginart, Melody Guan, Gregory Valiant and James Y Zou  “Making ai forget you: Data deletion in machine learning”  In _Advances in neural information processing systems_ 32, 2019
  * [13] Aditya Golatkar, Alessandro Achille and Stefano Soatto  “Eternal sunshine of the spotless net: Selective forgetting in deep networks”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9304–9312  URL: https://openaccess.thecvf.com/content_CVPR_2020/papers/Golatkar_Eternal_Sunshine_of_the_Spotless_Net_Selective_Forgetting_in_Deep_CVPR_2020_paper.pdf
  * [14] Jean-Bastien Grill et al.  “Bootstrap your own latent-a new approach to self-supervised learning”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 21271–21284
  * [15] Chuan Guo, Tom Goldstein, Awni Hannun and Laurens Van Der Maaten  “Certified data removal from machine learning models”  In _arXiv preprint arXiv:1911.03030_ , 2019
  * [16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick  “Masked autoencoders are scalable vision learners”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16000–16009
  * [17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie and Ross Girshick  “Momentum contrast for unsupervised visual representation learning”  In _CVPR_ , 2020
  * [18] Alvin Heng and Harold Soh  “Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models”, 2023  arXiv:2305.10120 [cs.LG]
  * [19] Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu and Min Zhang  “Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2024, pp. 18252–18260  URL: https://arxiv.org/abs/2308.08090
  * [20] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri and James Zou  “Approximate data deletion from machine learning models”  In _International Conference on Artificial Intelligence and Statistics_ , 2021, pp. 2008–2016  PMLR  URL: https://proceedings.mlr.press/v130/izzo21a/izzo21a.pdf
  * [21] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran and Minjoon Seo  “Knowledge unlearning for mitigating privacy risks in language models”  In _arXiv preprint arXiv:2210.01504_ , 2022  URL: https://arxiv.org/abs/2210.01504
  * [22] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma and Sijia Liu  “Model Sparsity Can Simplify Machine Unlearning”  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023  URL: https://openreview.net/forum?id=0jZH883i34
  * [23] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang and Jun-Yan Zhu  “Ablating Concepts in Text-to-Image Diffusion Models”, 2023  arXiv:2303.13516 [cs.CV]
  * [24] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes and Eleni Triantafillou  “Towards Unbounded Machine Unlearning”  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023  URL: https://openreview.net/forum?id=OveBaTtUAT
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C Lawrence Zitnick  “Microsoft coco: Common objects in context”  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 2014, pp. 740–755  Springer  URL: https://cocodataset.org/
  * [26] Hongbin Liu, Jinyuan Jia, Wenjie Qu and Neil Zhenqiang Gong  “EncoderMI: Membership inference against pre-trained encoders in contrastive learning”  In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021, pp. 2081–2095  URL: https://dl.acm.org/doi/pdf/10.1145/3460120.3484749
  * [27] Yiwei Lu, Guojun Zhang, Sun Sun, Hongyu Guo and Yaoliang Yu  “$f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning”  In _Transactions on Machine Learning Research_ , 2023
  * [28] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn and Christopher D Manning  “Fast Model Editing at Scale”  In _International Conference on Learning Representations_ , 2022  URL: https://openreview.net/forum?id=0DcZxeWfOPt
  * [29] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning and Chelsea Finn  “Memory-based model editing at scale”  In _International Conference on Machine Learning_ , 2022, pp. 15817–15831  PMLR  URL: https://arxiv.org/abs/2206.06520
  * [30] Seth Neel, Aaron Roth and Saeed Sharifi-Malvajerdi  “Descent-to-delete: Gradient-based methods for machine unlearning”  In _Algorithmic Learning Theory_ , 2021, pp. 931–962  PMLR
  * [31] Aaron Oord, Yazhe Li and Oriol Vinyals  “Representation learning with contrastive predictive coding”  In _arXiv preprint arXiv:1807.03748_ , 2018
  * [32] Alec Radford et al.  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer  “High-resolution image synthesis with latent diffusion models”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10684–10695
  * [34] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak and Hrishikesh Khandeparkar  “A Theoretical Analysis of Contrastive Unsupervised Representation Learning”  In _ICML_ , 2019  URL: http://proceedings.mlr.press/v97/saunshi19a.html
  * [35] Christoph Schuhmann et al.  “Laion-5b: An open large-scale dataset for training next generation image-text models”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 25278–25294  URL: https://arxiv.org/abs/2210.08402
  * [36] Ayush Sekhari, Jayadev Acharya, Gautam Kamath and Ananda Theertha Suresh  “Remember what you want to forget: Algorithms for machine unlearning”  In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 18075–18086
  * [37] Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Tony Chen and Miao Xu  “Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=SIZWiya7FE
  * [38] Liwei Song and Prateek Mittal  “Systematic evaluation of privacy risks of machine learning models”  In _30th USENIX Security Symposium (USENIX Security 21)_ , 2021, pp. 2615–2632  URL: https://www.usenix.org/system/files/sec21fall-song.pdf
  * [39] Anvith Thudi, Hengrui Jia, Ilia Shumailov and Nicolas Papernot  “On the necessity of auditable algorithmic definitions for machine unlearning”  In _31st USENIX Security Symposium (USENIX Security 22)_ , 2022, pp. 4007–4022  URL: https://www.usenix.org/system/files/sec22-thudi.pdf
  * [40] Christopher Tosh, Akshay Krishnamurthy and Daniel Hsu  “Contrastive learning, multi-view redundancy, and linear models”  In _Algorithmic Learning Theory_ , 2021  PMLR
  * [41] Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency and Ruslan Salakhutdinov  “Self-supervised Representation Learning with Relative Predictive Coding”  In _ICLR_ , 2021
  * [42] Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi and Raman Arora  “Machine unlearning via algorithmic stability”  In _Conference on Learning Theory_ , 2021, pp. 4126–4142  PMLR
  * [43] Tongzhou Wang and Phillip Isola  “Understanding contrastive representation learning through alignment and uniformity on the hypersphere”  In _ICML_ , 2020  PMLR
  * [44] Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot and Franziska Boenisch  “Memorization in Self-Supervised Learning Improves Downstream Generalization”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=KSjPaXtxP8
  * [45] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen and Ningyu Zhang  “Editing Large Language Models: Problems, Methods, and Opportunities”  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023  URL: https://openreview.net/forum?id=NZZB3UGcd8
  * [46] Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples and Xiwei Xu  “Right to be forgotten in the era of large language models: Implications, challenges, and solutions”  In _arXiv preprint arXiv:2307.03941_ , 2023  URL: https://arxiv.org/abs/2307.03941
  * [47] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang and Humphrey Shi  “Forget-me-not: Learning to forget in text-to-image diffusion models”  In _arXiv preprint arXiv:2303.17591_ , 2023
  * [48] Qiuchen Zhang, Carl Yang, Jian Lou and Li Xiong  “Contrastive Unlearning: A Contrastive Approach to Machine Unlearning” arXiv preprint arXiv:2401.10458  In _arXiv preprint arXiv:2401.10458_ , 2024  URL: https://arxiv.org/html/2401.10458v1

##  Appendix A Experiment Details

###  A.1 Datasets

#### CIFAR-10/100.

Both datasets consist of 50K training images and 10K test images. All the
images are 32x32 colored. CIFAR-100 has 100 categories and CIFAR-10 has 10
categories. In unimodal contrastive learning, the augmentations for training
encoders include random resizing and cropping, random grayscale, random color
jitter, and horizontal flipping. We split the 50K training images into a
validation set of 5K images and a training set of 45K images. For example,
when the unlearning task is to forget 10% of training data, the unlearn
dataset ${\mathcal{D}}_{\texttt{unlearn}}$ has 4.5K images and the retain
dataset ${\mathcal{D}}_{\texttt{retain}}$ has 4.05K images.

#### MS-COCO.

COCO is a large-scale object detection, segmentation, and captioning dataset.
Its training set contains 118,287 images and 591,753 captions. Each image has
several objects and corresponds to at least 5 captions. Different from
unimodal contrastive learning which uses strong augmentations, CLIP employs
only resizing, center cropping and horizontal flipping to make images of
224x224 pixels.

###  A.2 Methods

#### MoCo and SimCLR:

For the Pre-Trained models, we train the encoder for 800 epochs using an SGD
optimizer with cosine-scheduled learning rate initialized at 0.06, momentum of
0.9, and weight decay of 0.0005. The Retrain method uses the same training
strategy as Pre-Training. For the Fine-Tune and NegGrad methods, we update the
pre-trained encoder for 10 epochs with a learning rate searched in [0.003,
0.03]. For the Gradient-Ascent method, we update the pre-trained encoder using
reversed stochastic gradient descent for 5 epochs with a learning rate
searched in [$10^{-6}$, $10^{-4}$]. For the $l_{1}$-Sparsity method, we set
the learning rate as 0.006 and implement $l_{1}$ regularization with a
coefficient searched in [$10^{-6}$, $10^{-3}$]. For our _Alignment
Calibration_ method, we update the pre-trained encoder for 10 epochs and
search the learning rate in [0.003, 0.03] and the tunable parameter $\beta$ in
[0, 20] for different unlearning tasks. If not otherwise stated, we adopt
$\alpha=\gamma=1$. For simplicity, in our reported results on CIFAR-10/100, we
use a learning rate of 0.006 for 10% forgetting, and 0.02 for 50% forgetting.
The linear probing stage trains a linear classifier head for 100 epochs using
an SGD optimizer with a cosine-scheduled learning rate initialized at 1.0, and
a momentum of 0.9. The batch size for both encoder and linear head training is
512.

#### CLIP:

For the Pre-Trained CLIP, we train the model for 35 epochs on 2 NVIDIA RTX
4090 GPUs using an AdamW optimizer with a warm-up cosine-scheduled learning
rate initialized at 5e-4 and momentum of 0.9. The total batch size is 256 (128
on each GPU). The Retrain method uses the same training strategy as Pre-
Training. For the Fine-Tune method, we update the pre-trained model for 8
epochs with a fixed learning rate searched in [5e-5,5e-4]. For the NegGrad
method, we update the pre-trained model for 8 epochs with a fixed learning
rate searched in [$10^{-5}$, $10^{-4}$]. For the Gradient Ascent method, we
update the pre-trained model for 4 epochs with a fixed learning rate searched
in [5e-6, 5e-4]. For the $l_{1}$-Sparsity method, we update the pre-trained
model for 8 epochs with a learning rate of 0.0005 and a regularization
coefficient searched in [$10^{-9}$, $10^{-4}$]. For our _Alignment
Calibration_ method, we update the pre-trained model for 8 epochs with a fixed
learning rate of 0.0002. We search $\alpha=\gamma$ in [0.5, 1] and $\beta$ in
[0, 1].

###  A.3 Evaluation

#### CMIA efficacy.

Given an unlearned encoder $\mathbf{g}$, we execute linear probing on it and
denote the whole classifier by $f$. Following the implementation of [22, 10],
we evaluate the attack successful rate (ASR) on the unlearn dataset
${\mathcal{D}}_{\texttt{unlearn}}$ of a confidence-based membership inference
attack [38] $f$. The formal definition of CMIA efficacy is given by:

$\displaystyle\texttt{CMIA-
Efficacy}\coloneqq\frac{TN_{\texttt{CMIA}}}{|{\mathcal{D}}_{\texttt{unlearn}}|},$
(6)

where $TN_{\texttt{CMIA}}$ is the number of true negatives predicted by the
CMIA attack.

#### EMIA efficacy.

We implement the alignment-based EncoderMI-T attack [26] in an adapted white-
box setting. Given an unlearned encoder $g$ with its retain dataset
${\mathcal{D}}_{\texttt{retain}}$ and test dataset
${\mathcal{D}}_{\texttt{test}}$, we denote ${\mathcal{D}}_{\texttt{non-
member}}\coloneqq{\mathcal{D}}_{\texttt{test}}$ sample a subset
${\mathcal{D}}_{\texttt{member}}$ of ${\mathcal{D}}_{\texttt{retain}}$ such
that $|{\mathcal{D}}_{\texttt{non-member}}|={\mathcal{D}}_{\texttt{member}}$.
For each data in ${\mathcal{D}}_{\texttt{non-member}}$ and
${\mathcal{D}}_{\texttt{member}}$, we first augment it 10 times and compute
features of these 10 views via $g$. Then compute the cosine similarity between
each pair of features, _i.e._ 45 pairs, and then take the average of these
similarity values. Now we get a membership feature dataset and a non-
membership feature dataset whose data points are just scalar values. The
EncoderMI-T attack then searches for an optimal threshold to classify
membership features and non-membership features. Similar to MIA efficacy, the
formal definition of EMIA efficacy is given by:

$\displaystyle\texttt{EMIA-
Efficacy}\coloneqq\frac{TN_{\texttt{EMIA}}}{|{\mathcal{D}}_{\texttt{unlearn}}|},$
(7)

where $TN_{\texttt{EMIA}}$ is the number of true negatives predicted by the
EncoderMI attack.

##  Appendix B Additional Experiments

###  B.1 Unlearning Performance for More Tasks

We present experiment results on CIFAR-100 in Tables 6 and 7. Across these
different tasks, our proposed _Alignment Calibration_ method achieves the
lowest average gap compared to the Retrain method.

Table 6: Performance of methods on randomly forgetting 10% of CIFAR-100
training data. EMIA is evaluated on the unlearned encoder, while RA, TA, UA,
and MIA are evaluated after linear probing. We report the average gap (Avg.
Gap) over these 5 metrics between methods and Retrain. The results are
averaged over 5 random trials.

Methods | EMIA | RA | TA | UA | CMIA | Avg. Gap | RTE
---|---|---|---|---|---|---|---
MoCo
Retrain | 56.24 | 62.23 | 58.6 | 58.43 | 59.53 | - | 109.47
Fine-Tune | 46.05 (10.19) | 63.49 (1.27) | 58.88 (0.29) | 59.8 (1.37) | 48.81 (10.72) | 4.77 | 1.42
Grad. Ascent | 44.01 (12.23) | 62.56 (0.33) | 59.00 (0.41) | 60.65 (2.22) | 53.28 (6.25) | 3.96 | 0.17
NegGrad | 53.58 (2.66) | 63.70 (1.47) | 58.78 (0.19) | 58.85 (0.42) | 48.68 (10.84) | 3.12 | 1.70
$l_{1}$-Sparsity | 45.68 (10.56) | 60.89 (1.34) | 57.4 (1.20) | 58.66 (0.23) | 52.48 (7.04) | 4.07 | 1.43
Ours | 50.17 (6.07) | 63.2 (0.97) | 58.56 (0.04) | 58.44 (0.00) | 54.15 (5.38) | 2.49 | 1.87
SimCLR
Retrain | 51.2 | 57.76 | 56.25 | 55.86 | 65.60 | - | 151.77
Fine-Tune | 40.85 (10.35) | 57.29 (0.48) | 54.96 (1.29) | 55.85 (0.00) | 60.61 (4.99) | 3.42 | 1.93
Grad. Ascent | 34.00 (17.21) | 62.12 (4.36) | 59.58 (3.32) | 61.08 (5.23) | 54.21 (11.39) | 8.30 | 0.19
NegGrad | 46.39 (4.81) | 56.52 (1.24) | 54.30 (1.95) | 55.00 (0.86) | 60.04 (5.56) | 2.89 | 2.34
$l_{1}$-Sparsity | 40.55 (10.66) | 57.85 (0.09) | 55.76 (0.49) | 56.68 (0.83) | 58.46 (7.15) | 3.84 | 1.96
Ours | 46.72 (4.48) | 57.11 (0.65) | 54.70 (1.55) | 55.23 (0.63) | 59.78 (5.83) | 2.63 | 3.00

Table 7: Performance of methods on randomly forgetting 50% of CIFAR-100
training data. EMIA is evaluated on the unlearned encoder, while RA, TA, UA,
and MIA are evaluated after linear probing. We report the average gap (Avg.
Gap) over these 5 metrics between methods and Retrain. The results are
averaged over 5 random trials.

Methods | EMIA | RA | TA | UA | CMIA | Avg. Gap | RTE
---|---|---|---|---|---|---|---
MoCo
Retrain | 60.40 | 57.72 | 52.58 | 52.32 | 67.30 | - | 66.71
Fine-Tune | 53.58 (6.81) | 61.9 (4.18) | 56.07 (3.48) | 56.87 (4.55) | 52.27 (15.03) | 6.81 | 0.86
Grad. Ascent | 43.76 (16.63) | 60.91 (3.19) | 56.2 (3.62) | 57.48 (5.16) | 55.64 (11.66) | 8.05 | 0.45
NegGrad | 38.95 (21.44) | 60.94 (3.22) | 57.01 (4.43) | 58.21 (5.89) | 57.64 (9.66) | 8.93 | 1.66
$l_{1}$-Sparsity | 49.97 (10.43) | 58.89 (1.17) | 53.07 (0.49) | 53.66 (1.33) | 56.27 (11.03) | 4.89 | 0.87
Ours | 56.22 (4.18) | 59.53 (1.81) | 53.37 (0.79) | 53.69 (1.37) | 52.27 (15.03) | 4.63 | 1.84
SimCLR
Retrain | 56.00 | 50.40 | 48.46 | 47.68 | 69.47 | - | 89.74
Fine-Tune | 53.89 (2.12) | 52.82 (2.42) | 49.87 (1.41) | 51.04 (3.36) | 60.06 (9.41) | 3.74 | 1.17
Grad. Ascent | 40.28 (15.72) | 56.32 (5.92) | 54.12 (5.66) | 55.22 (7.54) | 61.43 (8.04) | 8.58 | 0.59
NegGrad | 46.83 (9.17) | 50.60 (0.20) | 48.20 (0.26) | 49.29 (1.62) | 58.01 (11.47) | 4.54 | 2.32
$l_{1}$-Sparsity | 45.12 (10.89) | 52.62 (2.22) | 50.09 (1.62) | 50.98 (3.30) | 65.25 (4.22) | 4.45 | 1.19
Ours | 54.98 (1.02) | 49.28 (1.12) | 46.8 (1.66) | 47.00 (0.68) | 57.78 (11.69) | 3.24 | 3.07

###  B.2 More Visual Auditing Results

In Figure 5, we report the AGM of Retrain, Fine-Tune, Gradient Ascent,
NegGrad, $l_{1}$-Sparisty on CIFAR-10, as a complement to Figure 3. For the
unlearning task on CLIP, we check the ASM of our _AC_ and other baseline
methods in Figure 6.

Figure 5: Alignment Gap Matrices of 8 unlearn images for Retrain, Fine-Tune,
Gradient Ascent, NegGrad, $l_{1}$-Sparsity, and our _Alignment Calibration_.
The unlearning task is to forget 10% of CIFAR-10 training data from a MoCo
encoder.

Figure 6: Alignment Gap Matrices of 6 unlearn image-text pairs for Retrain,
Fine-Tune (FT), Gradient Ascent (GA), NegGrad (NG), $l_{1}$-Sparsity, and
_Alignment Calibration (AC)_. The unlearning task is to forget 10% of MS-COCO
training data from a CLIP encoder.