# CLIPping Privacy: Identity Inference Attacks on Multi-Modal Machine Learning
Models

Dominik Hintersdorf Contact<EMAIL_ADDRESS>Department
of Computer Science, Technical University of Darmstadt, Germany  Lukas
Struppek  Department of Computer Science, Technical University of Darmstadt,
Germany  Kristian Kersting  Department of Computer Science, Technical
University of Darmstadt, Germany  Centre for Cognitive Science, TU Darmstadt,
Germany  Hessian Center for AI (hessian.AI), Germany

###### Abstract

As deep learning is now used in many real-world applications, research has
focused increasingly on the privacy of deep learning models and how to prevent
attackers from obtaining sensitive information about the training data.
However, image-text models like CLIP have not yet been looked at in the
context of privacy attacks. While membership inference attacks aim to tell
whether a specific data point was used for training, we introduce a new type
of privacy attack, named identity inference attack (IDIA), designed for multi-
modal image-text models like CLIP. Using IDIAs, an attacker can reveal whether
a particular person, was part of the training data by querying the model in a
black-box fashion with different images of the same person. Letting the model
choose from a wide variety of possible text labels, the attacker can probe the
model whether it recognizes the person and, therefore, was used for training.
Through several experiments on CLIP, we show that the attacker can identify
individuals used for training with very high accuracy and that the model
learns to connect the names with the depicted people. Our experiments show
that a multi-modal image-text model indeed leaks sensitive information about
its training data and, therefore, should be handled with care.

##  1 Introduction

Figure 1: Identity Inference Attacks. Depiction of the workflow of our IDIA.
Given different images and the name of a person, the adversary queries CLIP
with the images and multiple possible names, including the name of the person.
The person is predicted to be in the training set if the name is correctly
predicted for more than half the images. (Best Viewed in Color) Figure 2:
Identity Inference Attack Examples. True positive (green border) and true
negative (red border) prediction examples of our IDIA on a model trained on
the LAION-400M dataset using individuals from the FaceScrub Dataset [39] and
German celebrities [25, 24].

For a long time, machine learning models were trained on single modalities. In
recent years, however, research has started to focus on the combination of
multiple modalities, such as text and images, to create even more powerful
models. Large multi-modal models are pushing the benchmarks in many
applications like text-guided image generation [47, 40, 49, 65], visual
question answering [11], image captioning[38] or even content moderation [51].

One of the milestones for multi-modal machine learning was the introduction of
Contrastive Language-Image Pre-training [44], or CLIP in short, which has
opened up a whole new area of unsupervised pre-training. Instead of requiring
large datasets of labeled images to pre-train a model, CLIP uses image-text
pairs, collected from the internet, to train a vision model and a text
transformer simultaneously in a contrastive learning fashion. During training,
the visual and text embeddings of images and their captions are forced to be
close to each other, while maximizing the distance to other embeddings of text
examples in the training set. As a result, CLIP is learning the visual
concepts depicted on the images through the accompanied texts.

Trained on massive data, the resulting multi-modal models are quite powerful.
However, even though the data is publicly accessible, it also covers sensitive
information like names, addresses, and phone numbers crawled from the internet
at scale. Unfortunately, as we will show, this makes multi-modal models not
only more powerful but also easier to attack than single modality approaches.
When attacking a large language model such as GPT-2 [7], the attacker has to
make hundreds or thousands of queries to the model, whereas when attacking a
multi-modal model one can simply ask the model to name the person in an image.
In other words, attackers being able to combine different modalities are even
more powerful and, in turn, the models pose an even more severe risk to
privacy.

To demonstrate this trade-off, we propose a novel attack, called Identity
Inference Attacks (IDIAs). While in a membership inference attack the attacker
wants to infer whether a sample was used for training the model, an identity
inference attack takes a broader view and ask whether a particular person was
part of the training data.

As researchers are developing more and more capable models, with even the goal
of artificial general intelligence (AGI), one has to think about and discuss
possible trade-offs between utility and user privacy. In particular, if a
model has a high level of knowledge and can answer even complicated questions,
an attacker, if posing the right questions, can extract sensitive information.

On the other hand, we should be aware of “Clearview AI” [19] and “Ever” [32]
moments for large multi-modal models. Clearview AI is scraping billions of
images of individuals, and offers its service to law enforcement for police
investigations, to prevent human trafficking, or offers identity verification
for other businesses, while also failing to comply with the data protection
laws in Canada, Australia, and parts of Europe and has therefore been fined by
multiple countries [35, 33, 34]. The company Everalbum, provider of the now
shut down cloud photo storage app ”Ever”, was training facial recognition
models on users’ photos without their consent and was ordered by the US
Federal Trade Commission to delete all models and algorithms trained on this
user data [32, 62]. This shows that it is crucial to check where data is
coming from and whether the individual data owners have given consent before
training a model.

Likewise for multi-modal models like CLIP [44], DALL-E [47] or Imagen [49],
even if the data from large-scale datasets are publicly accessible, it is
still questionable whether the individuals depicted have given their consent
to be used for training a machine learning model. Therefore, it is critical to
investigate whether multi-modal models leak information about their training
data. If this should turn out to be the case, those models could not only be
misused to identify or infer sensitive information about individuals, but
would also potentially violate data protection laws. This shows that attacks
on multi-modal models have a severe social impact and go beyond the academic
world. Information leakage like for example names, addresses, phone numbers or
emails, and connections with other modalities like images could be exploited
by an attacker for various malicious purposes.

To summarize, this paper focuses on the privacy risks of multi-modal image-
text systems and investigates whether multi-modal models leak sensitive
information:

  1. 1.

We propose a new kind of privacy attack, called identity inference attack
(IDIA), which aims to extract sensitive information about specific identities
from a trained multi-modal zero-shot classifier.

  2. 2.

Exhaustive experiments demonstrate that an attacker can infer with very high
accuracy whether a person was used for training and that multi-modal models
indeed leak information about their training data.

  3. 3.

IDIA shows that image-text models are so powerful that they easily learn to
connect the names with the depicted individuals during training, even if the
person appears very few times in the training data. This may imply a novel
trade-off: the more “emergent” abilities, the more privacy leakage.

We proceed as follows. We start off by touching upon related work and
background in Sec. 2. In Sec. 3 we introduce identity inference attacks and
present our exhaustive empirical evaluation in Sec. 5. Before concluding, we
are discussing our results and possible implications.

##  2 Background and Related Work

To begin, we start by giving an overview over multi-modal systems,
particularly CLIP. We further introduce related attacks on machine learning
systems and discuss research on the privacy of multi-modal systems.

###  2.1 Multi-Modal Systems as Zero-Shot Classifiers

Multi-modal learning describes the process of training machine learning models
to use information from different modalities at the same time, e.g., written
text, images, or audio. Early approaches trained models based on deep
Boltzmann machines [56] or LSTM networks [46]. Instead of training a standard
image classifier directly on a labelled training set, CLIP [44] introduced a
novel contrastive pre-training scheme. The approach of Radford et al. [44]
relies on public data scraped from millions of websites. More specifically,
the training data consists of image-text pairs, where the images are
accompanied by their corresponding captions. CLIP is trained by jointly
optimizing an image feature extractor $M_{img}$ and a text encoder $M_{text}$
to learn to match images with their corresponding textual description. While
the image feature extractor can be any standard neural network for computer
vision tasks, for example, a ResNet [18] or a vision transformer (ViT) [10], a
transformer is usually used as the text encoder. The original CLIP model was
trained on a non-public dataset with about 400 million image-text pairs,
scraped from public websites.

After training the multi-modal model on these image-text pairs, the authors
demonstrated that CLIP achieves high zero-shot prediction accuracies on
various tasks. Zero-shot prediction in this context means to use the model for
predictions on unseen datasets and tasks. Given an image and multiple text
prompts, the model will predict the text prompt, best matching the visual
concept on the image.

To achieve this, the model is queried with an image $x$ to make a prediction
on, together with multiple text prompts $Z=[z_{1},z_{2},...,z_{n}]$ which
contain descriptions of possible visual concepts. Based on the computed
embeddings of both inputs, the pairwise cosine similarity score between every
text prompt $z_{i}$ and the input image $x$ is computed and scaled by a
temperature term $\tau$:

$sim(x,z_{i})=\frac{M_{img}(x)*M_{text}(z_{i})}{\|M_{img}(x)\|_{2}*\|M_{text}(z_{i})\|_{2}}*e^{\tau}.$
(1)

Finally, the cosine similarities are normalized using a softmax function and
the text prompt with the highest probability is predicted.

This makes CLIP a very versatile zero-shot classifier, since the labels and
the number of possible classes for the classification can be chosen freely.
Radford et al. [44] further demonstrated that carefully selected text prompts,
also known as prompt engineering, often improve the zero-shot prediction
accuracy. For example, instead of using cat as a text prompt, embedding it
into the prompt a photo of a cat helps the model to understand the context
better.

###  2.2 Privacy and Security of Machine Learning Models

Various privacy and security attacks against machine learning systems have
been proposed. Arguably best known in the field of security attacks are
adversarial attacks [59, 17]. These attacks introduce fine-tuned noise to
images to manipulate a model’s prediction, while avoiding humans being able to
detect these manipulations by visual inspection. Subsequent research in this
area adapted adversarial attacks to other kinds of models, such as NLP [43,
68] or multi-modal [13, 41] models, but also demonstrated their applicability
and security risks in production-ready systems [58, 37].

Related to adversarial attacks but more focused on the privacy instead of
security aspect of machine learning models are model inversion attacks [14].
Similar to adversarial attacks, most of the approaches are altering input
images, maximizing the output probabilities of a certain class and are trying
to reconstruct specific training samples [67] or class representatives [63,
57] based on a trained model’s learned knowledge. But leakage of training data
is not only limited to vision models. Carlini et al. [7] demonstrated that
large natural language models like GPT-2 memorize training data and leak
sensitive information. Other attacks that try to gain information about the
training data are property inference attacks [2, 15], inferring properties
about the training data, or model extraction attacks [27, 22], which aim to
create a clone of a trained model by querying it with black box access only.

Most related to our approach, however, are membership inference attacks
(MIAs), first introduced by Shokri et al. [54]. By performing MIAs, the
adversary tries to identify the specific samples used to train a machine
learning model. Successfully inferring this information might indeed lead to a
privacy breach, for example in domains like medicine or facial recognition.
If, for example, a model was trained on health data from a certain hospital,
inferring membership of a data sample would leak the information that the
person to which this data sample belongs, was once a patient in this hospital.
This could leak private information such as where the person lives or whether
the person has had certain illnesses.

Describing MIAs more formally, the adversary has access to some data point $x$
and a target model $M_{target}$, which has been trained on an unknown training
set $S_{training}^{target}$. The goal of the attack is then to answer whether
the data point was in the training data $x\in S_{training}^{target}$ (member)
or was not in the training data $x\notin S_{training}^{target}$ (non-member).

In the literature, it is usually assumed that the attacker has access to a
non-overlapping shadow dataset [54] $S_{training}^{shadow}$, which the
attacker can use to prepare the attack. Most proposed attacks [54, 50, 29, 9]
train so-called shadow models $M_{shadow}$ on the shadow dataset
$S_{training}^{shadow}$. These models are used to mimic the behavior of
$M_{target}$ but with the advantage that the adversary knows the training data
$S_{training}^{shadow}$ used to train $M_{shadow}$. With this knowledge, an
inference model $h$, which can be a simple threshold or a more complex model,
is trained on the outputs of $M_{shadow}$ on training and non-training
samples. The trained inference model $h$ is then applied to $M_{target}$ and
some unseen data $x$ to predict the membership status.

As one of the first MIAs, Shokri et al. [54] trained multiple shadow models to
fit $h$. The proposed attack only requires access to the prediction scores of
$M_{target}$. Later, Salem et al. [50] improved the attack process and
demonstrated that training a single shadow model is sufficient to perform the
attacks while achieving comparable results. While the attacks proposed by
Salem et al. use the prediction scores of $M_{target}$, Yeom et al. [64]
exploited the fact that neural networks for image classification are trained
to minimize some loss on their training samples and, therefore, propose to
distinguish between members and non-members based on their loss. Carlini et
al. [5] recently proposed a novel likelihood ratio attack that trains multiple
models, half of them on data sets which contain the data sample $x$ to infer
its membership status, and half of them without. Subsequently, the models are
used to approximate the loss distribution on the models’ predictions on $x$
and perform a likelihood ratio test to predict the membership.

However, in reality, often the attacker has only label-only access to the
model and cannot get the prediction scores for each class or calculate the
model’s loss for a sample. Therefore, Choquette-Choo et al. [9] and Li and
Zhang [29] introduced label-only MIAs, which rely solely on the predicted
labels to infer the membership status. The intuition behind these attacks is
that trained models are seemingly more robust in predicting samples from the
training data than predicting unseen data.

Whereas the literature on MIAs using shadow models attest high success rates
for inferring the membership status of samples, they all make the assumption
that the attacker knows the data distribution of the training data.
Hintersdorf et al. [20] recently argued that this information is usually not
available to the adversary and that the actual privacy risk resulting from
MIAs under realistic conditions, where the attacker does not know the
distribution of the training data, is overestimated. The authors demonstrated
that the overconfidence of neural networks acts as an implicit defense against
MIAs, and that there exists a trade-off between mitigating overconfidence and
the susceptibility against MIAs.

Moving slightly beyond MIA but staying uni-modal, Li et al. [28] observed that
embeddings of a person used for training, say, a LuNet for image
classification, result in more compact clusters than images of other
individuals in metric embedding learning. Likewise, other approaches on text
[55] and audio data [36] exploit the uniqueness of the data samples of a
person to check whether their data was used for training.

Finally, investigating the privacy of vision models trained with contrastive-
learning, Liu et al. [30] have shown that these models are susceptible to
membership inference attacks. The proposed attack exploits the fact that
during training, the augmentations of a single image are close to each other
in the embedding space, while the augmented images of samples not used for
training are more distant to each other. Even though they also looked at
vision models pre-trained using CLIP, they are only investigating the privacy
of the learned vision model with an attacker that has access to the calculated
embeddings of the images.

To summarize, IDIA is the first image-text attack exploiting that CLIP models
easily learn to connect the names with the depicted individuals during
training, even if the person appears very few times in the training data.

###  2.3 Privacy and Security of Multi-Modal Systems

The idea to combine different modalities to perform privacy attacks is not
new. Jaiswal and Provost [23], for example, have shown that multi-modal
representations unintentionally encode additional information like gender or
identity. Removing the gender and speaker information from the representations
using adversarial learning seems to reduce the information leakage. Similarly,
Rahman et al. [45] have shown that combining different modalities like image
and text information, locations, and hashtags in social media posts, an
attacker can achieve higher accuracy in inferring relationships in social
networks than when just using a single modality.

As multi-modal systems can also be used as data retrieval models, Zhang et al.
[66] have proposed a privacy protection method to prevent an adversary to
search for and combine private information in released data such as photos.
While training a retrieval system, the authors propose to modify the original
data using adversarial perturbations to prevent images from being retrieved by
an unauthorized party, not having access to the trained system.

In the area of security of multi-modal models, Carlini et al. [6] demonstrated
that image-text systems trained in a contrastive manner are susceptible
against simple backdoor and data poisoning attacks. The authors demonstrated
that by adding a small trigger to only 0.0001% of the dataset, causes the
model to misclassify its input when presented with the trigger.

To analyze how CLIP is learning to abstract the visual concepts during
training, Goh et al. [16] have examined how the neurons of the vision model of
CLIP react to certain stimuli. They have discovered that the model has
multimodal neurons that are activated by different images that describe the
same visual concept. Apparently, there are neurons that react to the visual
concept that are attributed to certain people. As an example, the “Jesus
Christ neuron” is activated by pictures of Jesus Christ but also by pictures
of other Christian symbols like crosses or crowns of thorns.

Related to the analysis of the neurons of CLIP, Thies [61] demonstrated with a
very limited example in a short blog post that CLIP does not only store
general concepts of images and texts, but also some information about people’s
identities in the training data. Similarly, the original CLIP paper also noted
that the model can be used to classify celebrities from the CelebA dataset
[31].

However, to date, no systematically in-depth research has been conducted in
the privacy context of zero-shot multi-modal classifiers such as CLIP. With
our work, we want to close this research gap and demonstrate that new privacy
concerns go hand in hand with the raise of multi-modal models trained on large
amount of public data from the internet.

##  3 Identity Inference Attacks

Now we are ready to introduce our identity inference attack (IDIA) as depicted
in Fig. 1. In an IDIA, the adversary is given black-box access to a multi-
modal image-text model together with some pictures of a person whose name is
known. Additional information, for example about the architecture or the
training algorithm used to train the multi-modal model, is not available to
the attacker. The goal of the attacker is to infer whether the model was
trained using some images of this specific person.

More formally, given images $I$ of an individual and its name $N$, the
attacker can query a target model $M$ with the images $x_{i}\in I$ and text
prompts $P$ with $N\in P$ containing potential names. After receiving the
prediction $z_{x,i}$ from the model, the attacker applies the decision
function $f:z_{x,i}\times P\rightarrow\\{0,1\\}$, which outputs $1$ if the
correct name for the person depicted on $x_{i}$ was predicted and $0$
otherwise. The identity inference attack score $S(I,P)$ is then calculated
using the following function:

$\displaystyle S(I,P)$
$\displaystyle=\frac{1}{|I|}\sum\nolimits_{i=0}^{|I|}f(M(x_{i},P),P)$ (2)
$\displaystyle=\frac{1}{|I|}\sum\nolimits_{i=0}^{|I|}f(z_{x,i},P).$ (3)

To infer whether the person was in the training set, the attacker applies the
identity inference attack prediction function $Q$, which takes the attack
score, applies a threshold $\tau$ to it and outputs $1$ if the person is
predicted to be in the training set and $0$ otherwise:

$\displaystyle Q(I,P)=\begin{cases}1&\text{if}\;S(I,P)>\tau\\\
0&\text{otherwise,}\end{cases}$ (4)

Setting, e.g., the threshold as $\tau=0.5$, will cause $Q$ predicting the
person to be in the training set, if more than half of the images available to
the attacker are predicted correctly. In Sec. 6 we will discuss how a
different threshold might influence the results of the attack.

##  4 Attack Protocol

Identity inference attacks (IDIAs) assume a threat model that we describe now
in detail and that also serves as our experimental setup.

Attacker’s Capabilities. In our attack, we consider an attacker who has black-
box label-only access to a trained CLIP model[44]. Performing privacy attacks
in a black-box label-only setting is usually the hardest and most challenging
setting for the adversary. The only information given to the attacker are a
certain number of images of an individual and the name of the person.

Datasets. To evaluate our attack, we are attacking CLIP models that were pre-
trained on the LAION-400M [52, 21] dataset and the Conceptual Captions 3M [53]
dataset. For the identities for which we want to infer whether they are in the
training data, we have taken images of actors and actresses from the FaceScrub
dataset [39], consisting of facial images of 530 actresses and actors. Both
the LAION-400M dataset and the Conceptual Captions 3M dataset contain images
and text pairs extracted from millions of webpages. While the LAION-400M
dataset contains 400 million image-text pairs, the Conceptual Captions dataset
contains 3.3 million image-text pairs. Because only the URLs of the images in
the Conceptual Captions dataset are provided and some of these images are not
available anymore, we were able to download 2.8 million images out of the 3.3
images URLs given. Because there is also a Conceptual Captions dataset
containing 12 million images [8], we are referring to the dataset used in this
work as Conceptual Captions 3M (CC3M).

Architecture |  Number of Image-Text Pairs per Person | ImageNet Val Top-1 Acc | ImageNet Val Top-5 Acc |  Val Image-to-Text Rank@1 |  Val Image-to-Text Rank@5
---|---|---|---|---|---
ViT-B/32 | 75 | 14.3% | 29.8% | 21.2% | 39.9%
50 | 14.3% | 29.8% | 21.0% | 39.7%
25 | 14.7% | 30.1% | 21.2% | 39.9%
10 | 14.5% | 30.0% | 21.1% | 39.8%
| 1 | 14.3% | 29.6% | 21.2% | 39.6%
ResNet-50 | 75 | 19.6% | 37.4% | 30.9% | 53.4%
50 | 19.9% | 37.9% | 30.9% | 53.4%
25 | 20.0% | 38.0% | 31.3% | 53.4%
10 | 19.7% | 38.0% | 31.3% | 53.6%
| 1 | 19.3% | 38.2% | 30.9% | 52.7%
ResNet-50x4 | 75 | 21.0% | 39.8% | 33.7% | 56.4%
50 | 21.7% | 40.2% | 33.6% | 56.2%
25 | 20.7% | 39.5% | 33.1% | 56.2%
10 | 21.2% | 39.9% | 33.4% | 55.7%
| 1 | 21.3% | 39.9% | 33.5% | 55.8%

Table 1: CLIP performs well on zero-shot predictions, here trained on the CC3M
dataset, and does not overfit on individuals. A random classifier would
achieve a top-1 accuracy (the higher, the better) of $0.1\%$ as there are
$1000$ classes. Image-to-Text Rank (the higher, the better) is the fraction of
images where the rank is zero, meaning that the image and the according
caption are indeed the most similar in the dataset. Moreover, the number of
image-text pairs added to the CC3M training set has no influence on the
predictive performance, showing that they do not overfit to individuals.

Target Models. In our experiments, we evaluate our IDIA on several CLIP models
with different image feature extractors. In our experiments on the LAION-400M
dataset, we attack models with the three vision transformer architectures
ViT-B/32, ViT-B/16 and ViT-L/14 [10] used as image feature extractors, which
were all pre-trained by the OpenCLIP project[21]. The different image feature
extractors used for the experiments on the CC3M dataset are ResNet-50 [18],
ViT-B/32 [10] and a ResNet-50x4. ResNet-50x4 is a ResNet-50, which was scaled
accordingly using EfficientNet-style scaling techniques[60]. During all the
experiments, the text encoder of the CLIP model is identical to the original
paper [44].

With this at hand, one can prepare attacks and their evaluation. This includes
the dataset analysis and training the target models. Our code, as well as the
pre-trained models, are available on
GitHub111https://github.com/D0miH/clipping_privacy.

Analyzing and Creating Datasets. To evaluate the attack, we first have to
analyze which entities are in the dataset and which are not. This enables the
correct calculation of false-positive and false-negative predictions. To
search for entities within such large dataset with millions of image-text
pairs, we used clip-retrieval[3], an open-source implementation for image
retrieval based on the image and text embeddings calculated by CLIP.

To analyze the LAION-400M dataset, we retrieved the 200 most similar images
together with their caption for each image in the FaceScrub dataset, resulting
in roughly 7.5 million retrieved image-text pairs. We then checked whether the
name of the actor or actress is present in one of the captions of the
retrieved images. If at least one caption of the 200 captions per person
contained the name, we concluded that the individual is present in the
LAION-400M dataset. Because the adversary needs different images of the same
person for the attack, we used only images of individuals in the FaceScrub
dataset which had at least 30 images available, resulting in 507 individuals.
The result of the analysis is that 504 of the 507 individuals of the FaceScrub
dataset are present in the LAION-400 dataset, while 3 of them are not. To
obtain a larger number of individuals that were not used for training, we
manually searched for 30 images of 8 German actors and 8 German actresses.
Because the LAION-400M dataset was primarily created by scraping English
websites, choosing actors and actresses not present in English-speaking TV
Series or movies reduces the likelihood of them being in the dataset. As with
the individuals of the FaceScrub dataset, we retrieved similar images from the
LAION-400M dataset for each German actor and actress and checked the captions
for the names. Our analysis has shown that 1 actress is contained in the
LAION-400M dataset, while 15 of the selected German actors and actresses are
not. For evaluating the attacks on the models trained on the LAION-400M
dataset, we combine both the analyzed identities of the FaceScrub dataset and
the analyzed identities of the German actors and actresses which in total
results in 523 individuals, 505 of which are present and 18 of which were not
present in the LAION-400M dataset.

For analyzing the Conceptual Captions 3M (CC3M) dataset, a similar procedure
as for analyzing the LAION-400M dataset was used. However, while creating the
CC3M dataset, Sharma et al. [53] have anonymized entity names in the image
captions by rephrasing the captions or by replacing them, e.g., names with a
<PERSON> tag. Therefore, it is not possible to search for the individual names
in the image captions. To still be able to analyze which individuals are
already present in the dataset, we trained a ResNet-50 on the FaceScrub
dataset to classify the faces of the 530 identities. The ResNet-50 used, was
trained with a batch size of 128 for 100 epochs on a single A100 GPU using the
cropped images of the FaceScrub dataset. 90% of the dataset was used to train
the model, while the other 10% was used as validation set. All images were
resized to 224x224 pixels and center cropped. During training, randomly
resized cropping, color jitter and horizontal flipping were applied as data
augmentation techniques. We used Adam [26] with a learning rate of $0.003$,
which was reduced by $0.1$ after 75 and 90 epochs.

The trained ResNet-50 achieved a test accuracy of 91% on the FaceScrub test
set. We used OpenCV’s [4] Haar cascade face detector to detect faces in the
similar images which were retrieved from the CC3M dataset and used the trained
ResNet-50 to predict whether the person in question was present in the images.
In addition to the trained ResNet-50 for filtering the images, we used the
CLIP model pre-trained on LAION-400M to predict whether the person is present
in the retrieved images of the CC3M dataset. We then chose the 200 individuals
with the lowest number of correct predictions and mixed the first 100 of these
individuals with the CC3M dataset, while the other half of the individuals was
not used for training the target models of our experiments.

| ViT-B/32 | ViT-B/16 | ViT-L/14
---|---|---|---
Number of Parameters Vision Model | 87,849,216 | 86,192,640 | 303,966,208
Number of Parameters Text Transformer | 37,828,608 | 37,828,608 | 85,054,464
ImageNet Top-1 Accuracy | 62.9% | 67.1% | 72.77%
TPR | $91.99\%\pm 0.31\%$ | $93.87\%\pm 0.22\%$ | $95.06\%\pm 0.25\%$
TNR | $100\%\pm 0\%$ | $94.44\%\pm 0\%$ | $77.78\%\pm 0\%$
FPR | $0\%\pm 0\%$ | $5.56\%\pm 0\%$ | $22.22\%\pm 0\%$
FNR | $8.01\%\pm 0.31\%$ | $6.13\%\pm 0.22\%$ | $4.94\%\pm 0.25\%$
Table 2: Attackers can be confident in their predictions. The mean true-
positive rate (TPR), true-negative rate (TNR), false-positive rate (FPR) and
false-negative rate (FNR) and their standard deviation of the identity
inference attack on the CLIP models [21] pre-trained on the LAION-400M
dataset.

Training Target Models on CC3M. To train the CLIP models on the CC3M dataset,
we used the code of the open-source project OpenCLIP [21]. As can be seen in
Tab. 1, we trained CLIP models with three different vision model
architectures. The hyperparameters of the vision and text model architectures
are the same as in the original CLIP paper [44]. While the text transformers
used as an encoder for the ResNet-50 and the ViT-B/32 architecture models use
12 layers, 8 heads and have a width of 512, the text transformer for the
ResNet-50x4 architecture is slightly larger and uses 12 layers, 10 heads and a
width of 640. Because we want to investigate the influence of the number of
occurrences of a person on the success of the attack, we trained five models
with 75, 50, 25, 10 and 1 data sample occurrences per person for each of the
architectures. All the models were trained on 8 A100 GPUs for 50 epochs on the
CC3M dataset with a per GPU batch size of 128, a learning rate of $1e^{-3}$
and a weight decay parameter of $0.1$.

The top-1 and top-5 ImageNet zero-shot prediction accuracy of the trained
models can be seen in Tab. 1. While the two ResNets achieve comparable results
of a top-1 accuracy of around 20%, the vision transformer ViT-B/32 achieves a
top-1 accuracy of 14.3% on the ImageNet [48] validation set. We suspect that
2.8 million data points is not sufficient for the vision transformer to fully
learn the visual concepts of the images. Even though the ResNet-50 is much
smaller, it seems to generalize well and performs even better than the bigger
vision transformer model. The image-to-text rank of the models describes how
often the model is predicting the correct text prompt for a given validation
image. Rank@1 means that the ground truth caption for the image was the
predicted one, while rank@5 means that the ground truth predicted caption was
under the top-5 predicted ones. As with the zero-shot accuracy, one can see
that the ResNet-50 and the ResNet-50x4 achieve comparable results. While the
ResNet-50 achieves an image-to-text rank@1 of around 30%, the ResNet-50x4
achieves an image-to-text rank@1 of around 33%, meaning that in about a third
of the cases the correct caption to a given image in the validation set was
predicted.

##  5 Attack Results

In this section, we will present the results of our experiments conducted on
the models trained on the LAION-400M and the CC3M dataset.

(a) ViT-B/32

(b) ViT-B/16

(c) ViT-L/14

Figure 3: IDIA can be executed with only a few samples. Depicted is the
influence of the number of samples the attacker has during the IDIA on the
ViT-B/32, ViT-B/16 and ViT-L/14 model pre-trained on the LAION-400M dataset
[21, 52]. Plotted are the mean and standard deviation of the true-positive
rate (TPR), false-negative rate (FNR), false-positive rate (FPR), true-
negative rate (TNR) and accuracy (Acc) of the identity inference attack.

###  5.1 Experimental Results on LAION-400M

We evaluated the attack on three models with different sizes. The top-1
accuracies of the models on the validation set of ImageNet [48] can be seen in
Tab. 2. The zero-shot top-1 accuracy of the models are all above 60%, which
indicates that the models have learned meaningful visual concepts during
training and can connect the visual concepts depicted on the images with given
text descriptions. The results of the top-1 accuracy are comparable with those
reported in the original CLIP paper [44]. For performing the attacks, the
adversary has the name and 30 images of each person. During each attack, the
adversary queries the model with each image and provides the model with 546
names as text prompts, 530 of which are the names of the entities in the
FaceScrub dataset and 16 of which are the names of the German actors and
actresses. In all our experiments, we have used only the names of the
individuals as text prompts and conducted no prompt engineering.

The outcome of IDIA on the CLIP models pre-trained on the LAION-400M dataset
can be seen in Tab. 2. The mean true-positive rate (TPR), true-negative rate
(TNR), false-positive rate (FPR) and false-negative rate (FNR) together with
the standard deviation for the IDIA with 30 samples per person available to
the attacker is depicted. Because some individuals have more than 30 images
available, the attack was run 20 times, with randomly sampling 30 images from
all available images for each person.

Looking at the results of the attacks, one can see that the models learn to
recognize individual people. As can be seen, the mean true-positive rate of
the attack on all models is consistently above 91% while the mean false-
positive rate is quite low on the ViT-B/32 and ViT-B/16 models. The mean true-
positive rate and the mean false-positive rate seems to increase, with the
model size. Accordingly, the mean true-negative rate and mean false-negative
rate decrease with increased model size.

The lower the false-positive rate, the higher the confidence of the adversary
can be in the positive predictions. In a setting of a privacy attack, this is
especially interesting since low false-positive rates are increasing the
potential leakage of sensitive information. High false-positive rates, on the
other hand, are reducing the risks of privacy attacks since the attacker
cannot be confident in the results of the attacks. For the smaller ViT-B/32
and ViT-B/16 models, the attacker can be sure that the entities with positive
predictions are indeed in the training data because of a false-positive rate
below 6%.

Interestingly, and against our initial expectation, the false-positive rate
seems to considerably increase with the model size. Our intuition is that the
larger a model’s capacity, the higher the chance that the model is remembering
the images and the names of a person. However, we suspect that the increase in
the false-positive rate is due to false positives in our analysis of the
LAION-400M dataset and incorrectly labelling entities as not occurring in the
dataset during our analysis. It is very unlikely that the model is predicting
the correct name out of 546 names more than half of the time if it has never
seen the person during training. Because of the considerable size of this
dataset, we unfortunately cannot verify this assumption. However, the low
false-positive rate in the experiments on the CC3M dataset indicate that this
assumption is most likely correct.

Interestingly, sample selection does not seem to have a high impact on the
results. While the standard deviation of the false-positive and true-negative
rates is 0%, the standard deviation of the false-negative rate and the true-
positive rate is below 0.31% in all three experiments. This indicates that the
selection of an individual’s images, used for the attack, has almost no effect
on the success of the attack.

Influence of Number of Attack Samples. To investigate how the number of
samples available to an adversary influences the success of the attack, we
varied the number of images available to the attacker. The results for all the
models pre-trained on LAION-400M can be seen in Fig. 3.

As can be seen in all three plots, the number of samples does not make a big
difference when having more than 5 images of a person available for the
attack. An adversary with 5 samples achieves similar attack results in terms
of true-positive rate as an adversary with 30 samples. However, it even seems
that having more samples increases the number of false-positive predictions
for the attack on the largest model ViT-L/14. As can be seen in Fig. 3(c), the
false-positive rate is increasing with the number of samples, while the true-
positive rate plateaus. As explained earlier, this is most likely caused by
incorrectly labelled entities during the dataset analysis and might indicate
that having more samples increases the effectiveness of the attack on larger
models.

As for the results of the other two models seen in 3(a) and Fig. 3(b), the
false-positive rate as well as the false-negative rate stay very low for all
sample sizes. This shows that an attacker with only a few images of a person
can perform the attack and be confident in the positive predictions, which
indicates that the adversary can confidently infer information about the
training data.

In all three plots, there can be seen a dip in the true-positive rate with a
sample size of 2. We assume that this is due to the attack predicting
individuals to be in the training set only when for more than half of the
images, the name of the person was correctly predicted. As a result, when
having two images to perform the attack, in numerous instances there are
conflicting predictions causing the IDIA to predict the individuals not to be
in the training set. This causes the mean true-positive rate to decrease
rapidly, while the mean false-negative rate increases.

###  5.2 Experimental Results on CC3M

Although the experiments with the models pre-trained on the LAION-400M dataset
show impressive results, due to the size of the dataset, the analysis of the
dataset is error-prone and the number of entities included and not included in
the training set is very unbalanced. Therefore, we conducted additional
results on the CC3M dataset, where the setting of the experiments can be
controlled more thoroughly and the circumstances under which the attack is
successful can be investigated more clearly.

| ResNet-50 | ResNet-50x4 | ViT-B/32
---|---|---|---
Number of Parameters Vision Model | 38,316,896 | 87,137,080 | 87,849,216
Number of Parameters Text Transformer | 37,828,608 | 59,082,240 | 37,828,608
ImageNet Top-1 Accuracy | 19.6% | 21.0% | 14.3%
TPR | $63.40\%\pm 2.14\%$ | $73.40\%\pm 2.56\%$ | $68.15\%\pm 3.76\%$
TNR | $99.75\%\pm 0.44\%$ | $100\%\pm 0\%$ | $100\%\pm 0\%$
FPR | $0.25\%\pm 0.44\%$ | $0\%\pm 0\%$ | $0\%\pm 0\%$
FNR | $36.60\%\pm 2.14\%$ | $26.60\%\pm 2.56\%$ | $31.85\%\pm 3.76\%$
Table 3: Attackers can be confident in their predictions. The mean true-
positive rate (TPR), true-negative rate (TNR), false-positive rate (FPR) and
false-negative rate (FNR) and their standard deviation of the identity
inference attack on the CLIP models trained on CC3M.

We are especially interested in answering the following two research
questions:

  1. 1.

What is the influence of the number of images the attacker has of the
individuals to perform the attack?

  2. 2.

How does the number of images per person in the training dataset affect the
success of the attack?

The results of IDIA on the CLIP models trained on the CC3M dataset can be seen
in Tab. 3. As with the results for the models trained on the LAION-400M
dataset, the values in the table are the mean and standard deviation values of
the attack with 30 samples per person over 20 attacks.

As stated, the mean true-positive rate is consistently above 63% for all
models, while the mean false-positive rate is always below 1%. While the mean
true-positive rate seems to increase, the mean false-negative rate seems to
decrease with the model size. Interestingly, the size of the model seems to
have no effect on the true-negative rate or the false-positive rate. Even
though the number of parameters for the vision model has almost tripled and
the number of parameters for the text encoder almost doubled from the
ResNet-50 to the ResNet-50x4, both the true-negative rate and the false-
positive rate have not changed very much, which is why the behavior of IDIA
for unseen individuals stays the same.

In contrast to the experiments on the LAION-400M dataset, the false-positive
rate is not increasing with the model size. This indicates that our assumption
about incorrect labelled entities during the analysis of the LAION-400M
dataset is most likely correct.

(a) ResNet-50

(b) ResNet-50x4

(c) ViT-B/32

Figure 4: Influence of the number of samples the attacker has during IDIA on
the models trained on the CC3M dataset. Depicted are the mean and standard
deviation of the true-positive rate (TPR), false-negative rate (FNR), false-
positive rate (FPR), true-negative rate (TNR) and accuracy (Acc) of the
attack.

The number of parameters of the vision model seems to have a positive effect
on the true-positive rate of the attack. Comparing the mean true-positive rate
of the ResNet-50 with the mean true-positive rate of the ViT-B/32 model, one
can see that the number of trainable parameters has more than doubled and as a
result, the mean true-positive rate increased by 3%. However, comparing the
results of the ViT-B/32 with the ResNet-50x4, one can see that the mean false-
negative rate decreased by more than 8% while the true-positive rate increased
by 8%. It seems that some samples that were incorrectly classified as not
being in the training set, are now correctly predicted. We have two possible
explanations on why this might be the case.

Our first theory concerns the number of parameters in the text encoder.
Comparing the number of parameters of the ViT-B/32 model with the number of
trainable parameters of the scaled ResNet-50x4 model, one can see that while
the parameters of the vision model stays roughly the same, the number of
parameters of the text encoder is increased by a factor of only more than 1.5.
This might have a significant effect on the true-positive and false-negative
rate. We suspect that the size of the text encoder has higher impact on the
attack than the vision model, as a main component of the attack are the names
of the individuals. While larger vision models can differentiate faces of
different individuals better, a larger text encoder can recognize and memorize
names better.

A second possible explanation for the higher true-positive rate and lower
false-negative rate might be that the dataset of 2.8 million image-text pairs
was too small for the vision transformer to learn the visual concepts well
enough. As a result, the image features might not be extracted as well as with
the ResNet-50x4 model, which results in a lower true-positive rate. The lower
zero-shot top-1 accuracy on ImageNet might be one indication of this.

Investigating the cause for this behavior and the importance of the complexity
of the vision and text models, using other architectures, is an interesting
avenue for future work.

(a) ResNet-50

(b) ResNet-50x4

(c) ViT-B/32

Figure 5: Influence of the number of samples of each person included in the
CC3M dataset. Depicted are the mean and standard deviation of the true-
positive rate (TPR), false-negative rate (FNR), false-positive rate (FPR),
true-negative rate (TNR) and accuracy (Acc) of the attack.

Influence of Number of Attack Samples. As with the experiments on the
LAION-400M dataset, we are interested in what influence the number of samples
available to the attacker has on the success of the attack. The results for
the models trained with 75 image-text pairs for each person contained in the
training set can be seen in Fig. 4. Similar to the experiments on the
LAION-400M dataset, the accuracy and mean true-positive rate are increasing
with more samples being available for the attacker. In contrast to the
experiments on the LAION-400M dataset, the number of samples is influencing
the success of the attack even more. On all models, the accuracy as well as
the true-positive rate are increasing with the number of samples and does not
plateau with more than 5 samples as with the LAION-400M experiments. The
number of samples available to the attacker does not seem to effect the false-
positive rate. Even if the attacker has only one sample, the false-positive
rate is zero. As a result, the attacker can be confident in the positive
predictions the attack is doing, even when performing the attack with only a
single image per person.

To answer our first research question, it seems that the number of samples
available is indeed influencing the success of the attack. However, because
the false-positive rate is zero, an adversary with only one image per person
could still learn valuable information about the training data. The worst
thing that could happen for the attacker in this scenario is that samples who
actually were in the training data are incorrectly predicted. However, if the
attacker achieves only a few positive predictions, little but still reliable
information about the training data can be obtained.

(a) ResNet-50

(b) ResNet-50x4

(c) ViT-B/32

Figure 6: Influence of the number of samples available to the attacker and the
number of samples per person in the training data on the true-positive rate
(TPR) of the attack. With a low number of training samples per individual, the
advantage of having more samples available for the attack is less pronounced
for each of the models trained on the CC3M dataset.

Influence of Number of Training Samples. Because the number of samples used of
each person to train the models can be controlled more thoroughly in contrast
to the LAION-400M experiments, we want to investigate whether the number of
image-text pairs per individual influences the success of the attack. For this
experiment, for each of the three architectures we trained five models with 1,
10, 25, 50 and 75 image-text pairs per person mixed with the 2.8 million
image-text pairs of the Conceptual Captions 3M dataset.

Fig. 5 plots the mean true-positive, false-negative, false-positive and true-
negative rate as well as the accuracy of the attacks and their standard
deviation for different number of training samples per person. As can be seen
in all three plots, the attack still achieves reasonable results when there
are only 10 image-text pairs per person used for training. Even though the
true-positive rate is just above 20%, the mean false-positive rate as well as
the mean true-negative rate are 0% and 100%, respectively. As in the previous
experiments, this indicates that the adversary still can learn information
about the training data even if the person is present only 10 times in the
training data. Reducing the number of training samples per person to only 1
among the 2.8 million training samples and using 30 samples to attack each
identity, leads to a true-positive rate of 3.75% on the ResNet-50, 4.85% on
the ViT-B/32 model and even of 5.7% on the ResNet-50x4, as can be seen in Fig.
5. Even though these true-positive values are very low, it shows that it is
theoretically possible to identify a person on a single image among 2.8
million images.

Increasing the number of samples per person seems to have diminishing
influence on the success of the attacks for the ResNet-50 and the ViT-B/32
model. While the mean TPR is increasing by as much as 18% when increasing the
number of samples from 10 to 25, the true-positive rate is increasing only
slightly for most of the models when the number of samples is increased from
50 to 75. However, this effect of diminishing returns is not as pronounced
with the ResNet-50x4 model seen in Fig. 5(b). Contrary to the other two
models, the true-positive rate and the false-negative rate seem to improve
with more training samples available without diminishing returns. While both
the ResNet-50 and the ViT-B/32 have the same number of parameters for the text
encoder, the ResNet-50x4 model has a much bigger text encoder. This supports
our hypothesis that the text encoder is an important part for the success of
the attack. The bigger text encoder being able to memorize the names of the
individuals better might be a possible explanation for the improvements of the
attack with increasing the number of training samples per person.

This leaves the question open, what influence the number of samples available
to the attacker has on the success of the attack, depending on the number of
samples per person that were present in the training data. In Fig. 6, the
influence of the number of samples available to the attacker and the number of
samples in the training data per person on the true-positive rate can be seen.
As illustrated in the heatmaps of all three architectures trained on the CC3M
dataset, the number of samples available to the attacker has the highest
influence on the true-positive rate of IDIA when the identities were present
75 times in the training data set. Even though the TPR is increasing with the
number of samples when the individuals appeared only 10 times in the training
set, the effect is much less pronounced. As a general observation, one can say
that the more often a person appears in the training set, the higher the
influence of the number of samples of a person available to the attacker on
the success of the attack.

##  6 Discussion

Choosing the Threshold. Our experiments have shown that multi-modal models
recognize and memorize people and their names. In our experiments, we have
chosen the threshold for predicting when a person was part of the training
data as $\tau=0.5$. Therefore, if the name of the individual is predicted
correct more than half of the times, the attacker predicts that the person was
part of the training data. We selected this threshold because we wanted to
ensure a very strict attack scenario, with the attacker having as few as
possible information. However, the attacker could also search for a more
suitable threshold. In our attack scenario, the only information the attacker
has, are some images of a person and the name. In many privacy attacks, it is
assumed that the attacker has much more information than that. For membership
inference attacks, for example, it is assumed that the adversary has access to
data from the same data distribution as the training data to train a shadow
model. During the preparation of the attack, this shadow model is used to
choose a threshold for predicting members and non-members. Assuming a similar
scenario for our identity inference attack, where the attacker has additional
information, one could fine-tune the threshold $\tau$ to even further reduce
the false-negative rate. For example, the attacker could train an additional
multi-modal model and use it as a shadow model, or the adversary has
information about some entities being part of the training data. As a result,
the privacy leakage of IDIA could be even more severe.

Trade-Off between Power and Privacy. To put it simple, in our proposed
identity inference attack, the adversary is asking whether the model knows the
person. Using the prediction, the attacker can reveal whether the individual
was part of the training data. As researchers around the world are striving
for artificial general intelligence (AGI) machine learning models are
projected to be even more capable. However, as our experiments on the
LAION-400M dataset have shown, multi-modal models capable of learning visual
concepts and performing high accuracy zero-shot classification tasks are very
susceptible to identity inference attacks. While the ViT-B/32 model trained on
the LAION-400M dataset achieved a zero-shot top-1 accuracy of 62.9%, the true-
positive rate of the attack is above 90% while the false-positive rate is 0%.
Even though our experiments are not conclusive and future work has to
investigate this further, we assume that there is a trade-off between the
“emergent” abilities of machine learning models and their susceptibility to
privacy attacks.

Possible Countermeasures Our identity inference attack is exploiting the fact
that the model is memorizing the visual appearance of a person together with
its name. A possible counter measure to prevent these privacy attacks is to
mask out all names in the dataset, similar to the approach applied during the
creation of the Conceptual Captions 3M dataset [53]. However, it is
questionable if this will fully prevent privacy attacks. If the name of a
person is memorized, other attributes about this individual are most likely
memorized too. One could think of other textual attributes than the name of a
person. Taking celebrities, for example, one could think of films or TV-series
in which they played a role. Instead of using the name of the person, one
could use the name of the TV-series to infer whether the celebrity was in the
training data.

With models getting more powerful, one should think about whether some kind of
decency could be taught to these multi-modal models. If the model is aware
that the information it is about to share is private, it could refuse to give
an answer, similar to a doctor refusing to talk about the health status of
other patients. With the DALL-E [47] API, OpenAI is already doing a first step
in this direction. According to the content policy [42] of DALL-E, it is
prohibited to generate public figures like celebrities or politicians. Trying
to do so, the image generation is refused.

##  7 Conclusion

We have introduced a new type of privacy attack on multi-modal image-text
models, called identity inference attack (IDIA). Through several experiments,
we have demonstrated that using images and the name of the people, an attacker
can predict with very high accuracy whether a person was used to train the
model, to which (s)he has black-box access. Specifically, our evaluation on
models trained on the LAION-400M dataset has shown that even models trained on
400 million data points and generalized well enough to achieve a zero-shot
accuracy of more than 60% on the ImageNet validation dataset still leak
information. Our experiments on the CC3M dataset have shown that because of
the very low false-positive rate, the adversary can be confident in the
results of the attack, causing privacy leakage even if the identity appeared
as few as only a single time in a training set of 2.8 million images-text
pairs.

Our experimental results suggest that posing the right questions, an attacker
can extract sensitive information about the training data of a multi-modal
system and that there might exist a trade-off between the “emergent” abilities
of a machine learning model and privacy. Overall, our results suggest that
multi-modal image-text models indeed leak sensitive information about the
training data, and that the models should be handled with care.

IDIAs provide a number of interesting avenues for future work. While the
evaluation of the IDIAs in this work was limited to 3 model architectures on
two datasets each, it is important to investigate under which circumstances
the attacks are successful. The complexity and the interplay between the
vision model and the text encoder are most likely vital to the success of
IDIAs. Investigating the influence of the size of the vision and the text
model would be an interesting avenue for future work.

Even though we have shown that multi-modal zero-shot models like CLIP are
susceptible to privacy attacks, these kinds of attacks are not limited to
multi-modal modals with zero-shot capabilities. As text-to-image generation
models like DALL-E [47] or Imagen [49] are built on the same principle to
learn the visual concepts and connect them to texts, it is very likely that
those models as well suffer from training data leakage. Attacking such a
generative model, the adversary would need even less information. Looking at
the generated images would suffice because if the model does not know a
certain concept it cannot generate it. Similarly, with visual question
answering models like MAGMA [12] or Flamingo [1] the attacker could simply ask
the model the right questions like “Who can be seen on the photo?” or “What is
the person’s name?”.

## Acknowledgements

This work was supported by the German Ministry of Education and Research
(BMBF) within the framework program “Research for Civil Security” of the
German Federal Government, project KISTRA (reference no. 13N15343).

## References

  * [1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan.  Flamingo: a visual language model for few-shot learning.  CoRR, abs/2204.14198, 2022.
  * [2] G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici.  Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers.  International Journal of Security and Networks (IJSN), 10:137–150, 2015.
  * [3] R. Beaumont.  clip-retrieval, 2021.
  * [4] G. Bradski.  The OpenCV Library.  Dr. Dobb’s Journal of Software Tools, 2000.
  * [5] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramèr.  Membership inference attacks from first principles.  In Symposium on Security and Privacy (S&P), pages 1897–1914, 2022.
  * [6] N. Carlini and A. Terzis.  Poisoning and backdooring contrastive learning.  In International Conference on Learning Representations, (ICLR), 2022.
  * [7] N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel.  Extracting training data from large language models.  In USENIX Security Symposium, pages 2633–2650, 2021.
  * [8] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568, 2021.
  * [9] C. A. Choquette-Choo, F. Tramèr, N. Carlini, and N. Papernot.  Label-only membership inference attacks.  In International Conference on Machine Learning (ICML), pages 1964–1974, 2021.
  * [10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021.
  * [11] C. Eichenberg, S. Black, S. Weinbach, L. Parcalabescu, and A. Frank.  MAGMA - multimodal augmentation of generative models through adapter-based finetuning.  CoRR, abs/2112.05253, 2021.
  * [12] C. Eichenberg, S. Black, S. Weinbach, L. Parcalabescu, and A. Frank.  MAGMA - multimodal augmentation of generative models through adapter-based finetuning.  CoRR, abs/2112.05253, 2021.
  * [13] S. Fort.  Pixels still beat text: Attacking the openai clip model with text patches and adversarial pixel perturbations, 2021.  Online; accessed 24-August-2022.
  * [14] M. Fredrikson, S. Jha, and T. Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In Conference on Computer and Communications Security (CCS), pages 1322–1333, 2015.
  * [15] K. Ganju, Q. Wang, W. Yang, C. A. Gunter, and N. Borisov.  Property inference attacks on fully connected neural networks using permutation invariant representations.  In Conference on Computer and Communications Security (CCS), page 619–633, 2018.
  * [16] G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah.  Multimodal neurons in artificial neural networks.  Distill, 2021.  https://distill.pub/2021/multimodal-neurons.
  * [17] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  In International Conference on Learning Representations (ICLR), 2015\.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [19] K. Hill.  The secretive company that might end privacy as we know it.  https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html.  Online; accessed 24-August-2022.
  * [20] D. Hintersdorf, L. Struppek, and K. Kersting.  To trust or not to trust prediction scores for membership inference attacks.  In International Joint Conference on Artificial Intelligence (IJCAI), pages 3043–3049, 2022.
  * [21] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt.  OpenCLIP, 2021.
  * [22] M. Jagielski, N. Carlini, D. Berthelot, A. Kurakin, and N. Papernot.  High accuracy and high fidelity extraction of neural networks.  In USENIX Security Symposium, pages 1345–1362, 2020.
  * [23] M. Jaiswal and E. M. Provost.  Privacy enhanced multimodal neural representations for emotion recognition.  In AAAI Conference on Artificial Intelligence (AAAI), pages 7985–7993, 2020.
  * [24] JCS.  Goldene kamera 2012 - bastian pastewka 1.jpg, 2012.  https://commons.wikimedia.org/wiki/File:Goldene_Kamera_2012_-_Bastian_Pastewka_1.jpg, accessed Sep. 02, 2022, image cropped.
  * [25] JCS.  Goldene kamera 2012 - kai pflaume.jpg, 2013.  https://commons.wikimedia.org/wiki/File:Goldene_Kamera_2012_-_Kai_Pflaume.jpg, accessed Sep. 02, 2022, image cropped.
  * [26] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015.
  * [27] K. Krishna, G. S. Tomar, A. P. Parikh, N. Papernot, and M. Iyyer.  Thieves on sesame street! model extraction of bert-based apis.  In International Conference on Learning Representations (ICLR), 2020.
  * [28] G. Li, S. Rezaei, and X. Liu.  User-level membership inference attack against metric embedding learning.  In ICLR Workshop on Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2022.
  * [29] Z. Li and Y. Zhang.  Membership leakage in label-only exposures.  In Conference on Computer and Communications (CCS), pages 880–895, 2021.
  * [30] H. Liu, J. Jia, W. Qu, and N. Z. Gong.  Encodermi: Membership inference against pre-trained encoders in contrastive learning.  In Conference on Computer and Communications Security (CCS), pages 2081–2095, 2021.
  * [31] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In International Conference on Computer Vision (ICCV), 2015.
  * [32] N. Lomas.  Ftc settlement with ever orders data and ais deleted after facial recognition pivot.  https://techcrunch.com/2021/01/12/ftc-settlement-with-ever-orders-data-and-ais-deleted-after-facial-recognition-pivot/.  Online; accessed 01-September-2022.
  * [33] N. Lomas.  Italy fines clearview ai €20m and orders data deleted.  https://techcrunch.com/2022/03/09/clearview-italy-gdpr/.  Online; accessed 01-September-2022.
  * [34] N. Lomas.  Selfie scraping clearview ai hit with another €20m ban order in europe.  https://techcrunch.com/2022/07/13/clearview-greek-ban-order/.  Online; accessed 01-September-2022.
  * [35] N. Lomas.  Uk fines clearview just under $10m for privacy breaches.  https://techcrunch.com/2022/05/23/clearview-uk-ico-fine/.  Online; accessed 01-September-2022.
  * [36] Y. Miao, M. Xue, C. Chen, L. Pan, J. Zhang, B. Z. H. Zhao, D. Kaafar, and Y. Xiang.  The audio auditor: User-level membership inference in internet of things voice services.  Privacy Enhancing Technologies, 2021(1):209–228, 2021.
  * [37] R. Millière.  Adversarial attacks on image generation with made-up words.  CoRR, abs/2208.04135, 2022.
  * [38] R. Mokady, A. Hertz, and A. H. Bermano.  Clipcap: CLIP prefix for image captioning.  CoRR, abs/2111.09734, 2021.
  * [39] H. Ng and S. Winkler.  A data-driven approach to cleaning large face datasets.  In International Conference on Image Processing (ICIP), pages 343–347, 2014.
  * [40] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning (ICML), pages 16784–16804, 2022.
  * [41] D. A. Noever and S. E. M. Noever.  Reading isn’t believing: Adversarial attacks on multi-modal neurons.  CoRR, abs/2103.10480, 2021.
  * [42] OpenAI.  Content policy.  https://labs.openai.com/policies/content-policy.  Online; accessed 01-September-2022.
  * [43] N. Papernot, P. D. McDaniel, A. Swami, and R. E. Harang.  Crafting adversarial input sequences for recurrent neural networks.  In Military Communications Conference (MILCOM), pages 49–54, 2016\.
  * [44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning (ICML), pages 8748–8763, 2021.
  * [45] T. A. Rahman, M. Fritz, M. Backes, and Y. Zhang.  Everything about you: A multimodal approach towards friendship inference in online social networks.  CoRR, abs/2003.00996, 2020.
  * [46] S. S. Rajagopalan, L. Morency, T. Baltrusaitis, and R. Goecke.  Extending long short-term memory for multi-view structured learning.  In European Conference on Computer Vision (ECCV), pages 338–353, 2016.
  * [47] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning (ICML), pages 8821–8831, 2021.
  * [48] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [49] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  CoRR, abs/2205.11487, 2022.
  * [50] A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes.  Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models.  In Annual Network and Distributed System Security Symposium (NDSS), 2019.
  * [51] P. Schramowski, C. Tauchmann, and K. Kersting.  Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?  In Conference on Fairness, Accountability, and Transparency (FAccT), pages 1350–1361, 2022.
  * [52] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki.  LAION-400M: open dataset of clip-filtered 400 million image-text pairs.  CoRR, abs/2111.02114, 2021.
  * [53] P. Sharma, N. Ding, S. Goodman, and R. Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
  * [54] R. Shokri, M. Stronati, C. Song, and V. Shmatikov.  Membership inference attacks against machine learning models.  In Symposium on Security and Privacy (S&P), pages 3–18, 2017.
  * [55] C. Song and V. Shmatikov.  Auditing data provenance in text-generation models.  In A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis, editors, International Conference on Knowledge Discovery & Data Mining (KDD), pages 196–206, 2019.
  * [56] N. Srivastava and R. Salakhutdinov.  Multimodal learning with deep boltzmann machines.  In Conference on Neural Information Processing Systems (NeurIPS), pages 2231–2239, 2012.
  * [57] L. Struppek, D. Hintersdorf, A. D. A. Correia, A. Adler, and K. Kersting.  Plug & play attacks: Towards robust and flexible model inversion attacks.  In International Conference on Machine Learning (ICML), pages 20522–20545, 2022.
  * [58] L. Struppek, D. Hintersdorf, D. Neider, and K. Kersting.  Learning to break deep perceptual hashing: The use case neuralhash.  In Conference on Fairness, Accountability, and Transparency (FAccT), pages 58–69, 2022.
  * [59] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus.  Intriguing properties of neural networks.  In International Conference on Learning Representations (ICLR), 2014.
  * [60] M. Tan and Q. V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning (ICML), pages 6105–6114, 2019.
  * [61] J. Thies.  Clipme if you can!, 2022.  https://justusthies.github.io/posts/clipme/, accessed Aug. 10, 2022\.
  * [62] J. Trilling and R. Wetherill.  California company settles ftc allegations it deceived consumers about use of facial recognition in photo storage app.  https://www.ftc.gov/news-events/news/press-releases/2021/01/california-company-settles-ftc-allegations-it-deceived-consumers-about-use-facial-recognition-photo.  Online; accessed 01-September-2022.
  * [63] K. Wang, Y. Fu, K. Li, A. Khisti, R. S. Zemel, and A. Makhzani.  Variational model inversion attacks.  In Conference on Neural Information Processing Systems (NeurIPS), pages 9706–9719, 2021.
  * [64] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In Computer Security Foundations Symposium (CSF), pages 268–282, 2018.
  * [65] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu.  Scaling autoregressive models for content-rich text-to-image generation.  CoRR, abs/2206.10789, 2022.
  * [66] P. Zhang, Y. Li, Z. Huang, and H. Yin.  Privacy protection in deep multi-modal retrieval.  In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 634–643, 2021.
  * [67] Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song.  The secret revealer: Generative model-inversion attacks against deep neural networks.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 250–258, 2020.
  * [68] Z. Zhao, D. Dua, and S. Singh.  Generating natural adversarial examples.  In International Conference on Learning Representations (ICLR), 2018.


# You do not have to train Graph Neural Networks at all on text-attributed
graphs

Kaiwen Dong  University of Notre Dame<EMAIL_ADDRESS>,  Zhichun Guo
University of Notre Dame<EMAIL_ADDRESS>and  Nitesh V. Chawla  University of
Notre Dame<EMAIL_ADDRESS>
###### Abstract.

Graph structured data, specifically text-attributed graphs, effectively
represent relationships among varied entities such as social media users,
academic articles, researchers, and e-commerce items. Such graphs are
essential for semi-supervised node classification tasks where the goal is to
predict labels of unlabeled nodes using a sparse set of labeled references.
Graph Neural Networks (GNNs) have emerged as a powerful tool for handling this
graph-structured data. Although gradient descent is commonly utilized for
training GNNs for node classification, this study ventures into alternative
methods, eliminating the conventional training processes. We introduce
TrainlessGNN, a linear GNN model capitalizing on the observation that text
encodings from the same class often cluster together in a linear subspace.
This model constructs a weight matrix to represent each class’s node attribute
subspace, offering an efficient approach to semi-supervised node
classification on text-attributed graphs. Extensive experimental assessments
reveal that our trainless models can either match or even surpass their
conventionally trained counterparts, demonstrating the possibility of
refraining from gradient descent in certain configurations.

Graph Neural Networks, Semi-supervised Learning, Multiclass Classification

††conference: Proceedings of the ACM Web Conference 2023; June 03–05, 2018;
Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information
systems Data mining††ccs: Computing methodologies Supervised learning by
classification