#  A Data Centric Approach for Unsupervised Domain Generalization
via Retrieval from Web Scale Multimodal Data

Christopher Liao  Theodoros Tsiligkaridis  Brian Kulis

###### Abstract

Domain generalization (DG) is an important problem that learns a model that
can generalize to unseen test domains leveraging one or more source domains,
under the assumption of shared label spaces. However, most DG methods assume
access to abundant source data in the target label space, a requirement that
proves overly stringent for numerous real-world applications, where acquiring
the same label space as the target task is prohibitively expensive. For this
setting, we tackle the multimodal version of the _unsupervised domain
generalization_ (UDG) problem, which uses a large _task-agnostic unlabeled_
source dataset, such as LAION-2B during finetuning. Our framework does not
explicitly assume any relationship between the source dataset and target task.
Instead, it relies only on the premise that the source dataset can be
efficiently searched in a joint vision-language space. For this multimodal UDG
setting, we propose a novel method to build a small ($<$100K) subset of the
source data in three simple steps: (1) diversified retrieval using label names
as queries, (2) rank pseudo-labeling, and (3) clustering to find
representative samples. To demonstrate the value of studying the multimodal
UDG problem, we compare our results against state-of-the-art source-free DG
and zero-shot (ZS) methods on their respective benchmarks and show up to 10%
improvement in accuracy on 20 diverse target datasets. Additionally, our
multi-stage dataset construction method achieves 3% improvement on average
over nearest neighbors retrieval. Code is available:
https://github.com/Chris210634/mudg

Machine Learning, ICML

Figure 1: Proposed multimodal unsupervised domain generalization (MUDG) setup.
For finetuning, we are given: a large indexed unlabeled source image dataset,
the CLIP model used to search the index, and the target task definition (i.e.
label names). We follow a 3-step experimental procedure: (A) construct a
pseudo-labeled subset of the source data, (B) train the student CLIP model,
and (C) evaluate on the target task. We focus on _step A_ using a _small-
scale_ subset of source data.

##  1 Introduction

Domain generalization (DG) is widely studied in the computer vision literature
because the train and test image data distributions are different for many
applications. However, traditional DG methods assume access to labeled task-
specific source data, which is expensive for many real-world applications.
Consequently, more recent studies have tackled the unsupervised DG (UDG)
problem, where source labels are not used during finetuning (Zhang et al.,
2022; Narayanan et al., 2022; Harary et al., 2022). Unfortunately, this
experimental procedure is fairly restrictive and impractical, since it still
assumes that the source and target label spaces are identical. To address the
shortcoming, we propose to study a more realistic multimodal UDG (MUDG)
setting, where the source data is both unlabeled and “task-agnostic”, i.e. we
do not assume any relationship between the source and target label spaces.
When the task-specific assumption is relaxed, large-scale source data can be
leveraged, as it can be crawled cheaply from the web.

#### Multimodal unsupervised domain generalization (MUDG)

In order to leverage publicly available unlabeled web-crawled image data, such
as LAION (Schuhmann et al., 2022), we propose MUDG, a more general variation
of the UDG classification problem. “Multimodal” refers to the requirement for
the source dataset to be efficiently searchable in a joint vision-language
space using a pretrained CLIP model. Using this searchable index, our goal is
to build a pseudo-labeled subset of the source data to train a student CLIP
model for a given target classification task. See Figure 1 and Table 1 for a
concise illustration of our problem setting. In particular, our work can be
viewed as an extension of the recent source free domain generalization (SFDG)
problem (Cho et al., 2023), which only uses the target task definition during
finetuning. Compared to SFDG, finding a suitable subset of the source data
adds complexity to algorithm design, but our results show that the margin for
accuracy improvement is much larger under our MUDG setting.

#### Data-centric focus

As Figure 1 illustrates, we focus on developing a method to construct a
pseudo-labeled subset of the source data for finetuning. For training, we use
simple selective finetuning and prompt tuning, using the cross-entropy loss,
as this is not the focus of our work. In contrast, a large body of work under
the umbrella of “webly supervised” classification (Chen & Gupta, 2015; Li et
al., 2020) focuses almost exclusively on developing training algorithms to
learn from noisy web-crawled data, using the same fixed datasets collected
from a search engine (Li et al., 2017b; Sun et al., 2021). We hope to
complement webly supervised methods by redirecting some focus toward how the
finetuning dataset can be built from large open-source datasets (Schuhmann et
al., 2022). This “data-centric” perspective on method development is growing
in popularity in the pretraining literature, especially with last year’s
DataComp initiative (Gadre et al., 2023). However, most data-centric works
currently focus on distilling datasets to build more robust foundational
models at an industrial scale, whereas our paper focuses on task-specific
finetuning using a much smaller subset of images.

Method overview Toward our central goal of developing a strong method for
task-specific dataset construction, we propose a pipeline of three steps in
Figures 2 and 3. (1) _Diversified retrieval_ : augment the text prototypes
using pseudo-random tokens according to waffleCLIP (Roth et al., 2023), e.g.
“a photo of a chicken, $\langle$random$\rangle$.” Retrieve the nearest
neighbors of each augmented text prototype. The purpose of this step is to
diversify the retrieved images; for example, a dataset with only one type of
chicken is undesirable. (2) _Rank pseudo-labeling_ : label each retrieved
image with the closest text prototype in the CLIP embedding space. We use
retrieval rank to determine the closeness of image and text feature pairs, as
opposed to using the cosine similarity directly. This mitigates the well-known
“hubness” effect in the retrieval literature (Radovanovic et al., 2010;
Bogolin et al., 2022). (3) _Clustering_ : cluster image embeddings into $k$
clusters within each label group and randomly select one sample from each
cluster. The purpose of this step is twofold: to build a balanced dataset,
with $k$ samples per label; and to assure that no two images are semantically
similar. Section 3 explains the reasoning for each step in more detail. We use
the same finetuning procedure for all experiments to focus our study on
evaluating the proposed dataset building pipeline.

Our main contributions are:

  1. 1.

We introduce the multimodal UDG problem, a more modern variation on the
traditional UDG problem and a more tractable problem than source free DG.

  2. 2.

We present an intuitive data-centric method for MUDG.

  3. 3.

We present results on 20 diverse image classification datasets. We observe up
to 10% improvements in target accuracy compared to the SFDG and ZS baselines.
This shows that there is promise in studying the MUDG problem.

  4. 4.

We compare our proposed dataset construction pipeline with baseline retrieval
methods, achieving 3% improvement on average over nearest neighbor retrieval.

| Task-agnostic | Task-specific | Unlabeled | Task
---|---|---|---|---
Setting | Source | Source | Target | Definition
DA | - | labeled | $\checkmark$ | $\checkmark$
SFDA | - | - | $\checkmark$ | $\checkmark$
ZS | - | - | - | -
DG | - | labeled | - | $\checkmark$
UDG | - | unlabeled | - | $\checkmark$
SFDG | - | - | - | $\checkmark$
MUDG | $\checkmark$ | - | - | $\checkmark$
Table 1: Comparison of problem settings based on the information available at
training time. DA - domain adaptation; DG - domain generalization; SF - source
free; ZS - zero-shot. UDG - unsupervised DG. MUDG - multimodal UDG is our
setting. Compared to SFDG, MUDG requires an additional large unlabeled
dataset, which is not task-specific, such as LAION-2B.  Figure 2: Illustration
of our three step pipeline for constructing a finetuning dataset.  Figure 3:
Continuation of Figure 2. Figure 4:  Qualitative results for step 1:
diversified retrieval. Left: nearest neighbors to text query in LAION-2B.
Right: images retrieved using diversified text features. Images retrieved
using diverse queries cover a broader spectrum of appearances in the wild.

##  2 Related Works

#### Multimodal foundational models

Multimodal foundational models such as CLIP (Radford et al., 2021), ALIGN (Jia
et al., 2021), BLIP (Li et al., 2022b), and CoCa (Yu et al., 2022) use
separate image and language encoders to embed the two modalities into a joint
space. Once pretrained, these embeddings can be used to create an open-sourced
database searchable by both image and text (Schuhmann et al., 2022). Efficient
search on a scale of billions of images is enabled by approximate nearest
neighbor search libraries such as FAISS (Douze et al., 2024). A recent work
(Gadre et al., 2023) achieved the latest state-of-the-art on ZS ImageNet by
cleaning LAION-5B with a teacher CLIP model. Another recent work (Sun et al.,
2023) uses a large CLIP model and unpaired web-crawled data to train a smaller
foundational model in a distillation-inspired manner. The above works focus on
generalist pretraining from scratch, which remains out-of-reach of most
academic researchers. We focus instead on task-specific finetuning using a
constructed dataset of up to 100K samples. Our work is most similar to React
(Liu et al., 2023), which tackles the “model customization” problem. In
comparison, our work is more focused on the source subset construction portion
of the finetuning pipeline, and consequently, we achieve similar accuracy
improvements as React with a 100$\times$ smaller retrieved dataset.

#### Flavors of domain generalization

Table 1 is a non-exhaustive summary of variations on generalization settings
studied in recent literature. Domain adaptation (Sun et al., 2016; Saito et
al., 2019; Acuna et al., 2021) aims to leverage out-of-distribution (OOD) but
task-specific source data in conjunction with unlabeled target data.
Traditional DG (Muandet et al., 2013; Cha et al., 2022; Min et al., 2022)
trains on OOD task-specific source data from multiple domains, without
knowledge of target data. A more recent flavor of DG (Shu et al., 2023;
Khattak et al., 2023) trains on generic labeled source data (e.g. ImageNet)
with the goal of generalizing to any classification task, by leveraging
transferability of the image-text alignment in CLIP. Unsupervised DG (Zhang et
al., 2022; Narayanan et al., 2022; Harary et al., 2022) trains on unlabeled
task-specific source data. Source-free DG (SFDG) (Cho et al., 2023) aims to
increase pretrained accuracy with only the target task information, but the
improvement over ZS methods is not consistent empirically. ZS methods (Menon &
Vondrick, 2022; Pratt et al., 2023; Roth et al., 2023) improve accuracy by
ensembling multiple text features. Our problem setting, multimodal UDG, takes
advantage of plentiful unlabeled non-task-specific image data, which offers
more leverage than the SFDG setting, while not relying on any task-specific or
labeled images contrary to the DA and DG studies.

#### Webly supervised, open world, and open set

The webly supervised literature (Chen & Gupta, 2015; Li et al., 2020) focuses
on learning from a noisy web-crawled dataset (Li et al., 2017b; Sun et al.,
2021) and is very closely related to the large body of work on noisy
supervised learning, see survey (Song et al., 2022). These works focus on the
finetuning algorithm given a dataset, rather than the construction of the
training data, unlike our work. Another popular research direction focuses on
generalization to unseen classes given a certain set of (possibly related)
training classes; these works fall under open-set (Saito et al., 2021; Du et
al., 2023; Panareda Busto & Gall, 2017) open-world (Bendale & Boult, 2015;
Boult et al., 2019; Cao et al., 2022) or base-to-novel (Zhou et al., 2022;
Khattak et al., 2023; Kan et al., 2023) semi-supervised learning. Finally,
some works selectively retrieve from an unlabeled data pool to expand a
smaller set of labeled training samples (Sener & Savarese, 2017; Killamsetty
et al., 2021; Kim et al., 2023), referred to as core-set sampling. Contrary to
these works, we assume no labeled data of any kind at training time. Retrieval
augmentation (Blattmann et al., 2022; Gur et al., 2021; Long et al., 2022;
Iscen et al., 2023) is a related line of work which requires a retrieval
system at test time, adding substantially heavier evaluation overhead.

Figure 5: t-SNE plots of image features in the indexing model’s embedding
space, showing the benefits of steps 1 and 3. Simple nearest neighbor
retrieval (blue circle) covers only a small portion of the image distribution
for each label. Diversified retrieval (orange dot) covers a broader portion of
the image distribution, but contains semantically-redundant samples. After the
clustering step (green diamond), the selected image samples are evenly spaced
across the entire distribution, and thus the best representation for each
label.

##  3 Method

Our main method consists of a three-stage pipeline presented in Figures 2 and
3 used to build a pseudo-labeled subset of the source data. We then use the
standard cross entropy loss to finetune the last three layers of the student
CLIP encoders along with the text prompt. This finetuning procedure is not the
focus of our work, but details are included in the Experiments section.

#### Assumptions and notation.

We are given an unlabeled source dataset $\mathcal{X}_{s}$ (e.g. LAION-2B
English, with text labels discarded). $\mathcal{X}_{s}$ must be indexed in a
joint image-text embedding space by a pair of CLIP encoders
$f_{\text{index},\text{text}}$ and $f_{\text{index},\text{image}}$. Both are
frozen. We are also given label tokens for the target classification task,
formatted as “a photo of a $\langle$class name$\rangle$”, and denoted as
$\\{\mathbf{t}_{1},...,\mathbf{t}_{c}\\}$ where $c$ is the number of classes.
The goal is to optimize a “student” CLIP model $f_{\text{student}}$ to
classify images from the given classes. Note that $f_{\text{student}}$ and
$f_{\text{index}}$ can be the same or different models, and we experiment with
both possibilities.

###  3.1 Step 1: Diversified Retrieval

_Goal:_ Retrieve a diverse set of image data for training.

The simplest way to build a dataset from the list of class names is to
calculate the text feature for each class and retrieve the nearest neighbors
from $\mathcal{X}_{s}$. This is straightforward, but the results are not
promising as shown in the left of Figure 4. The retrieved images are not
identical, but contain very little variation. For instance, images of wallets
only contain one possible orientation; images of couches only contain stock
photos of a perfect couch. Figure 6 (the line with blue x) shows that when
trained on these images, the model severely overfits to the retrieved dataset.
To diversify the dataset, we augment the query text tokens using WaffleCLIP
(Roth et al., 2023), which appends random characters to the end of the query,
conceptually similar to adding random tokens. Concretely, let
$\\{\mathcal{A}_{1},...\mathcal{A}_{m}\\}$ denote a set of $m$ text
augmentation functions. We construct a dataset with $mc$ queries:
$\\{f_{\text{index},\text{text}}(\mathcal{A}_{j}(\mathbf{t}_{i}))|\forall
i=1:c,j=1:m\\}$. We retrieve the $n$ nearest neighbors to each query in
$\mathcal{X}_{s}$ and remove redundancies, resulting in a preliminary dataset
size of at most $mcn$.

For choosing the augmentation function, we consider two factors: (1) the
augmented query text does not change the label of the original text; and (2)
the resulting distribution of augmented queries covers the entire concept of
the class. Intuitively, WaffleCLIP is likely to preserve the label of the
original text, since it only appends random tokens to the end of the class
name. From inspecting Figure 4 right, WaffleCLIP also seems to capture a broad
range of visual variation within each class. We also demonstrate this
diversity using t-SNE plots in Figure 5.

Figure 6:  Target accuracy vs. training iterations for the datasets
corresponding to Figure 5 (colors match). This confirms our intuition that
both the diversified retrieval and clustering steps are necessary. Figure 7:
Hubness effect and the value of pseudo-labeling based on rank (step 2). The
x-axis labels are the label names for Terra Incognita. The label “empty” is a
hub because about 50 more images were labeled as empty when cosine similarity
is used instead of rank (left bar plot). The right two bar plots show the
precision and recall improvement _of rank labeling over cosine similarity
labeling_ , after clustering and training. Rank labeling improves precision
for images labeled as empty while improving the recall for most animal images.
_This is desireable_ : The cost of mislabeling an animal image as empty is
much greater than the cost of mislabeling an empty image.  Figure 8: A
selection of images from the 50 that were labeled as “empty” by cosine
similarity but as one of the animals by rank. Figure 9:  Target accuracy vs.
training iterations for the two datasets constructed for Figure 7 left. Colors
match. This confirms our intuition that rank labeling improves overall target
accuracy in addition to the good precision-recall properties in Figure 7
right.

Input: Source dataset $\mathcal{X}_{s}$, searchable in logarithmic time with
indexing model $f_{\text{index},\text{image}}$, $m$ augmentation functions
$\mathcal{A}_{1}...\mathcal{A}_{m}$, $n$, $k$, pretrained $f_{\text{index}}$
and $f_{\text{student}}$, class name tokens
$\\{\mathbf{t}_{1},...,\mathbf{t}_{c}\\}$.
Step 1: Let
$\mathcal{Q}=\\{f_{\text{index},\text{text}}(\mathcal{A}_{j}(\mathbf{t}_{i}))|\forall
i=1:c,j=1:m\\}$ denote the query set. For each $q\in\mathcal{Q}$, retrieve $n$
nearest neighbors in $\mathcal{X}_{s}$. Combine retrieved samples from all
queries and denote as $\mathcal{X}_{1}$.
Step 2: For each query $q\in\mathcal{Q}$, sort image samples
$\mathbf{x}\in\mathcal{X}_{1}$ by decreasing cosine similarity between $q$ and
$f_{\text{index, image}}(\mathbf{x})$. Denote the rank of $\mathbf{x}$
relative to $q$ as $\operatorname{rank}(\mathbf{x},q)\geq 1$. Then, assign
each $\mathbf{x}\in\mathcal{X}_{1}$ the label corresponding to the closest
query in terms of rank, i.e.
$\operatorname*{arg\,min}_{q}\operatorname{rank}(\mathbf{x},q)$. Denote the
labeled set as $\mathcal{X}_{2}$.
Step 3: Initialize an empty labeled dataset $\mathcal{X}_{3}$. For each label
$y\in 1:c$: Find the subset of $\mathcal{X}_{2}$ with label $y$. Calculate
their embeddings with $f_{\text{index},\text{image}}$. Cluster into $k$
clusters, using k-means. Randomly select one sample from each cluster and
append to $\mathcal{X}_{3}$.
$\mathcal{X}_{3}$ contains $ck$ samples.
Step 4: Finetune $f_{\text{student}}$ on $\mathcal{X}_{3}$ for $N$ iterations,
using standard cross entropy loss.
Output: finetuned $f_{\text{student}}$

Algorithm 1 MUDG

###  3.2 Step 2: Rank Pseudo-labeling

_Goal:_ Mitigate hubness effect.

This step is a small digression from the main story, please skip to step 3 for
high-level understanding. If each image sample is only retrieved by queries
from one label, then pseudo-labeling is trivial. However, there is a large
amount of overlap between retrievals from different labels, especially for
datasets with a large number of classes or fine-grained concepts. For each
image that is retrieved by multiple queries, we can assign it either (1) the
label of the closest text feature as measured by their cosine similarity, or
(2) the label of the text feature to which it is ranked the highest. We choose
the latter option (detailed concretely in Algorithm 1) to address the well-
known hubness effect (Radovanovic et al., 2010). In simple terms, hubs are
samples in the dataset which tend to be closer to other samples in a high-
dimensional embedding space, regardless of relevance. Specific to our
application, a “hub” text feature is one that is close to a disproportionately
large number of image samples, resulting in a large number of image samples
being assigned the hub label. In other words, the pseudo-label is biased
towards any hubs in the label space when cosine similarity is used directly.
However, when we use rank to assign labels, the hub label cannot be overused
because closeness to the hub is determined by rank relative to other image
samples.

Not all datasets have hubs, but we found that the Terra Incognita dataset
illustrates the effect perfectly. This dataset contains camera trap images of
different animals, and the labels are the animal names along with “empty” for
empty images. As a case study, we retrieve images from LAION-2B using step 1
and the query: “a photo of a $\langle$class name$\rangle$, from a camera
trap.”. We then compare pseudo-labeling using cosine similarity versus using
rank. The left bar plot in Figure 7 shows that cosine similarity pseudo-
labeling assigns some images the “empty” label, which are labeled as one of
the animals when using rank. Figure 8 displays examples of these images. For
this dataset, “empty” likely functions as a hub, since many camera-trap images
are mostly empty, especially if the animal is small. We verify in the two
right bar plots of Figure 7 that using rank pseudo-labeling improves the
recall of most animal images at the expense of decreasing the recall of empty
images. This is a favorable trade-off for this application. We further verify
in Figure 9 that rank pseudo-labeling improves the overall accuracy as well,
compared to cosine similarity labeling.

| Setting | ImageNet | Caltech | Pets | Cars | Flowers | Food | Aircraft | SUN | DTD | EuroSAT | UCF | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Open-AI CLIP ViT-B/16
CLIP ZS (Radford et al., 2021) | ZS | 67.1 | 93.3 | 89.0 | 65.4 | 71.0 | 85.7 | 24.9 | 63.2 | 43.6 | 46.6 | 67.4 | 65.2
waffleCLIP (Roth et al., 2023) | ZS | 68.2 | 93.5 | 88.1 | 65.5 | 72.1 | 85.9 | 25.6 | 66.2 | 44.3 | 47.3 | 68.1 | 65.9
Random Descriptors (Roth et al., 2023) | ZS | 68.1 | 94.3 | 87.7 | 65.7 | 71.7 | 85.7 | 25.2 | 66.2 | 44.7 | 47.7 | 67.3 | 65.8
Handcrafted Ensemble (Radford et al., 2021) | ZS | 68.4 | 93.5 | 88.8 | 66.0 | 71.1 | 86.0 | 24.9 | 66.0 | 43.9 | 45.0 | 68.0 | 65.6
GPT Descriptors (Menon & Vondrick, 2022) | ZS | 68.6 | 93.7 | 89.0 | 65.1 | 72.1 | 85.7 | 23.9 | 67.4 | 43.9 | 46.4 | 66.8 | 65.7
Nearest neighbors | MUDG | 69.4 | 93.9 | 93.4 | 70.2 | 75.8 | 86.3 | 27.2 | 67.4 | 52.4 | 41.2 | 69.9 | 67.9
Skip clustering step | MUDG | 69.1 | 94.4 | 93.1 | 73.0 | 76.4 | 86.2 | 33.7 | 68.4 | 54.0 | 57.9 | 71.8 | 70.7
MUDG (ours) | MUDG | 69.9 | 94.1 | 93.4 | 73.7 | 76.8 | 86.6 | 33.7 | 68.8 | 54.1 | 58.5 | 72.8 | 71.1
gain over ZS |  | +2.8 | +0.8 | +4.5 | +8.3 | +5.8 | +0.9 | +8.8 | +5.6 | +10.6 | +11.9 | +5.4 | +5.9
gain over waffleCLIP |  | +1.7 | +0.7 | +5.4 | +8.1 | +4.7 | +0.7 | +8.0 | +2.6 | +9.8 | +11.1 | +4.6 | +5.2
Open-AI CLIP ViT-L/14
CLIP ZS (Radford et al., 2021) | ZS | 73.8 | 94.6 | 93.6 | 76.9 | 79.4 | 90.9 | 32.8 | 68.0 | 52.7 | 56.2 | 74.7 | 72.1
waffleCLIP (Roth et al., 2023) | ZS | 75.0 | 96.1 | 93.5 | 77.1 | 78.8 | 90.9 | 33.6 | 69.3 | 54.3 | 57.7 | 75.3 | 72.9
Random Descriptors (Roth et al., 2023) | ZS | 75.1 | 96.9 | 93.4 | 76.7 | 78.5 | 90.7 | 33.6 | 70.1 | 54.5 | 59.3 | 75.5 | 73.1
Handcrafted Ensemble (Radford et al., 2021) | ZS | 75.6 | 95.6 | 94.0 | 78.1 | 79.8 | 91.2 | 32.7 | 70.5 | 54.0 | 55.2 | 75.0 | 72.9
GPT Descriptors (Menon & Vondrick, 2022) | ZS | 75.3 | 96.7 | 93.8 | 77.4 | 79.3 | 90.9 | 34.8 | 71.0 | 56.4 | 62.8 | 73.9 | 73.8
Nearest neighbors | MUDG | 76.2 | 95.8 | 95.3 | 78.0 | 80.2 | 91.3 | 33.3 | 71.7 | 56.2 | 61.6 | 75.6 | 74.1
Skip clustering step | MUDG | 75.5 | 96.0 | 95.3 | 79.3 | 79.7 | 91.2 | 36.6 | 72.2 | 56.3 | 69.5 | 76.5 | 75.3
MUDG (ours) | MUDG | 76.3 | 95.9 | 95.5 | 79.9 | 80.2 | 91.3 | 36.6 | 72.8 | 57.2 | 67.9 | 76.5 | 75.5
gain over ZS |  | +2.5 | +1.3 | +1.9 | +3.0 | +0.8 | +0.5 | +3.8 | +4.8 | +4.5 | +11.7 | +1.8 | +3.3
gain over waffleCLIP |  | +1.3 | –0.1 | +1.9 | +2.8 | +1.4 | +0.4 | +3.0 | +3.5 | +3.0 | +10.2 | +1.2 | +2.6
Table 2: Comparison of our MUDG baseline with ZS baselines on 11 diverse
datasets. Average of three experiments. For MUDG rows, dataset construction
and model training is separate for each dataset. “Nearest neighbors” refers to
simple nearest neighbors retrieval; “Skip clustering step” refers to omitting
step 3 in our pipeline. Figure 10: Label distribution of datasets constructed
before and after clustering (step 3). For this figure, $m=n=k=16$.

###  3.3 Step 3: Clustering

_Goal:_ Select representative samples and balance the label distribution.

Referring back to Figure 6, note that the dataset resulting from diversified
retrieval (in orange) actually lowers target accuracy on ImageNet when used to
train the student CLIP model, despite containing a large number of samples
($\mathcal{O}(mcn)$). This stems from two problems: (1) Some images are
semantic-duplicates as evident by the small clusters of orange dots in Figure
5, e.g. pictures of the same object in different orientations. (2) The dataset
is imbalanced as shown by the orange distribution over labels in Figure 10.
This is simply caused by asymmetries in the retrieval and download process
(e.g. dead links, linked image changed since dataset creation, etc.). As a
result, the training process overfits to dominant semantic-duplicate images
and the pseudo-label distribution; both are artifacts of the dataset
construction process.

To address both of the above issues, we first use k-means clustering to
cluster the image features in the embedding space of the indexing model into
$k<<mcn$ clusters, then randomly select an image from each cluster. If $k$ is
chosen conservatively, semantic duplicates fall into a single cluster, and
only one can be selected for the final training set. Additionally, each label
should have $k$ training samples. Figure 10 illustrates the final balanced
label distribution in green, and Figure 6 shows the corresponding target
accuracy improvements in matching colors. For reference, ImageNet-1K has
$c=1000$ labels, and we found $m=16$, $n=64$ and $k=48$ to yield good results.

|  | ImageNet | Office Home | Terra Incognita
---|---|---|---|---
|  | V2 | Sketch | A | R | Mean | A | C | P | R | Mean | 100 | 38 | 43 | 46 | Mean
Open-AI CLIP ViT-B/16
CLIP ZS | ZS | 60.9 | 46.6 | 47.2 | 74.1 | 57.2 | 82.6 | 67.2 | 88.8 | 89.6 | 82.1 | 51.5 | 26.1 | 34.1 | 29.3 | 35.2
waffleCLIP | ZS | 61.8 | 48.5 | 50.0 | 76.3 | 59.2 | 83.1 | 68.2 | 89.7 | 90.4 | 82.9 | 54.2 | 29.5 | 36.4 | 30.6 | 37.7
Random Descriptors | ZS | 61.7 | 48.8 | 49.9 | 76.6 | 59.2 | 83.0 | 69.1 | 89.5 | 90.2 | 83.0 | 51.3 | 21.7 | 36.7 | 28.8 | 34.6
Handcrafted Ensemble | ZS | 61.9 | 48.5 | 49.2 | 77.9 | 59.4 | 84.3 | 67.7 | 89.3 | 90.2 | 82.9 | 55.4 | 28.5 | 33.4 | 31.0 | 37.1
GPT Descriptors | ZS | 61.8 | 48.1 | 48.6 | 75.2 | 58.4 | - | - | - | - | - | - | - | - | - | -
PromptStyler $\dagger$ | SFDG | - | - | - | - | - | 83.8 | 68.2 | 91.6 | 90.7 | 83.6 | - | - | - | - | -
MUDG (ours) | MUDG | 63.0 | 49.9 | 50.6 | 78.6 | 60.5 | 85.4 | 72.8 | 91.9 | 91.3 | 85.4 | 53.3 | 32.3 | 37.7 | 25.3 | 37.2
gain over ZS |  | +2.1 | +3.4 | +3.4 | +4.5 | +3.3 | +2.8 | +5.6 | +3.2 | +1.8 | +3.3 | +1.8 | +6.2 | +3.6 | –4.0 | +1.9
gain over waffle |  | +1.2 | +1.4 | +0.6 | +2.3 | +1.4 | +2.2 | +4.6 | +2.3 | +0.9 | +2.5 | –0.9 | +2.9 | +1.3 | –5.3 | –0.5
Open-AI CLIP ViT-L/14
CLIP ZS | ZS | 68.0 | 57.9 | 68.3 | 85.5 | 69.9 | 87.1 | 74.8 | 93.1 | 93.4 | 87.1 | 46.3 | 50.9 | 43.0 | 32.4 | 43.1
waffleCLIP | ZS | 68.8 | 58.7 | 70.1 | 87.1 | 71.2 | 87.7 | 78.2 | 93.8 | 94.4 | 88.5 | 45.6 | 45.2 | 43.7 | 31.4 | 41.4
Random Descriptors | ZS | 69.2 | 59.1 | 70.5 | 87.1 | 71.5 | 88.2 | 78.4 | 94.4 | 94.0 | 88.7 | 40.9 | 36.3 | 38.5 | 26.3 | 35.5
Handcrafted Ensemble | ZS | 69.9 | 59.7 | 70.2 | 87.8 | 71.9 | 88.5 | 76.9 | 93.8 | 94.5 | 88.4 | 47.5 | 50.9 | 41.8 | 30.5 | 42.7
GPT Descriptors | ZS | 69.4 | 58.8 | 69.6 | 86.4 | 71.1 | - | - | - | - | - | - | - | - | - | -
PromptStyler $\dagger$ | SFDG | - | - | - | - | - | 89.1 | 77.6 | 94.8 | 94.8 | 89.1 | - | - | - | - | -
MUDG (ours) | MUDG | 70.4 | 60.6 | 71.5 | 88.7 | 72.8 | 90.3 | 80.9 | 94.8 | 94.8 | 90.2 | 49.3 | 54.4 | 44.6 | 32.5 | 45.2
gain over ZS |  | +2.4 | +2.7 | +3.2 | +3.2 | +2.9 | +3.2 | +6.2 | +1.7 | +1.4 | +3.1 | +3.0 | +3.5 | +1.6 | +0.1 | +2.1
gain over waffle |  | +1.6 | +1.9 | +1.4 | +1.6 | +1.6 | +2.6 | +2.8 | +1.1 | +0.5 | +1.7 | +3.7 | +9.3 | +0.9 | +1.1 | +3.8
Table 3: Comparison of our MUDG baseline with ZS baselines and PromptStyler on DG benchmarks. Average of three trials. Dataset construction and model training is performed once and evaluated on all domains for Office Home, Terra Incognita, and DomainNet; but we perform the steps separately for each ImageNet domain, due to differences in label spaces. PromptStyler (Cho et al., 2023) $\dagger$ results are those reported by the authors; all other results are our reproductions. | DomainNet
---|---
| C | I | P | Q | R | S | Mean
Open-AI CLIP ViT-B/16
CLIP ZS | 71.4 | 47.1 | 66.2 | 13.8 | 83.4 | 63.4 | 57.6
waffleCLIP | 73.0 | 52.0 | 68.3 | 14.0 | 84.9 | 65.8 | 59.7
Random Desc. | 73.5 | 51.0 | 67.6 | 14.6 | 84.7 | 65.9 | 59.6
Ensemble | 73.7 | 51.2 | 69.3 | 16.0 | 85.0 | 66.2 | 60.2
PromptStyler $\dagger$ | 73.1 | 50.9 | 68.2 | 13.3 | 85.4 | 65.3 | 59.4
MUDG (ours) | 74.6 | 53.6 | 68.9 | 15.4 | 85.2 | 66.1 | 60.6
gain over ZS | +3.2 | +6.5 | +2.7 | +1.6 | +1.8 | +2.6 | +3.1
gain over waffle | +1.5 | +1.7 | +0.6 | +1.4 | +0.3 | +0.3 | +1.0
Open-AI CLIP ViT-L/14
CLIP ZS | 79.5 | 52.2 | 70.9 | 22.5 | 86.8 | 71.5 | 63.9
waffleCLIP | 80.4 | 56.5 | 72.8 | 22.0 | 88.1 | 73.0 | 65.4
Random Desc. | 80.6 | 56.0 | 73.4 | 23.3 | 87.9 | 73.2 | 65.7
Ensemble | 81.1 | 55.8 | 73.9 | 24.2 | 87.9 | 73.7 | 66.1
PromptStyler $\dagger$ | 80.7 | 55.6 | 73.8 | 21.7 | 88.2 | 73.2 | 65.5
MUDG (ours) | 81.5 | 58.6 | 74.4 | 23.1 | 88.4 | 74.0 | 66.7
gain over ZS | +2.0 | +6.4 | +3.5 | +0.6 | +1.6 | +2.5 | +2.8
gain over waffle | +1.1 | +2.1 | +1.6 | +1.1 | +0.4 | +1.0 | +1.2
Table 4: DomainNet Results, continued from Table 3.

##  4 Experiments

We experiment with the ViT B/16 and ViT L/14 pretrained weights released by
Radford et al. (2021) and available through the Python openclip package
(Ilharco et al., 2021). The indexing model is ViT L/14; we use an index
released by the LAION team built using the FAISS package (Douze et al., 2024).
For details, please see the GitHub repository. The source dataset is
LAION-2B-en (Schuhmann et al., 2022). We experiment with two model sizes to
show that we achieve large gains in target accuracies even when the indexing
model and the student model are identical.

#### Datasets

We experiment with a diverse set of target classification tasks. ImageNet-1K
(Russakovsky et al., 2015), Caltech-101 (Li et al., 2022a), Oxford-Pets
(Parkhi et al., 2012), Stanford-Cars (Krause et al., 2013), Flowers-102
(Nilsback & Zisserman, 2008), Food-101 (Bossard et al., 2014), FGVC-Aircraft
(Maji et al., 2013), SUN-397 (Xiao et al., 2010), Describable-Textures (DTD)
(Cimpoi et al., 2013), EuroSAT (Helber et al., 2019), UCF-101 (an action
recognition dataset) (Soomro et al., 2012) in Table 2 and ImageNet-V2 (Recht
et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (natural
adversarial examples) (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et
al., 2021a) in Table 3 are commonly used by zero-shot papers, while Office
Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018),
DomainNet (Peng et al., 2019), VLCS (Torralba & Efros, 2011), and PACS (Li et
al., 2017a) are common DG and DA datasets. VLCS and PACS results are in
Appendix A, since they are less challenging.

#### Baselines

We compare to both ZS and SFDG baselines, since they achieve similar results
on average, despite SFDG having stronger assumptions. Note that all ZS and
SFDG baselines (other than CLIP ZS) ensemble multiple text features at test
time; MUDG baselines do not require ensembling at test time, and this is a
slight advantage. The published ZS baseline methods are: waffleCLIP (Roth et
al., 2023), handcrafted ensemble (Radford et al., 2021) and GPT descriptors
(Menon & Vondrick, 2022). We also include a strong random descriptor baseline
which ensembles randomly selected GPT descriptors, introduced by waffleCLIP.
Note that the handcrafted ensemble is manually engineered for standard visual
domains (e.g. painting and sketch), so achieves the highest accuracy on some
DomainNet subsets, but under-performs significantly on fine-grained tasks. To
the best of our knowledge, PromptStyler (Cho et al., 2023) is the only current
SFDG baseline.

#### Ablations

We provide two ablation comparisons across 11 datasets in Table 2 in the MUDG
setting. We first show that our retrieval pipeline improves average accuracy
by 3.2% over nearest neighbors retrieval for the B/16 model size. We then show
that omitting the clustering step decreases average accuracy by 0.4%. Also
importantly, the clustering step decreases the training dataset size
significantly.

#### Hyperparameters

See Tables 6 and 7 of the Appendix. An ablation study on $m$, $n$, and $k$ is
included in Figure 11.

##  5 Conclusion

This work introduced the multimodal unsupervised domain generalization
problem, an extension of the zero-shot and source-free domain generalization
problems. MUDG uses a searchable, large, non-task-specific, unlabeled dataset,
which adds substantial leverage over existing variations of the DG/ZS problem.
We proposed an intuitive three-stage process for creating a task-specific
subset of the source data, and reported promising accuracy improvements across
20 diverse benchmarks. This work highlights the importance of the dataset
construction step of the finetuning pipeline, and further complements existing
work on finetuning methods.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here.

## Acknowledgments

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
unlimited.

This material is based upon work supported by the Under Secretary of Defense
for Research and Engineering under Air Force Contract No. FA8702-15-D-0001.
Any opinions, findings, conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the Under Secretary of Defense for Research and Engineering.

## References

  * Acuna et al. (2021) Acuna, D., Zhang, G., Law, M. T., and Fidler, S.  f-domain adversarial learning: Theory and algorithms.  In _International Conference on Machine Learning_ , pp. 66–75. PMLR, 2021.
  * Beery et al. (2018) Beery, S., Van Horn, G., and Perona, P.  Recognition in terra incognita.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 456–473, 2018.
  * Bendale & Boult (2015) Bendale, A. and Boult, T.  Towards open world recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1893–1902, 2015.
  * Blattmann et al. (2022) Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B.  Retrieval-augmented diffusion models.  _Advances in Neural Information Processing Systems_ , 35:15309–15324, 2022.
  * Bogolin et al. (2022) Bogolin, S.-V., Croitoru, I., Jin, H., Liu, Y., and Albanie, S.  Cross modal retrieval with querybank normalisation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5194–5205, 2022.
  * Bossard et al. (2014) Bossard, L., Guillaumin, M., and Van Gool, L.  Food-101 – mining discriminative components with random forests.  In _European Conference on Computer Vision_ , 2014.
  * Boult et al. (2019) Boult, T. E., Cruz, S., Dhamija, A. R., Gunther, M., Henrydoss, J., and Scheirer, W. J.  Learning and the unknown: Surveying steps toward open world recognition.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 9801–9807, 2019.
  * Cao et al. (2022) Cao, K., Brbic, M., and Leskovec, J.  Open-world semi-supervised learning.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Cha et al. (2022) Cha, J., Lee, K., Park, S., and Chun, S.  Domain generalization by mutual-information regularization with pre-trained models.  In _European Conference on Computer Vision_ , pp. 440–457. Springer, 2022.
  * Chen & Gupta (2015) Chen, X. and Gupta, A.  Webly supervised learning of convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1431–1439, 2015.
  * Cho et al. (2023) Cho, J., Nam, G., Kim, S., Yang, H., and Kwak, S.  Promptstyler: Prompt-driven style generation for source-free domain generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 15702–15712, 2023.
  * Cimpoi et al. (2013) Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A.  Describing textures in the wild.  _CoRR_ , abs/1311.3618, 2013.  URL http://arxiv.org/abs/1311.3618.
  * Douze et al. (2024) Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H.  The faiss library.  _arXiv preprint arXiv:2401.08281_ , 2024.
  * Du et al. (2023) Du, P., Zhao, S., Sheng, Z., Li, C., and Chen, H.  Semi-supervised learning via weight-aware distillation under class distribution mismatch.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 16410–16420, 2023.
  * Gadre et al. (2023) Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al.  Datacomp: In search of the next generation of multimodal datasets.  _arXiv preprint arXiv:2304.14108_ , 2023.
  * Gur et al. (2021) Gur, S., Neverova, N., Stauffer, C., Lim, S.-N., Kiela, D., and Reiter, A.  Cross-modal retrieval augmentation for multi-modal classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 111–123, 2021.
  * Harary et al. (2022) Harary, S., Schwartz, E., Arbelle, A., Staar, P., Abu-Hussein, S., Amrani, E., Herzig, R., Alfassy, A., Giryes, R., Kuehne, H., et al.  Unsupervised domain generalization by learning a bridge across domains.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5280–5290, 2022.
  * Helber et al. (2019) Helber, P., Bischke, B., Dengel, A., and Borth, D.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7):2217–2226, 2019.
  * Hendrycks et al. (2021a) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8340–8349, 2021a.
  * Hendrycks et al. (2021b) Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15262–15271, 2021b.
  * Ilharco et al. (2021) Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.  If you use this software, please cite it as below.
  * Iscen et al. (2023) Iscen, A., Caron, M., Fathi, A., and Schmid, C.  Retrieval-enhanced contrastive vision-text models.  _arXiv preprint arXiv:2306.07196_ , 2023.
  * Jia et al. (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pp. 4904–4916. PMLR, 2021.
  * Kan et al. (2023) Kan, B., Wang, T., Lu, W., Zhen, X., Guan, W., and Zheng, F.  Knowledge-aware prompt tuning for generalizable vision-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 15670–15680, 2023.
  * Khattak et al. (2023) Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19113–19122, 2023.
  * Killamsetty et al. (2021) Killamsetty, K., Zhao, X., Chen, F., and Iyer, R.  Retrieve: Coreset selection for efficient and robust semi-supervised learning.  _Advances in Neural Information Processing Systems_ , 34:14488–14501, 2021.
  * Kim et al. (2023) Kim, S., Bae, S., and Yun, S.-Y.  Coreset sampling from open-set for fine-grained self-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7537–7547, 2023.
  * Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pp. 554–561, 2013.
  * Li et al. (2017a) Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5542–5550, 2017a.
  * Li et al. (2022a) Li, F.-F., Andreeto, M., Ranzato, M., and Perona, P.  Caltech 101, Apr 2022a.
  * Li et al. (2020) Li, J., Xiong, C., and Hoi, S. C.  Mopro: Webly supervised learning with momentum prototypes.  _arXiv preprint arXiv:2009.07995_ , 2020.
  * Li et al. (2022b) Li, J., Li, D., Xiong, C., and Hoi, S.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pp. 12888–12900. PMLR, 2022b.
  * Li et al. (2017b) Li, W., Wang, L., Li, W., Agustsson, E., and Van Gool, L.  Webvision database: Visual learning and understanding from web data.  _arXiv preprint arXiv:1708.02862_ , 2017b.
  * Liu et al. (2023) Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., and Li, C.  Learning customized visual models with retrieval-augmented knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15148–15158, 2023.
  * Long et al. (2022) Long, A., Yin, W., Ajanthan, T., Nguyen, V., Purkait, P., Garg, R., Blair, A., Shen, C., and van den Hengel, A.  Retrieval augmented classification for long-tail visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6959–6969, 2022.
  * Maji et al. (2013) Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., and Vedaldi, A.  Fine-grained visual classification of aircraft.  _CoRR_ , abs/1306.5151, 2013.  URL http://arxiv.org/abs/1306.5151.
  * Menon & Vondrick (2022) Menon, S. and Vondrick, C.  Visual classification via description from large language models.  _arXiv preprint arXiv:2210.07183_ , 2022.
  * Min et al. (2022) Min, S., Park, N., Kim, S., Park, S., and Kim, J.  Grounding visual representations with texts for domain generalization.  In _European Conference on Computer Vision_ , pp. 37–53. Springer, 2022.
  * Muandet et al. (2013) Muandet, K., Balduzzi, D., and Schölkopf, B.  Domain generalization via invariant feature representation.  In _International conference on machine learning_ , pp. 10–18. PMLR, 2013.
  * Narayanan et al. (2022) Narayanan, V., Deshmukh, A. A., Dogan, U., and Balasubramanian, V. N.  On challenges in unsupervised domain generalization.  In _NeurIPS 2021 Workshop on Pre-registration in Machine Learning_ , pp. 42–58. PMLR, 2022.
  * Nilsback & Zisserman (2008) Nilsback, M.-E. and Zisserman, A.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing_ , pp. 722–729, 2008.  doi: 10.1109/ICVGIP.2008.47.
  * Panareda Busto & Gall (2017) Panareda Busto, P. and Gall, J.  Open set domain adaptation.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 754–763, 2017.
  * Parkhi et al. (2012) Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , pp. 3498–3505. IEEE, 2012.
  * Peng et al. (2019) Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1406–1415, 2019.
  * Pratt et al. (2023) Pratt, S., Covert, I., Liu, R., and Farhadi, A.  What does a platypus look like? generating customized prompts for zero-shot image classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 15691–15701, 2023.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision, 2021.
  * Radovanovic et al. (2010) Radovanovic, M., Nanopoulos, A., and Ivanovic, M.  Hubs in space: Popular nearest neighbors in high-dimensional data.  _Journal of Machine Learning Research_ , 11(sept):2487–2531, 2010.
  * Recht et al. (2019) Recht, B., Roelofs, R., Schmidt, L., and Shankar, V.  Do imagenet classifiers generalize to imagenet?  _CoRR_ , abs/1902.10811, 2019.  URL http://arxiv.org/abs/1902.10811.
  * Roth et al. (2023) Roth, K., Kim, J. M., Koepke, A., Vinyals, O., Schmid, C., and Akata, Z.  Waffling around for performance: Visual classification with random words and broad concepts.  _arXiv preprint arXiv:2306.07282_ , 2023.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Saito et al. (2019) Saito, K., Kim, D., Sclaroff, S., Darrell, T., and Saenko, K.  Semi-supervised domain adaptation via minimax entropy.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 8050–8058, 2019.
  * Saito et al. (2021) Saito, K., Kim, D., and Saenko, K.  Openmatch: Open-set consistency regularization for semi-supervised learning with outliers.  _arXiv preprint arXiv:2105.14148_ , 2021.
  * Schuhmann et al. (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Sener & Savarese (2017) Sener, O. and Savarese, S.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_ , 2017.
  * Shu et al. (2023) Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., and Long, M.  Clipood: Generalizing clip to out-of-distributions, 2023.
  * Song et al. (2022) Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.-G.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Soomro et al. (2012) Soomro, K., Zamir, A. R., and Shah, M.  UCF101: A dataset of 101 human actions classes from videos in the wild.  _CoRR_ , abs/1212.0402, 2012.  URL http://arxiv.org/abs/1212.0402.
  * Sun et al. (2016) Sun, B., Feng, J., and Saenko, K.  Return of frustratingly easy domain adaptation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30, 2016.
  * Sun et al. (2023) Sun, X., Zhang, P., Zhang, P., Shah, H., Saenko, K., and Xia, X.  Dime-fm: Distilling multimodal and efficient foundation models.  _arXiv preprint arXiv:2303.18232_ , 2023.
  * Sun et al. (2021) Sun, Z., Yao, Y., Wei, X.-S., Zhang, Y., Shen, F., Wu, J., Zhang, J., and Shen, H. T.  Webly supervised fine-grained recognition: Benchmark datasets and an approach.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 10602–10611, 2021.
  * Torralba & Efros (2011) Torralba, A. and Efros, A. A.  Unbiased look at dataset bias.  In _CVPR 2011_ , pp. 1521–1528. IEEE, 2011.
  * Venkateswara et al. (2017) Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5018–5027, 2017.
  * Wang et al. (2019) Wang, H., Ge, S., Xing, E. P., and Lipton, Z. C.  Learning robust global representations by penalizing local predictive power.  _CoRR_ , abs/1905.13549, 2019.  URL http://arxiv.org/abs/1905.13549.
  * Xiao et al. (2010) Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 3485–3492, 2010.  doi: 10.1109/CVPR.2010.5539970.
  * Yu et al. (2022) Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y.  Coca: Contrastive captioners are image-text foundation models, 2022.
  * Zhang et al. (2022) Zhang, X., Zhou, L., Xu, R., Cui, P., Shen, Z., and Liu, H.  Towards unsupervised domain generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4910–4920, 2022.
  * Zhou et al. (2022) Zhou, K., Yang, J., Loy, C. C., and Liu, Z.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, jul 2022.  doi: 10.1007/s11263-022-01653-1.  URL https://doi.org/10.1007%2Fs11263-022-01653-1.

## Limitations

Even though we do not explicitly assume any relationships between the source
and target data, our work may not be applicable to problems where the target
visual concepts are either not present in the source dataset or completely
misaligned with corresponding language concepts. Possible examples include
synthetic aperture radar images, images of tissue samples, or medical scans.
Additionally, our method may not improve results on datasets where the zero-
shot accuracy is already saturated. For example, the VLCS dataset contains 5
classes: bird, car, chair, dog, and person. Our method does not achieve any
meaningful improvement over the ZS baseline on these simple classification
tasks.

##  Appendix A Experimental Details and Additional Results

#### Domain abbreviations

Office Home domains: A - art; C - clipart; P - product; R - real. Terra
Incognita domain names are anonymous location identifiers for camera traps.
DomainNet domains: C - clipart; I - infograph; P - painting; Q - quickdraw; R
- real; S - sketch. PACS domains: A - art; C - cartoon; P - photo; S - sketch.
VLCS domain names are dataset names.

#### Hyperparameters

The finetuning parameters are displayed in Table 6. The training set
construction parameters $n$, $m$, and $k$ are dataset-specific and listed in
Table 7. Moreover, the number of training iterations $N$ and query/prompt
template also varies with the dataset, as listed in Table 7.

#### Ablation study

An ablation study on the training set construction parameters $n$, $m$, and
$k$ are included in Figure 11. We perform these experiments for ImageNet,
DomainNet, and Office Home. When varying the values of $n$ and $m$, we scale
the value of $k$ by the same amount. Note that changing the values of these
hyperparameters changes the size of the training dataset. For example, scaling
$k$ by 2 scales the number of training samples by the same amount. The main
take-away from Figure 11 is that increasing the number of samples in the
training data improves the target accuracy, but only up to a point. The target
accuracy saturates at some point, and it is not beneficial to increase $n$,
$m$, or $k$ further.

#### Additional Notes

We do not verify the check-sums of the downloaded images, instead we filter
out retrieved images where the cosine similarity between the image embedding
and query text embedding is very low ($<$0.25). The size of the retrieved
datasets is listed in Table 7, and we emphasize again that our framework
achieves impressive improvements in accuracy with a small number of retrieved
image samples ($<$100K).

Figure 11: Ablation experiments for varying values of $n$, $m$ and $k$. Reference Algorithm 1 in the main paper and Table 7 in the Appendix for default values. Top row: ImageNet; middle row: DomainNet; bottom row: Office Home. Increasing either $n$, $m$ or $k$ improves the target accuracy by retrieving a larger training set, but these plots show that the accuracy saturates at a certain value.  | PACS | VLCS
---|---|---
| A | C | P | S | Mean | Caltech | Labelme | SUN | VOC | Mean
Open-AI CLIP ViT-B/16
CLIP ZS | 97.1 | 99.0 | 99.9 | 88.0 | 96.0 | 99.9 | 68.3 | 75.3 | 85.5 | 82.2
waffleCLIP | 97.3 | 99.0 | 99.9 | 90.3 | 96.6 | 99.9 | 68.6 | 74.4 | 86.3 | 82.3
Random Descriptors | 97.1 | 99.2 | 99.9 | 89.2 | 96.4 | 99.9 | 70.3 | 77.9 | 87.0 | 83.8
Ensemble | 97.6 | 99.2 | 99.9 | 89.9 | 96.7 | 99.9 | 69.1 | 76.4 | 84.2 | 82.4
PromptStyler $\dagger$ | 97.6 | 99.1 | 99.9 | 92.3 | 97.2 | 99.9 | 71.5 | 73.9 | 86.3 | 82.9
MUDG (ours) | 97.9 | 99.2 | 99.9 | 90.7 | 96.9 | 99.9 | 65.5 | 78.5 | 86.3 | 82.6
Open-AI CLIP ViT-L/14
CLIP ZS | 98.8 | 99.6 | 99.9 | 95.6 | 98.5 | 99.9 | 70.7 | 73.8 | 85.7 | 82.5
waffleCLIP | 99.1 | 99.7 | 100.0 | 95.7 | 98.6 | 99.9 | 70.8 | 74.1 | 87.1 | 83.0
Random Descriptors | 98.9 | 99.6 | 100.0 | 95.6 | 98.5 | 99.9 | 67.6 | 78.0 | 86.4 | 83.0
Ensemble | 98.8 | 99.6 | 100.0 | 95.7 | 98.5 | 99.9 | 65.5 | 76.1 | 85.1 | 81.7
PromptStyler $\dagger$ | 99.1 | 99.7 | 100.0 | 95.5 | 98.6 | 99.9 | 71.1 | 71.8 | 86.8 | 82.4
MUDG (ours) | 98.8 | 99.6 | 100.0 | 95.8 | 98.6 | 99.9 | 70.0 | 75.5 | 86.0 | 82.9
Table 5: Comparison of our MUDG method with ZS baselines and PromptStyler on
PACS and VLCS. Average of three trials. Dataset construction and model
training is performed once and evaluated on all domains. $\dagger$ denotes
author reported numbers; all other results are our reproductions. Finetuning
Parameters
---
| ViT-B/16 | ViT-L/14
Finetune last 3 layers of text and vision encoders
batch size | 128 | 64
learning rate | 0.00064 | 0.00016
weight decay | 1e-5
number of iterations ($N$) | dataset dependent
learning rate decay | none
softmax temperature | 25
optimizer | SGD momentum=0.9
label smoothing | 0
EMA weight averaging $\beta$ | 0.995
text prompt length | 3
text prompt initialization | “a photo of”
text prompt learning rate multiplier | 10 $\times$
Parameters for Baselines
WaffleCLIP ensemble size | 8
Table 6: Training hyperparameters. |  |  |  |  | Actual training |
---|---|---|---|---|---|---
Dataset | $n$ | $m$ | $k$ | $N$ | dataset size | Query template
ImageNet | 64 | 16 | 96 | 300 | 96K | a photo of a {}.
Caltech | 64 | 64 | 384 | 100 | 38K | a photo of a {}.
Pets | 64 | 64 | 384 | 200 | 12K | a photo of a {}, a type of pet.
Cars | 64 | 16 | 96 | 1000 | 18K | a photo of a {}.
Flowers | 64 | 64 | 384 | 200 | 31K | a photo of a {}, a type of flower.
Food | 64 | 64 | 384 | 100 | 34K | a photo of a {}, a type of food.
Aircraft | 64 | 64 | 384 | 1000 | 26K | a photo of a {}, a type of aircraft.
SUN | 64 | 16 | 96 | 300 | 38K | a photo of a {}.
DTD | 64 | 64 | 384 | 200 | 18K | a photo of a {} texture.
EuroSAT | 64 | 64 | 384 | 200 | 3K | a photo of a {}, from a satellite.
UCF | 64 | 64 | 384 | 200 | 37K | a photo of a person doing {}.
ImageNet-V2 | 64 | 16 | 96 | 200 | 96K | a photo of a {}.
ImageNet-Sketch | 64 | 16 | 96 | 200 | 96K | a photo of a {}.
ImageNet-A | 64 | 16 | 96 | 200 | 19K | a photo of a {}.
ImageNet-R | 64 | 16 | 96 | 200 | 19K | a photo of a {}.
DomainNet | 64 | 16 | 96 | 200 | 33K | a photo of a {}.
Office Home | 64 | 64 | 384 | 200 | 25K | a photo of a {}.
PACS | 64 | 64 | 384 | 100 | 3K | a photo of a {}.
VLCS | 64 | 64 | 384 | 50 | 2K | a photo of a {}.
Terra Incognita | 64 | 64 | 384 | 100 | 3K | a photo of a {}, from a camera trap.
Table 7: Dataset-specific hyperparameters, reference Algorithm 1 in the main
paper. $n$ is number of nearest neighbors to be retrieved; $m$ is number of
text augmentations; $k$ is number of k-means clusters; $N$ is number of
training iterations.