# LanDA: Language-Guided Multi-Source Domain Adaptation

Zhenbin Wang
Sichuan University
<EMAIL_ADDRESS>Lei Zhang, Lituan Wang
Sichuan University
<EMAIL_ADDRESS>The corresponding author. Minjuan Zhu
Sichuan University
<EMAIL_ADDRESS>
###### Abstract

Multi-Source Domain Adaptation (MSDA) aims to mitigate changes in data
distribution when transferring knowledge from multiple labeled source domains
to an unlabeled target domain. However, existing MSDA techniques assume target
domain images are available, yet overlook image-rich semantic information.
Consequently, an open question is whether MSDA can be guided solely by textual
cues in the absence of target domain images. By employing a multimodal model
with a joint image and language embedding space, we propose a novel language-
guided MSDA approach, termed LanDA, based on optimal transfer theory, which
facilitates the transfer of multiple source domains to a new target domain,
requiring only a textual description of the target domain without needing even
a single target domain image, while retaining task-relevant information. We
present extensive experiments across different transfer scenarios using a
suite of relevant benchmarks, demonstrating that LanDA outperforms standard
fine-tuning and ensemble approaches in both target and source domains.

##  1 Introduction

Generalizing outside of the training domain is vital for constructing robust
computer vision models, and Domain Adaptation (DA) is an efficacious and
prevalent technique to achieve this capability. When multiple source domains
are available and multiple distribution shifts occur simultaneously, the
problem evolves into a more challenging scenario known as Multi-Source Domain
Adaptation (MSDA). Conventional MSDA techniques [31, 1, 44] rely on visual
backbones, often necessitating a substantial volume of additional target
domain image data to acquire the target domain’s probability distribution. In
many settings, obtaining target domain images may pose challenges. Collecting
data for an uncommon target domain is frequently a time-consuming and costly,
yet describing them linguistically can be relatively straightforward (_e.g_.,
domains characterized like nighttime, fog, painting, and infrared light).
Therefore, we propose a novel and unexplored problem: how to improve target
domain performance using solely language and multiple source domain images,
absence any target domain images.

Visual-Language Foundational Models (VLFMs), such as CLIP [36], ALING [20],
Flamingo [2], BLIP [25, 26], present significant potential for using language
to guide visual tasks by aligning text and images within shared embedding
spaces. However, the effective utilization of these powerful foundational
models is non-trivial in design. Moreover, recent research [48] has show that
the backbone network of VLFMs significantly outperform many backbone networks
exclusively trained on images, including ResNet [18], visual transformer [13],
and big transfer [23]. Indeed, VLFMs have gradually become a commonly used
feature extraction network in computer vision [38]. While VLFMs demonstrate
impressive generalization capabilities, zero-shot classifiers derived from
them frequently yield significantly poorer performance compared to models
specifically trained for downstream tasks [24, 36]. When multiple source
domain datasets are accessible, a common practice is to fine-tune models on
the source domains data. While this enhances accuracy on the source domains,
it often diminishes performance on unseen target domains.

Currently, only limited works explored VLFMs for distribution migration
problems. While given that full parameter fine-tuning may distort the
favorable attributes of pre-trained features [24], prior research has
primarily followed two main routes, i.e., designing template prompts then
adapting prompts to suit downstream tasks [52, 51, 4, 38], and implementing a
linear probe on the features then fine-tuning the model’s backbone [14].
Manual prompt template design lacks clarity and scalability, rendering it
unsuitable for widespread adoption. In contrast, linear probing has been shown
to yield more resilient classifiers [36, 24]. However, those methods primarily
focus on single-source domain settings.

In this work, we present LanDA, a language-guided MSDA method that utilizes
CLIP or similar models as the backbone network, harnessing their extensive
domain-level knowledge (_e.g_., realistic, painting and sketch) within a
shared image-language embedding space. Our approach requires only source
domain images and textual descriptions of both the target and source domains
(_e.g_., the description of source domains is a realistic photo of a [CLS], a
painting of a [CLS], and the target domain is a sketch of a [CLS]).

Specifically, we freeze CLIP’s parameters and incorporate lightweight domain-
specific augmenters (composed of multi-layer perceptron) into the image
embedding for each source domain, the resulting augmentered embeddings are
termed as extended domains. To align the distribution and obtain domain-
invariant features, we introduce a method for computing the Wasserstein cost
matrix within a visual-language model that includes both image and text
information. While minimizing the Wasserstein distance between each extended
domain, the inter-class distance of the text embedding space is considered.
Finally, the convex combination of joint distributions from multiple extended
domains not only retain the class-specific information of the source domains
while eliminating class-irrelevant information. We support this novel
conceptual approach by deriving a generalization bound on the target error.

Our main contributions are (1) the first introduction of the MSDA with
Language problem, (2) proposing LanDa, a novel language-guided MSDA method
that leverages VLFMs to achieve domain adaptation without any target domain
image data, (3) we propose a cost function suitable for VLFMs, and evaluate
our method across several benchmarks, the results show that LanDA achieves the
highest accuracy in both target and source domain samples.

##  2 Related Work

#### Multi-Source Domain Adaptation (MSDA).

MSDA address unsupervised domain adaptation by employing source dataset
containing multiple domains, so it should consider both domain divergences
among various sources and domain shifts between the source and target. MSDA
mainly have two strategies, i.e., statistical discrepancy combination [30,
19], and cross-domain feature extraction [42, 29]. The first strategy computes
the statistical discrepancy between each source domain and target domain, and
then combines all predictions. For pairwise training, methods such as
adversarial learning [46, 49] and moment matching [34] have been utilized. For
determining weight assignments, various techniques are commonly used,
including weighted averages [34], perplexity scores [46] and Wasserstein
distances [49]. The second strategy employing multiple classifiers while
sharing one feature extractor, all domain distributions are aligned
implicitly. However, these conventional methods make the assumption that the
target domain image is readily available and depend on the visual backbone.
Unlike these, our work assumes no access to any target domain image data, with
only multiple labeled source domains available. Our approach accomplishes
domain adaptation through language-based guidance and adopts large VLFMs like
CLIP as the backbone.

#### Visual-Language Foundational Models (VLFMs).

In recent years, with the advancement of transformer for both vision [13] and
language [41] tasks, large-scale pre-trained models [45, 36, 5, 9, 20, 40]
have gained popularity. They serve as valuable, task-agnostic foundational
tools for various downstream applications in computer vision and natural
language processing, consistently set new state-of-the-art benchmarks across
multiple tasks. VLFMs bridge the gap between images and text by utilizing a
shared embedding space to achieve cross-modal learning. Various pre-training
schemes have been adopted in VLFMs, including contrastive learning [36, 28],
masked language modeling [39], and masked region modeling [7]. While VLFMs
demonstrate great potential, their effectiveness for domain adaptation remains
to be comprehensively investigated.
Numerous vision models are pretrained on large-scale datasets like ImageNet to
obtain initialized parameters, and all parameters are updated during transfer
learning. In the case of large-scale VLFMs, we can still obtain the network’s
initialization parameters from the image encoder. However, updating all
parameters can lead to a disparity between image embedding and text embedding.
This misalignment adversely affects the applicability of downstream tasks. As
one of the most representative VLFMs, CLIP [36] has been rarely leveraged for
domain generalization or domain adaptation, with existing efforts [14, 8, 47,
37] utilized fine-tuning techniques and confined to single-source settings.
Our work aims to advance the capability of CLIP for language-guided MSDA. To
our best knowledge, this direction has remained unexplored in prior arts.

#### Optimal Transport (OT).

OT theory [21, 35] defines distance metrics between probability distributions,
furnishing a theoretical framework to investigate optimal transport plans for
mapping one distribution onto another. Having matured as a mathematical field,
OT has recently been proposed as a probabilistic framework in machine learning
for comparing and manipulating probability distributions [32]. OT is also
known by names such as the Wasserstein distance, Dudley metric, Kantorovich
metric, or earth mover’s distance.
The cost function is a fundamental mental concept in OT, as it quantifies the
distance between samples from distinct distributions. Compared to other
distance metrics, the Wasserstein distance captures the geometric structure
between distribution supports and exhibits greater sensitivity to feature
distribution shifts across domains. This provides a mathematically grounded
distance measure, with notable empirical performance. In machine learning, the
Wasserstein distance serves both as an adaptation loss function, minimizing
the distance to align distributions [27, 6], and as a means to map features
from different domains into a Wasserstein space for adaptive classification
[50, 10]. However, existing cost functions are inadequate for VLFMs due to
their failure to account for the text embeddings.

Figure 1: An overview of the proposed two-stage framework. The training
dataset comprises images from multiple source domains and various categories.
In the absence of any image samples from the target domain, our goal is to
generalize our model to the unseen clipart domain. To achieve this, we employ
VLFMs and learn domain-level knowledge from the text. Each domain-specific
augmenter utilizes the domain-class alignment loss to align the image
embeddings from the source domain to the unseen target domain, while
preserving their class information. To effectively leverage the knowledge from
multiple source domains, we propose a cost matrix function that projects the
extended domains and text embeddings of class name into the Wasserstein space
to learn domain-invariant information.

##  3 Language-Guided MSDA

We tackle the MSDA problem utilizing solely the linguistic descriptions of the
multiple source domains and the foreseen but unseen target domain. More
formally,

there are $N$ different underlying source distributions denoted as
$\\{Q_{k}\\}_{k=1}^{N}$, and one underlying target distribution denoted as
$\\{P\\}$. It is important to note that $Q_{k}$ is not equal to $P$, leading
to domain shift. Additionally, we have the following data:

  * •

Visual-Language Foundational Models $\\{f_{V},f_{L}\\}$, with text encoder
$f_{L}$ and visual encoder $f_{V}$.

  * •

Image-label pair
$\mathcal{D}^{(Q_{k})}=\\{(x_{i}^{(Q_{k})},y_{i}^{(Q_{k})})\\}_{i=1}^{n_{Q_{k}}}$
from multiple source domains, where $n_{Q_{k}}$ denotes the number of samples
in the $k$-th source domain.

  * •

The domain descriptions $\\{\mathcal{T}_{Q_{k}}\\}_{k=1}^{N}$ of the multiple
source domains, and domain description $\\{\mathcal{T}_{P}\\}$ of the target
domain. Let $\\{\\{\mathcal{T}_{C}^{k,i}\\}_{i=1}^{n_{Q_{k}}}\\}_{k=1}^{N}$ to
represent the class name. Furthermore,
$\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i}$ and
$\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,i}$ denote the composition of
the domain name and the class name. For example, if $\mathcal{T}_{Q_{k}}=$
painting, $\mathcal{T}_{C}^{k,i}=$ cat, the composition
$\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i}=$ a painting of a cat.

Our approach can be decomposed into two stages. The first stage entails
learning multiple domain-specific augmenters to ”place” each source domain
image embedding into its respective extended domain. This serves to (1) align
with the unseen target domain, and (2) remove source domain class-irrelevant
information, retaining only domain-invariant features. The second stage
performs linear probing on the expanded domains and image embeddings to
generate a linear classifier. Fig.1 presents an overview of the proposed
framework.

###  3.1 Training Domain-Specific Augmenters

We choose CLIP [36] as our visual-language foundational model. For a given
$k$-th source domain image $x_{i}^{(Q_{k})}$ and its corresponding text prompt
$\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i}$,
$f_{V}(x_{i}^{(Q_{k})})$ and
$f_{L}(\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i})$ denote the image
embedding and text embedding respectively.

The initial stage of LanDA is dedicated to the acquisition of domain augmented
network, which is composed by $N$ domain-specific augmenters
$\\{f_{\textrm{aug}}^{k}\\}_{k=1}^{N}$. Each $f_{\textrm{aug}}^{k}$ transform
the image embedding $f_{V}(x_{i}^{(Q_{k})})$ into its respective extended
domain. It is crucial to highlight that these domain-specific augmenters do
not share weights. Nonetheless, the decision boundaries among the various
extended domains may not align perfectly (refer to Fig.2(c)). We posit that
this discrepancy arises from variations in the probability distributions among
the different extended domains, thus necessitating the imposition of
constraints on the domain-specific augmenters. Consequently, we train
$f_{\textrm{aug}}^{k}$ using a combination of two losses: Domain-Class
Alignment and Distribution Consistency.

Figure 2: t-SNE visualizations on the Office-Home dataset for transfer task
A,C,R→P. In figure (a), the image embeddings of CLIP are depicted, with
different colors indicating distinct classes. The remaining figure represent
the extended domains, where each color corresponds to a different extended
domain image embeddings. Three approaches to train domain-specific augmenters
are compared: (b) using only $\mathcal{L}_{DA}$; (c) using $\mathcal{L}_{DA}$
and $\mathcal{L}_{CA}$; and (d) using $\mathcal{L}_{DA}$, $\mathcal{L}_{CA}$
and $\mathcal{L}_{DC}$.

#### Domain-Class Alignment.

The domain alignment loss utilizes the difference between the source and
target domain text embeddings to steer the source domain image embeddings
toward the target domain image embeddings. Some VLFMs such as CLIP employs
cosine similarity to quantify the correlation between images and text. The
model is trained by maximizing the cosine similarity within the embedding
space of similar images and text, while minimizing it within the embedding
space of unrelated images and text. Although CLIP establishes a correspondence
between the image embeddings and the text embeddings, the precise mapping
mechanism remains elusive, making it challenging to describe the mapping
relationship between them with a specific function. Previous works [33, 16]
assumed the existence of a ”global space” in which the direction from the
source to target domain text embeddings aligns with the source to target
domain image embeddings. We follow this assumption. Consequently, we train the
$k$-th domain-specific augmenter using

$\displaystyle\small\begin{split}\mathcal{L}_{DA}(f_{\mathrm{aug}}^{k})=\sum_{i=1}^{n_{Q_{k}}}1-(\frac{f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(Q_{k})}))-f_{V}(x_{i}^{(Q_{k})})}{\|f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(Q_{k})}))-f_{V}(x_{i}^{(Q_{k})})\|}\\\
\cdot\frac{f_{L}(\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,i})-f_{L}(\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i})}{\|f_{L}(\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,i})-f_{L}(\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i})\|}).\end{split}$
(1)

Although Eq.(1) aligns the source domain image embeddings towards the unseen
target domain, prior work [14] indicates that optimizing $\mathcal{L}_{DA}$
degrades class-specific representation, resulting in extended domains that
fail to retain class-specific traits and exhibit only minor differences of
distinct images. Fig.2(b) displays the visualization of the training results
solely obtained using Eq.(1). It is evident that numerous clusters are closely
situated and lack distinct separability, suggesting a limited classification
capability of the model. Fortunately, by leveraging CLIP’s zero-shot capacity
to accurately classify image categories, we can add an auxiliary optimization
objective to enhance discrimination during alignment,

$\displaystyle\small\begin{split}\mathcal{L}_{CA}(f_{\mathrm{aug}}^{k})\\!=\\!\sum_{i=1}^{n_{Q_{k}}}\\!\mathcal{H}(\mathcal{S}(f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(Q_{k})}))\cdot
f_{L}(\mathcal{T}_{C}^{k,i})),y_{i}^{(Q_{k})}).\end{split}$ (2)

Here $\mathcal{H}$ is cross-entropy and $\mathcal{S}$ is softmax.

Eq.(2) provides standard CLIP supervision during extended domains learning to
retain vital class-specific knowledge by clarifying the classification
capabilities of the extended domain. Rather than using the composition of the
target-domain description and the class name (i.e.,
$\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,i}$), we utilize isolated class
names (i.e., $\mathcal{T}_{C}^{k,i}$), aspiring to impart domain invariance
and bolster generalization for the extended domains. The extended domains
devoid of domain idiosyncrasies facilitates robust transfer to novel
distributions, allowing the model to perform well not only in the unseen
target domain. We refer to Eq.(1) as the domain alignment loss and to Eq(2) as
the class alignment loss.

#### Distribution Consistency.

While the above Domain-Class Alignment induces convergence of the source
embeddings towards the unseen target domain, inspection of Eq.(1) reveals it
merely enforces orientation alignment, without constraining the embedding
magnitudes. Furthermore, each extended domain has the potential to include
class-independent information derived from its respective source domain.
Consequently, this fails to mitigate distributional divergence across the
resulting augmented domains from disparate sources (Fig.2(c)), yielding
decision boundaries not robust to examples proximate to the data manifold
boundaries. We add a distribution consistency loss which force the
distribution alignment of the extended domains and remove class-irrelevant
information.

The Wasserstein distance provides a geometrically principled metric for
comparing probability distributions. Central to its computation is the cost
function, which prescribes the expense of transferring unit mass between all
locations in the two distributions. We denote $\\{\bar{Q}_{k}\\}_{k=1}^{N}$ as
the true probability distribution of the extended domains. With
$\\{\Omega_{k}\\}_{k=1}^{N}$ represents the sample space of
$\\{\bar{Q}_{k}\\}_{k=1}^{N}$, and $\\{\bar{\mathbf{x}}_{k}\\}_{k=1}^{N}$
represents sample drawn from $\\{\Omega_{k}\\}_{k=1}^{N}$. Therefore, the
Wasserstein distance between distributions $\bar{Q}_{i}$ and $\bar{Q}_{j}$ is
expressed mathematically as

$\displaystyle\small\begin{split}\mathcal{W}(\bar{Q}_{i},\bar{Q}_{j})\overset{\mathrm{def}}{=}&\underset{\mathcal{J}\in\Pi(\bar{Q}_{i},\bar{Q}_{j})}{\mathrm{inf}}\int_{\Omega_{i}\times\Omega_{j}}C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})d\mathcal{J}(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})\\\
=&\underset{\mathcal{J}\in\Pi(\bar{Q}_{i},\bar{Q}_{j})}{\mathrm{inf}}\left\langle\mathcal{C},\mathcal{J}\right\rangle_{F},\end{split}$
(3)

where $\mathcal{J}\in\Pi(\bar{Q}_{i},\bar{Q}_{j})$ denotes a joint probability
measure of $\Pi(\bar{Q}_{i},\bar{Q}_{j})$ with marginals $\bar{Q}_{i}$ and
$\bar{Q}_{j}$, $\left\langle\cdot,\cdot\right\rangle_{F}$ is the Frobenius
inner product of matrices, and $\Omega_{i}\times\Omega_{j}$ represents the
Cartesian product of the sample spaces corresponding to the two probability
distributions. $C(\cdot,\cdot)$ is the cost function. $\mathcal{C}$ is the
cost matrix, where
$\mathcal{C}_{ij}=C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})$. Eq.(3) states
that we only need to sample one joint distribution $\mathcal{J}$ that
minimizes the equation, and the minimum value obtained acts as the Wasserstein
distance. In traditional MSDA, commonly used cost functions is Minkowski
distance cost, i.e.,
$C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})=\left\|\bar{\mathbf{x}}_{i}-\bar{\mathbf{x}}_{j}\right\|_{p}^{p}$.
However, these cost functions do not consider the cost of inter-class
transport. Intuitively, the cost of sample transport within and between
classes should be different. To address this problem, a few existing works
[15, 31] incorporate the labels into the cost function. However, these
approaches still have limitations as they treat the cost of transport between
different classes equally. For instance, transport dog to wolf incurs the same
cost as dog to bird. However, CLIP’s text embeddings situate dog and wolf in
proximity, while positioning dog and bird more distantly. This uniform
treatment of transport cost between classes ultimately impedes the alignment
of image and text embeddings. Therefore, we propose a cost function
computation method tailored for visual-language models,

$\displaystyle\small\begin{split}C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})\\!=\\!\mathrm{exp}\big{(}\frac{\|\bar{\mathbf{x}}_{i}-\bar{\mathbf{x}}_{j}\|^{2}\\!+\\!\lambda\|f_{L}(\mathcal{T}_{C}^{i,\cdot})\\!-\\!f_{L}(\mathcal{T}_{C}^{j,\cdot})\|^{2}}{2\sigma^{2}}\big{)}.\end{split}$
(4)

Here $\mathcal{T}_{C}^{i,\cdot}$ and $\mathcal{T}_{C}^{j,\cdot}$ denotes the
class name corresponding to $\bar{\mathbf{x}}_{i}$ and $\bar{\mathbf{x}}_{j}$,
respectively.

The cost function $C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})$ is defined as
a multivariate Gaussian distribution to enable weighted smoothing of cost
between sample points from disparate distributions, enhancing robustness
against outliers and noise during distance calculations. $\lambda>0$ controls
the degree of emphasis placed on inter-class discrepancies, and $\sigma$
controls the Gaussian width, modulating the smoothness of the distance
computations. $C(\bar{\mathbf{x}}_{i},\bar{\mathbf{x}}_{j})$ incorporates
considerations for varying inter-class transfer cost.

In Eq(3), the solution is guaranteed to be at some vertex in the feasible set
$\Pi(\bar{Q}_{i},\bar{Q}_{j})$, indicating that the solution is a sparse
matrix, leading to a highly imbalanced transportation plan. We add an entropy
regularization term to address this issue. A larger entropy value indicates
higher uncertainty in the solution, which leads to a closer approximation to a
uniform distribution and a less sparse solution. Importantly, by introducing
the entropy regularization, the original problem can be solved approximately
using the Sinkhorn algorithm [12], significantly reducing the computational
cost. Thus, the distribution consistency loss is

$\displaystyle\small\begin{split}\mathcal{L}_{DC}(f_{\mathrm{aug}}^{k})=\sum_{j=1,j\neq
k}^{N}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j})-\frac{1}{\zeta}\mathcal{H}_{\mathrm{reg}}(\mathcal{J}).\end{split}$
(5)

Here entropy regularization
$\mathcal{H}_{\mathrm{reg}}(\mathcal{J})=-\sum_{i}\mathcal{J}_{i}\log(\mathcal{J}_{i})$,
and $\zeta$ is a hyperparameter. As the first stage in LanDA, our final
objective $f_{\textrm{aug}}^{k}$ for training the domain-specific augmenter
network combines the domain-class alignment loss and distribution consistency
loss:

$\displaystyle\small\begin{split}\mathcal{L}_{DCA}(f_{\mathrm{aug}}^{k})\\!=\\!\alpha\mathcal{L}_{DA}(f_{\mathrm{aug}}^{k})\\!+\\!\beta\mathcal{L}_{CA}(f_{\mathrm{aug}}^{k})\\!+\\!\gamma\mathcal{L}_{DC}(f_{\mathrm{aug}}^{k}).\end{split}$
(6)

$\alpha$, $\beta$ and $\gamma$ is the hyperparameters.

###### Theorem 1

Let $\epsilon_{T}(f_{\mathrm{aug}})$ and $\epsilon_{S}(f_{\mathrm{aug}})$
represent the target domain error and source domain error, respectively.
$\\{\hat{Q}_{k}\\}_{k=1}^{N}$ denotes the associated empirical measure from
$\\{\bar{Q}_{k}\\}_{k=1}^{N}$. The kernel function $\varphi(\cdot,\cdot)$
solely depends on the utilized VLFMs. $z$ represents the hypothetical
empirical measurement of the unseen target domain, while $\mathcal{T}_{C}^{z}$
denotes its corresponding class. Let $\vartheta$ represents the combined error
of the ideal hypothesis $f_{\mathrm{aug}}^{*}$ that minimizes the combined
error of $\epsilon_{S}(f_{\mathrm{aug}})+\epsilon_{T}(f_{\mathrm{aug}})$.
$w_{i}$ is the weight of the $i$-th extended domain. For any $\delta>0$,
$\varsigma^{{}^{\prime}}<\sqrt{2}$, and any $f_{\mathrm{aug}}$, the following
formula holds:

$\displaystyle\small\begin{split}&\epsilon_{T}(f_{\mathrm{aug}})\leq\epsilon_{S}(f_{\mathrm{aug}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+\vartheta+\quad\end{split}$
(7)
$\displaystyle\footnotesize\begin{split}{\sum_{k=1}^{N}\sum_{j=k+1}^{N}\\!w_{k}w_{j}}\big{[}\mathcal{W}(\hat{Q}_{k},\\!\hat{Q}_{j})\\!+\\!\exp(\\!\frac{(1\\!+\\!\lambda)\\!\log(\\!\frac{1}{\delta}\\!)}{\sigma^{2}n_{Q_{k}}\varsigma^{{}^{\prime}}}\\!)\\!+\\!\exp(\\!\frac{(1\\!+\\!\lambda)\\!\log(\\!\frac{1}{\delta}\\!)}{\sigma^{2}n_{Q_{j}}\varsigma^{{}^{\prime}}}\\!)\big{]}.\end{split}$

Theorem.1 indicates that by minimizing both domain-class alignment loss and
distribution consistency loss, we can effectively reduce the error of domain-
specific augmenters in the unseen target domain. _For the proof and
explanation of Theorem. 1, refer to the suppl. material._

###  3.2 Training Linear Classifier

After all the domain-specific augmenters is trained, we train a linear probe
$f_{C}$ on all original image embeddings along with the extended domains.
Mathematically,

$\displaystyle\small\begin{split}\mathcal{L}_{C}(f_{C})=\sum_{k=1}^{N}&\sum_{i=1}^{n_{Q_{k}}}\varepsilon\mathcal{H}(\mathcal{S}(f_{C}(f_{V}(x_{i}^{(Q_{k})}))),y_{i}^{(Q_{k})})\\\
+&(1\\!-\\!\varepsilon)\mathcal{H}(\mathcal{S}(f_{C}(f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(Q_{k})})))),y_{i}^{(Q_{k})}).\end{split}$
(8)

$\varepsilon$ is a hyperparameter. Our proposed vision model consists of three
components, including (a) a visual feature extractor $f_{V}$ instantiated from
the image decoder of VLFMs, (b) $N$ distinct domain-specific augmenters
$\\{f_{\mathrm{aug}}^{k}\\}_{k=1}^{N}$, and (c) a linear classifier $f_{C}$.
We train only domain-specific augmenters and the linear classifier, which have
fewer parameters, and freeze the visual feature extractor, which has large
parameters, thereby preserving the feature extraction capability of the pre-
trained large-scale VLFMs. Our work can also be seen as a novel approach to
fine-tuning large models.

###  3.3 Aggregated Target Prediction

In the testing phase, the goal is to accurately classify a given target image
$x_{i}^{P}$. We combine the extended domains to obtain the final prediction.
The key challenge lies in determining the weights for each extended domain.
For instance, consider the sunny domain, which is closer to the cloudy domain
than the night domain. Thus, it is necessary to assign different weights for
each source and target to highlight more relevant sources and suppress
irrelevant ones. Fortunately, the difference between each source and target
can be assessed using the text embedding of the domain name. Ultimately
yielding the final prediction:

$\displaystyle\small\begin{split}&\mathrm{Prediction}(x_{i}^{(P)})=f_{C}(\sum_{k=1}^{N}w_{k}f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(P)}))),\quad\mathrm{where}\end{split}$
(9)
$\displaystyle\footnotesize\begin{split}w_{k}\\!=\\!\frac{\mathcal{W}(\bar{Q}^{\mathcal{T}_{Q_{k}}},\bar{P}^{\mathcal{T}_{P}})}{\sum_{j=1}^{N}\\!\mathcal{W}(\\!\bar{Q}^{\mathcal{T}_{Q_{j}}}\\!,\\!\bar{P}^{\mathcal{T}_{P}}\\!)},~{}~{}\mathrm{and}~{}~{}C\big{(}\\!f_{L}\\!(\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i}),\\!f_{L}\\!(\\!\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,j})\big{)}\\!=\\!\\\
\end{split}$
$\displaystyle\footnotesize\begin{split}\mathrm{exp}\big{(}\frac{\|f_{L}\\!(\mathcal{T}_{Q_{k}}\\!\circ\\!\mathcal{T}_{C}^{k,i})\\!-\\!f_{L}\\!(\mathcal{T}_{P}\\!\circ\\!\mathcal{T}_{C}^{k,j})\|^{2}\\!+\\!\lambda\|f_{L}\\!(\mathcal{T}_{C}^{k,i})\\!-\\!f_{L}\\!(\mathcal{T}_{C}^{k,j})\|^{2}}{2\sigma^{2}}\big{)}.\end{split}$

Here, $\bar{P}^{\mathcal{T}_{P}}$ is the probability distribution of the VLFMs
text embeddings for the combined set of target domain names and class names.
Similarly, $\bar{Q}^{\mathcal{T}_{Q_{k}}}$ represents the probability
distribution of the VLFMs text embeddings for the combined set of source
domain names and class names.

Figure 3: Image generation and nearest neighbor results of different methods
in different scenarios. (a) the source domain images in the order of real,
painting, sketch and clipart, they are then transferred to painting, sketch,
clipart and real domains, respectively. (b) and (d) display the target domain
image generation results of VQGAN+CLIP and Diffusion+CLIP methods. For both
methods, text prompts incorporate the target domain name and class name (e.g.,
a painting of a dog), and image prompts (taken from (a)) are inputted to
generate the ”target style” images. (c) and (e) present the nearest neighbor
results of the generate images within the correct target domain in the CLIP
image embedding space. (f) displays the nearest neighbor result obtained from
the LADS method within the correct target domain in its augmented embedding
space. (g) shows the nearest neighbor result obtained from our method within
the correct target domain in the combine extended domain space. We have also
indicated the distance of the nearest neighbor.

##  4 Experiments

#### Architecture Detail.

For our experiments, we employ three pre-trained vision encoders from CLIP as
$f_{V}$: ViT-L/14, ViT-B/16 [17] and ResNet-50 [18] for validation. Here
ViT-L/14 boasts a larger parameter count, whereas ViT-B/16 and ResNet-50 have
fewer parameters. Additionally, we utilize a transformer-based text encoder as
$f_{L}$. In this section, we present the primary experiments and results
conducted on ViT-L/14. We keep all of the visual/text encoder parameters
frozen and only update the domain-specific augmenters network parameters, each
of which is a 2-layer MLP with input and output dimensions of 768 and a hidden
dimension of 384. The comprehensive introduction of the datasets, experiments
and corresponding discussions regarding ViT-B/16 and ResNet-50, are deferred
to suppl. material.

Scenario | R, P, S to C | R, P, S to I | R, C, S to P
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | R | P | S | C | EXT | R | P | S | I | EXT | R | C | S | P | EXT
CLIP ZS(G) [36] |  96.65 |  94.40 |  92.54 |  94.86 |  94.70 |  96.65 |  94.40 |  92.54 |  79.08 |  86.81 |  96.65 |  94.86 |  92.54 |  94.40 |  94.54
CLIP ZS(A) [36] |  96.80 |  95.32 |  92.21 |  94.39 |  94.58 |  96.80 |  95.32 |  92.21 |  80.90 |  87.84 |  96.80 |  94.39 |  92.21 |  95.32 |  94.89
CLIP LP |  97.21 |  96.12 |  95.33 |  94.62 |  95.42 |  96.80 |  95.05 |  94.62 |  80.03 |  87.76 |  97.28 |  95.30 |  95.46 |  94.16 |  95.09
VQGAN+CLIP [11] |  97.02 |  96.30 |  94.96 |  94.65 |  95.37 |  96.91 |  95.30 |  94.68 |  80.13 |  87.88 |  97.40 |  94.98 |  95.64 |  94.93 |  95.47
Diffusion+CLIP [22] |  97.18 |  96.27 |  95.24 |  94.71 |  95.47 |  96.86 |  95.22 |  94.53 |  81.16 |  88.35 |  97.18 |  95.11 |  95.73 |  95.80 |  95.90
LADS [14] |  97.30 |  96.19 |  95.15 |  94.74 |  95.48 |  97.04 |  95.12 |  94.70 |  81.22 |  88.42 |  97.52 |  94.89 |  95.74 |  96.00 |  96.03
LanDA(Ours) |  97.49 |  96.67 |  95.41 |  95.28 |  95.90 |  97.36 |  95.60 |  95.17 |  82.06 |  89.05 |  97.99 |  95.40 |  95.95 |  96.63 |  96.54
Table 1: Quantitative evaluation in the three-source domain scenario of the Mini-DomainNet dataset. The “EXT” column is derived as follows: 50% of the average predictions from the ID domain, added to 50% of the predicted value from the OD domain. Scenario | C, P, R to A | A, C, P to R | A, C, R to P
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | C | P | R | A | EXT | A | C | P | R | EXT | A | C | R | P | EXT
CLIP ZS(G) [36] |  73.13 |  90.96 |  91.94 |  84.97 |  85.16 |  84.97 |  73.13 |  90.96 |  91.94 |  87.48 |  84.97 |  73.13 |  91.94 |  90.96 |  87.15
CLIP ZS(A) [36] |  75.34 |  87.73 |  92.10 |  86.34 |  85.70 |  86.34 |  75.34 |  87.73 |  92.10 |  87.62 |  86.34 |  75.34 |  92.10 |  87.73 |  86.16
CLIP LP |  83.89 |  96.08 |  95.32 |  87.02 |  89.39 |  88.97 |  82.75 |  95.18 |  92.55 |  90.76 |  90.34 |  83.59 |  95.78 |  92.70 |  91.30
LADS [14] |  83.13 |  96.10 |  95.24 |  87.71 |  89.60 |  90.07 |  82.52 |  95.03 |  93.86 |  91.53 |  91.86 |  85.44 |  95.51 |  93.00 |  91.97
LanDA(Ours) |  85.27 |  96.31 |  94.93 |  88.83 |  90.50 |  90.48 |  85.73 |  95.63 |  94.09 |  92.35 |  94.90 |  85.34 |  95.47 |  93.22 |  92.56
Table 2: Quantitative evaluation in the three-source domain scenario of the
Office-Home dataset.

#### Baselines.

Considering the absence of established approaches for language-guided MSDA, we
opt to compare against the following methods: (1) CLIP ZS(G) [36] uses the
class name alone as the text prompt (_e.g_., car), then compute the similarity
between the image embedding and the text embedding for each category, and the
image is predicted to be the class with the highest similarity to its text
embedding; (2) CLIP ZS(A) [36] uses the composition of class name and domain
name as the text prompt (_e.g_., a sketch of a car); (3) CLIP LP fit a linear
classifier on top of the CLIP image embeddings; (4) VQGAN+CLIP [11] Using a
text prompt and an source domain image, we conduct ”style transfer” in order
to transfer the source image data to the unseen target domain, thereby
augmenting the training data. Subsequently, a linear probe is trained on both
the augmented and non-augmented CLIP embeddings. (5) Diffusion+CLIP [22] also
uses a text prompt and an image prompt to transfer a source domain style image
to a target domain style image, and then train a linear classifier. (6) LADS
[14] is a language-guided domain generalization method applicable to a single
source domain. The LADS method is conducted independently for each source
domain to the target domain, followed by averaging the prediction results. Due
to the time-consuming nature of image generation, we limit the comparison of
VQGAN+CLIP and Diffusion+CLIP methods to the Mini-DomainNet dataset.

| ID | OD |
---|---|---|---
Methods | R | P | S | C | EXT
A)CLIP+$\mathcal{L}_{DA}$ |  70.34 |  74.77 |  70.86 |  75.93 | 73.96
B)w/ $\mathcal{L}_{CA}$ |  97.35 |  96.26 |  95.22 |  94.80 | 95.54
C)w/ $\mathcal{L}_{DC}$ |  97.40 |  96.58 |  95.39 |  95.11 | 95.78
D)w/ LP (FULL) |  97.49 |  96.67 |  95.41 |  95.28 | 95.90
Table 3: Ablation study in the R,P,S to C scenario of the Mini-DomainNet
dataset.

###  4.1 Quantitative Evaluation

Quantitative results on the two benchmark datasets are shown in Table.1 and
Table.2. CLIP LP exhibits satisfactory performance on in-domain (ID), but
limited performance on out-of-domain (OD). Table.1 includes additional pixel-
level augmentation baselines, i.e., VQGAN+CLIP and Diffusion+CLIP. However,
despite their utilization of CLIP’s language knowledge to generate unseen
target domain images, the experimental results indicate that Diffusion+CLIP
performs better than VQGAN+CLIP on OD. Compared with the image generation
baselines, LADS has better performance and less time-consuming training. The
proposed approach LanDA significantly improves over prior state-of-the-art
methods in terms of all metrics. Notably, it outperforms other comparative
methods not only in the unseen target domain but also exhibits significant
improvement in the source domain. This highlights the superiority of our
approach in achieving better performance across all domains.

###  4.2 Qualitative Evaluation

Fig.8 presents the visualization results obtained from different methods. The
generated images of VQGAN+CLIP not only exhibit poor quality but also fail to
transfer to the target domain in certain case (e.g., from painting to sketch),
thereby resulting in poor performance on OD. In contrast, the images generated
by the Diffusion+CLIP are closer to the correct target domain and have better
performance than the VQGAN+CLIP method on OD. However, it is worth noting that
the Diffusion+CLIP method requires significant time and computing resources.
Furthermore, we observed that the distance between the images generated by
VQGAN+CLIP and Diffusion+CLIP, and the nearest neighbor samples from the
correct target domain, still remains considerable in the CLIP image embedding
space. Both LADS and the proposed method, LanDA, exhibit strong performance in
the target domain without the need to generate additional images. However, the
nearest neighbor samples of LADS contain more class-independent information,
such as the word ”bike” in the bicycle picture and the water in the speedboat
picture in Fig.8(f). In contrast, LanDA focuses more on domain-invariant
features.

###  4.3 Ablation Study

An ablation study on the R,P,S to C scenario of the Mini-DomainNet Dataset is
conducted to validate the effectiveness of various components proposed in the
framework of LanDA. In Table.3, we present comparisons between our full method
and its variants: A) using only the domain alignment loss $\mathcal{L}_{DA}$
to train domain-specific augmenters; B) using both $\mathcal{L}_{DA}$ and the
class alignment loss $\mathcal{L}_{CA}$ to train domain-specific augmenters;
C) using $\mathcal{L}_{DA}$, $\mathcal{L}_{CA}$, and the distribution
consistency loss $\mathcal{L}_{DC}$ to train domain-specific augmenters,
without training linear classifier, then compute the consistency loss between
$\sum_{k=1}^{N}\omega_{k}f_{\mathrm{aug}}^{k}(f_{V}(x_{i}^{(P)}))$ and the
text embedding $f_{L}(\mathcal{T}_{C})$ to obtain prediction results; D) our
full method to train both domain-specific augmenters and a linear classifier.

Furthermore, we conduct t-SEN visualizations on the extended domain image
embeddings obtained from methods A, B, and C, corresponding to (a), (b), and
(c) in Fig.2, respectively. As depicted in Fig.2(a), when training the network
using only $\mathcal{L}_{DA}$, the class clusters of the extended domains are
close to each other, leading to a subpar classification outcome. When
employing both $\mathcal{L}_{DA}$ and $\mathcal{L}_{CA}$ during network
training, the obtained class clusters become more dispersed, indicating
improved classification effectiveness. However, similar to Fig.2(a), the three
extended domain distributions are situated in different positions in Fig.2(b).
This could be attributed to the fact that the extended domains contain class-
independent information from their respective source domains. While one
domain-specific augmenter performs well for a target distribution located at
the edge, the other two perform poorly, resulting in the underutilization of
the multi-source domain property. In Fig.2(c), when utilizing our
comprehensive loss for training the domain-specific augmenters, it is evident
that the boundaries of the extended domains distribution are not distinct, and
the class clusters are dispersed across different regions. It means that for a
target sample distribution, multiple extended domains are available to
validate it, allowing for the effective utilization of the multi-source domain
information. The nearest neighbor results presented in Fig.4 demonstrate that
$\mathcal{L}_{DC}$ is highly effective in eliminating class-irrelevant
information.

Figure 4: Nearest Neighbors for LanDA when ablating the loss. The labels below
each image represent their respective categories. Images outlined in red have
been augmented to error categories. Solely training using $\mathcal{L}_{DA}$
easily result in misclassification, transferring the image to the wrong
category. Even though combining $\mathcal{L}_{DA}$ and $\mathcal{L}_{CA}$
helps reduce classification errors, it still contains class-independent
information. Our comprehensive method specializes in learning domain-invariant
features.

##  5 Conclusion

We propose LanDA, a language-guided multi-source domain adaptation method. We
view LanDA as a jumping-off point for further exploration on leveraging the
powerful capabilities of large-scale multimodal models to improve multi-source
domain adaptation accuracy when only language descriptions are provided as
input. The method consists of two stages. In the first stage, multiple domain-
specific augmenters are trained to align each source domain to the target
domain separately in the VLFMs’ image and text embedding spaces. Subsequently,
the extended domains and class-specific text embeddings are projected into the
Wasserstein space for further alignment, extracting domain-invariant
information and eliminating class-irrelevant information. In the second stage,
we merge multiple source domain image embeddings with multiple extended
domains to train a shared classifier. The framework comprises a VLFMs vision
encoder, multiple domain-specific augmenters, and a linear classifier. During
testing, we assign weights by measuring the distance between the text
embedding of the source domain and the unseen target domain. Experimental
results demonstrate that LanDA achieves strong performance not only in the
target domain but also exhibits improvements in the source domains.

## References

  * Ahmed et al. [2021] Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy-Chowdhury.  Unsupervised multi-source domain adaptation without access to source data.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10103–10112, 2021.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bolley et al. [2007] François Bolley, Arnaud Guillin, and Cédric Villani.  Quantitative concentration inequalities for empirical measures on non-compact spaces.  _Probability Theory and Related Fields_ , 137:541–593, 2007\.
  * Bose et al. [2023] Shirsha Bose, Enrico Fini, Ankit Jha, Mainak Singha, Biplab Banerjee, and Elisa Ricci.  Stylip: Multi-scale style-conditioned prompt learning for clip-based domain generalization.  _arXiv preprint arXiv:2302.09251_ , 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2019a] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang.  Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation.  In _International conference on machine learning_ , pages 1081–1090. PMLR, 2019a.
  * Chen et al. [2019b] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Learning universal image-text representations.  2019b.
  * Cho et al. [2023] Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak.  Promptstyler: Prompt-driven style generation for source-free domain generalization.  _arXiv preprint arXiv:2307.15199_ , 2023.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Courty et al. [2017] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy.  Joint distribution optimal transportation for domain adaptation.  _Advances in neural information processing systems_ , 30, 2017.
  * Crowson et al. [2022] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  In _European Conference on Computer Vision_ , pages 88–105. Springer, 2022.
  * Cuturi [2013] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26, 2013.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dunlap et al. [2022] Lisa Dunlap, Clara Mohri, Devin Guillory, Han Zhang, Trevor Darrell, Joseph E Gonzalez, Aditi Raghunathan, and Anna Rohrbach.  Using language to extend to unseen domains.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Flamary et al. [2016] R Flamary, N Courty, D Tuia, and A Rakotomamonjy.  Optimal transport for domain adaptation.  _IEEE Trans. Pattern Anal. Mach. Intell_ , 1:1–40, 2016\.
  * Gal et al. [2021] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ , 2021.
  * Han et al. [2022] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):87–110, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hoffman et al. [2018] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang.  Algorithms and theory for multiple-source adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Kantorovich [1942] L Kantorovich.  On the transfer of masses (in russian).  In _Doklady Akademii Nauk_ , page 227, 1942.
  * Kim and Ye [2021] Gwanghyun Kim and Jong Chul Ye.  Diffusionclip: Text-guided image manipulation using diffusion models.  2021\.
  * Kolesnikov et al. [2020] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 491–507. Springer, 2020.
  * Kumar et al. [2022] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ , 2022.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Long et al. [2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32, 2019.
  * Mancini et al. [2018] Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci.  Boosting domain adaptation by discovering latent domains.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3771–3780, 2018.
  * Mansour et al. [2009] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh.  Domain adaptation: Learning bounds and algorithms.  _arXiv preprint arXiv:0902.3430_ , 2009.
  * Montesuma and Mboula [2021] Eduardo Fernandes Montesuma and Fred Maurice Ngole Mboula.  Wasserstein barycenter for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16785–16793, 2021.
  * Montesuma et al. [2023] Eduardo Fernandes Montesuma, Fred Ngole Mboula, and Antoine Souloumiac.  Recent advances in optimal transport for machine learning.  _arXiv preprint arXiv:2306.16156_ , 2023.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2085–2094, 2021.
  * Peng et al. [2019] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1406–1415, 2019.
  * Peyré et al. [2019] Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 11(5-6):355–607, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Shu et al. [2023] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long.  Clipood: Generalizing clip to out-of-distributions.  _arXiv preprint arXiv:2302.00864_ , 2023.
  * Singha et al. [2023] Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee.  Ad-clip: Adapting domains in prompt space using clip.  _arXiv preprint arXiv:2308.05659_ , 2023.
  * Su et al. [2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_ , 2019.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Venkat et al. [2020] Naveen Venkat, Jogendra Nath Kundu, Durgesh Singh, Ambareesh Revanur, et al.  Your classifier can secretly suffice multi-source domain adaptation.  _Advances in Neural Information Processing Systems_ , 33:4647–4659, 2020.
  * Venkateswara et al. [2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5018–5027, 2017.
  * Wang et al. [2022] Zhenbin Wang, Mao Ye, Xiatian Zhu, Liuhan Peng, Liang Tian, and Yingying Zhu.  Metateacher: Coordinating multi-model domain adaptation for medical image classification.  _Advances in Neural Information Processing Systems_ , 35:20823–20837, 2022.
  * Wei et al. [2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.
  * Xu et al. [2018] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin.  Deep cocktail network: Multi-source unsupervised domain adaptation with category shift.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3964–3973, 2018.
  * Yu et al. [2023] Qing Yu, Go Irie, and Kiyoharu Aizawa.  Open-set domain adaptation with visual-language foundation models.  _arXiv preprint arXiv:2307.16204_ , 2023.
  * Zhang et al. [2021] Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, and Yusuke Iwasawa.  Domain prompt learning for efficiently adapting clip to unseen domains.  _arXiv preprint arXiv:2111.12853_ , 2021.
  * Zhao et al. [2020] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer.  Multi-source distilling domain adaptation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12975–12983, 2020.
  * Zhou et al. [2020] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9719–9728, 2020.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16816–16825, 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022b.

Supplementary Material for
”LanDA: Language-Guided Multi-Source Domain Adaptation”

##  Appendix A Theoretical Analysis

In this section, we introduce error bounds for the proposed LanDA method.
Firstly, we decompose the cost function proposed in LanDA and derive a lemma
that establishes the relationship between the Wasserstein metric and the error
functions of the source and target. Subsequently, we demonstrate how the
target error can be bounded for empirical measures.

###### Definition 1

Consider $x$ and $z$ as images from distinct source domain, with
$\mathcal{T}_{C}^{x}$ and $\mathcal{T}_{C}^{z}$ representing the textual
descriptions of their respective class name. Let $\bar{x}$ and $\bar{z}$
denote their corresponding image embeddings. Since the image and text
embeddings output by CLIP are one-dimensional, and dimension $m=$768, 512, or
1024, the following equation holds

$\displaystyle\begin{split}&C(\bar{x},\bar{z})=\mathrm{exp}\big{(}\frac{\|\bar{x}-\bar{z}\|^{2}+\lambda\|f_{L}(\mathcal{T}_{C}^{x})-f_{L}(\mathcal{T}_{C}^{z})\|^{2}}{2\sigma^{2}}\big{)}\\\
=&\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{z}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(-\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}{\bar{z}}_{i}}}{\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(-\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}[f_{L}(\mathcal{T}_{C}^{z})]_{i}}}{\sigma^{2}})\\\
=&\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{z}}_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}\mathrm{exp}(-\frac{{\textstyle{\bar{x}}_{i}{\bar{z}}_{i}}}{\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}\mathrm{exp}(-\frac{{\textstyle\lambda[f_{L}(\mathcal{T}_{C}^{x})]_{i}[f_{L}(\mathcal{T}_{C}^{z})]_{i}}}{\sigma^{2}})\\\
=&\big{[}\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{z}}_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}\sum_{n=0}^{\infty}(\frac{{\textstyle{(-1)^{n}\bar{x}}_{i}^{n}{\bar{z}}_{i}^{n}}}{\sigma^{2n}n!})\big{]}\\\
&\cdot\big{[}\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\lambda\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}\sum_{n=0}^{\infty}(\frac{{\textstyle(-1)^{n}\lambda^{n}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{n}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{n}}}{\sigma^{2n}n!})\big{]}\\\
=&\big{[}\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{z}}_{i}^{2}}}{2\sigma^{2}})\sum_{n=0}^{\infty}{\textstyle}\prod_{i}^{m}(-1)^{n}\prod_{i}^{m}(\frac{{\textstyle{\bar{x}}_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\prod_{i}^{m}(\frac{{\textstyle{\bar{z}}_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\big{]}\\\
&\cdot\big{[}\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{2}}}{2\sigma^{2}})\sum_{n=0}^{\infty}{\textstyle}\prod_{i}^{m}(-1)^{n}\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\big{]}\\\
=&\Big{\\{}\sum_{n=0}^{\infty}\Big{[}\Big{(}\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{x}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}(\frac{{\textstyle{\bar{x}}_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\Big{)}\Big{(}\sum_{n=0}^{\infty}\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}{\bar{z}}_{i}^{2}}}{2\sigma^{2}})\mathrm{exp}(\frac{{\textstyle\sum_{i}^{m}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{2}}}{2\sigma^{2}})\prod_{i}^{m}(\frac{{\textstyle{\bar{z}}_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\Big{)}\Big{]}\Big{\\}}\\\
&\cdot\sum_{n=0}^{\infty}\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\\\
\stackrel{{\scriptstyle\triangle}}{{=}}&\phi(\varphi(x,\mathcal{T}_{C}^{x}),x)\phi(\varphi(z,\mathcal{T}_{C}^{z}),z)\sum_{n=0}^{\infty}\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\\\
=&\sum_{n=0}^{\infty}\Big{[}\Big{(}\phi(\varphi(x,\mathcal{T}_{C}^{x}),x)\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{x})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\Big{)}\Big{(}\sum_{n=0}^{\infty}\phi(\varphi(z,\mathcal{T}_{C}^{z}),z)\prod_{i}^{m}(\frac{{\textstyle\sqrt{\lambda}[f_{L}(\mathcal{T}_{C}^{z})]_{i}^{n}}}{\sigma^{n}\sqrt{n!}})\Big{)}\Big{]}\\\
\stackrel{{\scriptstyle\triangle}}{{=}}&\Phi(\phi(\varphi(x,\mathcal{T}_{C}^{x}),x),\mathcal{T}_{C}^{x})\Phi(\phi(\varphi(z,\mathcal{T}_{C}^{z}),z),\mathcal{T}_{C}^{z})\\\
\end{split}$

After normalizing, we can make $\bar{x}_{i}$, $\bar{z}_{i}$,
$[f_{L}(\mathcal{T}_{C}^{x})]_{i}$ and $[f_{L}(\mathcal{T}_{C}^{z})]_{i}$
smaller than 1, so that
$0\leq\Phi(\phi(\varphi(x,\mathcal{T}_{C}^{x}),x),\mathcal{T}_{C}^{x})\leq\phi(\varphi(x,\mathcal{T}_{C}^{x}),x)\leq\varphi(x,\mathcal{T}_{C}^{x})\leq
1$. Furthermore, it is evident that $\varphi(x,\mathcal{T}_{C}^{x})$ satisfies
the conditions of Mercer’s theorem and can therefore be considered a kernel
function. We denote the Hebraic space generated by
$\varphi(x,\mathcal{T}_{C}^{x})$ as $\left\langle\right\rangle_{\varphi}$.

###### Definition 2

Let’s assume that the probability of inconsistency between the image
embeddings and the text embeddings of the source domains is

$\displaystyle\begin{split}\epsilon_{S}(f_{\mathrm{aug}},f_{L})&=\sum_{k=1}^{N}w_{k}\epsilon_{S_{k}}(f^{k}_{{\mathrm{aug}}},f_{L})=\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[error(f_{\mathrm{aug}}(f_{V}(x)),f_{L}(\mathcal{T}_{C}^{x}))]=\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\left\langle\varphi(x,\mathcal{T}_{C}^{x}),error\right\rangle_{\varphi}].\end{split}$

We define the target error in the same manner:

$\displaystyle\begin{split}\epsilon_{T}(f_{\mathrm{aug}},f_{T})=\mathbb{E}_{z\sim
P}[error(f_{\mathrm{aug}}(f_{V}(z)),f_{T}(\mathcal{T}_{C}^{z}))]=\mathbb{E}_{z\sim
P}[\left\langle\phi(z,\mathcal{T}_{C}^{z}),error\right\rangle_{\varphi}].\end{split}$

When the source and target error functions are defined w.r.t.
$f_{\mathrm{aug}}$ and $f_{L}$, we use the shorthand
$\epsilon_{S}(f_{\mathrm{aug}},f_{L})=\epsilon_{S}(f_{\mathrm{aug}})$.

###### Lemma 1

Let cost function
$C(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})=\mathrm{exp}\big{(}\frac{\|\bar{\mathbf{x}}_{k}-\bar{\mathbf{x}}_{j}\|^{2}+\lambda\|f_{L}(\mathcal{T}_{C}^{k,\cdot})-f_{L}(\mathcal{T}_{C}^{j,\cdot})\|^{2}}{2\sigma^{2}}\big{)}$,
and $\bar{Q}_{k},\bar{Q}_{j}\in\mathcal{P}(\Omega)$ be two probability
measures on $\mathbb{R}^{m}$. For every hypothesis
$f_{\mathrm{aug}}^{{}^{\prime}}$, $f_{\mathrm{aug}}$, the following holds

$\displaystyle\begin{split}\epsilon_{T}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j}).\end{split}$

#### Proof.

For simplicity, we assume that $\|error\|_{\varphi}$ is bounded by 1. This
assumption can be verified by imposing appropriate bounds on the norms of
$f_{\text{aug}}$ and $f_{T}$. Furthermore, it can be easily extended to cases
where $\|error\|_{\varphi}\leq M$ by scaling, as explained in [30]. As the
above Lemma.1 plays a pivotal role in the subsequent section, we present its
proof here,

$\displaystyle\begin{split}&~{}~{}~{}~{}~{}\epsilon_{T}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})=\epsilon_{T}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})-{\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})}+\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})\\\
&=\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\mathbb{E}_{z\sim
P}[\big{\langle}\varphi(z,\mathcal{T}_{c}^{z}),error\big{\rangle}_{\varphi}]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\big{\langle}\varphi(x,\mathcal{T}_{C}^{x}),error\big{\rangle}_{\varphi}}]\\\
&=\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\big{\langle}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\varphi(x,\mathcal{T}_{C}^{x})]},error\big{\rangle}_{\varphi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}error\Big{\|}_{\varphi}\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\varphi(x,\mathcal{T}_{C}^{x})]}\Big{\|}_{\varphi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\varphi(x,\mathcal{T}_{C}^{x})]}\Big{\|}_{\varphi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\Phi(\phi(\varphi(x,\mathcal{T}_{C}^{x}),x),\mathcal{T}_{C}^{x})]}\Big{\|}_{\varphi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]-{\sum_{k=1}^{N}w_{k}\mathbb{E}_{x\sim
Q_{k}}[\Phi(\phi(\varphi(x,\mathcal{T}_{C}^{x}),x),\mathcal{T}_{C}^{x})]}{\sum_{j=k+1}^{N}w_{j}\mathbb{E}_{x\sim
Q_{j}}[\Phi(\phi(\varphi(x,\mathcal{T}_{C}^{x}),x),\mathcal{T}_{C}^{x})]}\Big{\|}_{\varphi}\\\
&=\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\\\
&\quad\quad\quad\quad\quad\quad\quad-{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\int_{\Omega\times\Omega}[\Phi(\phi(\varphi({{\mathbf{x}}_{k}},\mathcal{T}_{C}^{\mathbf{x}_{k}}),{\mathbf{x}}_{k}),\mathcal{T}_{C}^{{\mathbf{x}}_{k}})][\Phi(\phi(\varphi({{\mathbf{x}}_{j}},\mathcal{T}_{C}^{\mathbf{x}_{j}}),{\mathbf{x}}_{j}),\mathcal{T}_{C}^{{\mathbf{x}}_{j}})]~{}d\mathcal{J}(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})\Big{\|}_{\varphi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}\\\
&\quad\quad\quad\quad\quad\quad\quad+\Big{\|}{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\int_{\Omega\times\Omega}[\Phi(\phi(\varphi({{\mathbf{x}}_{k}},\mathcal{T}_{C}^{{\mathbf{x}}_{k}}),{\mathbf{x}}_{k}),\mathcal{T}_{C}^{{\mathbf{x}}_{k}})][\Phi(\phi(\varphi({{\mathbf{x}}_{j}},\mathcal{T}_{C}^{{\mathbf{x}}_{j}}),{\mathbf{x}}_{j}),\mathcal{T}_{C}^{{\mathbf{x}}_{j}})]~{}d\mathcal{J}(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})\Big{\|}_{\Phi}\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}\\\
&\quad\quad\quad\quad\quad\quad\quad+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\int_{\Omega\times\Omega}\Big{\|}[\Phi(\phi(\varphi({{\mathbf{x}}_{k}},\mathcal{T}_{C}^{{\mathbf{x}}_{k}}),{\mathbf{x}}_{k}),\mathcal{T}_{C}^{{\mathbf{x}}_{k}})][\Phi(\phi(\varphi({{\mathbf{x}}_{j}},\mathcal{T}_{C}^{{\mathbf{x}}_{j}}),{\mathbf{x}}_{j}),\mathcal{T}_{C}^{{\mathbf{x}}_{j}})]\Big{\|}_{\Phi}~{}d\mathcal{J}(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\int_{\Omega\times\Omega}C(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})~{}d\mathcal{J}(\bar{\mathbf{x}}_{k},\bar{\mathbf{x}}_{j})\\\
&=\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j}).\end{split}$

The second line is obtained by utilizing the reproducing property applied to
the function $error(\cdot,\cdot)$. The third line follows from the properties
of the expected value. Furthermore, the fourth, ninth, and tenth lines arise
from the properties of the inner-product. We note that the fifth line holds
due to the earlier discussion that $\|error\|\leq 1$. Additional lines are
derived through scaling and equivalent substitution. We present the Lemma.1
that introduces Wasserstein distance to relate the source and target error
functions in a shared space.

The convergence guarantee of this theorem can be further strengthened. To
establish the validity of our subsequent theorem, we use
$\hat{Q}_{k}=\frac{1}{n_{Q_{k}}}\sum_{i=1}^{n_{Q_{k}}}\delta_{i}$ denote the
empirical measure to its corresponding true measure $\bar{Q}_{k}$, with
respect to the Wasserstein metric. $\delta_{i}$ is the sampling probability of
the $i$-th sample. The work [3] show that the empirical measure $\hat{Q}_{k}$
and its true related measure $\bar{Q}_{k}$ converge in finite samples. For
traditional Wasserstein distance $\mathcal{W}_{1}$, there is an inequality

$\displaystyle\mathcal{W}_{1}(\bar{Q}_{k},\hat{Q}_{k})\leq\sqrt{2\log(\frac{1}{\delta})/n_{Q_{k}}\varsigma^{{}^{\prime}}},$

holds for any $\varsigma^{{}^{\prime}}<\sqrt{2}$ and $\delta>0$. In light of
this inequality, the upper convergence bound associated with the cost function
proposed by us can be deduced straightforwardly.

$\displaystyle\begin{split}\mathcal{W}(\bar{Q}_{k},\hat{Q}_{k})\leq\exp(\frac{(1+\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{k}}\varsigma^{{}^{\prime}}}),\end{split}$

We now utilize it in conjunction with the Lemma.1 to establish the following
theorem.

###### Theorem 2

Under the assumptions of Lemma.1, suppose we have $N$ source domains, let
$\\{\bar{Q}_{k}\\}_{k=1}^{N}$ represent the probability distribution of image
embedding, with $\\{n_{Q_{k}}\\}_{k=1}^{N}$ denoting the corresponding number
of samples. Let $\\{\hat{Q}_{k}\\}_{k=1}^{N}$ denote the associated empirical
measure, where
$\hat{Q}_{k}=\frac{1}{n_{Q_{k}}}\sum_{i=1}^{n_{Q_{k}}}\delta_{i}$, and
$\delta_{i}$ is the sampling probability of the $i$-th sample. For any
$\varsigma^{{}^{\prime}}<\sqrt{2}$, $\delta>0$, and any $f_{\mathrm{aug}}$,
the following formula holds:

$\displaystyle\begin{split}\epsilon_{T}(f_{\mathrm{aug}})\leq\epsilon_{S}(f_{\mathrm{aug}})\\!+\\!\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{c}^{z})]\Big{\|}_{\varphi}\\!+\\!\Big{(}{\sum_{k=1}^{N}\sum_{j=k}^{N}w_{k}w_{j}}[\mathcal{W}(\hat{Q}_{k},\hat{Q}_{j})\\!+\\!\exp(\frac{(1\\!+\\!\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{k}}\varsigma^{{}^{\prime}}})\\!+\\!\exp(\frac{(1\\!+\\!\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{j}}\varsigma^{{}^{\prime}}})]\Big{)}\\!+\\!\vartheta,\end{split}$

where $\vartheta$ is the combined error of the ideal hypothesis
$f_{\mathrm{aug}}^{*}$ that minimizes the combined error of
$\epsilon_{S}(f_{\mathrm{aug}})+\epsilon_{T}(f_{\mathrm{aug}})$.

#### Proof.

By performing the term shift operation on the formula of Lemma.1, we obtain
$\epsilon_{T}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})-\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{{}^{\prime}})\leq\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{c}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j})$.
For clarity and ease of expression, we use $\epsilon_{S}(f_{\mathrm{aug}})$
and $\epsilon_{T}(f_{\mathrm{aug}})$ as a representation of
$\epsilon_{S}(f_{\mathrm{aug}},f_{L})$ and
$\epsilon_{T}(f_{\mathrm{aug}},f_{L})$, avoiding any potential confusion.

$\displaystyle\begin{split}\epsilon_{T}(f_{\mathrm{aug}})&\leq\epsilon_{T}(f_{\mathrm{aug}}^{*})+\epsilon_{T}(f_{\mathrm{aug}}^{*},f_{\mathrm{aug}})\\\
&=\epsilon_{T}(f_{\mathrm{aug}}^{*})+\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{*})+\epsilon_{T}(f_{\mathrm{aug}}^{*},f_{\mathrm{aug}})-\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{*})\\\
&\leq\epsilon_{T}(f_{\mathrm{aug}}^{*})+\epsilon_{S}(f_{\mathrm{aug}},f_{\mathrm{aug}}^{*})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j})\\\
&\leq\epsilon_{T}(f_{\mathrm{aug}}^{*})+\epsilon_{S}(f_{\mathrm{aug}})+\epsilon_{S}(f_{\mathrm{aug}}^{*})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j})\\\
&=\epsilon_{S}(f_{\mathrm{aug}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\mathcal{W}(\bar{Q}_{k},\bar{Q}_{j})+\vartheta\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}(\mathcal{W}(\bar{Q}_{k},\hat{Q}_{k})+\mathcal{W}(\hat{Q}_{k},\bar{Q}_{j}))+\vartheta\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}})+\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{C}^{z})]\Big{\|}_{\varphi}+{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}\big{(}\mathcal{W}(\hat{Q}_{k},\hat{Q}_{j})+\mathcal{W}(\hat{Q}_{j},\bar{Q}_{j})+\exp(\frac{(1\\!+\\!\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{k}}\varsigma^{{}^{\prime}}})\big{)}+\vartheta\\\
&\leq\epsilon_{S}(f_{\mathrm{aug}})\\!+\\!\Big{\|}\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{c}^{z})]\Big{\|}_{\varphi}\\!+\\!\Big{(}{\sum_{k=1}^{N}\sum_{j=k+1}^{N}w_{k}w_{j}}[\mathcal{W}(\hat{Q}_{k},\hat{Q}_{j})\\!+\\!\exp(\frac{(1\\!+\\!\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{k}}\varsigma^{{}^{\prime}}})\\!+\\!\exp(\frac{(1\\!+\\!\lambda)\log(1/\delta)}{\sigma^{2}n_{Q_{j}}\varsigma^{{}^{\prime}}})]\Big{)}\\!+\\!\vartheta.\\\
\end{split}$

#### Remark.

Based on the theoretical analysis, the conclusion is evident. To optimize the
domain-specific augmenters, it is sufficient to focus solely on optimizing the
formula on the right side of the proposed theory. In other words, our
objective is to minimize (1) $\epsilon_{S}(f_{\mathrm{aug}})$, which
corresponds to the domain-class alignment loss, i.e., $\mathcal{L}_{DA}$ and
$\mathcal{L}_{CA}$, and (2)
${\sum_{k=1}^{N}\sum_{j=k+1}^{N}\omega_{k}\omega_{j}}\mathcal{W}(\hat{Q}_{k},\hat{Q}_{j})$,
which corresponds to distribution consistency loss, i.e., $\mathcal{L}_{DC}$.
In addition to the aforementioned items, the theory further reveals that the
learning bound is influenced by the following underlying factors: (1) Size of
source domain samples: the more source samples used for training, the higher
the accuracy; (2) The performance of the VLFMs is a crucial aspect. In
Theorem.2, we encounter the term $\|\mathbb{E}_{z\sim
P}[\varphi(z,\mathcal{T}_{c}^{z})]\|_{\varphi}$, which represents the
correlation between the image embedding and the text embedding of the target
domain. Since both image embeddings and text embeddings are generated by
VLFMs, this term is only relevant to the performance of VLFMs. The better
performance of VLFMs, the lower calculated value of this item, which further
reduces the value of the cost function and the calculated Wasserstein distance
$\mathcal{W}(\hat{Q}_{k},\hat{Q}_{j})$, leading to a narrower learning bound.

##  Appendix B Implementation Detail and Datasets

#### Implementation Detail.

We conducted training by normalizing all text and image embeddings and
resizing all images to $224\times 224$. Typically, the hyperparameters for
training the domain-specific augmenters are set as follows: $\alpha=\beta=1.0$
and $\gamma=0.05$. The fixed value for cost function: $\sigma=\lambda=1.0$.
$\varepsilon=0.1$ for training the linear classifier. In our experiments, we
used the Adam optimizer with an initial learning rate of 1e-2 to train the
domain-specific augmenters. For the Mini-DomainNet dataset, we employed a
MultiStepLR scheduler with milestones set at [2, 4, 7, 10, 15, 20, 30, 40]
epochs and a gamma of 0.5. For the Office-Home dataset, the milestones were
set at [5, 10, 15, 20, 30, 40] epochs. When training a linear classifier, we
set the initial learning rate to 1e-3, and the milestones are set at [1, 3, 5]
epochs with a gamma of 0.1. We train on four NVIDIA GeForce RTX 3090 Ti GPUs
with 24GB VRAM each.

#### Dataset.

We evaluated LanDA using two existing cross-class datasets, as follows: (1)
Mini-DomainNet is a subset of the original DomainNet [34] dataset existing
53,237 images, which contains the 40 most common classes from 5 domains:
sketch, infograph, clipart, painting and real; (2) Office-Home [43]
encompasses 15,500 images spanning 66 categories, comprising four domains:
art, clipart, product and real. We set up various MSDA transfer scenarios for
each datasets independently to assess the efficacy of our proposed approach.

##  Appendix C Additional Quantitative Evaluation on ViT-L/14

Table.4 presents supplementary results for the three-source domains transfer
scenario conducted on the Mini-DomainNet dataset. To further complement the
quantitative evaluation outlined in the main body, we conducted transfer
learning experiments in a two-source domains scenario by employing ViT-L/14 as
the visual backbone within the CLIP framework. The transfer experiments for
six scenarios on the Office-Home dataset are outlined in Table.5 and Table.6.
Additionally, the transfer experiments for eight scenarios on the Mini-
DomainNet dataset are presented in Table.7 and Table.8.

Scenario | R, P, C to S | C, P, S to R
---|---|---
| ID | OD |  | ID | OD |
Domains | R | P | C | S | EXT | C | P | S | R |  EXT
CLIP ZS(G) [36] |  96.65 |  94.40 |  94.86 |  92.54 |  93.21 |  94.86 |  94.40 |  92.54 |  96.65 |  95.29
CLIP ZS(A) [36] |  96.80 |  95.32 |  94.39 |  92.21 |  93.86 |  94.39 |  95.32 |  92.21 |  96.80 |  95.39
CLIP LP |  97.23 |  96.25 |  95.67 |  93.53 |  94.96 |  95.73 |  96.25 |  95.66 |  96.61 |  96.25
VQGAN+CLIP [11] |  97.36 |  96.27 |  95.82 |  93.60 |  95.05 |  95.90 |  96.36 |  95.71 |  96.65 |  96.32
Diffusion+CLIP [22] |  97.24 |  95.84 |  95.87 |  93.88 |  95.10 |  96.32 |  96.37 |  95.75 |  96.86 |  96.50
LADS [14] |  97.11 |  95.89 |  96.08 |  94.02 |  95.19 |  96.30 |  96.14 |  95.61 |  96.90 |  96.46
LanDA(Ours) |  97.47 |  96.05 |  96.04 |  94.33 |  95.43 |  96.23 |  96.46 |  95.79 |  97.02 |  96.59
Table 4: Quantitative evaluation in the three-source domains scenario of the
Mini-DomainNet dataset. Each of domain-specific augmenters is a 2-layer MLP
with input and output dimensions of 768 and a hidden dimension of 384. The
”EXT” column is derived as follows: 50% of the average predictions from the ID
domain, added to 50% of the predicted value from the OD domain. ”ID” column
denotes the source domains, ”OD” column denotes the unseen target domain.

Consistent with the experimental findings in the main body, CLIP LP and
VQGAN+CLIP [11] demonstrate satisfactory performance on the source domain test
set, however, they do not exhibit significant improvements on the unseen
target domain compared to CLIP zero shot in most cases. Conversely,
Diddusion+CLIP [22], LADS [14], and LandA all achieve superior performance to
CLIP zero shot in both the target and source domains. Notably, our method,
LandA, manifests as the most effective approach.

Scenario | C, R to A | P, R to A | A, R to P
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | C | R | A | EXT | P | R | A | EXT | A | R | P | EXT
CLIP ZS(G) [36] |  73.13 |  91.94 |  84.97 |  83.75 |  90.96 |  91.94 |  84.97 |  88.21 |  84.97 |  91.94 |  90.96 |  89.71
CLIP ZS(A) [36] |  75.34 |  92.10 |  86.34 |  85.03 |  87.73 |  92.10 |  86.34 |  88.13 |  86.34 |  92.10 |  87.73 |  88.48
CLIP LP |  83.13 |  94.09 |  86.62 |  87.62 |  95.63 |  92.33 |  86.07 |  90.03 |  90.76 |  94.70 |  91.29 |  92.01
LADS [14] |  82.97 |  93.94 |  87.42 |  87.94 |  94.94 |  93.06 |  86.45 |  90.23 |  91.03 |  95.00 |  92.61 |  92.81
LanDA(Ours) |  83.05 |  93.78 |  88.69 |  88.55 |  94.88 |  93.02 |  86.79 |  90.37 |  92.62 |  94.93 |  93.15 |  93.46
Table 5: Quantitative evaluation in two-source domains scenario of the Office-Home dataset. Scenario | C, R to P | A, P to R | C, P to R
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | C | R | P | EXT | A | P | R | EXT | C | P | R | EXT
CLIP ZS(G) [36] |  73.13 |  91.94 |  90.96 |  86.75 |  84.97 |  90.96 |  91.94 |  89.95 |  73.13 |  90.96 |  91.94 |  86.99
CLIP ZS(A) [36] |  75.34 |  92.10 |  87.73 |  85.73 |  86.34 |  87.73 |  92.10 |  89.57 |  75.34 |  87.73 |  92.10 |  86.82
CLIP LP |  81.53 |  94.86 |  90.81 |  89.50 |  89.38 |  95.78 |  92.32 |  92.45 |  82.67 |  95.48 |  92.47 |  90.62
LADS [14] |  81.44 |  94.51 |  91.27 |  89.72 |  89.33 |  95.94 |  93.06 |  92.85 |  82.10 |  94.96 |  92.82 |  90.68
LanDA(Ours) |  82.06 |  94.24 |  91.80 |  89.98 |  89.69 |  95.86 |  93.79 |  93.28 |  82.37 |  95.78 |  93.55 |  91.31
Table 6: Quantitative evaluation in two-source domains scenario of the Office-Home dataset. Scenario | R, S to C | R, P to C | R, S to I | R, P to I
---|---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |  | ID | OD |
Domains | R | S | C | EXT | R | P | C | EXT | R | S | I | EXT | R | P | I | EXT
CLIP ZS(G) [36] |  96.65 |  92.54 |  94.86 |  94.73 |  96.65 |  94.40 |  94.86 |  95.19 |  96.65 |  92.54 |  79.08 |  86.34 |  96.65 |  94.40 |  79.08 |  87.30
CLIP ZS(A) [36] |  96.80 |  92.21 |  94.39 |  94.45 |  96.80 |  95.32 |  94.39 |  95.23 |  96.80 |  92.21 |  80.90 |  87.70 |  96.80 |  95.32 |  80.90 |  88.48
CLIP LP |  97.34 |  96.08 |  94.49 |  95.60 |  97.55 |  96.73 |  94.68 |  95.91 |  97.31 |  95.46 |  79.25 |  87.82 |  97.58 |  95.01 |  79.79 |  88.04
VQGAN+CLIP [11] |  97.41 |  95.92 |  94.50 |  95.58 |  97.62 |  96.76 |  94.64 |  95.92 |  97.42 |  95.56 |  79.31 |  87.90 |  97.47 |  95.27 |  79.80 |  88.09
Diffusion+CLIP [22] |  97.60 |  95.83 |  94.68 |  95.70 |  97.56 |  96.80 |  94.70 |  95.94 |  97.48 |  95.58 |  79.92 |  88.23 |  97.50 |  95.35 |  80.87 |  88.65
LADS [14] |  97.48 |  95.90 |  94.64 |  95.67 |  97.44 |  96.75 |  94.72 |  95.91 |  97.37 |  95.60 |  80.94 |  88.71 |  97.34 |  95.46 |  81.03 |  88.72
LanDA(Ours) |  97.57 |  95.89 |  95.24 |  95.99 |  97.49 |  96.87 |  95.25 |  96.22 |  97.49 |  95.71 |  81.29 |  88.95 |  97.36 |  95.70 |  81.55 |  89.04
Table 7: Quantitative evaluation in two-source domains scenario of the Mini-DomainNet dataset. Scenario | R, C to S | R, P to S | R, S to P | R, C to P
---|---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |  | ID | OD |
Domains | R | C | S | EXT | R | P | S | EXT | R | S | P | EXT | R | C | P | EXT
CLIP ZS(G) [36] |  96.65 |  94.86 |  92.54 |  94.15 |  96.65 |  94.40 |  92.54 |  94.03 |  96.65 |  92.54 |  94.40 |  94.50 |  96.65 |  94.86 |  94.40 |  95.08
CLIP ZS(A) [36] |  96.80 |  94.39 |  92.21 |  93.90 |  96.80 |  95.32 |  92.21 |  94.14 |  96.80 |  92.21 |  95.32 |  94.91 |  96.80 |  94.39 |  95.32 |  95.46
CLIP LP |  97.34 |  95.61 |  92.61 |  94.54 |  96.79 |  95.57 |  92.58 |  94.38 |  97.10 |  95.41 |  93.95 |  95.10 |  97.45 |  95.98 |  93.88 |  93.30
VQGAN+CLIP [11] |  97.26 |  95.47 |  92.63 |  94.50 |  96.81 |  95.48 |  92.60 |  94.37 |  96.96 |  95.50 |  94.87 |  95.55 |  97.26 |  95.90 |  94.84 |  95.71
Diffusion+CLIP [22] |  97.29 |  95.55 |  93.14 |  94.78 |  96.93 |  95.55 |  92.97 |  94.61 |  97.02 |  95.48 |  95.64 |  95.95 |  97.38 |  96.07 |  95.68 |  96.20
LADS [14] |  97.32 |  95.76 |  93.22 |  94.88 |  96.77 |  95.60 |  92.84 |  94.51 |  96.89 |  95.26 |  95.68 |  95.88 |  97.36 |  96.12 |  95.79 |  96.27
LanDA(Ours) |  97.37 |  95.78 |  93.78 |  95.18 |  96.93 |  95.46 |  93.33 |  94.76 |  97.29 |  95.79 |  95.94 |  96.24 |  97.29 |  95.98 |  96.10 |  96.38
Table 8: Quantitative evaluation in two-source domains scenario of the Mini-
DomainNet dataset.

##  Appendix D Additional Quantitative Evaluation on ViT-B/16 and RN50

In previous experiments, ViT-L/14 was utilized as the visual backbone of CLIP,
employing its larger model size and capability to extract multiple
hierarchical features. However, in this section, we assess the effectiveness
of ViT-B/16 and RN50 (ResNet50) as alternative backbones. ViT-B/16, compared
to ViT-L/14, offers a smaller model size, making it well-suited for real-time
applications and providing faster inference speed. Detailed experimental
results can be found in Table.9.

Scenario | Mini-DomainNet: R, P, S to C | Mini-DomainNet: R, P, C to S | Office-Home: A, C, R to P
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | R | P | S | C | EXT | R | P | C | S | EXT | A | C | R | P | EXT
CLIP ZS(G) [36] |  95.85 |  93.09 |  90.12 |  92.39 |  92.70 |  95.85 |  93.09 |  92.39 |  90.12 |  91.95 |  79.45 |  63.13 |  87.72 |  86.14 |  81.45
CLIP ZS(A) [36] |  96.11 |  93.19 |  90.83 |  92.64 |  93.01 |  96.11 |  93.19 |  92.64 |  90.83 |  92.01 |  78.07 |  63.36 |  87.34 |  88.68 |  82.47
CLIP LP |  96.53 |  94.47 |  93.41 |  92.20 |  93.50 |  96.52 |  94.39 |  94.34 |  90.89 |  92.99 |  85.24 |  75.14 |  90.71 |  88.75 |  86.22
VQGAN+CLIP [11] |  96.57 |  94.52 |  93.33 |  92.28 |  93.54 |  96.80 |  94.68 |  94.42 |  90.99 |  93.15 |  N.A. |  N.A. |  N.A. |  N.A. |  N.A.
Diffusion+CLIP [22] |  96.66 |  94.51 |  93.28 |  92.75 |  93.78 |  96.77 |  94.25 |  94.16 |  91.43 |  93.25 |  N.A. |  N.A. |  N.A. |  N.A. |  N.A.
LADS [14] |  96.55 |  94.48 |  93.40 |  92.69 |  93.75 |  96.58 |  94.60 |  94.03 |  91.47 |  93.27 |  85.58 |  75.79 |  91.04 |  89.12 |  86.63
LanDA(Ours) |  96.60 |  94.33 |  93.45 |  93.03 |  93.91 |  96.70 |  94.71 |  94.31 |  91.79 |  93.52 |  85.79 |  77.94 |  91.20 |  89.96 |  87.47
Table 9: Quantitative evaluation in the three-source domains scenario. Each of
domain-specific augmenters is a 2-layer MLP with input and output dimensions
of 512 and a hidden dimension of 256.

RN50 (ResNet50) is commonly used as the visual backbone in traditional Multi-
Source Domain Adaptation (MSDA) methods due to its fewer parameters compared
to ViT-L/14 and ViT-B/16. The experimental results, presented in Table.10,
showcase the performance achieved when utilizing ResNet50 as the visual
backbone of CLIP.

Scenario | Mini-DomainNet: R, P, S to C | Mini-DomainNet: R, P, C to S | Office-Home: A, C, R to P
---|---|---|---
| ID | OD |  | ID | OD |  | ID | OD |
Domains | R | P | S | C | EXT | R | P | C | S | EXT | A | C | R | P | EXT
CLIP ZS(G) [36] |  93.20 |  85.05 |  79.41 |  80.94 |  83.41 |  93.20 |  85.05 |  80.94 |  79.41 |  82.90 |  67.59 |  45.88 |  78.97 |  77.71 |  70.93
CLIP ZS(A) [36] |  93.55 |  85.94 |  79.95 |  82.76 |  84.62 |  93.55 |  85.94 |  82.67 |  79.95 |  83.67 |  61.79 |  39.39 |  76.98 |  77.03 |  68.21
CLIP LP |  94.94 |  89.10 |  82.67 |  82.67 |  85.79 |  95.30 |  89.21 |  90.22 |  80.49 |  86.03 |  75.86 |  65.11 |  85.80 |  79.64 |  77.62
VQGAN+CLIP [11] |  95.01 |  88.95 |  82.63 |  82.80 |  85.83 |  95.36 |  88.89 |  90.25 |  80.53 |  86.02 |  N.A. |  N.A. |  N.A. |  N.A. |  N.A.
Diffusion+CLIP [22] |  94.88 |  88.97 |  83.52 |  83.28 |  86.20 |  95.28 |  88.94 |  90.30 |  81.02 |  86.26 |  N.A. |  N.A. |  N.A. |  N.A. |  N.A.
LADS [14] |  94.90 |  89.24 |  85.75 |  83.44 |  86.70 |  95.20 |  88.76 |  89.88 |  81.14 |  86.21 |  75.22 |  67.97 |  86.05 |  80.20 |  78.31
LanDA(Ours) |  95.00 |  89.48 |  87.33 |  84.03 |  87.32 |  95.38 |  88.83 |  89.79 |  81.66 |  86.50 |  75.07 |  69.16 |  86.33 |  80.87 |  78.86
Table 10: Quantitative evaluation in the three-source domains scenario. Each
of domain-specific augmenters is a 2-layer MLP with input and output
dimensions of 1024 and a hidden dimension of 512.

Upon comparing all experimental results in the main body and the outcomes
presented in Table.5 through 8 in the appendix, it is evident that when using
ViT-B/16 and RN50 as the visual backbone of CLIP, CLIP LP and VQGAN-CLIP [11]
struggle to surpass CLIP ZS [36] in OD in certain cases (_e.g_., A, C, R to P
in Table.9, and R, P, S to C in Table.10). The hypothesized reason is that a
reduced parameter count compromises the ability to extract accurate features
for each class, particularly when dealing with vast datasets. However, other
methods excluding CLIP ZS [36] exhibit substantial improvements in OD
performance. The results presented in Table.9 and Table.10 illustrate that the
proposed LanDA method continues to achieve superior outcomes, highlighting the
effectiveness of our approach within a lightweight general ViT and CNN
architecture.

##  Appendix E Additional t-SNE Visualizations

Figure.5 displays the t-SNE visualization results of the ablation experiment,
corresponding to Figure 2 in the main body. In Figure.5(a), the experiment was
conducted on the Mini-DomainNet dataset, while in Figure.5(b), it was
performed on the Office-Home dataset. The leftmost column of the figure
showcases the visualized image embeddings obtained from CLIP, where each color
represents a specific class. The remaining columns illustrate the extended
domains, with distinct colors indicating different image embeddings from these
domains. Our study compares three approaches for training domain-specific
augmenters: the second column utilizes $\mathcal{L}_{DA}$ alone, the third
column combines $\mathcal{L}_{DA}$ and $\mathcal{L}_{CA}$, and the fourth
column integrates $\mathcal{L}_{DA}$, $\mathcal{L}_{CA}$, and
$\mathcal{L}_{DC}$. Figure.5 provides an intuitive observation where the
second column exhibits a noticeable distance between domains, along with
relatively closely clustered points within each domain. In the third column,
the clusters within domains are more dispersed, while the inter-domain
distance remains substantial. The fourth column demonstrates a close
distribution of the three domains, suggesting the extraction of common
features, while also showcasing dispersed clusters within each domain,
indicating a strong classification ability.

Figure 5: t-SNE visualizations on the Mini-DomainNet and Office-Home datasets.

##  Appendix F Examples of the Generation Images Produced by VQGAN+CLIP

Figure 6: We give examples from the VQGAN+CLIP method. Beginning with a
realistic photo of a cake, a realistic photo of a parrot, a realistic photo of
an ice cream, a realistic photo of an apple and a realistic photo of a swan.
Then we transfer these images into the domains of painting, sketch, infograph,
and clipart. Simultaneously inputting a text prompt in a desired style and an
image prompt from a source domain can be employed to achieve a ”style
transfer” effect. For each pairing of source and target domains, we
systematically generate 200 images, amounting to an aggregate of 3000 images
for training. Subsequently, we train a classifier for CLIP using both the
source domains data and the generated data.

##  Appendix G Examples of the Generation Images Produced by Diffusion+CLIP

Figure 7: We give examples from the Diffusion+CLIP method. Beginning with a
painting of an airplane, a painting of a basketball, a painting of a cat, a
painting of a palm tree and a painting of a tiger. Then we transfer these
images into the domains of real, sketch, infograph, and clipart.
Simultaneously inputting a text prompt describing the target domain and an
image prompt of the source domain can promote the transfer of images from the
source domain to the target domain. Given the time-consuming nature of the
diffusion-based image generation process, we systematically generate 200
images for each source domain-target domain pairing, thereby accumulating a
total of 400 or 600 images for each transfer scenario. Consequently, our
experiments yield a comprehensive set of 3000 images. Analogously, we
simultaneously feed the source domains and generated target domain images into
CLIP to training the classifier.

##  Appendix H Compared with Conventional MSDA Methods

To ensure fairness, quantitative comparisons with the traditional MSDA method
[46, 42, 34] were not conducted in our previous experiments. In this section,
we outline the key differences between these two approaches.

  * •

Traditional MSDA methods, whether source-unfree or source-free, operate under
the assumption that the target domain is accessible (see Figure.8(a)). While
our method does not rely on any target domain images.

  * •

The majority of conventional MSDA methods necessitate training a substantial
number of parameters, typically ranging in the tens of millions. But our
method only requires training hundreds of thousands to millions of parameters.

  * •

Similar to previous experimental baselines, our method incorporates VLFMs as
prior knowledge. Additionally, the performance of LanDA is closely tied to the
capabilities of VLFMs.

  * •

Even when utilizing ResNet50 as the vision backbone, our method can achieve
comparable performance to traditional MSDA methods (see Figure.8(b)), even
without a single target domain image and using a reduced number of training
parameters. The performance of our proposed method significantly outperforms
the traditional MSDA method when utilizing ViT-L/14 or ViT-B/16 as the
backbone.

  * •

Our method demonstrates strong performance in both the target and source
domains, whereas traditional methods primarily focus on the performance solely
on the target domain.

Figure 8: Compared with Conventiona MSDA methods. (a) Most conventional MSDA
methods employ a shared feature extractor alongside domain-specific classifier
heads, and the target domain images is required. (b) We compare the A, C, R,
$\to$ P transfer scenario of various methods on the Office-Home dataset when
utilizing RN50 as the visual backbone. The trainable parameters of LanDA are
significantly less, ranging from 1/8 to 1/5, in comparison to the conventiona
MSDA method.

##  Appendix I Limitations and Future Work

Given the propose approach reliance on the rich concepts learned by pre-
trained visual-language models, its applicability is constrained to domains
that can be accurately depicted through textual descriptions and are
sufficiently represented in the data employed for training the pre-trained
visual-language models. Our proposed method has limitations when it comes to
domain transformations in complex situations. We hope that future research
will facilitate more sophisticated domain transformations that are not reliant
on the visual language pre-trained models’ accuracy in classifying existing
domains and classes.