# Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation

Zhekai Du1, Xinyao Li1, Fengling Li2, Ke Lu1, Lei Zhu3, Jingjing Li 1
1University of Electronic Science and Technology of China;
2University of Technology Sydney; 3Tongji University
{zhekaid<EMAIL_ADDRESS><EMAIL_ADDRESS>Jingjing Li is the
corresponding author.

###### Abstract

Conventional Unsupervised Domain Adaptation (UDA) strives to minimize
distribution discrepancy between domains, which neglects to harness rich
semantics from data and struggles to handle complex domain shifts. A promising
technique is to leverage the knowledge of large-scale pre-trained vision-
language models for more guided adaptation. Despite some endeavors, current
methods often learn textual prompts to embed domain semantics for source and
target domains separately and perform classification within each domain,
limiting cross-domain knowledge transfer. Moreover, prompting only the
language branch lacks flexibility to adapt both modalities dynamically. To
bridge this gap, we propose Domain-Agnostic Mutual Prompting (DAMP) to exploit
domain-invariant semantics by mutually aligning visual and textual embeddings.
Specifically, the image contextual information is utilized to prompt the
language branch in a domain-agnostic and instance-conditioned way. Meanwhile,
visual prompts are imposed based on the domain-agnostic textual prompt to
elicit domain-invariant visual embeddings. These two branches of prompts are
learned mutually with a cross-attention module and regularized with a
semantic-consistency loss and an instance-discrimination contrastive loss.
Experiments on three UDA benchmarks demonstrate the superiority of DAMP over
state-of-the-art approaches.

##  1 Introduction

Labeling scarcity is a perennial problem in deep learning, as collecting
abundant labeled data can be expensive, time-consuming, or even infeasible
[37, 68]. Unsupervised Domain Adaptation (UDA) serves as a promising approach
to leverage the knowledge from a well-labeled source domain to benefit the
task on an unlabeled target domain, where the two domains have similar
semantics but different data distributions [11, 52, 21].

Conventional UDA methods typically bridge the domain gap by minimizing the
distribution discrepancy, through either moment matching [32, 31, 39, 21, 26]
or adversarial learning [11, 33, 43, 7]. However, roughly aligning two domains
can result in distorted semantic structure and less class discriminability in
learned feature representations [49, 4]. Besides, prior works use numerical
labels for training and inference, which discard rich semantics behind
categories, leading to sub-optimal adaptation when handling complex categories
and domain shifts.

Figure 1: Top: exsiting prompt-based methods (e.g., DAPrompt [12]) only learn
textual prompts to embed semantics for each domain and perform classification
separately, which limits cross-domain knowledge transfer and feature
alignment. Bottom: our method learns both textual and visual prompts mutually
to make both modalities of embeddings domain-invariant, thus enabling better
utilization of source knowledge and flexibility alignment.

Recently, large-scale pre-trained Vision-Language Models (VLMs) have
demonstrated impressive successes in various downstream tasks [63, 18, 41,
15]. By pre-training on tremendous image-text pairs, these models learn
transferable multimodal representations that align images and texts in a joint
embedding space. In particular, the Contrastive Language-Image Pre-training
(CLIP) model [40] encodes rich semantic knowledge about visual concepts,
presenting new opportunities to address the domain gap by leveraging the pre-
trained vision and language knowledge. However, few attempts have been made to
leverage VLMs for UDA since two challenges stand in the way, namely, 1) how to
effectively take advantage of the rich pre-trained knowledge encoded in VLMs,
and 2) how to transfer the source knowledge to the target domain for better
adaptation.

Generally, there are two feasible routes for adapting large-scale pre-trained
VLMs. The first is to use the zero-shot prediction capacity of VLMs to obtain
pseudo-labels and fine-tune the image backbone with other UDA techniques [24].
While the source knowledge can be well-encoded by fine-tuning, the pseudo-
labels largely rely on manually designed textual descriptions and fine-tuning
the model may ruin the pre-trained knowledge. Another way is to freeze the
pre-trained model and only tune the input data (e.g., prompt) for model
adaptation, which only involves a small set of learnable parameters and
retains the pre-trained knowledge. For instance, DAPrompt [12] proposes to
embed domain semantics into domain-specific textual prompts for each domain,
which are then coupled with a domain-agnostic context for domain-specific
classification in the joint CLIP space. However, we argue that a large portion
of source knowledge is prone to be encoded in source-specific prompts, which
cannot be transfered to the target domain. For instance, we conduct an
experiment by disabling the source supervision loss in DAPrompt, and observe
marginal influence on the target performance (see Fig. 1).

In this work, we aim to learn transferable (domain-agnostic) prompts to
effectively leverage both pre-trained knowledge and source-knowledge for the
target domain using CLIP. However, directly learning such textual prompts in
UDA can be sub-optimal, as visual embeddings from different domains typically
encompass distinct, domain-biased information that conforms to different
distributions within the CLIP space. This is a key motivation behind domain-
specific prompting in previous methods [12, 45]. Inspired by the recent
success of visual prompting [19], we propose also adapting the visual
embeddings to elicit domain-invariant representations by prompting the vision
backbone, based on the domain-agnostic textual prompt. Meanwhile, domain-
invariant visual embeddings can still retain individual characteristics, e.g.,
object color and size. Such variations, even within the same category,
necessitate instance-conditioned textual prompts for better alignment, as
shown in Fig. 1. Given the interdependent nature of the two kinds of prompts,
we build a mutual learning framework based on a cross-attention mechanism
inspired by the Transformer decoder [54]. A semantic-consistency
regularization and an instance-discrimination contrastive loss are further
imposed to ensure that the learned prompts carry pure domain-agnostic and
instance-conditioned information.

In summary, the key contributions of this work are three-fold: 1) We propose a
novel framework termed DAMP to learn domain-agnostic prompts for transferring
pre-trained knowledge and source knowledge to the target domain using CLIP. 2)
DAMP mutually aligns textual and visual embeddings by prompting both
modalities to learn domain-invariant representations, which are optimized with
two elaborate regularizations. 3) Extensive experiments on three UDA
benchmarks validate that DAMP brings consistent and notable gains over state-
of-the-art approaches.

##  2 Related Works

Unsupervised Domain Adaptation. To enable effective knowledge transfer, modern
UDA methods typically fall into two technical routes. The first line of works
aim to reduce the domain shift by aligning the feature distributions across
domains. Common techniques include minimizing statistical distribution
distances via moment matching [32, 39, 47] and learning domain-invariant
features via adversarial alignment [11, 67, 44, 43]. More recent methods focus
on disentangling domain-invariant and domain-specific factors for casual
invariance [36, 64] or self-training with elaborate pseudo labels [30, 35,
58]. The second line of works resort to more large-scale networks, e.g.,
Vision Transformer (ViT) [6], for more transferable features. For instance,
CDTrans [60] leverages the cross-attention mechanism in Transformer for cross-
domain feature alignment. TVT [61] introduces the evaluated transferabilities
into the Multi-head Self-Attention module to construct a transferable ViT.
SSRT [48] proposes to perturb the target features to refine the ViT and
designs a safe training mechanism.

Despite remarkable progresses, most existing UDA methods only operate in the
vision modality, discarding the rich semantics behind features and categories,
hindering effective adaptation for complex and large domain gaps.

Vision-Language Models and Prompt Learning. Recent large-scale pre-trained
VLMs have shown impressive performance on various vision-and-language tasks
[62, 63]. VLMs like CLIP [40] and ALIGN [18] learn joint representations of
images and texts by pre-training on large amounts of image-text pairs. A key
capability of VLMs is the zero-shot prediction, where the pre-trained model
can be applied to downstream tasks by simply conditioning on a textual prompt
like “a photo of a [CLS]”. This avoids costly fine-tuning and preserves the
original knowledge in VLMs. However, manually designing effective prompts can
be challenging. Prompt learning has thus become a popular VLMs adaptation
technique. CoOp [72] first uses learnable context tokens to prompt the
language encoder of CLIP for visual classification. Later, CoCoOp [71] learns
instance-conditioned prompts with a two-layer network for more generalizable
textual prompts. MaPLe [22] introduces multi-modal prompts to fine-tune both
modalities. Neverthless, these works do not consider the domain shift problem.

Figure 2: Overview of the proposed DAMP framework. Parameters of $f_{s}$ and
$f_{v}$ are frozen and only $\bm{p}_{1:N}$ and $G$ are tunable during
training. The blue arrows represent text data flows, while the green and
purple arrows are data flows for source and target images, respectively. We
only depict the prompting process for source weakly augmentated samples. All
other samples follow the same process. $\mathcal{L}_{sc}^{s}$
($\mathcal{L}_{sc}^{t}$), $\mathcal{L}_{idc}^{s}$ ($\mathcal{L}_{idc}^{t}$),
and $\mathcal{L}_{im}$ are regularizations to make the prompting domain-
agnostic, instance-conditioned and semantic-compatible, respectively.

To leverage VLMs and prompt learning for UDA, DAPrompt [12] introduces a set
of domain-specific textual tokens to encode domain semantics and perform
classification with target-specific prompts. AD-CLIP [45] learns both domain-
and image-specific tokens with feature statistics in the vision backbone.
However, learning prompts for different domains separately may limit cross-
domain knowledge transfer. Besides, prompting in a single modality cannot
fully adapt the multi-modal knowledge in VLMs.

##  3 Proposed Method

Let $\mathcal{D}_{s}=\\{x^{s}_{i},y^{s}_{i}\\}_{i=1}^{N_{s}}$ be the source
domain with $N_{s}$ labeled samples, where $x_{s}^{i}\sim P_{s}(X)$ is the
input and $y_{s}^{i}\in Y$ is the label. Meanwhile, we have a target domain
$\mathcal{D}_{t}=\\{x_{i}^{t}\\}_{i=1}^{N_{t}}$ with $N_{t}$ unlabeled samples
and $x_{t}\sim P_{t}(X)$. These two domains are assumed to have different
distributions in the data space $X$, but share the same label (semantic) set
$Y$. The goal of UDA is to learn a model $f:X\rightarrow Y$ with
$\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ that can perform well on the target
domain.

###  3.1 Domain-Agnostic Prompting with CLIP

Traditional UDA methods typically implement $f$ as a uni-modal neural network
and associate each category with a numerical label, which overlook the rich
semantics that could inform classification. In this work, we leverage CLIP
[40] to enable semantic-driven classification.

CLIP learns aligned visual and textual representations by pre-training an
image encoder $f_{v}$ and a text encoder $f_{s}$ on a large dataset of image-
text pairs. Specifically, $f_{v}$ can be a ResNet [14] or ViT [6] backbone
that extracts a visual embedding from an image. On the other hand, $f_{s}$
uses a Transformer [54] to encode the paired textual description into a
compact embedding. The two encoders are trained jointly with a contrastive
loss. The aligned joint space then allows zero-shot classification for
arbitrary input $\bm{x}$ 111We use bold font $\bm{x}$ to denote either a
source or a target sample. by comparing the visual embedding $f_{v}(\bm{x})$
to textual embeddings $\\{f_{s}(\bm{t}_{k})\\}_{k=1}^{K}$ correspond to $K$
classes in the joint space:

$P(\bm{y}=k\mid\bm{x})=\frac{\exp\left(cos\left(f_{v}(\bm{x}),f_{s}\left(\bm{t}_{k}\right)\right)/\tau\right)}{\sum_{k=1}^{K}\exp\left(cos\left(f_{v}(\bm{x}),f_{t}\left(\bm{t}_{k}\right)\right)/\tau\right)},$
(1)

where $\tau=0.01$ is the temperature coefficient learned by CLIP,
$cos(\cdot,\cdot)$ denotes the cosine similarity, and $\bm{t}_{k}$ is the
textual prompt of the $k$-th class, e.g., “a photo of a [CLS]”.

However, manually designed prompts can be naive and sub-optimal. A more
effective way is to make $\\{\bm{t}_{k}\\}_{k=1}^{K}$ learnable as in CoOp
[72]. In UDA, covariate shift [37] is a widely adopted assumption, which
indicates that the marginal distributions differ (i.e., $P_{s}(X)\neq
P_{t}(X)$) but the conditional distribution $P(Y|X)$ remains unchanged between
domains. This motivates using a shared set of textual prompts to model the
invariant $P(Y|X)$. In this work, we use a domain- and class-shared input
prompt template:

$\bm{t}_{k}:=[\bm{p}_{1}][\bm{p}_{2}]\dots[\bm{p}_{N}][\text{CLS}_{k}],$ (2)

where $\bm{p}_{1:N}\in\mathbb{R}^{N\times D}$ are learnable contexts with
length $N$ and dimension $D$, and $[\text{CLS}_{k}]$ is the $k$-th class name.

###  3.2 Mutual Prompt Learning with Cross-Attention

Directly learning domain-agnostic prompts as in Eq. (2) can be challenging in
UDA. First, $f_{v}$ is pre-trained without domain adaptation objectives,
yielding domain-biased visual embeddings that conform to different
distributions across domains [12]. Second, the instance diversity leads to
large intra-class variation, making it difficult to align all samples to a
class-level textual prompt. To address these issues, we propose to impose
visual prompts on $f_{v}$ to elicit more domain-agnostic visual
representations. Meanwhile, we also adjust the textual prompt on $f_{s}$
according to each image contextual information for better image-text paired
alignment, like in the original CLIP pre-training.

Language-Guided Visual Prompting. As $\\{\bm{t}_{k}\\}_{i=1}^{K}$ encode
domain-agnostic class semantics, we can exploit these semantics to guide the
generation of visual prompts that elicit domain-invariant visual
characteristics. To achieve this, we use the cross-attention [54] mechanism to
pass information between the two branches, which has shown great success in
modeling multimodal interactions [51, 16].

Given a textual prompt $\bm{t}_{k}$, the class name $[\text{CLS}_{k}]$ is
first tokenized and embedded into $\bm{r}_{k}\in\mathbb{R}^{L_{k}\times D}$,
where $L_{k}$ is the name length. The text encoder $f_{s}$ then extract
embeddings via $J$ Transformer encoder layers
$\\{\texttt{Enc}_{j}\\}_{j=1}^{J}$:

$\displaystyle
T^{k}_{j}=\texttt{Enc}_{j}([\bm{p}_{1:N},\bm{r}_{k}])~{}~{}~{}j=1.$ (3)
$\displaystyle T^{k}_{j}=\texttt{Enc}_{j}(T_{j-1})~{}~{}~{}j=2,3,\dots,J.$

Here $[\cdot,\cdot]$ stands for concatenation,
$T^{k}_{j}\in\mathbb{R}^{(N+L_{k})\times D}$ is the extracted embeddings in
layer $j$. CLIP only uses the embedding at the last position of layer $J$ as
the textual embedding, denoted as $\bm{s}_{k}$. However, we argue that
embeddings at other positions also encode rich contextual information due to
shared parameters among them. Therefore, we use the first $N$ embeddings of
$T^{k}_{J}$, denoted as
$\bm{\widetilde{s}}=T_{J}^{k}[1:N]\in\mathbb{R}^{N\times D}$, to guide the
generation of visual prompts, which generally encode domain- and class-
agnostic semantics.

For visual prompting, there are two widely used forms in the community, i.e.,
the pixel-level prompts [1, 10] and token-level prompts [19, 65]. These two
kinds of pre-model prompting poses challenges to prompt different vision
backbones in a unified manner. In this work, we adopt the post-model prompting
[41] strategy to prompt $f_{v}$ in the embedding space. Specifically, we first
obtain the visual embedding ${\bm{v}}=f_{v}(\bm{x})\in\mathbb{R}^{D}$ for an
input $\bm{x}$, and aggregate information from text contexts
$\bm{\widetilde{s}}$ via a cross-attention-based module $G$ with $L$
Transformer decoder layers $\\{\texttt{Dec}_{l}\\}_{l=1}^{L}$:

$\displaystyle[W^{v}_{0},W^{\widetilde{s}}]$
$\displaystyle=\texttt{InProj}([\bm{v},~{}\bm{\widetilde{s}}]),$ (4)
$\displaystyle W^{v}_{l}$
$\displaystyle=\texttt{Dec}_{l}(W^{v}_{l-1},W^{\widetilde{s}})~{}~{}~{}l=1,2,\dots,L,$
$\displaystyle\bm{v}^{*}$ $\displaystyle=\texttt{OutProj}(W^{v}_{L}).$

Here InProj and OutProj are two projection operations, and each token is
projected independently. We then obtain the final embedding $\bm{v}^{\prime}$
via a residual connection: $\bm{v}^{\prime}=\bm{v}+\gamma_{v}\bm{v}^{*}$,
where $\gamma_{v}$ controls the weight. This produces visual embeddings
$\bm{v}^{\prime}$ guided by the domain-agnostic texts.

Vision-Guided Language Prompting. To accommodate various visual backbones,
CLIP uses a modified version of ResNet to implement $f_{v}$ by replacing the
last Global Average Pooling (GAP) layer with an attention pooling layer.
Specifically, it first transforms an image $\bm{x}\in\mathbb{R}^{H\times
W\times 3}$ into a feature map
$\bm{z}\in\mathbb{R}^{\hat{H}\times\hat{W}\times C}$, where
$H(\hat{H}),W(\hat{W})$ and $C$ are the height, width and the channel number.
The original ResNet uses
$\bm{\overline{z}}=\texttt{GAP}(\bm{z})\in\mathbb{R}^{C}$ as the final visual
embedding. In CLIP, $\bm{z}$ and $\bm{\overline{z}}$ are further handled by a
Multi-Head Self-Attention (MHSA) layer:

$\bm{v},\bm{\widetilde{v}}=\texttt{MHSA}([\bm{\overline{z}},\bm{z}]),$ (5)

where $\bm{v}\in\mathbb{R}^{1\times D}$ and
$\bm{\widetilde{v}}\in\mathbb{R}^{\hat{H}\hat{W}\times D}$ are the embeddings
at the class token and other spatial positions, respectively, which are
consistent with the ones in ViT. Generally, CLIP only uses $\bm{v}$ as the
visual embedding and discards $\bm{\widetilde{v}}$. However,
$\bm{\widetilde{v}}$ can also preserve useful semantical and spatial
information that can be used as contextual information [41]. In this work, we
leverage $\bm{\widetilde{v}}$ to adjust the textual embeddings
$\\{\bm{s}_{k}\\}_{k=1}^{K}$ via the same post-model prompting strategy and
$G$. Specifically, for the $k$-th class,

$\displaystyle[W^{s}_{0},W^{\widetilde{v}}]$
$\displaystyle=\texttt{InProj}([\bm{s}_{k},\bm{\widetilde{v}}]),$ (6)
$\displaystyle W^{s}_{l}$
$\displaystyle=\texttt{Dec}_{l}(W^{s}_{l-1},W^{\widetilde{v}})~{}~{}~{}l=1,2,\dots,L,$
$\displaystyle\bm{s}^{*}_{k}$ $\displaystyle=\texttt{OutProj}(W^{s}_{L}).$

The final semantic embedding $\bm{s}_{k}^{\prime}$ is then obtained by:
$\bm{s}_{k}^{\prime}=\bm{s}_{k}+\gamma_{s}\bm{s}_{k}^{*}$, where $\gamma_{s}$
is weight coefficient for the text modality. Note that each
$\bm{s}_{k}^{\prime}$ is updated based on a specific $\bm{x}$, making it
instance-dependent, enabling better image-text alignment. These two branches
of prompting are guided from each other to ensure mutual synergy. As a result,
we use

$\hat{P}(\bm{y}=k\mid\bm{x})=\frac{\exp\left(cos(\bm{v}^{\prime},\bm{s}_{k}^{\prime})/\tau\right)}{\sum_{k=1}^{K}\exp\left(cos(\bm{v}^{\prime},\bm{s}_{k}^{\prime})/\tau\right)}$
(7)

in our method for classification in the CLIP space.

###  3.3 Auxiliary Regularizations

While the mutual prompting framework aims to generate domain-invariant visual
embeddings and instance-conditioned textual embeddings, directly optimizing
with the source classification loss cannot guarantee achieving this goal.
Hence, we design two auxiliary regularizations.

Instance-Discrimination Contrastive Loss. During the mutual prompting, the
updated textual embeddings may still encode some domain-specific semantics
from the image context, making $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$ domain-
biased and less capable for the target domain. To address this problem, we
design an instance-discrimination contrastive loss to prevent textual prompts
from learning domain-related cues from visual contexts. Our motivation is that
images from the same domain typically share the same domain-information.
Therefore, maximizing the difference in $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$
among them help remove the domain-specific information.

Specifically, given a batch of source or target samples $\mathcal{B}$, denote
$\\{\bm{s}_{a,k}^{\prime}\\}_{k=1}^{K}$ and $\bm{v}_{a}^{\prime}$ the textual
and visual embeddings after mutual prompting for $\bm{x}_{a}\sim\mathcal{B}$,
each $\bm{x}_{a}$ forms a positive pair for $\bm{v}_{a}^{\prime}$ and
$\\{\bm{s}_{a,k}^{\prime}\\}_{k=1}^{K}$ and forms negative pairs for
$\\{\bm{s}_{a,k}^{\prime}\\}_{k=1}^{K}$ and $\bm{v}_{b}^{\prime}$ from another
$\bm{x}_{b}$ within the same batch:

$\displaystyle\operatorname{sim}(\bm{x}_{a},\bm{x}_{b})=\frac{1}{K}\sum_{k=1}^{K}cos(\bm{s}_{a,k}^{\prime},\bm{v}_{b}^{\prime})/\tau,$
(8)
$\displaystyle\mathcal{L}_{idc}=-\log\frac{\exp\left(\operatorname{sim}(\bm{x}_{a},\bm{x}_{a})\right)}{\operatorname{sim}(\bm{x}_{a},\bm{x}_{a})+\sum_{b\neq
a}\operatorname{sim}(\bm{x}_{a},\bm{x}_{b})}.$

This contrastive loss forces $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$ to not
encode domain-specific cues while retaining pure instance-specific
information. Imagine that if $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$ contained
domain-related information, they would be more similar for different images
from the same domain, thus the domain-specific information can be further
removed by optimizing $\mathcal{L}_{idc}$. Meanwhile, this contrastive loss
can be optimized in an unsupervised way, thus providing regularizations for
both domains.

Semantic-Consistency Regularization. In addition to removing domain-specific
information in $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$, we also want to ensure
the prompted visual embedding $\bm{v}^{\prime}$ is domain-invariant. Inspired
by FixMatch [46], we aim to exploit domain-agnostic visual characteristics
with a semantic-consistency regularization. Concretely, we leverage
RandAugment [5] to obtain a strongly-augmentated version of $\bm{x}$, denoted
as $\mathcal{A}(\bm{x})$, and enforce it to be correctly classified. For
labeled source samples $\\{x^{s}_{i},y^{s}_{i}\\}_{i=1}^{N_{s}}$, we can
directly optimize with ground-truth labels via:

$\mathcal{L}^{s}_{sc}=-\sum_{i=1}^{N_{s}}\log\hat{P}(\bm{y}=y^{s}_{i}\mid\mathcal{A}(x^{s}_{i})).$
(9)

Meanwhile, we also obtain pseudo-labels $\\{\hat{y^{t}_{i}}\\}_{i=1}^{N_{t}}$
for target data $\\{x_{i}^{t}\\}_{i=1}^{N_{t}}$ and only involves confident
ones for training:

$\mathcal{L}^{t}_{sc}=-\sum_{i=1}^{N_{t}}\mathbb{I}\\{\hat{P}(\bm{y}=\hat{y^{t}_{i}}\mid
S(x^{t}_{i}))\geq
T\\}\log\hat{P}(\bm{y}=\hat{y^{t}_{i}}\mid\mathcal{A}(x^{t}_{i})),$ (10)

where $\mathbb{I}{\cdot}$ is an indication function, $T$ is the threshold for
filtering confident target samples.

However, the unconfident target sampels are still not well-exploited. To make
the updated target domain embeddings fit the learned semantic structure, we
leverage the information maximization [28, 17] technique to regularize the
unlabeled target data via an entropy-based loss:

$\mathcal{L}_{im}=\frac{1}{N_{t}}\sum_{i=1}^{N_{t}}\sum_{k=1}^{K}p_{ti}^{c}\log
p_{ti}^{c}-\sum_{k=1}^{K}\hat{p}_{t}^{k}\log\hat{p}_{t}^{k},$ (11)

where $p_{ti}^{k}=\hat{P}(\bm{y}=k\mid x^{t}_{i})$ and
$\hat{p}_{t}^{k}=\frac{1}{N_{t}}\sum_{j=1}^{N_{t}}p_{tj}^{k}$. Optimizing
$\mathcal{L}_{im}$ makes predictions globally diverse and locally confident,
thus avoiding category collapse and ambiguity.

###  3.4 Overall Training Objective

We train our DAMP with the supervised loss and above regularizations in an
end-to-end manner. For the source domain, the supervised loss can be expressed
by:

$\mathcal{L}^{s}_{sup}=-\sum_{i=1}^{N_{s}}\log\hat{P}(\bm{y}=y^{s}_{i}\mid\mathcal{W}(x^{s}_{i})),$
(12)

where $\mathcal{W}(\cdot)$ is a weak augmentation operation. Besides, we also
supervise target samples with confident pseudo-labels:

$\mathcal{L}^{t}_{sup}=-\sum_{i=1}^{N_{t}}\mathbb{I}\\{\hat{P}(\bm{y}=\hat{y^{t}_{i}}\mid\mathcal{W}(x^{t}_{i}))\geq
T\\}\log\hat{P}(\bm{y}=\hat{y^{t}_{i}}\mid\mathcal{W}(x^{t}_{i})).$ (13)

The final training objective $\mathcal{L}_{all}$ is formulated by

$\mathcal{L}_{all}=\mathcal{L}_{sup}+\mathcal{L}_{sc}+\lambda_{c}\mathcal{L}_{idc}+\lambda_{i}\mathcal{L}_{im},$
(14)

where $\lambda_{c}$ and $\lambda_{i}$ are trade-off weights,
$\mathcal{L}_{sc}=\mathcal{L}^{s}_{sc}+\mathcal{L}^{t}_{sc}$ and
$\mathcal{L}_{sup}=\mathcal{L}^{s}_{sup}+\mathcal{L}^{t}_{sup}$. We give equal
weights to $\mathcal{L}_{sup}$ and $\mathcal{L}_{sc}$ for treating different
augmentations equally. An overview of our method can be found in Fig. 2.

##  4 Experiments

Table 1: Classification accuracies (%) on Office-Home dataset for UDA. The
best and second best results within each backbone are highlighted in bold and
underline, respectively. $\dagger$ CDTrans uses DeiT-B [50] as the backbone.
Methods within each backbone are grouped into three categories, i.e., fine-
tuning, zero-shot and prompt learning (from top to bottom), respectively.

Method | $f_{v}$ | Ar$\rightarrow$Cl | Ar$\rightarrow$Pr | Ar$\rightarrow$Rw | Cl$\rightarrow$Ar | Cl$\rightarrow$Pr | Cl$\rightarrow$Rw | Pr$\rightarrow$Ar | Pr$\rightarrow$Cl | Pr$\rightarrow$Rw | Rw$\rightarrow$Ar | Rw$\rightarrow$Cl | Rw$\rightarrow$Pr | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ResNet-50 [14] |  | 34.9 | 50.0 | 58.0 | 37.4 | 41.9 | 46.2 | 38.5 | 31.2 | 60.4 | 53.9 | 41.2 | 59.9 | 46.1
SRDC [49] |  | 52.3 | 76.3 | 81.0 | 69.5 | 76.2 | 78.0 | 68.7 | 53.8 | 81.7 | 76.3 | 57.1 | 85.0 | 71.3
ToAlign [59] |  | 57.9 | 76.9 | 80.8 | 66.7 | 75.6 | 77.0 | 67.8 | 57.0 | 82.5 | 75.1 | 60.0 | 84.9 | 72.0
\+ FixMatch + EIDCo [68] |  | 63.8 | 80.8 | 82.6 | 71.5 | 80.1 | 80.9 | 72.1 | 61.3 | 84.5 | 78.6 | 65.8 | 87.1 | 75.8
PADCLIP [24] |  | 57.5 | 84.0 | 83.8 | 77.8 | 85.5 | 84.7 | 76.3 | 59.2 | 85.4 | 78.1 | 60.2 | 86.7 | 76.6
CLIP [40] |  | 51.6 | 81.9 | 82.6 | 71.9 | 81.9 | 82.6 | 71.9 | 51.6 | 82.6 | 71.9 | 51.6 | 81.9 | 72.0
DAPrompt [12] |  | 54.1 | 84.3 | 84.8 | 74.4 | 83.7 | 85.0 | 74.5 | 54.6 | 84.8 | 75.2 | 54.7 | 83.8 | 74.5
AD-CLIP [45] |  |  ResNet-50 55.4 | 85.2 | 85.6 | 76.1 | 85.8 | 86.2 | 76.7 | 56.1 | 85.4 | 76.8 | 56.1 | 85.5 | 75.9
DAMP (Ours) |  | 59.7 | 88.5 | 86.8 | 76.6 | 88.9 | 87.0 | 76.3 | 59.6 | 87.1 | 77.0 | 61.0 | 89.9 | 78.2
ViT-B [6] |  | 54.7 | 83.0 | 87.2 | 77.3 | 83.4 | 85.5 | 74.4 | 50.9 | 87.2 | 79.6 | 53.8 | 88.8 | 75.5
CDTrans $\dagger$ [60] |  | 68.8 | 85.0 | 86.9 | 81.5 | 87.1 | 87.3 | 79.6 | 63.3 | 88.2 | 82.0 | 66.0 | 90.6 | 80.5
TVT-B [61] |  | 74.9 | 86.8 | 89.5 | 82.8 | 88.0 | 88.3 | 79.8 | 71.9 | 90.1 | 85.5 | 74.6 | 90.6 | 83.6
SSRT-B [48] |  | 75.2 | 89.0 | 91.1 | 85.1 | 88.3 | 90.0 | 85.0 | 74.2 | 91.3 | 85.7 | 78.6 | 91.8 | 85.4
\+ FixMatch + EIDCo [68] |  | 76.9 | 90.3 | 91.3 | 86.5 | 90.5 | 90.0 | 86.3 | 75.5 | 91.7 | 88.1 | 77.1 | 92.3 | 86.4
PADCLIP [24] |  | 76.4 | 90.6 | 90.8 | 86.7 | 92.3 | 92.0 | 86.0 | 74.5 | 91.5 | 86.9 | 79.1 | 93.1 | 86.7
CLIP [40] |  | 67.8 | 89.0 | 89.8 | 82.9 | 89.0 | 89.8 | 82.9 | 67.8 | 89.8 | 82.9 | 67.8 | 89.0 | 82.4
DAPrompt [12] |  |  ViT-B/16 70.7 | 91.0 | 90.9 | 85.2 | 91.0 | 91.0 | 85.1 | 70.7 | 90.9 | 85.3 | 70.4 | 91.4 | 84.4
AD-CLIP [45] |  | 70.9 | 92.5 | 92.1 | 85.4 | 92.4 | 92.5 | 86.7 | 74.3 | 93.0 | 86.9 | 72.6 | 93.8 | 86.1
DAMP (Ours) |  | 75.7 | 94.2 | 92.0 | 86.3 | 94.2 | 91.9 | 86.2 | 76.3 | 92.4 | 86.1 | 75.6 | 94.0 | 87.1

Table 2: Per-class accuracies (%) on VisDA-17 dataset for UDA. Marks and symbols share the same meaning in Table 1.  Method | $f_{v}$ | plane | bicycle | bus | car | horse | knife | mcycl | person | plant | sktbrd | train | truck | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
RN-101 [14] |  | 55.1 | 53.3 | 61.9 | 59.1 | 80.6 | 17.9 | 79.7 | 31.2 | 81.0 | 26.5 | 73.5 | 8.5 | 52.4
CGDM [7] |  | 93.4 | 82.7 | 73.2 | 68.4 | 92.9 | 94.5 | 88.7 | 82.1 | 93.4 | 82.5 | 86.8 | 49.2 | 82.3
CAN [21] |  | 97.0 | 87.2 | 82.5 | 74.3 | 97.8 | 96.2 | 90.8 | 80.7 | 96.6 | 96.3 | 87.5 | 59.9 | 87.2
PADCLIP [24] |  | 96.7 | 88.8 | 87.0 | 82.8 | 97.1 | 93.0 | 91.3 | 83.0 | 95.5 | 91.8 | 91.5 | 63.0 | 88.5
CLIP [40] |  | 98.2 | 83.9 | 90.5 | 73.5 | 97.2 | 84.0 | 95.3 | 65.7 | 79.4 | 89.9 | 91.8 | 63.3 | 84.4
DAPrompt [12] |  | 97.8 | 83.1 | 88.8 | 77.9 | 97.4 | 91.5 | 94.2 | 79.7 | 88.6 | 89.3 | 92.5 | 62.0 | 86.9
AD-CLIP [45] |  |  ResNet-101 98.1 | 83.6 | 91.2 | 76.6 | 98.1 | 93.4 | 96.0 | 81.4 | 86.4 | 91.5 | 92.1 | 64.2 | 87.7
DAMP (Ours) |  | 97.3 | 91.6 | 89.1 | 76.4 | 97.5 | 94.0 | 92.3 | 84.5 | 91.2 | 88.1 | 91.2 | 67.0 | 88.4
ViT-B [6] |  | 99.1 | 60.7 | 70.6 | 82.7 | 96.5 | 73.1 | 97.1 | 19.7 | 64.5 | 94.7 | 97.2 | 15.4 | 72.6
CDTrans $\dagger$ [60] |  | 97.1 | 90.5 | 82.4 | 77.5 | 96.6 | 96.1 | 93.6 | 88.6 | 97.9 | 86.9 | 90.3 | 62.8 | 88.4
TVT-B [61] |  | 92.9 | 85.6 | 77.5 | 60.5 | 93.6 | 98.2 | 89.4 | 76.4 | 93.6 | 92.0 | 91.7 | 55.7 | 83.9
SSRT-B [48] |  | 98.9 | 87.6 | 89.1 | 84.8 | 98.3 | 98.7 | 96.3 | 81.1 | 94.9 | 97.9 | 94.5 | 43.1 | 88.8
PADCLIP [24] |  | 98.1 | 93.8 | 87.1 | 85.5 | 98.0 | 96.0 | 94.4 | 86.0 | 94.9 | 93.3 | 93.5 | 70.2 | 90.9
CLIP [40] |  | 99.1 | 91.7 | 93.8 | 76.7 | 98.4 | 91.7 | 95.3 | 82.7 | 86.5 | 96.0 | 94.6 | 60.5 | 88.9
DAPrompt [12] |  | 99.2 | 92.5 | 93.3 | 75.4 | 98.6 | 92.8 | 95.2 | 82.5 | 89.3 | 96.5 | 95.1 | 63.5 | 89.5
AD-CLIP [45] |  |  ViT-B/16 99.6 | 92.8 | 94.0 | 78.6 | 98.8 | 95.4 | 96.8 | 83.9 | 91.5 | 95.8 | 95.5 | 65.7 | 90.7
DAMP (Ours) |  | 98.7 | 92.8 | 91.7 | 80.1 | 98.9 | 96.9 | 94.9 | 83.2 | 93.9 | 94.9 | 94.8 | 70.2 | 90.9
Table 3: Classification accuracies (%) on Mini-DomainNet dataset for UDA.
Marks and symbols share the same meaning in Table 1.

Method | $f_{v}$ | Cl$\rightarrow$Pn | Cl$\rightarrow$Rl | Cl$\rightarrow$Sk | Pn$\rightarrow$Cl | Pn$\rightarrow$Rl | Pn$\rightarrow$Sk | Rl$\rightarrow$Cl | Rl$\rightarrow$Pn | Rl$\rightarrow$Sk | Sk$\rightarrow$Cl | Sk$\rightarrow$Pn | Sk$\rightarrow$Rl | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ResNet-50 [14] |  | 52.1 | 63.0 | 49.4 | 55.9 | 73.0 | 51.1 | 56.8 | 61.0 | 50.0 | 54.0 | 48.9 | 60.3 | 56.3
CLIP [40] |  | 67.9 | 84.8 | 62.9 | 69.1 | 84.8 | 62.9 | 69.2 | 67.9 | 62.9 | 69.1 | 67.9 | 84.8 | $71.2$
DAPrompt [12] |  | 72.4 | 87.6 | 65.9 | 72.7 | 87.6 | 65.6 | 73.2 | 72.4 | 66.2 | 73.8 | 72.9 | 87.8 | $74.8$
AD-CLIP [45] |  |  ResNet-50 71.7 | 88.1 | 66.0 | 73.2 | 86.9 | 65.2 | 73.6 | 73.0 | 68.4 | 72.3 | 74.2 | 89.3 | 75.2
DAMP (Ours) |  | 76.7 | 88.5 | 71.7 | 74.2 | 88.7 | 70.8 | 74.4 | 75.7 | 70.5 | 74.9 | 76.1 | 88.2 | 77.5
ViT-B [6] |  | 63.3 | 79.0 | 56.4 | 62.6 | 83.3 | 55.4 | 62.0 | 70.3 | 53.5 | 63.0 | 63.6 | 75.8 | 65.7
CLIP [40] |  | 80.3 | 90.5 | 77.8 | 82.7 | 90.5 | 77.8 | 82.7 | 80.3 | 77.8 | 82.7 | 80.3 | 90.5 | $82.8$
DAPrompt [12] |  | 83.3 | 92.4 | 81.1 | 86.4 | 92.1 | 81.0 | 86.7 | 83.3 | 80.8 | 86.8 | 83.5 | 91.9 | $85.8$
AD-CLIP [45] |  |  ViT-B/16 84.3 | 93.7 | 82.4 | 87.5 | 93.5 | 82.4 | 87.3 | 84.5 | 81.6 | 87.9 | 84.8 | 93.0 | 86.9
DAMP (Ours) |  | 86.4 | 93.3 | 83.5 | 87.2 | 93.4 | 84.1 | 87.2 | 86.5 | 82.5 | 87.3 | 86.6 | 93.4 | 87.6

In this section, we mainly verify the effectiveness of our method on UDA
tasks. More evaluation on Multi-Source UDA [39] and Domain Generalization (DG)
[25] tasks and more analytical experiments can be found in Appendix.

###  4.1 Experimental Setup

Datasets. We evaluate our method on three widely used UDA datasets. Office-
Home [56] consists of images from 4 different domains: Art (Ar), Clipart (Cl),
Product (Pr) and Real-World (Rw). There are 65 object categories and around
15,500 images in total. VisDA-17 [38] contains synthetic images to real images
across 12 categories. The synthetic source domain has 152,397 images generated
from 3D models. The real target domain has 55,388 real images. Mini-DomainNet
is a subset of the most challenging dataset DomainNet [39]. We use a subset
with 4 domains, i.e., Clipart (Cl), Painting (Pn), Real (Rl) and Sketch (Sk),
across 126 categories following previous works [70, 45].

Training Configuration. We evaluate DAMP with both ResNet-50 [14] and ViT-B/16
[6] as the visual encoder $f_{v}$. The text encoder $f_{s}$ is a pretrained
CLIP text encoder with depth $J=12$. During training, we freeze these encoders
and tune the input textual prompts $\bm{p}_{1:N}$ and the prompting module
$G$. The learnable token length $N$ is set to 32 and we use $L=2$ Transformer
decoder layers in the mutual prompting module $G$. For training, we use the
Adam optimizer [23] with an initial learning rate of 3e-3 for all datasets,
and adjust it with a cosine annealing scheduler [34]. Our model is trained for
30 epochs in total (for Mini-DomainNet, we train 500 iterations per epoch).
The batch size is set to 32 for each domain. We set the confidence threshold
$T=0.6$ for Office-Home and $0.5$ for VisDA-17 and Mini-DomainNet. For
hyperparameters, we use $\lambda_{c}=\lambda_{i}=1.0$. Due to the modality gap
in CLIP [20, 29], the visual and textual embeddings need different updating
magnitudes, making manually searching $\gamma_{v}$ and $\gamma_{s}$
challenging. Therefore, we make them learnable parameters. More implementation
details about network architectures and pseudo-labels can be found in
Appendix.

###  4.2 Comparasion with State-of-the-Arts

We report the results on Office-Home in Table 1. Our DAMP outperforms all
competitors on most tasks, especially the challenging ones like
Ar$\rightarrow$Cl and Cl$\rightarrow$Ar. Besides, DAMP brings substantial
gains over strong baselines, improving the average accuracy over PADCLIP by
1.6% with ResNet-50 and 0.4% with ViT-B. Compared to prompt-based methods like
DAPrompt and AD-CLIP, DAMP also shows superiority by mutually aligning both
modalities. For example, it surpasses DAPrompt by 3.7% with ResNet-50 and 3.1%
with ViT-B. The improvements are more significant with ResNet-50. This
indicates DAMP can better exploit ViT’s intrinsic transferability while
effectively prompting ResNet for inspiring improvements.

For VisDA-17 (Table 2), DAMP demonstrates competitive performance compared to
other methods. It achieves 88.4% average accuracy with ResNet-101. Although it
is slightly worse than the best competitor PADCLIP (88.5%), we show in Sec.
4.3 that our method involves much less learnable parameters. When using ViT-B,
DAMP obtains the highest average accuracy (90.9%), comparable to PADCLIP. The
results validate the consistently strong performance of DAMP across different
vision backbones.

As shown in Table 3, DAMP sets new state-of-the-art on Mini-DomainNet. It
brings significant improvements over baseline methods like standard CLIP and
other prompt learning methods like DAPrompt and AD-CLIP. For example, with
ResNet-50, DAMP improves over CLIP by 6.3%, DAPrompt by 2.7%, and AD-CLIP by
2.3%. Similar gains are observed when using ViT backbone. In addition, it
achieves especially large gains on challenging tasks like Cl$\rightarrow$Pn,
Cl$\rightarrow$Sk and Sk$\rightarrow$Pn. The consistent and substantial
improvements of DAMP over strong baselines highlight the benefits of mutually
aligning textual and visual prompts in a domain-agnostic and instance-
conditioned manner.

(a)

(b)

Figure 3: Visualization of (a) visual embeddings and (b) textual embeddings
using t-SNE [53] on task Ar $\rightarrow$ Pr (Office-Home). Light and dark
colors represent embeddings before and after our mutual prompting,
respectively. Red and blue points are source and target samples, respectively.
Orange stars denote the class-level domain-agnostic textual embeddings
$\\{\bm{s}_{k}\\}_{k=1}^{K}$.

###  4.3 Analytical Experiments

Visualization of Embeddings. Fig. 3 visualizes the visual and textual
embeddings learned by DAMP. In Fig. 3(a), we see that before visual prompting,
the source and target domains form distinct distributions, indicating a large
domain gap. After prompting, the visual features become better aligned across
domains and form more clear clusters, suggesting a reduced domain gap and a
discriminative semantic structure. Fig. 3(b) shows that instance-conditioned
textual prompting increases within-class semantic diversity. This enables
better pairing of text and images in both domains. Through mutual alignment of
visual and textual embeddings, the two prompts make representations more
domain-invariant to facilitate cross-domain knowledge transfer.

Model Capacity Analysis. Fig. 4 compares different UDA methods regarding the
number of tunable parameters versus accuracy. Traditional methods like DANN
and CDAN fine-tune the full model, requiring extensive parameters yet
achieving relatively low accuracy. PADCLIP [24] fine-tunes the CLIP visual
backbone, introducing many parameters. In contrast, our DAMP only tunes the
textual prompts $\bm{p}_{1:N}$ and the prompting module $G$, which just adds a
few parameters over DAPrompt yet achieves comparable accuracy (88.4%) to
PADCLIP.

Figure 4: Comparasion between different UDA methods regarding tunable parameters and accuracies on VisDA-17 (ResNet-101). DAMP only use 11.9% parameters compared with PADCLIP. Table 4: Comparasion between different prompting strategies on Office-Home (ViT-B/16). MP denotes mutual prompting. Prompting Strategy | Office-Home
---|---
w/o MP (CoOp) | 85.0
Independent prompting | 85.4
MP w/ simple synergy | 85.9
MP w/ cross-attention | 86.1

(a)

(b)

(c)

(d)

Figure 5: Hyperparameter analysis. (a) Performance under different learnable
token length $N$ on VisDA-17 dataset. (b) Values of learnable hyperparameters
$\gamma_{v}$ and $\gamma_{t}$ during training on task Cl $\rightarrow$ Sk
(Mini-DomainNet). (c) The influence of different choices of $T$ on Office-Home
dataset. (d) parameter sensitivities of $\lambda_{c}$ and $\lambda_{i}$ on
task Cl $\rightarrow$ Ar (Office-Home).

Analysis of the Mutual Prompting Strategy. To further explore the
effectiveness of the cross-attention-based mutual prompting framework, we
conduct experiments comparing it to other prompting strategies in Table 4. For
fair comparison, all regularizations are removed in this experiment and only
$\mathcal{L}_{sup}$ is optimized. Specifically, we evaluate an independent
prompting strategy by replacing the cross-attention with self-attention in
$G$, which results in separate prompting modules for each modality without any
interaction. It shows that without mutual synergy, the improvement is limited
over the baseline CoOp. We also examine a simple synergy strategy inspired by
MaPLe [22], where we use a linear projection layer to obtain the prompted
textual embedding $\bm{s}^{*}_{k}$ from the image context embedding
$\bm{\widetilde{v}}$ (and vice versa for the visual embedding). However, this
uni-directional projection does not fully capture the complex interaction
between the modalities in the prompting process. In contrast, the cross-
attention module in our framework allows bi-directional interaction, which
enables more effective fusion of information from both modalities, resulting
in better domain-agnostic and instance-specific prompts.

Table 5: Ablation study on Office-Home (ResNet-50) and VisDA-17 (ResNet-101).
ITP and VP refer to instance-level textual prompting and visual prompting,
respectively.

Method | Prompting | Loss | Mean Accuracy
---|---|---|---
ITP | VP | $\mathcal{L}_{sc}$ | $\mathcal{L}_{idc}$ | $\mathcal{L}_{im}$ | Office-Home | VisDA-17
Baseline (CoOp) | ✗ | ✗ | ✗ | ✗ | ✗ | 75.3 | 86.1
Uni-modal Prompting | ✗ | ✓ | ✗ | ✗ | ✗ | 75.7 | 86.5
✓ | ✗ | ✗ | ✗ | ✗ | 75.8 | 87.1
Mutual Prompting | ✓ | ✓ | ✗ | ✗ | ✗ | 76.1 | 87.2
✓ | ✓ | ✓ | ✗ | ✗ | 76.9 | 87.8
✓ | ✓ | ✓ | ✓ | ✗ | 77.3 | 88.0
✓ | ✓ | ✓ | ✓ | ✓ | 78.2 | 88.4

Hyperparameter Analysis. There are four categories of hyperparameters in our
method, i.e., number of learnable tokens $N$, learnable weights $\gamma_{v}$
and $\gamma_{s}$, confidence threshold $T$, and loss weights $\lambda_{c}$ and
$\lambda_{i}$. As shown in Fig. 5(a), the performance on VisDA-17 improves as
$N$ increases from 4 to 32, and then remains relatively stable with larger
$N$. This indicates that a moderate length is enough for encoding rich
semantic information. However, further increasing $N$ does not bring
substantial gains.

Fig. 5(b) plots the values of the learnable weight coefficients $\gamma_{v}$
and $\gamma_{s}$ during training. We can see that $\gamma_{v}$ converges to a
relatively small value around 0.1, while $\gamma_{s}$ converges to a larger
value around 0.5. This aligns with the intuition that a smaller perturbation
is needed on the visual embeddings compared to the textual embeddings, due to
the inherent modality gap between vision and language in CLIP. The learnable
nature of $\gamma_{v}$ and $\gamma_{s}$ provides the flexibility to adapt the
updating magnitudes for both modalities.

Fig. 5(c) shows that the performance is relatively stable across different
choices of $T$ from 0.5 to 1.0. Setting $T$ too small (0.1) deteriorates the
performance. This indicates that the confidence threshold is not too
sensitive, while using very unconfident pseudo-labels can hurt the
performance. A moderate threshold between 0.5 and 1.0 works reliably.

Fig. 5(d) studies $\lambda_{c}$ and $\lambda_{i}$ on task Cl$\rightarrow$Ar.
The results show DAMP is relatively robust to different choices of
$\lambda_{c}$, and a little sensitive to the variation of $\lambda_{c}$.
Overally, trivially setting both of them to 1.0 offers a good trade-off.

Ablation Study. To validate the efficacy of each component in our method, we
conduct an ablation study on Office-Home and VisDA-17 datasets. As shown in
Table 5, we first evaluate a baseline model that directly optimizes the
supervised loss $\mathcal{L}_{sup}$ on source and confident target samples to
fine-tune $\bm{p}_{1:N}$ without further prompting (analogous to CoOp), which
gives the lowest performance. Adding either instance-level textual prompting
(ITP) or visual prompting (VP) brings gains over the baseline, showing the
benefits of adapting either modality with prompting. Further prompting both
modalities together with the mutual prompting framework leads to additional
performance boosts, which mainly benefits from the flexibility to adapt both
language and vision branches to the target domain. Finally, regularizations
$\mathcal{L}_{sc}$, $\mathcal{L}_{idc}$ and $\mathcal{L}_{im}$ all contribute
to the superior performance in a collaborative manner. The step-wise
improvements support the rationality of our design.

##  5 Conclusion

We propose DAMP, a novel framework to address UDA using VLMs. DAMP mutually
aligns the visual and textual modalities via prompting to elicit domain-
agnostic embeddings. The prompts are optimized together through cross-
attention and regularized with elaborate losses. Extensive experiments
validate that DAMP brings substantial and consistent improvements over strong
baselines on three benchmarks. DAMP provides an effective approach to harness
both source and pre-trained VLMs knowledge for UDA.

## References

  * Bahng et al. [2022] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Exploring visual prompts for adapting large-scale models.  _arXiv preprint arXiv:2203.17274_ , 2022.
  * Beery et al. [2018] Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In _ECCV_ , pages 456–473, 2018.
  * Chen et al. [2022] Haoran Chen, Zuxuan Wu, Xintong Han, and Yu-Gang Jiang.  Multi-prompt alignment for multi-source unsupervised domain adaptation.  _arXiv preprint arXiv:2209.15210_ , 2022.
  * Chen et al. [2019] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang.  Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation.  In _ICML_ , pages 1081–1090. PMLR, 2019.
  * Cubuk et al. [2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _CVPR workshops_ , pages 702–703, 2020.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Du et al. [2021] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu.  Cross-domain gradient discrepancy minimization for unsupervised domain adaptation.  In _CVPR_ , pages 3937–3946, 2021.
  * Fang et al. [2013] Chen Fang, Ye Xu, and Daniel N Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In _ICCV_ , pages 1657–1664, 2013.
  * Fu et al. [2021] Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu.  Partial feature selection and alignment for multi-source domain adaptation.  In _CVPR_ , pages 16654–16663, 2021.
  * Gan et al. [2023] Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo.  Decorate the newcomers: Visual domain prompt for continual test time adaptation.  In _AAAI_ , pages 7595–7603, 2023.
  * Ganin and Lempitsky [2015] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In _ICML_ , pages 1180–1189. PMLR, 2015.
  * Ge et al. [2023] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang.  Domain adaptation via prompt learning.  _TNNLS_ , pages 1–11, 2023.
  * Gulrajani and Lopez-Paz [2020] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In _ICLR_ , 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * He et al. [2023] Wenbin He, Suphanut Jamonnak, Liang Gou, and Liu Ren.  Clip-s4: Language-guided self-supervised semantic segmentation.  In _CVPR_ , pages 11207–11216, 2023.
  * Hu and Singh [2021] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In _ICCV_ , pages 1439–1449, 2021.
  * Hu et al. [2017] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama.  Learning discrete representations via information maximizing self-augmented training.  In _ICML_ , pages 1558–1567. PMLR, 2017.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , pages 4904–4916. PMLR, 2021.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _ECCV_ , pages 709–727. Springer, 2022.
  * Jiang et al. [2023] Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi.  Understanding and constructing latent modality structures in multi-modal representation learning.  In _CVPR_ , pages 7661–7671, 2023.
  * Kang et al. [2019] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In _CVPR_ , pages 4893–4902, 2019.
  * Khattak et al. [2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _CVPR_ , pages 19113–19122, 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Lai et al. [2023] Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah.  Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation.  In _ICCV_ , pages 16155–16165, 2023.
  * Li et al. [2017] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In _ICCV_ , pages 5542–5550, 2017.
  * Li et al. [2020] Jingjing Li, Erpeng Chen, Zhengming Ding, Lei Zhu, Ke Lu, and Heng Tao Shen.  Maximum density divergence for domain adaptation.  _TPAMI_ , 43(11):3918–3930, 2020.
  * Li et al. [2021] Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu.  T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation.  In _ICCV_ , pages 9991–10000, 2021.
  * Liang et al. [2020] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In _ICML_ , pages 6028–6039. PMLR, 2020.
  * Liang et al. [2022] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  In _NeurIPS_ , pages 17612–17625, 2022.
  * Liu et al. [2021] Hong Liu, Jianmin Wang, and Mingsheng Long.  Cycle self-training for domain adaptation.  In _NeurIPS_ , pages 22968–22981, 2021.
  * Long et al. [2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In _ICML_ , pages 97–105. PMLR, 2015.
  * Long et al. [2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Deep transfer learning with joint adaptation networks.  In _ICML_ , pages 2208–2217. PMLR, 2017.
  * Long et al. [2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In _NeurIPS_ , 2018.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Mei et al. [2020] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  In _ECCV_ , pages 415–430. Springer, 2020.
  * Moraffah et al. [2019] Raha Moraffah, Kai Shu, Adrienne Raglin, and Huan Liu.  Deep causal representation learning for unsupervised domain adaptation.  _arXiv preprint arXiv:1910.12417_ , 2019.
  * Pan and Yang [2009] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  _TKDE_ , 22(10):1345–1359, 2009.
  * Peng et al. [2017] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  _arXiv preprint arXiv:1710.06924_ , 2017.
  * Peng et al. [2019] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _ICCV_ , pages 1406–1415, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763. PMLR, 2021.
  * Rao et al. [2022] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In _CVPR_ , pages 18082–18091, 2022.
  * Ren et al. [2022] Chuan-Xian Ren, Yong-Hui Liu, Xi-Wen Zhang, and Ke-Kun Huang.  Multi-source unsupervised domain adaptation via pseudo target domain.  _TIP_ , 31:2122–2135, 2022.
  * Saito et al. [2018] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In _CVPR_ , pages 3723–3732, 2018.
  * Sankaranarayanan et al. [2018] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa.  Generate to adapt: Aligning domains using generative adversarial networks.  In _CVPR_ , pages 8503–8512, 2018.
  * Singha et al. [2023] Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee.  Ad-clip: Adapting domains in prompt space using clip.  In _ICCV_ , pages 4355–4364, 2023.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In _NeurIPS_ , pages 596–608, 2020.
  * Sun and Saenko [2016] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _ECCV_ , pages 443–450. Springer, 2016.
  * Sun et al. [2022] Tao Sun, Cheng Lu, Tianshuo Zhang, and Haibin Ling.  Safe self-refinement for transformer-based domain adaptation.  In _CVPR_ , pages 7191–7200, 2022.
  * Tang et al. [2020] Hui Tang, Ke Chen, and Kui Jia.  Unsupervised domain adaptation via structurally regularized deep clustering.  In _CVPR_ , pages 8725–8735, 2020.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , pages 10347–10357. PMLR, 2021.
  * Tsai et al. [2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In _ACL_ , page 6558, 2019.
  * Tzeng et al. [2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  _arXiv preprint arXiv:1412.3474_ , 2014.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _JMLR_ , 9(11), 2008.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Venkat et al. [2020] Naveen Venkat, Jogendra Nath Kundu, Durgesh Singh, Ambareesh Revanur, et al.  Your classifier can secretly suffice multi-source domain adaptation.  pages 4647–4659, 2020.
  * Venkateswara et al. [2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _CVPR_ , pages 5018–5027, 2017.
  * Wang et al. [2020] Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang.  Learning to combine: Knowledge aggregation for multi-source domain adaptation.  In _ECCV_ , pages 727–744. Springer, 2020.
  * Wang et al. [2022] Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu.  Debiased learning from naturally imbalanced pseudo-labels.  In _CVPR_ , pages 14647–14657, 2022.
  * Wei et al. [2021] Guoqiang Wei, Cuiling Lan, Wenjun Zeng, Zhizheng Zhang, and Zhibo Chen.  Toalign: task-oriented alignment for unsupervised domain adaptation.  In _NeurIPS_ , pages 13834–13846, 2021.
  * Xu et al. [2021] Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin.  Cdtrans: Cross-domain transformer for unsupervised domain adaptation.  _arXiv preprint arXiv:2109.06165_ , 2021.
  * Yang et al. [2023] Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang.  Tvt: Transferable vision transformer for unsupervised domain adaptation.  In _WACV_ , pages 520–530, 2023.
  * Yao et al. [2021] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Cpt: Colorful prompt tuning for pre-trained vision-language models.  _arXiv preprint arXiv:2109.11797_ , 2021.
  * Yu et al. [2023] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Bo Ren, and Xiang Bai.  Turning a clip model into a scene text detector.  In _CVPR_ , pages 6978–6988, 2023.
  * Yue et al. [2023] Zhongqi Yue, Hanwang Zhang, and Qianru Sun.  Make the u in uda matter: Invariant consistency learning for unsupervised domain adaptation.  _arXiv preprint arXiv:2309.12742_ , 2023.
  * Zha et al. [2023] Yaohua Zha, Jinpeng Wang, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia.  Instance-aware dynamic prompt tuning for pre-trained point cloud models.  _arXiv preprint arXiv:2304.07221_ , 2023.
  * Zhang et al. [2023a] Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, and Yusuke Iwasawa.  Domain prompt learning for efficiently adapting clip to unseen domains.  _TJSAI_ , 38(6):B–MC2_1, 2023a.
  * Zhang et al. [2019] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan.  Domain-symmetric networks for adversarial domain adaptation.  In _CVPR_ , pages 5031–5040, 2019.
  * Zhang et al. [2023b] Yixin Zhang, Zilei Wang, Junjie Li, Jiafan Zhuang, and Zihan Lin.  Towards effective instance discrimination contrastive loss for unsupervised domain adaptation.  In _ICCV_ , pages 11388–11399, 2023b.
  * Zhao et al. [2020] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer.  Multi-source distilling domain adaptation.  In _AAAI_ , pages 12975–12983, 2020.
  * Zhou et al. [2021] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain adaptive ensemble learning.  _TIP_ , 30:8008–8018, 2021.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _CVPR_ , pages 16816–16825, 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _IJCV_ , 130(9):2337–2348, 2022b.
  * Zhu et al. [2019] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang.  Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources.  In _AAAI_ , pages 5989–5996, 2019.

Supplementary Material

## Appendix Contents

  * A.

More Implementation Details

  * B.

Algorithm

  * C.

Experiments on Multi-Source UDA

  * D.

Experiments on Doamin Generalization

  * E.

Additional Analytical Experiments for UDA

##  Appendix A More Implementation Details

We implement our method with the Pytorch framework. Our code is built on the
Dassl.pytorch 222https://github.com/KaiyangZhou/Dassl.pytorch platForm, which
is a principled implementation and evaluation platForm For DA and DG tasks. We
train DAMP with a single NVIDIA GeForce RTX 3090 GPU. Details about network
architecture, data augmentation and pseudo-labels are described as follows.

Network Architecture. The text encoder $f_{s}$ and the mutual prompting module
$G$ we used are mainly comprised of a TransFormer encoder and a TransFormer
decoder, respectively. Specifically, the standard dot-product attention is
leveraged. Given a set of queries $Q\in\mathbb{R}^{N_{q}\times d_{k}}$, keys
$K\in\mathbb{R}^{N_{k}\times d_{k}}$ and values $V\in\mathbb{R}^{N_{k}\times
d_{v}}$, the attentional outputs For all queries can be calculated by:

$\texttt{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V\in\mathbb{R}^{N_{q}\times
d_{v}}.$ (15)

For Self-Attention (SA), $Q,K,V$ obtained from the same input sequence
$I\in\mathbb{R}^{N\times D}$, through three projection matrixes
$M_{q}\in\mathbb{R}^{D\times d_{k}},M_{k}\in\mathbb{R}^{D\times
d_{k}},M_{v}\in\mathbb{R}^{D\times d_{v}}$,

$\texttt{SA}(I)=\texttt{Attention}(IM_{q},IM_{k},IM_{v})\in\mathbb{R}^{N\times
d_{v}}.$ (16)

Multi-Head Self-Attention (MHSA) extends SA by using multiple attention heads,

$\texttt{MHSA}(I)=\operatorname{Concat}\left(\operatorname{head}_{1},\ldots,\operatorname{head}_{\mathrm{h}}\right)M_{O},\\\
$ (17)

where
$\operatorname{head}_{i}=\texttt{Attention}\left(IM^{i}_{q},IM^{i}_{k},IM^{i}_{v}\right)$,
$h$ is the head number, and $M_{O}\in\mathbb{R}^{hd_{v}\times D}$ maps the
intermediate embeddings to match the input dimension.

For $f_{s}$, each encoder layer $\texttt{Enc}_{j}$ comprised of a MHSA block
and a feed-Forward block $\texttt{FD}(\cdot)$, with $h=8$,
$d_{k}=d_{v}=D=512$. A residual connection and LayerNorm operation
$\texttt{LN}(\cdot)$ is employed after each of them, i.e.,

$\displaystyle\texttt{Enc}_{j}(I)=\texttt{LN}(\texttt{FD}(I^{\prime})+I^{\prime}),$
(18) $\displaystyle I^{\prime}=\texttt{LN}(\texttt{MHSA}(I)+I)$

For $G$, a Masked MHSA (M-MHSA) block and Multi-Head Cross-Attention (MHCA)
block is used in each $\texttt{Dec}_{l}$. Specifically, the M-MHSA alters MHSA
by imposing a mask on the attention scores, i.e.,

$\displaystyle\texttt{M-MHSA}(I)=\operatorname{Concat}\left(\operatorname{head}_{1},\ldots,\operatorname{head}_{\mathrm{h}}\right)M_{O},$
(19)
$\displaystyle\operatorname{head}_{i}=\texttt{MaskedAttention}\left(IM^{i}_{q},IM^{i}_{k},IM^{i}_{v}\right),$
$\displaystyle\texttt{MaskedAttention}(Q,K,V)=\operatorname{softmax}\left(\frac{S\odot
QK^{T}}{\sqrt{d_{k}}}\right),$

where $S\in\mathbb{R}^{N_{q}\times N_{k}}$ is the mask. The MHCA block is a
variation of MHSA by using different source of $Q$ and $K,V$, i.e.,

$\displaystyle\texttt{MHCA}(I_{1},I_{2})=\operatorname{Concat}\left(\operatorname{head}_{1},\ldots,\operatorname{head}_{\mathrm{h}}\right)M_{O},$
(20)
$\displaystyle\operatorname{head}_{i}=\texttt{Attention}\left(I_{1}M^{i}_{q},I_{2}M^{i}_{k},I_{2}M^{i}_{v}\right).$

Each decoder layer $\texttt{Dec}_{l}$ can be Formulated as,

$\displaystyle\texttt{Dec}_{l}(I_{1},I_{2})=\texttt{LN}(\texttt{FD}(I_{1}^{\prime\prime})+I_{1}^{\prime\prime}),$
(21) $\displaystyle
I_{1}^{\prime\prime}=\texttt{LN}(\texttt{MHCA}(I_{1},I_{2})+I_{1}),$
$\displaystyle I_{1}^{\prime}=\texttt{LN}(\texttt{M-MHSA}(I_{1})+I_{1}).$

We use $d_{k}=d_{v}=D=256$ and $h=4$ For the decoder. The InProj and OutProj
are two linear layers with LayerNorm operations. In this work, we use the same
$G$ For language-guided visual prompting and vision-guided language prompting
since the two modalities are aligned in the CLIP embedding space. Therefore,
they can attend to each other by freely change the position in
$\texttt{Dec}_{l}$.

Data augmentations. In this work, we use random flip as a simple weak
augmentation operation. For strong augmentation, we select two random
operations For each sample from the RandAugment library [5], which includes
invert, rotation, color enhancing, auto contrast and other transFormations.
Randomly selecting and combining these strong augmentations is intended to
simulate the diverse domain shifts that can occur in real-world data.

Pseudo-label. For previous methods that incorporate domain-specific tokens in
the textual prompt, obtaining pseudo-labels for the target domain is
challenging. On one hand, the domain-specific prompts are not transferable,
i.e., we cannot obtain high-quality target pseudo-labels with the learned
source-specific prompts. On the other hand, without high-quality pseudo-
labels, learning target-specific prompts becomes an ill-posed problem. To
circumvent this problem, DAPrompt only [12] uses a naive textual prompt, i.e.,
“a photo of a [CLS]. a [Domain] image.” to obtain pseudo-labels, where [CLS]
and [Domain] are the name of each class and each domain, respectively. This
resuls in knowledge isolation between the two domains.

In this work, we learn shared prompts For both domains. This allows us to
leverage the rich source domain knowledge to pseudo-label the target domain.
However, we found that in the early stages of training, the source domain
model is not yet well-trained, resulting in low-quality pseudo-labels. To
address this, we propose combining prior knowledge with the source knowledge
to obtain better pseudo-labels. Specifically, we first generate a naive
textual prompt to produce naive soft pseudo-labels $\tilde{y}^{t}_{i}$ for
each target sample $y^{t}_{i}$. We also generate source-enabled soft pseudo-
labels $\dot{y}^{t}_{i}$ from the model outputs according to Eq. (7). The
final pseudo label is an ensemble of both:

$\hat{y}^{t}_{i}=(1-\alpha)\tilde{y}^{t}_{i}+\alpha\dot{y}^{t}_{i}.$ (22)

The weight $\alpha$ is gradually increased from 0 to 1 during training.
Weighting the naive and source-enabled pseudo-labels via the $\alpha$ enables
a smooth transition from relying more on the prior knowledge to relying more
on the source-knowledge as training progresses.

Algorithm 1 Training Procedure of DAMP For UDA.

1:Labeled source dataset $\mathcal{D}_{s}$, unlabeled target dataset
$\mathcal{D}_{t}$, total training epochs $E$, iteration number per epoch
$N_{e}$.

2:Optimal $\bm{p}_{1:N}$, $\gamma_{v}$, $\gamma_{s}$ and $G$.

3:Initialize parameters for $\bm{p}_{1:N}$, $\gamma_{v}$, $\gamma_{s}$ and
$G$.

4:for $t=1$ to $E$ do

5: for $i=1$ to $N_{e}$ do

6: Sample a source batch $\mathcal{B}_{s}\sim\mathcal{D}_{s}$ and a target
batch $\mathcal{B}_{t}\sim\mathcal{D}_{t}$

7: obtain $\\{\bm{s}_{k}^{\prime}\\}_{k=1}^{K}$ and $\bm{v}^{\prime}$ for each
$\bm{x}\in\mathcal{B}_{s}\cup\mathcal{B}_{t}$ according to Eq. (6) and (4).

8: Calculate $\mathcal{L}^{s}_{sup}$ and $\mathcal{L}^{t}_{sup}$ according to
Eq. (12) and (13).

9: Calculate $\mathcal{L}^{s}_{sc}$ and $\mathcal{L}^{t}_{sc}$ according to
Eq. (9) and (10).

10: Calculate $\mathcal{L}_{idc}$ within $\mathcal{B}_{s}$ and
$\mathcal{B}_{t}$ respectively according to Eq. (8) and sum them up.

11: Calculate $\mathcal{L}_{im}$ according to Eq. (11).

12: Update parameters via optimizing $\mathcal{L}_{all}$.

13: end for

14:end for

15:Return final model parameters $\bm{p}_{1:N}$, $\gamma_{v}$, $\gamma_{s}$
and $G$.

##  Appendix B Algorithm

To better understand our method, we summarize the training procedure of DAMP
for UDA in Algorithm 1.

##  Appendix C Experiments on Multi-Source UDA

Table 6: Classification accuracies (%) on DomainNet for MSDA with ResNet-101. * Prompt learning-based methods. Method |  $\rightarrow$Clipart |  $\rightarrow$Infograph |  $\rightarrow$Painting |  $\rightarrow$Quickdraw |  $\rightarrow$Real |  $\rightarrow$Sketch | Avg.
---|---|---|---|---|---|---|---
Zero-Shot |  |  |  |  |  |  |
CLIP [40] | 61.3 | 42.0 | 56.1 | 10.3 | 79.3 | 54.1 | 50.5
Source Combined |  |  |  |  |  |  |
DANN [11] | 45.5 | 13.1 | 37.0 | 13.2 | 48.9 | 31.8 | 32.6
MCD [43] | 54.3 | 22.1 | 45.7 | 7.6 | 58.4 | 43.5 | 38.5
DAPrompt * [12] | 62.4 | 43.8 | 59.3 | 10.6 | 81.5 | 54.6 | 52.0
CoOp * [72] | 63.1 | 41.2 | 57.7 | 10.0 | 75.8 | 55.8 | 50.6
Multi-Source |  |  |  |  |  |  |
$\text{M}^{3}$SDA-$\beta$ [39] | 58.6 | 26.0 | 52.3 | 6.3 | 62.7 | 49.5 | 42.6
SImpA$\text{I}_{101}$ [55] | 66.4 | 26.5 | 56.6 | 18.9 | 68.0 | 55.5 | 48.6
LtC-MSDA [57] | 63.1 | 28.7 | 56.1 | 16.3 | 66.1 | 53.8 | 47.4
T-SVDNet [27] | 66.1 | 25.0 | 54.3 | 16.5 | 65.4 | 54.6 | 47.0
PFSA [9] | 64.5 | 29.2 | 57.6 | 17.2 | 67.2 | 55.1 | 48.5
PTMDA [42] | 66.0 | 28.5 | 58.4 | 13.0 | 63.0 | 54.1 | 47.2
MPA * [3] | 65.2 | 47.3 | 62.0 | 10.2 | 82.0 | 57.9 | 54.1
DAMP * (Ours) | 69.7 | 51.0 | 67.5 | 14.7 | 82.5 | 61.5 | 57.8
Table 7: Classification accuracies (%) on Office-Home for MSDA with ResNet-50. * Prompt learning-based methods. Method |  $\rightarrow$Ar |  $\rightarrow$Cl |  $\rightarrow$Pr |  $\rightarrow$Rw | Avg.
---|---|---|---|---|---
Zero-Shot |  |  |  |  |
CLIP [40] | 71.5 | 50.2 | 81.3 | 82.4 | 71.4
Source Combined |  |  |  |  |
DAN [31] | 68.5 | 59.4 | 79.0 | 82.5 | 72.4
DANN [11] | 68.4 | 59.1 | 79.5 | 82.7 | 72.4
CORAL [47] | 68.1 | 58.6 | 79.5 | 82.7 | 72.2
DAPrompt * [12] | 72.8 | 51.9 | 82.6 | 83.7 | 72.8
CoOp * [72] | 70.7 | 52.9 | 82.9 | 83.9 | 72.4
Multi-Source |  |  |  |  |
MDDA [69] | 66.7 | 62.3 | 79.5 | 79.6 | 71.0
SImpA$\text{I}_{50}$ [55] | 70.8 | 56.3 | 80.2 | 81.5 | 72.2
MFSAN [73] | 72.1 | 62.0 | 80.3 | 81.8 | 74.1
MPA * [3] | 74.8 | 54.9 | 86.2 | 85.7 | 75.4
DAMP * (Ours) | 77.7 | 61.2 | 90.1 | 87.7 | 79.2

To evaluate the versatility of our method in various domain adaptation
scenarios, we extend our method to the multi-source domain adaptation (MSDA)
setting.

Datasets. We evaluate our method on two widely used MSDA datasets.
Specifically, we reuse the Office-Home [56] dataset for MUDA by combining
arbitrary 3 domains as source domains and regard the rest domain as the target
domain, which forms 4 adaptation tasks ($\rightarrow$Ar, $\rightarrow$Cl,
$\rightarrow$Pr, $\rightarrow$Rw). DomainNet [39] is the lagest and the most
challenging dataset for domain adaptation. It consists of 6 diverse domains,
including Clipart, Painting, Real, Sketch, Quickdraw and Infograph. These
domains encompass a wide range of visual styles, making the dataset
challenging for domain adaptation tasks. Similar to Office-Home, 6 tasks are
constructed for MSDA.

Experimental Setup. It is a natural extension to apply our method to the MSDA
setting, where the goal is to learn a shared set of prompts across all source
domains and the target domain. Specifically, we extend $\mathcal{L}_{sup}$ and
$\mathcal{L}_{sc}$ to include losses on all source domains. We treat each
domain with equal importance. For $\mathcal{L}_{idc}$, we obtain a batch of
samples from each domain in each iteration, and compute $\mathcal{L}_{idc}$
within each domain batch. The utilization of domain labels in this process
distinguishes our method from other single-source domain adaptation methods
which simply mix the source domains. For convenient comparison with previous
methods, we use the ResNet-50 backbone on Office-Home and ResNet-101 on
DomainNet. Other training configurations remain consistent with those employed
in single-source UDA, as detailed in Sec. 4.

Experimental Results. The results on DomainNet are reported in Table 6. DAMP
achieves the best average accuracy of 57.8%, outperforming the previous state-
of-the-art MPA by 3.7%. This demonstrates the effectiveness of DAMP on multi-
source domain adaptation. Compared to single-source methods like DANN, MCD and
DAPrompt, DAMP brings substantial gains, improving over DAPrompt by 5.8%. This
shows the benefits of domain-agnostic prompts and exploiting multiple source
domains in our method. Besides, DAMP also surpasses other multi-source domain
adaptation methods such as $\text{M}^{3}$SDA-$\beta$, SImpA, LtC-MSDA and
T-SVDNet by a large margin. Notably, DAMP achieves the best performance on 5
out of 6 tasks. The consistent improvements over competitive baselines
validate the robustness of DAMP.

On Office-Home (Table 7), we can observe that DAMP again achieves state-of-
the-art accuracy (79.2%), outperforming the closest competitor MPA by 3.8%.
Compared to single-source methods, DAMP brings significant gains over 6.4%
over DAPrompt, showing the benefit of learning domain-agnostic prompts in the
multi-source setting. DAMP surpasses other multi-source domain adaptation
methods like MDDA, SImpA and MFSAN by solid margins. Furthermore, by comparing
with the results for single-source UDA (Table 1), we can observe that for any
target domain, using multiple source domain data is better than using only one
source domain. This validates that our method indeed utilizes source domain
knowledge effectively.

##  Appendix D Experiments on Doamin Generalization

Table 8:  Classification accuracies (%) on VLCS, PACS, Office-Home, and TerraIncognita for domain generalization. The best results are highlighted in bold. All compared methods are implemented based on CLIP with ViT-B/16 backbone. Method | VLCS | PACS | Office-Home | TerraInc | Avg
---|---|---|---|---|---
Fine-tuning (CLIP) |  |  |  |  |
ERM |  82.7 $\pm$ 0.3 |  92.9 $\pm$ 1.9 |  78.1 $\pm$ 2.1 |  50.2 $\pm$ 1.7 | 75.9
CORAL [47] |  82.0 $\pm$ 0.2 |  93.2 $\pm$ 1.1 |  78.9 $\pm$ 1.9 | 53.5 $\pm$ 0.7 | 76.9
DANN [11] |  83.2 $\pm$ 1.2 |  93.8 $\pm$ 1.3 |  78.8 $\pm$ 1.1 |  52.2 $\pm$ 2.0 | 77.0
Zero-shot |  |  |  |  |
CLIP [40] |  82.3 $\pm$ 0.1 |  96.1 $\pm$ 0.1 |  82.3 $\pm$ 0.2 |  34.1 $\pm$ 0.1 | 73.7
Prompt Learning |  |  |  |  |
DPL [66] |  84.3 $\pm$ 0.4 |  97.3 $\pm$ 0.2 |  84.2 $\pm$ 0.2 |  52.6 $\pm$ 0.6 | 79.6
DAMP (Ours) | 84.5 $\pm$ 0.3 | 97.4 $\pm$ 0.2 | 85.0 $\pm$ 0.4 | 53.7 $\pm$ 0.2 | 80.2

We show that with only minor changes, our method can also be used for domain
generalization tasks.

Datasets. We use four popular DG datasets in this experiment, namely, VLCS
[8], PACS [25], Office-Home [56] and TerraIncognita [2]. VLCS contains images
from PASCAL VOC 2007 (V), LabelMe (L), Caltech (C) and SUN (S). There are 5
object categories shared by all domains: bird, car, chair, dog and person.
PACS collects totally 9,991 images from Photo (P), Art painting (A), Cartoon
(C) and Sketch (S) with 7 common categories. Office-Home is originally used in
domain adaptation, which contains images from Art (Ar), Clipart (Cl), Product
(Pr) and Real-World (Rw) across 65 categories. TerraIncognita comprises a
collection of wildlife photographs captured by cameras at various locations.
We follow [13] to use 4 locations, i.e., {L38, L43, L46, L100}, for the DG
task, which have totally 24,788 samples of 5 classes.

Experimental Setup. For the DG task, we implement our method on DomainBed
333https://github.com/facebookresearch/DomainBed, a standard DG benchmark in
the community. We strictly follow [13] to split each domain into 80% training
data and 20% validation data, and use standard training-domain validation for
model selection. The results are obtained by three trials with seed={1,2,3}.

Different from domain adaptation, in DG we cannot access any target sample
during training. Therefore, we remove $\mathcal{L}_{im}$ and target-related
terms in $\mathcal{L}_{sup}$, $\mathcal{L}_{sc}$ and $\mathcal{L}_{idc}$ for
optimization. In this scenario, our method aims to elicit domain-invariant
visual embeddings from multiple source domains and instance-compatible text
embeddings for classification. Due to the absence of the target domain, large-
scale pre-trained knowledge becomes more important in DG. Therefore, CLIP-
based methods will have significantly better results than traditional ones.
For fair comparison, all compared baselines are built on CLIP, and we use the
ViT-B/16 vision backbone for all datasets following [66].

Specifically, there are three categories of baselines for comparison. The
first category of methods fine-tune the image encoder of CLIP using common DG
algorithms (e.g., like ERM, DANN) and freeze the text encoder for
classification. The second category directly use the zero-shot ability of CLIP
and prompt the text encoder with manually designed prompts (‘a photo of
[CLS]’) for classification. The third category resorts to learnable prompts
for adapt the pre-trained CLIP to specific domains. For our method, the
hyperparameter configuration is consistent with the ones in UDA and MSDA.

Experimental Results. We report the mean accuracies as well as standard
derivation on four datasets in Table 8. Our DAMP achieves the best average
accuracy of 80.2% across all datasets. Compared to fine-tuning-based methods
like ERM, CORAL and DANN, DAMP brings significant improvements of 4.3%, 3.3%
and 3.2% respectively in average accuracy. We conjecture the reason is that
large-scale pre-training is a very effective approach to bridge the domain
gap. However, fine-tuning the image encoder is prone to destroy the pre-
trained knowledge encoded in CLIP. This demonstrates the superiority of
adapting pre-trained models via prompt learning over fine-tuning for domain
generalization. On the other hand, DAMP also surpasses the vanilla zero-shot
CLIP model by 6.5%, showing the benefits of learning adaptive prompts compared
to relying solely on pre-trained knowledge. Compared with DPL that only
prompts the text encoder, our method prompts both vision and textual
modalities for generalizing both visual images and textual semantics to unseen
domains. Besides, two regularizations (i.e., $\mathcal{L}_{idc}$ and
$\mathcal{L}_{sc}$) encourage the embeddings to be more domain-agnostic, thus
outperforming DPL on all datasets. Even though the margins appear small, i.e.,
0.6% over DPL, it is well-recognized that further advancing the state-of-the-
art on DG benchmarks is extremely challenging. Even slight gains of 1% are
considered significant and difficult in the DG community, which typically
indicate non-trivial improvements in the robustness and generalization
abilities of the model across diverse domains.

##  Appendix E Additional Analytical Experiments for UDA

###  E.1 Confusion Matrix Visualization

Figure 6: Visualization of confusion matrixes yield by different methods on
VisDA-17 dataset with ViT-B/16 backbone.

To illustrate how our method benefits UDA, we visualize the confusion matrixes
obtained by different methods in Fig. 6. We can observe that directly using
the zero-shot classification capability of CLIP can easily lead to confusion
between categories. For instance, the model may easily predict ”car” as ”bus”
or ”truck,” or predict ”bicycle” as ”motorcycle”, because these categories are
conceptually similar. In contrast, DAPrompt adjusts the textual semantics of
categories specifically for each domain (dataset), making it clearer to
distinguish the semantic differences between categories and to some extent
alleviating the confusion problem. However, DAPrompt does not perform any
adaptation in the visual modality, making it susceptible to domain shifts in
the visual modality. Additionally, DAPrompt uses class-level semantic
embeddings for classification, which does not take into account variations
within categories. Imagine if the visual embedding of a truck in the
multimodal space is closer to the semantic representation of ”bus” than
”truck.” In this case, DAPrompt would have no way of classifying it as a
truck. In contrast, our method allows the semantic embeddings to dynamically
adjust their positions based on visual cues for each sample, providing a
customized set of semantic embeddings for classification, and therefore
performs better in disambiguation compared to CLIP and DAPrompt.

###  E.2 Effectiveness of the Pseudo-label Strategy

(a)

(b)

(c)

(d)

Figure 7: Pseudo-label accuracies of different strategies. We choose four
tasks on Office-Home as examples (ResNet-50). Table 9: Classification
accuracies of our method with (w/) and without (w/o) Parameter-Sharing (PS)
strategy in the mutual prompting module. The results are obtained on Office-
Home dataset.

Method | $f_{v}$ |  Ar$\rightarrow$Cl |  Ar$\rightarrow$Pr |  Ar$\rightarrow$Rw |  Cl$\rightarrow$Ar |  Cl$\rightarrow$Pr |  Cl$\rightarrow$Rw |  Pr$\rightarrow$Ar |  Pr$\rightarrow$Cl |  Pr$\rightarrow$Rw |  Rw$\rightarrow$Ar |  Rw$\rightarrow$Cl |  Rw$\rightarrow$Pr | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
DAMP w/o PS |  ResNet-50 | 59.5 | 88.6 | 86.4 | 76.5 | 89.0 | 86.8 | 76.6 | 59.8 | 86.9 | 77.1 | 60.4 | 89.5 | 78.1
DAMP w/ PS | 59.7 | 88.5 | 86.8 | 76.6 | 88.9 | 87.0 | 76.3 | 59.6 | 87.1 | 77.0 | 61.0 | 89.9 | 78.2
DAMP w/o PS |  ViT-B/16 | 75.9 | 93.7 | 92.2 | 86.1 | 94.3 | 91.7 | 85.7 | 76.1 | 92.0 | 85.5 | 76.1 | 93.6 | 86.9
DAMP w/ PS | 75.7 | 94.2 | 92.0 | 86.3 | 94.2 | 91.9 | 86.2 | 76.3 | 92.4 | 86.1 | 75.6 | 94.0 | 87.1

To evaluate the effectiveness of our ensemble-based pseudo-label strategy
described in Appendix. A, we conduct experiments to compare with other two
pseudo-label strategies, i.e., using the naive prompts (‘a photo of a [CLS]’)
for zero-shot prediction and using the learned prompts $\bm{p}_{1:N}$ with
post-model multual prompting. The former strategy is used in DAPrompt [12] and
the latter is the outputs of our model. As shown in Fig. 7, we can observe a
common phenomenon across all tasks. Initially, due to the zero-shot capability
of CLIP, it can provide high-quality pseudo-labels for the target domain
samples, resulting in high accuracy achieved in the first epoch for learned
prompts. However, as the proportion of zero-shot pseudo-labels is still high
at this point, the accuracy of the ensemble pseudo-labels remains lower than
that of the learned prompt-based pseudo-labels. As training progresses (after
10 epochs), the ensemble pseudo-labels outperforms the other two strategies.
We conjecture the reason is that the incorporation of certain prior knowledge
(naive prompts) helps alleviate the risk of overfitting the learned prompts to
the source domain. This enables the model to achieve accuracy that cannot be
attained by relying solely on learned prompts for pseudo-labeling. Finally,
the accuracy of the ensemble gradually converges towards the accuracy of the
learned prompts.

###  E.3 Effectiveness of Parameter-Sharing Strategy

Table 9 shows the impact of parameter-sharing $G$ on the performance of our
method. It turns out that the parameter-sharing strategy (w/ PS) slightly
outperforms the version without parameter-sharing (w/o PS) on both vision
backbones. Parameter-sharing enables a single prompting module $G$ to
transform both visual and textual embeddings bidirectionally. This allows
richer cross-modal interactions and fusion between the modalities, eliciting
better domain- and modality-shared representations. In contrast, without
parameter-sharing, the promptings of vision of vision and text modalities will
be more independent, and more tunable parameters make it difficult to train
and less effective. The consistently positive gains across various backbones
indicate that parameter-sharing is an effective and generalizable design
choice for mutual prompting in DAMP. It enables a single compact module to
prompt both modalities flexibly for domain adaptation.