# HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain
Generalization

Guanglin Zhou, Zhongyi Han, Shiming Chen, Biwei Huang, Liming Zhu, Tongliang
Liu, , Lina Yao, , Kun Zhang  Guanglin Zhou is with the University of New
South Wales. Email<EMAIL_ADDRESS>Zhongyi Han and Shiming Chen are
with Carnegie Mellon University and Mohamed bin Zayed University of Artificial
Intelligence. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Biwei
Huang is with University of California, San Diego. Email<EMAIL_ADDRESS>Liming Zhu is with CSIRO’s Data61. Email<EMAIL_ADDRESS>Tongliang
Liu is with the University of Sydney and Mohamed bin Zayed University of
Artificial Intelligence. Email<EMAIL_ADDRESS>Lina Yao is with
CSIRO’s Data61, the University of New South Wales and Macquarie University.
Email<EMAIL_ADDRESS>Kun Zhang is with Carnegie Mellon University and
Mohamed bin Zayed University of Artificial Intelligence. Email<EMAIL_ADDRESS>0000-0002-1305-2622 0000-0003-2851-193X 0000-0001-9633-3392
0000-0002-9640-6472 0000-0002-4149-839X

###### Abstract

Domain Generalization (DG) endeavors to create machine learning models that
excel in unseen scenarios by learning invariant features. In DG, the prevalent
practice of constraining models to a fixed structure or uniform
parameterization to encapsulate invariant features can inadvertently blend
specific aspects. Such an approach struggles with nuanced differentiation of
inter-domain variations and may exhibit bias towards certain domains,
hindering the precise learning of domain-invariant features. Recognizing this,
we introduce a novel method designed to supplement the model with domain-level
and task-specific characteristics. This approach aims to guide the model in
more effectively separating invariant features from specific characteristics,
thereby boosting the generalization. Building on the emerging trend of visual
prompts in the DG paradigm, our work introduces the novel Hierarchical
Contrastive Visual Prompt (HCVP) methodology. This represents a significant
advancement in the field, setting itself apart with a unique generative
approach to prompts, alongside an explicit model structure and specialized
loss functions. Differing from traditional visual prompts that are often
shared across entire datasets, HCVP utilizes a hierarchical prompt generation
network enhanced by prompt contrastive learning. These generative prompts are
instance-dependent, catering to the unique characteristics inherent to
different domains and tasks. Additionally, we devise a prompt modulation
network that serves as a bridge, effectively incorporating the generated
visual prompts into the vision transformer backbone. Experiments conducted on
five DG datasets demonstrate the effectiveness of HCVP, outperforming both
established DG algorithms and adaptation protocols.

###### Index Terms:

Domain Generalization, Visual Prompt, Contrastive Learning

Figure 1: Motivation illustration. (a) Traditional DG methods, employing
universal parameters across the entire dataset, often struggle to distinguish
between invariant shape attributes (e.g., large ears, long nose) and domain-
specific texture attributes (e.g., thick grey skin). This difficulty leads to
a blending of features, thereby diminishing the model’s capacity for
generalization. (b) Our approach introduces hierarchical visual prompts that
encapsulate domain-level and task-specific characteristics, enabling the model
to better differentiate and understand both invariant and specific attributes,
thereby contributing to more effective generalization across different
domains.

##  I Introduction

Despite the remarkable successes of machine learning models, particularly deep
neural networks (DNNs), in various areas, they often exhibit unexpected
failures when the test data distribution deviates from the training data [1,
2, 3]. For instance, adversarial inputs can lead to misclassifications [4],
performance on digit recognition may deteriorate under rotations [5], and DNNs
trained to recognize pneumonia may fail when applied to scans from new
hospitals [6]. Addressing this, domain generalization (DG) seeks to create
robust models that can effectively adapt and perform on unseen, yet related,
domains by leveraging invariant features learned across multiple source
domains [1, 2].

Domain generalization has traditionally relied on techniques such as data
manipulation and representation learning [1, 2], giving rise to classic DG
algorithms [7, 8, 9, 10, 11, 12]. Despite these efforts, several large-scale
empirical studies [13, 14] have highlighted limited performance improvements
in current DG algorithms. Alongside these traditional methods, recent
advancements in visual foundation models [15], such as vision transformer
(ViT) backbone [16], have been leveraged for DG tasks [17, 18]. Their ability
to capture a broad spectrum of diverse features has opened up new avenues for
effective DG. Several tuning protocols have been proposed to adapt these
foundation models to downstream tasks [19, 20, 21, 22, 19, 23].

The fundamental objective in DG lies in effectively discerning and leveraging
domain-invariant features from specific characteristics. Nevertheless, the
challenge lies in accomplishing this separation within a unified modeling
framework without extra, tailored information. The inherent complexity of
handling diverse and often subtle differences between domains necessitates
additional context beyond what a single model can provide. This lack of
specificity leads to a DG approach that unintentionally blends domain-specific
aspects, making it challenging to differentiate nuanced inter-domain
variations [6]. For example, the DG dataset depicted in Figure 1, encompasses
the classification tasks like classifying elephants across diverse domains
such as the cartoon, photo, and sketch. Shape attributes like large ears and
texture attributes such as thick grey skin are crucial for prediction.
However, the significance of the thick grey skin might diminish across
domains. Without specific adjustments, a conventional model might struggle to
discern these features, leading to the blending of domain-specific attributes
in predictions on unseen domains. This complexity highlights the need for a
more nuanced approach to handle the unique challenges.

In this work, we introduce a novel method that supplements raw images with
domain-level and task-specific characteristics, aiming to guide the model in
better learning invariant features from specific aspects. Such insight paves
the way for our exploration of visual prompts, which have served as contextual
anchors, enriching models with a tailored understanding within specific
contexts [24, 25, 26]. Building on the emerging trend of visual prompts in the
DG paradigm [27, 28], we introduce the novel Hierarchical Contrastive Visual
Prompt (HCVP) methodology, representing a significant advancement in this
field. Differing from traditional visual prompts that are often shared across
the entire dataset, HCVP employs a two-tier hierarchical prompt generation
network along with a prompt contrastive learning strategy [29, 30, 31]. Such
explicit model structure and specialized loss functions guide the visual
prompt learning process to capture both domain-level and task-specific
details. Moreover, we devise a prompt modulation network that acts as a
bridge, effectively incorporating our generated visual prompts into the ViT
backbone. In order to enhance visual features with the discriminative power
between inter-classes, we introduce class-conditioned contrastive invariance.
The experiments, conducted on several DG datasets [13, 32], demonstrate the
effectiveness of our approach.

The main contributions of this paper are summarized as:

  * •

We introduce a novel approach that integrates domain and task information in
the DG process, enhancing the distinction of invariant features and those
specific to individual domains.

  * •

Our HCVP method innovatively utilizes generative visual prompts tailored for
DG, explicitly employing a hierarchical prompt generation network and prompt
contrastive learning, to craft prompts that encapsulate relevant domain-level
and task-specific characteristics.

  * •

Through extensive experiments, HCVP is demonstrated to achieve state-of-the-
art performance on five real-world datasets in DG, surpassing both established
DG algorithms and adaptation protocols.

##  II Related Work

###  II-A Domain Generalization

Domain generalization traditionally relied on data manipulation and
representation learning [1]. Techniques such as data augmentation [33] and
generation [34] were utilized to diversify training data. Representation
learning [35, 36, 37, 38, 39, 40, 41, 42] used strategies ranging from feature
alignment [43], domain adversarial learning [44, 7], to causality-inspired
methods [8, 9, 10]. Recent studies suggest limited improvement in DG methods
over empirical risk minimization (ERM) [13, 14], indicating a need for
innovation. While visual foundation models are employed in modern DG
techniques [17, 15], they often necessitate extensive model selection [20,
21]. Various protocols have been proposed for adaptation, such as linear
probing [45, 22], LP-FT (linear probing and then full fine-tuning) [19], and
gradual unfreeze (fine-tuning last $k$ layers) [23, 46]. However, conventional
DG methods typically rely on a universal parameter set and my blend domain-
specific features, and thus hinder generalization. In contrast, our approach
focuses on enriching the model with domain-level and task-specific
characteristics to enhance separation of invariant features.

###  II-B Visual Prompt Tuning

Prompt tuning acts as a powerful tool that furnishes transformer-based large
models with the necessary semantic context for particular tasks [47, 48]. Such
a principle has successfully transitioned to the visual realm, culminating in
visual prompt tuning (VPT) [26]. Traditional visual prompt methods [26, 49]
typically implement visual prompts using parameter placeholders, where a
single set of prompts is shared across various tasks. While effective for
specific downstream tasks [50, 51], they are not specifically tailored for OoD
scenarios, potentially hindering their ability to bridge distribution shifts
within the DG context. Interestingly, the potential benefits of more
customized prompts have been observed in other areas, such as the use of
class-specific prompts for certain fine-grained categories [25]. This
observation highlights the potential of adapting visual prompts to embody
domain-level and task-specific details within DG. Recent works, namely
DoPrompt [27] and CSVPT [28], have begun to explore the application of visual
prompting within DG paradigms. However, our HCVP presents distinct
advancements in this area. Unlike DoPrompt [27], which uses prompts shared
across each domain and leads to challenges in prompt selection or fusion, HCVP
utilizes generative prompts that encapsulate more nuanced characteristics.
Besides, HCVP eliminates the need for the extra adapters required in DoPrompt.
Moreover, contrasting with CSVPT [28], which also adopts generative prompting
but lacks additional regularization mechanisms, HCVP introduces an explicit
model structure and specialized loss functions. These are meticulously
designed to guide the visual prompt learning process, ensuring a more targeted
and effective integration of domain-specific and task-specific information
within the DG framework.

Figure 2:  The architecture overview of the proposed Hierarchical Contrastive
Visual Prompt (HCVP) model. HCVP comprises two key components: the
Hierarchical Prompt Generation Network (HPGN) and the Prompt Modulation
Network (PMN). The HPGN first uses a pretrained encoder to extract feature
maps. These feature maps are then processed through a dual-level generation
module for domain-level and task-specific prompt generation, respectively. The
PMN serves as a conduit, integrating the prompts generated by the HPGN into
the ViT layers. Additionally, HCVP incorporates two contrastive learning
strategies: Prompt Contrastive Learning (PCL), which optimizes the generation
of both domain and task-specific visual prompts, and Class-conditional
Contrastive Invariance (CCI), which enhances the model’s class-specific
discriminative power.

##  III Preliminaries and Problem Formulation

In this section, we establish the foundational notations and define the
objective of our study, particularly focusing on the role of generative visual
prompts in DG. We also introduce the concept of mutual information as a key
metric in our approach.

###  III-A Notations

To establish a clear understanding of our approach, we define several critical
variables that are central to our methodology: $Y$ (the class label we aim to
predict), $D$ (the domain index), $Z$ (the learned invariant features crucial
for domain generalization), $X$ (the raw input features), and $P$ (the learned
prompts representing specific characteristics extracted from $X$). Following
[1, 2], we denote the input feature space as $\mathcal{X}$ and the target
label space as $\mathcal{Y}$. A domain comprises data from a joint
distribution $P_{XY}$ in the space $\mathcal{X}\times\mathcal{Y}$. For a given
domain $P_{XY}$, $P_{X}$, $P_{Y|X}$, and $P_{X|Y}$ represent the marginal,
posterior, and class-conditional distributions, respectively. We have access
to $M$ source domains
$\mathcal{S}_{train}=\\{\mathcal{S}^{i}|i=1,\cdots,M\\}$, each
$\mathcal{S}^{i}=\\{(\bm{x}_{j}^{i},y_{j}^{i})\\}_{j=1}^{n_{i}}$, where
$\bm{x}$ and $y$ denote the vectors of input features and class labels. The
joint distributions between any two domains are not equal, i.e.,
$P_{XY}^{i}\neq P_{XY}^{j},1\leq i\neq j\leq M$. The objective in DG is to
learn a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ using the $M$ training
domains to minimize prediction error on a new test domain
$\mathcal{T}=\\{\bm{x}^{\mathcal{T}},y^{\mathcal{T}}\\}$. In our work, this
function $f$ is defined as $f(\bm{x},\bm{p})$, where $\bm{p}$ represents
generative visual prompts that are integrated into the model to enhance its
learning capability:

$\underset{f}{\text{min}}\mathbb{E}_{(\bm{x},y)\in\mathcal{T}}[l(f(\bm{x},\bm{p}),y)]$
(1)

This formulation reflects our novel approach of enriching the ViT model with
visual prompts $\bm{p}$ generated from $\bm{x}$.

###  III-B Role of Generative Visual Prompts in Mutual Information

Mutual information is a measure from information theory that quantifies the
amount of information obtained about one random variable through observing
another random variable [52, 53]. In the context of DG, we consider two key
mutual information terms [54]:

  * •

$I(Z,Y)$ quantifies how much information the learned representation $Z$ shares
with the class label $Y$. A higher value indicates that $Z$ is more
informative about $Y$.

  * •

$I(Z,D|Y)$ measures the information that $Z$ contains about the domain $D$,
given the label $Y$. In DG, we aim to reduce this term, indicating that the
learned representation is less influenced by domain-specific features and more
by features that generalize across domains.

We hypothesize that the incorporation of visual prompts $P$ into the learning
process will lead to a reduction of $I(Z,D|Y)$, indicating a more invariant
representation. This hypothesis is grounded on the intuition that $P$, by
providing additional context, assists the model in better isolating specific
variations from invariant features. Formally, we aim to show:

$I(Z^{{}^{\prime}},D|Y)<I(Z,D|Y)$ (2)

Here, $Z^{{}^{\prime}}$ is the representation learned with the inclusion of
$P$. Our subsequent analysis and empirical results aim to support this
hypothesis. Furthermore, our model architecture and objective functions,
detailed in the following methodology section, are explicitly designed to
align with such two mutual information terms.

##  IV Methodology

Our proposed HCVP methodology, as depicted in Figure 2, illustrates the
comprehensive architecture tailored for DG. Specifically, the hierarchical
prompt generation network and prompt modulation network, complemented with
prompt contrastive learning, are designed to ensure that the generated prompt
vectors align precisely with specific characteristics. The cross-entropy loss
($\mathcal{L}_{cls}$) directly supports the mutual information term $I(Z,Y)$,
while the class-conditioned contrastive invariance loss ($\mathcal{L}_{CCI}$)
is designed to align with $I(Z,D|Y)$ respectively. This alignment with the
mutual information framework is instrumental in achieving our goal of
enhancing DG capabilities of the model.

###  IV-A Hierarchical Prompt Generation Network

As discussed in Sec. II-B, although traditional visual prompt methods [26] are
effective for specific downstream tasks, they are not specifically tailored
for OoD scenarios. While merely replicating visual prompts for individual
domain can allow prompts to mirror domain-level information, prompt selection
or fusion becomes challenging in DG settings where the target domain is
unknown or inaccessible [27]. To tackle these challenges, our Hierarchical
Prompt Generation Network (HPGN) offers a sophisticated solution. While CSVPT
[28] also utilizes generative visual prompts, HPGN, goes a step further by
integrating prompt contrastive learning. In our work, the model structure and
loss functions are specifically designed to generate visual prompts that
adeptly encapsulate domain-level and task-specific information.

####  IV-A1 Domain-level Prompt Generation

In the first level of our hierarchical model, we design a mechanism to
generate domain-level prompts. These prompts are aimed at encapsulating
coarse-grained and high-level features that are common across a particular
domain. The process begins by taking input features, denoted as $\bm{x}$, and
passing them through a pre-trained and frozen ResNet encoder [55], resulting
in the output feature maps represented as $F=R(\bm{x})$111The domain index is
omitted for simplicity.. To further distill these feature maps, we append a
global average pooling (GAP) layer, denoted as $G(\cdot)$, followed by a fully
connected (FC) layer, which is a Multi-Layer Perceptron (MLP) and is denoted
as $FC(\cdot)$. The GAP layer functions to reduce the spatial dimensions of
the feature maps $F$, effectively encapsulating the entire spatial content
into a single value per feature map. This operation retains the dominant
global features across the spatial domain. The subsequent MLP is capable of
learning non-linear combinations of these global features, thus allowing for
the capture of intricate global patterns in the data. The resulting domain
prompt vector for an instance $\bm{x}$ is:

$C(\bm{x})=FC(G(F))$ (3)

where $F=R(\bm{x})$. This structure captures the overall domain
characteristics, setting the stage for further refinement in the subsequent
hierarchical level.

####  IV-A2 Task-specific Prompt Generation

The second level of the hierarchy focuses on creating task-specific prompts to
capture task-specific features. By appending a set of convolutional layers,
denoted as $\Phi(\cdot)$, to the ResNet model, we facilitate the extraction of
localized and intricate patterns from the feature maps. In the context of DG,
where each classification task operates within a unique domain, it becomes
pertinent to refine task-specific prompts by integrating the domain-level
prompts as inputs. This integration forms a hierarchical structure,
synergizing the broad domain-level insights with a refined understanding of
task-specific characteristics. The task-specific prompt for an instance
$\bm{x}$ is formally given by:

$P(\bm{x})=\Phi(F,C(\bm{x}))$ (4)

where $F=R(\bm{x})$ and $C(\bm{x})$ is the domain-level prompt in Eq. (3).
These two prompt vectors are concatenated to form the final prompt vector,
represented as $\bm{p}=\text{concat}(C(\bm{x}),P(\bm{x}))$, where
$\text{concat}(\cdot)$ denotes the concatenation operation. The resulting
prompt vector is subsequently fed into the PCL module for optimization and
combined with image patch embeddings as the input for ViT layers, shown in
Figure 2.

###  IV-B Prompt Modulation Network

Within our architecture, the Prompt Modulation Network (PMN) serves as a
conduit to effectively integrate the generated visual prompts into the ViT
backbone. Unlike traditional visual prompt methods [26] that rely on fixed
prompt placeholders, our use of dynamically generated prompts necessitates
this specially designed integration mechanism. Specifically, the PMN receives
the two-tier generated visual prompts as input and transforming them through a
MLP. Importantly, the MLP is configured with a number of layers that is
identical to the layer count in the ViT. This transformation process is
conducted during training, where the transformed prompts are systematically
integrated into each ViT layer $L_{i}$, to facilitate a nuanced adaptation of
the visual content. The class token vector and patch embeddings after $i$-th
ViT layer are represented by $\mathbf{x}_{i}$ and $\mathbf{E}_{i}$, and the
integration is formalized as:

$\displaystyle[\mathbf{x}_{i},\\_,\\_,\mathbf{E}_{i}]$
$\displaystyle=L_{i}([\mathbf{x}_{i-1},C_{i-1},P_{i-1},\mathbf{E}_{i-1})$ (5)
$\displaystyle[C_{i},P_{i}]$
$\displaystyle=\text{PMN}(C_{i-1}),\text{PMN}(P_{i-1})$ (6)

This modulation mechanism, drawing on principles from batch normalization [56]
and attention mechanisms [57, 58], allows the PMN to adapt the ViT’s internal
representations, effectively utilizing the generated visual prompts. This
enhances the model’s ability to accommodate both domain-level and task-
specific details.

###  IV-C Prompt Contrastive Learning

We introduce a prompt contrastive learning strategy, which acts as the
supplementary to HPGN and is tailored to learn both domain and task prompts,
thereby guiding the learning process for visual prompts effectively.

Domain-level prompt contrastive learning focuses on producing similar domain-
level prompts for instances within the same domain, while ensuring distinct
ones from different domains. As a result, the generated domain-level prompts
accurately encapsulate the unique characteristics associated with each domain.
The loss for a positive pair and the total loss are defined as follows:

$l_{c}^{ij}=-\log\frac{\exp({C(\bm{x}_{i})\cdot
C(\bm{x}_{j})}/{\tau})}{\sum_{k}\mathbbm{1}_{[k\neq
i]}\exp({C(\bm{x}_{i})\cdot C(\bm{x}_{k})}/{\tau})}$ (7)

$\mathcal{L}_{domain}=\frac{1}{n_{i}}\sum_{i=1}^{n_{i}}l_{c}^{ij}$ (8)

where $C(\bm{x}_{i})$ represents the domain-level prompt for the $i$-th
instance in the sampled batch, $j$ denotes a positive sample of $i$,
specifically from the same domain. The temperature parameter is denoted by
$\tau$ and dot product is utilized as the similarity measure.

Task-specific prompt contrastive learning aims to encourage the model to
generate similar task prompts for instances of the same class within a domain,
and distinct ones for instances from different classes within a domain. The
loss for a positive pair is defined as follows:

$l_{p}^{ij}=-\log\frac{\exp\left({P(\bm{x}_{i})\cdot
P(\bm{x}_{j})}/{\tau}\right)}{\sum_{k}\mathbbm{1}_{[k\neq
i]}\exp({P(\bm{x}_{i})\cdot P(\bm{x}_{k})}/{\tau})}$ (9)

$\mathcal{L}_{task}=\frac{1}{n_{i}}\sum_{i=1}^{n_{i}}l_{p}^{ij}$ (10)

Here, $P(\bm{x}_{i})$ represents the task prompt for the $i$-th instance, $j$
denotes a positive sample of $i$, specifically from the same class within the
same domain. The overall prompt contrastive learning loss
$\mathcal{L}_{\text{{PCL}}}$ is a combination of
$\mathcal{L}_{\text{{domain}}}$ and $\mathcal{L}_{\text{{task}}}$ with equal
weight.

###  IV-D Class-conditioned Contrastive Invariance

Our approach introduces Class-conditioned Contrastive Invariance (CCI) as a
key component to augment the discriminative ability between different classes
while promoting invariance to domain-specific variations. CCI focuses on
promoting invariance within the same class, regardless of the domain they
originate from, while maintaining sensitivity to differences between distinct
classes. The CCI loss function is designed to encourage representations to be
domain-invariant in the context of class labels, thereby decreasing
$I(Z,D|Y)$.

Given $\mathbf{x}_{N}$ as the class token embedding after the last ViT layer,
and $y$ as the class label of the instance, the CCI loss can be defined as:

$\mathcal{L}_{\text{CCI}}=-\mathbb{E}\left[\log\frac{\exp(\mathbf{x}_{N}\cdot\mathbf{x}_{N^{\prime}}/\tau)}{\sum_{k\neq
N}\exp(\mathbf{x}_{N}\cdot\mathbf{x}_{k}/\tau)}\right]$ (11)

where $\mathbf{x}_{N^{\prime}}$ represents a positive sample of
$\mathbf{x}_{N}$ (same class regardless of the domains), and $\mathbf{x}_{k}$
signifies a negative sample of $\mathbf{x}_{N}$ (different class). Through
this formulation, CCI effectively reinforces the desired domain invariance
while maintaining robust class discriminability within the learned
representations.

###  IV-E End-to-End Training Strategy

Our training strategy combines multiple loss components: prompt contrastive
learning loss $\mathcal{L}_{\text{PCL}}$, class-conditioned contrastive
invariance loss $\mathcal{L}_{\text{CCI}}$, and classification loss
$\mathcal{L}_{\text{cls}}$. The classification loss is defined using cross-
entropy between the output from the classification head applied to
$\mathbf{x}_{N}$ and the ground-truth label $y$, aiming to increase the mutual
information term $I(Z,Y)$:

$\mathcal{L}_{\text{cls}}=-\sum_{i=1}^{C}y_{i}\log(\text{Head}(\mathbf{x}_{N})_{i})$
(12)

We then formulate the total loss $\mathcal{L}_{\text{total}}$:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{cls}}+\lambda_{PCL}\mathcal{L}_{\text{PCL}}+\lambda_{CCI}\mathcal{L}_{\text{CCI}}$
(13)

Here, $\lambda_{PCL}$ and $\lambda_{CCI}$ are balancing coefficients for each
loss component respectively. These coefficients are determined through a
systematic grid search.

##  V Experiments

###  V-A Experimental Setup

####  V-A1 Datasets

We evaluate on five benchmark datasets in DG [13, 32]. PACS [59]: encapsulates
four distinctive visual domains: Art Paintings, Cartoon, Photos, and Sketches;
contains 9,991 instances across 7 classes, each being a (3, 224,
224)-dimensional image. VLCS [60]: an amalgamation of four photographic
domains: Caltech-101, LabelMe, SUN09, and VOC2007; includes 10,729 instances
in five classes, each a (3, 224, 224)-dimensional image. Office-Home [61]:
comprises four domains: Art, Clipart, Product, and Real-World Images, with
15,588 images in 65 classes, each of dimension (3, 224, 224). TerraIncognita
[62]: features wildlife images from four camera trap locations (L100, L38,
L43, L46), each representing a distinct domain; has 24,788 (3, 224,
224)-dimensional images across 10 classes. CelebA [63]: focuses on “hair
color” classification against “gender” as a spurious attribute [64, 32]. Our
subset has 27,040 images in three environments, akin to the Colored MNIST
dataset’s correlation shift [65, 32].

####  V-A2 Baselines

We compare HCVP with two categories of DG methods: DG Algorithms include ERM
[66], IRM [65], MLDG [67], MMD [43], DANN [7], CDANN [11], VREx [68], MTL
[69], SagNet [70], RSC [71], IB_ERM, IB_IRM [72], SWAD [36], MIRO [37] and
POEM [35]; Tuning Protocols include full fine-tuning, linear probing (LP)—a
frozen backbone with a tuned linear head [19], partial-k (fine-tuning last k
layers). We also acknowledge related visual prompt methods like DoPrompt [27].
A direct comparison with CSVPT [28] is not feasible, as its implementation is
not open-sourced.

####  V-A3 Implementation Details

We adopt unified settings for all methods: ViT-B/16
backbone222https://huggingface.co/google/vit-base-patch16-224-in21k pre-
trained on ImageNet-21k [16, 73]; AdamW optimizer [74] with learning rate
1$e$-5 and weight decay 0.01; 80%:20% train-validation split, batch size of
32, 3000 steps per trial; averages reported over 3 runs with different seeds.
Training-domain validation strategy is used for model selection. We use
PyTorch to implement all experiments on NVIDIA A100-40GB and A5000-24GB GPUs.

TABLE I: Performance comparison of HCVP with existing DG algorithms and tuning protocols on five DG datasets. The best and second-best results are marked in Red and Blue respectively.  Methods | PACS | VLCS | OfficeHome | TerraIncognita | CelebA | Avg
---|---|---|---|---|---|---
| DG Algorithms
ERM | 89.3 | 80.0 | 82.5 | 54.7 | 84.6 | 78.2
IRM | 82.9 | 78.1 | 73.6 | 38.9 | 82.1 | 71.1
MLDG | 89.9 | 79.8 | 82.5 | 52.9 | 85.0 | 78.0
MMD | 88.4 | 79.9 | 82.1 | 53.0 | 85.7 | 77.8
DANN | 84.2 | 79.0 | 80.1 | 47.8 | 85.5 | 75.3
CDANN | 84.3 | 78.4 | 80.4 | 49.4 | 84.5 | 75.4
VREx | 87.3 | 79.4 | 80.3 | 53.1 | 84.5 | 76.9
MTL | 87.2 | 80.1 | 80.7 | 55.0 | 84.5 | 77.5
SagNet | 83.5 | 80.1 | 77.6 | 48.3 | 83.8 | 74.7
RSC | 89.8 | 79.9 | 82.5 | 55.5 | 85.0 | 78.5
IB_ERM | 82.2 | 78.5 | 71.6 | 33.6 | 83.4 | 69.9
IB_IRM | 81.4 | 78.3 | 58.4 | 37.7 | 82.6 | 67.7
SWAD | 90.1 | 79.3 | 79.7 | 53.0 | 84.0 | 77.2
MIRO | 83.6 | 77.4 | 66.0 | 47.7 | 81.4 | 71.2
POEM | 84.3 | 76.1 | 73.3 | 39.3 | 83.7 | 71.3
| Tuning Protocols
Full | 89.3 | 80.0 | 82.5 | 54.7 | 84.6 | 78.2
LP | 71.6 | 76.9 | 75.6 | 35.2 | 71.0 | 66.1
Partial_1 | 74.5 | 79.4 | 76.1 | 36.7 | 79.7 | 69.3
Partial_2 | 76.7 | 79.9 | 76.8 | 38.4 | 81.1 | 70.6
Partial_4 | 80.5 | 80.3 | 77.7 | 38.8 | 84.6 | 72.4
DoPrompt | 89.8 | 80.3 | 82.7 | 54.5 | 85.3 | 78.5
| Our Method
HCVP | 90.2 | 81.1 | 82.5 | 55.1 | 86.6 | 79.1

###  V-B Main Results

In our evaluation, HCVP is benchmarked against established DG algorithms and
various tuning protocols. The comparative results are presented in Table I.
For a more detailed analysis, particularly focusing on performance across each
unseen domain, refer to Tables II, III, IV, and V. Our comparison spans two
primary types of distribution shifts: diversity shift observed in the PACS,
VLCS, OfficeHome and TerraIncognita datasets, and the spurious shift in the
CelebA dataset [32].

####  V-B1 DG Algorithms

As illustrated in Table I, our HCVP achieves the highest overall accuracy of
79.1%, and achieves the best or second-best performance on each dataset.
Specifically, HCVP attains the best average accuracies of 90.2%, 81.1% and
86.6% on PACS, VLCS and CelebA datasets, respectively. These results
demonstrate that HCVP effectively learns invariant features that are
informative for prediction across domains. In the case of the OfficeHome
dataset, our HCVP still achieves a comparable performance of 82.5%, only 0.2%
lower than the best performance achieved by MLDG. Compared with other baseline
DG algorithms designed to capture invariant features, such as MMD and DANN for
domain-invariant features, and IRM, VREx, CDANN for conditional invariant
features, HCVP achieves significant gains, with an average improvement of 3.8%
in overall performance across five datasets. This underscores the fact that,
although several nuanced constraints are proposed for invariant features, a
single set of model parameters inadvertently incorporates domain-specific
features, thereby limiting the model’s generalization ability. In contrast,
our HCVP, which augments domain-level and task-specific characteristics to
guide the model to learn invariant features, exhibits robust performance
across diverse distribution shifts.

TABLE II: Performance comparison on PACS dataset. The average accuracy across all domains is reported. “$\rightarrow$” denotes the unseen domain.  Methods |  $\rightarrow$ Art |  $\rightarrow$ Cartoon |  $\rightarrow$ Photo |  $\rightarrow$ Sketch | Avg
---|---|---|---|---|---
| DG Algorithms
ERM | 94.85 | 87.56 | 99.68 | 75.13 | 89.31
IRM | 91.11 | 76.69 | 99.35 | 64.33 | 82.87
MLDG | 95.08 | 87.76 | 99.38 | 77.30 | 89.88
MMD | 93.35 | 86.55 | 99.70 | 73.91 | 88.38
DANN | 89.20 | 81.79 | 99.00 | 66.60 | 84.15
CDANN | 89.73 | 81.59 | 98.80 | 66.96 | 84.27
VREx | 92.68 | 84.15 | 99.45 | 72.81 | 87.27
MTL | 92.48 | 84.90 | 99.55 | 71.68 | 87.15
SagNet | 90.10 | 80.95 | 99.48 | 63.50 | 83.51
RSC | 95.28 | 87.42 | 99.68 | 76.83 | 89.80
IB_ERM | 89.69 | 78.00 | 99.43 | 61.58 | 82.18
IB_IRM | 90.10 | 78.27 | 98.73 | 58.51 | 81.40
SWAD | 93.23 | 85.93 | 99.18 | 82.03 | 90.10
MIRO | 83.41 | 78.95 | 93.19 | 78.91 | 83.61
POEM | 86.21 | 79.74 | 97.16 | 74.17 | 84.32
| Tuning Protocols
Full | 94.85 | 87.56 | 99.68 | 75.13 | 89.31
Linear Probing | 84.75 | 68.85 | 92.54 | 40.41 | 71.64
Partial_1 | 87.80 | 70.52 | 98.03 | 41.79 | 74.54
Partial_2 | 89.77 | 73.93 | 98.30 | 44.88 | 76.72
Partial_4 | 91.64 | 77.90 | 99.00 | 53.41 | 80.49
DoPrompt | 95.04 | 86.35 | 99.63 | 78.20 | 89.81
| Our Method
HCVP | 93.17 | 86.89 | 99.33 | 81.30 | 90.17
TABLE III: Performance comparison on VLCS dataset. The average accuracy across all domains is reported. “$\rightarrow$” denotes the unseen domain. Methods |  $\rightarrow$ Caltech101 |  $\rightarrow$ LabelMe |  $\rightarrow$ SUN09 |  $\rightarrow$ VOC2007 | Avg
---|---|---|---|---|---
| DG Algorithms
ERM | 96.50 | 65.54 | 78.32 | 79.76 | 80.03
IRM | 97.67 | 63.81 | 73.86 | 77.11 | 78.11
MLDG | 96.82 | 64.52 | 78.47 | 79.55 | 79.84
MMD | 96.70 | 66.37 | 78.23 | 78.37 | 79.92
DANN | 94.05 | 65.68 | 76.96 | 79.33 | 79.01
CDANN | 95.17 | 65.05 | 75.97 | 77.32 | 78.38
VREx | 96.55 | 65.40 | 76.29 | 79.38 | 79.41
MTL | 96.76 | 65.43 | 78.93 | 79.29 | 80.10
SagNet | 97.44 | 64.96 | 77.99 | 80.16 | 80.14
RSC | 96.41 | 65.27 | 78.10 | 79.93 | 79.93
IB_ERM | 97.59 | 62.76 | 75.41 | 78.22 | 78.50
IB_IRM | 98.47 | 64.83 | 74.51 | 75.27 | 78.27
SWAD | 98.49 | 63.86 | 75.40 | 79.49 | 79.31
MIRO | 98.85 | 63.44 | 69.31 | 78.08 | 77.42
POEM | 96.91 | 58.40 | 73.31 | 75.60 | 76.05
| Tuning Protocols
Full | 96.50 | 65.54 | 78.32 | 79.76 | 80.03
Linear Probing | 98.70 | 65.33 | 71.71 | 71.79 | 76.88
Partial_1 | 98.97 | 65.10 | 76.25 | 77.26 | 79.40
Partial_2 | 98.41 | 64.44 | 77.55 | 79.07 | 79.87
Partial_4 | 98.03 | 64.44 | 78.26 | 80.33 | 80.27
DoPrompt | 96.70 | 66.53 | 78.28 | 79.39 | 80.23
| Our Method
HCVP | 96.32 | 66.26 | 80.08 | 81.65 | 81.08
TABLE IV: Performance comparison on OfficeHome dataset. The average accuracy across all domains is reported. “$\rightarrow$” denotes the unseen domain. Methods |  $\rightarrow$ Art |  $\rightarrow$ Clipart |  $\rightarrow$ Product |  $\rightarrow$ Real | Avg
---|---|---|---|---|---
| DG Algorithms
ERM | 81.38 | 69.78 | 88.72 | 90.24 | 82.53
IRM | 71.88 | 58.39 | 80.97 | 82.99 | 73.56
MLDG | 80.71 | 70.56 | 88.25 | 90.50 | 82.51
MMD | 80.45 | 68.92 | 88.96 | 90.19 | 82.13
DANN | 78.39 | 66.28 | 87.09 | 88.80 | 80.14
CDANN | 78.77 | 65.85 | 87.63 | 89.22 | 80.37
VREx | 79.09 | 65.56 | 87.33 | 89.36 | 80.34
MTL | 79.71 | 66.51 | 87.37 | 89.18 | 80.69
SagNet | 74.34 | 62.55 | 85.64 | 88.03 | 77.64
RSC | 80.50 | 70.12 | 89.02 | 90.45 | 82.52
IB_ERM | 69.91 | 56.55 | 78.46 | 81.47 | 71.60
IB_IRM | 56.56 | 47.22 | 61.42 | 68.23 | 58.36
SWAD | 76.26 | 68.87 | 86.74 | 87.03 | 79.73
MIRO | 61.48 | 50.03 | 75.56 | 76.91 | 66.00
POEM | 70.13 | 59.88 | 80.41 | 82.59 | 73.25
| Tuning Protocols
Full | 81.38 | 69.78 | 88.72 | 90.24 | 82.53
Linear Probing | 75.49 | 56.04 | 83.59 | 87.45 | 75.64
Partial_1 | 75.58 | 56.55 | 83.92 | 88.33 | 76.10
Partial_2 | 76.35 | 58.33 | 84.37 | 88.20 | 76.81
Partial_4 | 77.14 | 60.20 | 84.88 | 88.42 | 77.66
DoPrompt | 80.95 | 70.88 | 88.94 | 90.10 | 82.72
| Our Method
HCVP | 81.77 | 69.76 | 88.01 | 90.62 | 82.54

####  V-B2 Tuning Protocols

We outline tuning protocols for adapting visual foundation models to
distribution shifts in downstream tasks. As shown in Table I, full fine-
tuning, akin to ERM, yields a robust average accuracy of 78.2%. In contrast,
less computationally intensive methods such as linear probing and gradual
unfreezing (Partial_1, Partial_2, Partial_4) achieve relatively lower average
accuracies, almost all falling below 72%. This trend clearly illustrates a
correlation between the number of tuned parameters and the average performance
under distribution shifts. In this context, our HCVP integrates domain and
task-specific information into the model, by augmenting visual prompts to the
ViT backbone. This enhancement not only aligns with the observed trend of
increased performance with broader parameter tuning but also contributes to
the model’s robustness. For example, compared to the best tuning protocol—full
fine-tuning—our HCVP outperforms it on all five datasets. This advantage
becomes particularly pronounced when facing substantial distribution shifts.
For example, when adapting to the sketch domain on the PACS dataset, denoted
by "$\rightarrow$ Sketch ", the performance is 81.30% compared to 75.13%, as
shown in Table II. Similarly, when encountering spurious distribution shifts
on the CelebA dataset, the performance is 86.6% versus 84.6%.

####  V-B3 Significance of Incremental Gains in DG

In the field of DG, even marginal performance improvements carry considerable
significance. This is largely attributed to the inherent complexity and the
unpredictable nature of generalizing effectively to unseen domains. The
previous large-scale evaluations [13, 32, 14] indicate that no single DG
algorithm consistently outperforms over ERM across all benchmark datasets.
However, our HCVP method demonstrates this remarkable capability. Therefore,
the incremental gains, such as the 1% improvement reported by HCVP, should be
recognized for their substantial value.

TABLE V: Performance comparison on TerraIncognita dataset. The average accuracy across all domains is reported. “$\rightarrow$” denotes the unseen domain. Methods |  $\rightarrow$ L100 |  $\rightarrow$ L38 |  $\rightarrow$ L43 |  $\rightarrow$ L46 | Avg
---|---|---|---|---|---
| DG Algorithms
ERM | 63.01 | 46.28 | 62.15 | 47.50 | 54.74
IRM | 36.07 | 43.43 | 39.04 | 36.92 | 38.87
MLDG | 60.37 | 42.74 | 61.40 | 47.10 | 52.90
MMD | 62.09 | 43.64 | 59.04 | 47.33 | 53.03
DANN | 52.31 | 42.52 | 55.67 | 39.77 | 47.57
CDANN | 56.31 | 39.59 | 56.05 | 45.74 | 49.42
VREx | 58.19 | 47.03 | 59.48 | 47.50 | 53.05
MTL | 60.01 | 47.12 | 61.24 | 51.80 | 55.04
SagNet | 58.42 | 41.94 | 54.28 | 38.64 | 48.32
RSC | 65.70 | 43.78 | 62.50 | 50.18 | 55.54
IB_ERM | 32.82 | 21.21 | 41.72 | 38.64 | 33.60
IB_IRM | 38.15 | 35.27 | 40.43 | 36.90 | 37.69
SWAD | 59.29 | 47.84 | 61.97 | 42.75 | 52.96
MIRO | 61.80 | 46.55 | 54.72 | 27.90 | 47.74
POEM | 48.75 | 27.54 | 53.40 | 27.43 | 39.28
| Tuning Protocols
Full | 63.01 | 46.28 | 62.15 | 47.50 | 54.74
Linear Probing | 30.64 | 35.13 | 41.53 | 33.38 | 35.17
Partial_1 | 35.35 | 31.16 | 43.26 | 36.82 | 36.65
Partial_2 | 39.15 | 26.36 | 46.00 | 41.96 | 38.37
Partial_4 | 43.11 | 26.33 | 44.46 | 41.45 | 38.84
DoPrompt | 64.51 | 40.93 | 63.00 | 49.37 | 54.45
| Our Method
HCVP | 62.06 | 51.84 | 58.91 | 47.61 | 55.11
Figure 3: Comparison of inter-domain feature distances for ERM and HCVP across
multiple datasets. It illustrates the effectiveness of HCVP in achieving lower
inter-domain distances compared to ERM, suggesting a stronger capability for
domain-invariant feature learning.

###  V-C Quantifying Domain-Invariance

We initially explored the possibility of directly measuring the conditional
mutual information $I(Z,D|Y)$ to assess our model’s capacity to learn
invariant features [54]. This measurement is consistent with Eq. 2 to evaluate
the hypothesis that the incorporation of $P$ leads to the reduction of
$I(Z,D|Y)$. However, given the high-dimensional nature of the latent
representation $Z$, we encountered significant computational challenges in
reliably estimating this metric. To circumvent this issue, we opted to measure
inter-domain feature distances as an alternative metric. This approach is
grounded in the theory that lower distances across domains indicate more
domain-invariant feature learning [75, 7]. In essence, closer feature
representations across domains suggest that the model is not overfitting to
domain-specific characteristics and is, instead, capturing more generalizable
aspects of the data. Specifically, we regard the last domain in each dataset
as the unseen domain. We then train both ERM and HCVP models on this unseen
domain and save the iterations of the models that demonstrate the best
performance. Using the saved best model, we calculate the average inter-domain
distance by utilizing the latent representations generated by the models.

As shown in Figure 3, our HCVP consistently achieves smaller inter-domain
distances compared to the ERM baseline across all evaluated datasets. This
observation, together with our main results on the last unseen domain shown in
Tables II, III, IV, and V, strongly indicates that HCVP is effective in
extracting domain-invariant features. While inter-domain distances offer a
more indirect measure of domain invariance compared to mutual information,
their consistency with our primary results lends feasibility and reliability
to this approach.

Figure 4: The t-SNE visualizations of visual features in our HCVP, for the
last unseen domain on PACS, VLCS, OfficeHome, CelebA, and TerraIncognita
datasets. Forty instances are sampled within each class. Additionally, we
select the first ten classes in the OfficeHome dataset.

###  V-D Ablation Study

In this section, we perfrom ablation study to evaluate the individual
contribution of each loss in our HCVP model. As shown in Table VI, where “P,
V, O, T, C” represents PACS, VLCS, OfficeHome, TerraIncognita and CelebA
respectively, the complete model "HCVP (full)" achieves the highest accuracy
on each dataset and the best agerage accuracy of 79.10. Removing each loss
sequentially leads to "HCVP w/o $\mathcal{L}_{PCL}$" and "HCVP w/o
$\mathcal{L}_{CCI}$", which exhibit slightly reduced accuracies of 78.59 and
78.21, respectively. This underscores the importance of these losses to the
model’s performance. Removing both losses results in "HCVP w/o
$\mathcal{L}_{CCI}$ and $\mathcal{L}_{PCL}$", further reducing the accuracy to
77.69.

TABLE VI:  This ablation study evaluates the individual contributions of various losses in the HCVP model across four datasets. "HCVP (full)" refers to the complete model. We sequentially remove each loss to create "HCVP w/o $\mathcal{L}_{PCL}$" and "HCVP w/o $\mathcal{L}_{CCI}$". Removing both losses results in "HCVP w/o $\mathcal{L}_{CCI}$ and $\mathcal{L}_{PCL}$", the vanilla HCVP variant with only the classification loss $\mathcal{L}_{cls}$. Values are shown with two decimal place; columns P, V, O, T, C correspond to the PACS, VLCS, OfficeHome, TerraIncognita, and CelebA datasets.  Methods | P | V | O | T | C | Avg
---|---|---|---|---|---|---
HCVP w/o $\mathcal{L}_{CCI}$ and $\mathcal{L}_{PCL}$ | 89.43 | 80.30 | 82.16 | 51.05 | 85.50 | 77.69
HCVP w/o $\mathcal{L}_{CCI}$ | 89.97 | 79.73 | 82.19 | 53.42 | 85.76 | 78.21
HCVP w/o $\mathcal{L}_{PCL}$ | 89.67 | 80.51 | 82.52 | 54.21 | 86.02 | 78.59
HCVP (full) | 90.17 | 81.08 | 82.54 | 55.11 | 86.60 | 79.10

###  V-E Quanlitative Evaluations

Figure 5: The t-SNE visualizations of domain-level and task-specific prompts
on the PACS dataset, representing four domains and seven class labels. Thirty
instances are sampled within each class for visualization.

####  V-E1 The t-SNE Visualizations of Visual Features

Figure 4 illustrates the t-SNE [76] visualizations of visual features in our
HCVP, for the last unseen domain on the PACS, VLCS, OfficeHome, CelebA and
TerraIncognita datasets. Overall, the visual embeddings of HCVP are clearly
clustered in accordance with their class labels across five datasets. This
clear clustering is indicative of the robustness and effectiveness of our HCVP
model in capturing invariant features for task prediction. In generalizing to
the last unseen domain on PACS, namely “Sketch”, where the test scenario may
diverge largely from our pretraining and training features in “Art, Photo,
Cartoon”, an overlap between several classes, such as dogs with elephants and
horses, is observed in Figure 4 (a). Despite this, our HCVP still shows the
best generalization ability, as evidenced in Tables I and II.

####  V-E2 The t-SNE Visualizations of Visual Prompts

To investigate the relationship between visual prompts with domain and task
specific characteristics, we also leverage t-SNE visualizations, as visual
prompts typically lack clear meanings [25, 26]. As illustrated in Figure 5,
our visualization study on the PACS dataset reveals: (a) domain prompt vectors
corresponding to the same domains are clustered together, these vectors are
distinctly separate from other domains, validating that they carry domain-
level characteristics; (b) task prompts are similarly clustered by specific
task labels. These findings collectively affirm that the visual prompts
generated by our method effectively capture domain-level and task-specific
characteristics, contributing to HCVP’s superior performance in DG tasks.

###  V-F Hyperparameter Tuning

We investigate the effects of two loss weights, $\lambda_{PCL}$ and
$\lambda_{CCI}$, on the VLCS and OfficeHome datasets. We explore a wide range
of values for both weights, specifically $\\{$0.001, 0.01, 0.1, 0.5, 1.0$\\}$
for $\lambda_{PCL}$ and $\\{$0.01, 0.1, 0.3, 0.6, 1.0$\\}$ for
$\lambda_{CCI}$. Given that the original value of $\mathcal{L}_{PCL}$ is
almost four times larger than $\mathcal{L}_{\text{cls}}$, a large weight for
$\lambda_{PCL}$ could impact classification. As depicted in Figure 6, when
$\lambda_{PCL}$ is set to 0.1, our HCVP achieves optimal performance.
Additionally, we find that 1.0 serves as an effective weight for
$\mathcal{L}_{CCI}$.

Figure 6: Analysis of the effects of two loss weights, $\lambda_{PCL}$ and
$\lambda_{CCI}$, on the VLCS and OfficeHome datasets.

##  VI Conclusion

In this work, we introduced the Hierarchical Contrastive Visual Prompt (HCVP)
methodology, a novel approach to DG that effectively separates invariant
features from domain-specific aspects. By integrating domain-level and task-
specific characteristics through a two-tier hierarchical prompt generation
network coupled with prompt contrastive learning, HCVP enhances adaptability
to unseen domains. Extensive experiments on five benchmark datasets
demonstrate the superiority of HCVP over state-of-the-art DG algorithms and
adaptation protocols. Despite its successes, HCVP may not achieve optimal
performance across all unseen domains, owing to the inherent complexity of
these domains. The method’s reliance on extracting solely domain-level and
task-specific characteristics from visual information may prove insufficient
in certain contexts, thereby inspiring consideration for the incorporation of
cross-modal information. Overall, we believe that this work marks a meaningful
advancement in the field of DG, offering a more nuanced approach for learning
invariant features.

## References

  * [1] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, “Generalizing to unseen domains: A survey on domain generalization,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [2] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [3] G. Zhou, S. Xie, G. Hao, S. Chen, B. Huang, X. Xu, C. Wang, L. Zhu, L. Yao, and K. Zhang, “Emerging synergies in causality and deep generative models: A survey,” 2023.
  * [4] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [5] V. Piratla, P. Netrapalli, and S. Sarawagi, “Efficient domain generalization via common-specific low-rank decomposition,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 7728–7738.
  * [6] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” _Nature Machine Intelligence_ , vol. 2, no. 11, pp. 665–673, 2020.
  * [7] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [8] M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters, “Invariant models for causal transfer learning,” _The Journal of Machine Learning Research_ , vol. 19, no. 1, pp. 1309–1342, 2018.
  * [9] X. Sun, B. Wu, X. Zheng, C. Liu, W. Chen, T. Qin, and T.-Y. Liu, “Recovering latent causal factor for generalization to distributional shifts,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 16 846–16 859, 2021.
  * [10] R. Christiansen, N. Pfister, M. E. Jakobsen, N. Gnecco, and J. Peters, “A causal framework for distribution generalization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 10, pp. 6614–6630, 2021.
  * [11] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao, “Deep domain generalization via conditional invariant adversarial networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 624–639.
  * [12] X. Wang, M. Saxon, J. Li, H. Zhang, K. Zhang, and W. Y. Wang, “Causal balancing for domain generalization,” in _The Eleventh International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=F91SROvVJ_6
  * [13] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in _International Conference on Learning Representations_ , 2020.
  * [14] O. Wiles, S. Gowal, F. Stimberg, S. Alvise-Rebuffi, I. Ktena, K. Dvijotham, and T. Cemgil, “A fine-grained analysis on distribution shift,” _arXiv preprint arXiv:2110.11328_ , 2021.
  * [15] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [17] C. Zhang, M. Zhang, S. Zhang, D. Jin, Q. Zhou, Z. Cai, H. Zhao, X. Liu, and Z. Liu, “Delving deep into the generalization of vision transformers under distribution shifts,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7277–7286.
  * [18] Y. Chen, T. Hu, F. Zhou, Z. Li, and Z.-M. Ma, “Explore and exploit the diverse knowledge in model zoo for domain generalization,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 4623–4640.
  * [19] A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” _arXiv preprint arXiv:2202.10054_ , 2022.
  * [20] B. Li, Y. Shen, J. Yang, Y. Wang, J. Ren, T. Che, J. Zhang, and Z. Liu, “Sparse mixture-of-experts are domain generalizable learners,” _arXiv preprint arXiv:2206.04046_ , 2022.
  * [21] A. Ramé, K. Ahuja, J. Zhang, M. Cord, L. Bottou, and D. Lopez-Paz, “Model ratatouille: Recycling diverse models for out-of-distribution generalization,” 2023.
  * [22] S. S. Du, W. Hu, S. M. Kakade, J. D. Lee, and Q. Lei, “Few-shot learning via learning the representation, provably,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=pW2Q2xLwIMD
  * [23] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn, “Surgical fine-tuning improves adaptation to distribution shifts,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [25] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” _International Journal of Computer Vision_ , vol. 130, no. 9, pp. 2337–2348, 2022.
  * [26] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in _European Conference on Computer Vision_. Springer, 2022, pp. 709–727.
  * [27] Z. Zheng, X. Yue, K. Wang, and Y. You, “Prompt vision transformer for domain generalization,” _arXiv preprint arXiv:2208.08914_ , 2022.
  * [28] A. Li, L. Zhuang, S. Fan, and S. Wang, “Learning common and specific visual prompts for domain generalization,” in _Proceedings of the Asian Conference on Computer Vision_ , 2022, pp. 4260–4275.
  * [29] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [30] G. Zhou, C. Huang, X. Chen, X. Xu, C. Wang, L. Zhu, and L. Yao, “Contrastive counterfactual learning for causality-aware interpretable recommender systems,” 2023.
  * [31] X. Li and L. Yao, “Contrastive individual treatment effects estimation,” in _2022 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2022, pp. 1053–1058.
  * [32] N. Ye, K. Li, H. Bai, R. Yu, L. Hong, F. Zhou, Z. Li, and J. Zhu, “Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7947–7958.
  * [33] X. Yue, Y. Zhang, S. Zhao, A. Sangiovanni-Vincentelli, K. Keutzer, and B. Gong, “Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2100–2110.
  * [34] F. Qiao, L. Zhao, and X. Peng, “Learning to learn single domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 556–12 565.
  * [35] S.-Y. Jo and S. W. Yoon, “Poem: Polarization of embeddings for domain-invariant representations,” _arXiv preprint arXiv:2305.13046_ , 2023.
  * [36] J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park, “Swad: Domain generalization by seeking flat minima,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 22 405–22 418, 2021.
  * [37] J. Cha, K. Lee, S. Park, and S. Chun, “Domain generalization by mutual-information regularization with pre-trained models,” in _European Conference on Computer Vision_. Springer, 2022, pp. 440–457.
  * [38] Y. Liu, Z. Xiong, Y. Li, X. Tian, and Z.-J. Zha, “Domain generalization via encoding and resampling in a unified latent space,” _IEEE Transactions on Multimedia_ , 2021.
  * [39] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Style normalization and restitution for domain generalization and adaptation,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 3636–3651, 2021.
  * [40] X. Ma, T. Zhang, and C. Xu, “Deep multi-modality adversarial networks for unsupervised domain adaptation,” _IEEE Transactions on Multimedia_ , vol. 21, no. 9, pp. 2419–2431, 2019.
  * [41] Y. Luo, G. Kang, K. Liu, F. Zhuang, and J. Lü, “Taking a closer look at factor disentanglement: Dual-path variational autoencoder learning for domain generalization,” _IEEE Transactions on Multimedia_ , 2023.
  * [42] Z. Niu, J. Yuan, X. Ma, Y. Xu, J. Liu, Y.-W. Chen, R. Tong, and L. Lin, “Knowledge distillation-based domain-invariant representation learning for domain generalization,” _IEEE Transactions on Multimedia_ , 2023.
  * [43] H. Li, S. J. Pan, S. Wang, and A. C. Kot, “Domain generalization with adversarial feature learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5400–5409.
  * [44] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [45] S. Wu, H. R. Zhang, and C. Ré, “Understanding and improving information transfer in multi-task learning,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=SylzhkBtDB
  * [46] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16 000–16 009.
  * [47] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” _arXiv preprint arXiv:2101.00190_ , 2021.
  * [48] X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” _ArXiv_ , vol. abs/2110.07602, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:238857040
  * [49] A. Chen, Y. Yao, P.-Y. Chen, Y. Zhang, and S. Liu, “Understanding and improving visual prompting: A label-mapping perspective,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 19 133–19 143.
  * [50] K. Sohn, Y. Hao, J. Lezama, L. F. Polanía, H. Chang, H. Zhang, I. Essa, and L. Jiang, “Visual prompt tuning for generative transfer learning,” _ArXiv_ , vol. abs/2210.00990, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252683603
  * [51] R. Das, Y. Dukler, A. Ravichandran, and A. Swaminathan, “Learning expressive prompting with residuals for vision transformers,” _ArXiv_ , vol. abs/2303.15591, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257771595
  * [52] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _arXiv preprint physics/0004057_ , 2000.
  * [53] N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in _2015 ieee information theory workshop (itw)_. IEEE, 2015, pp. 1–5.
  * [54] B. Li, Y. Shen, Y. Wang, W. Zhu, D. Li, K. Keutzer, and H. Zhao, “Invariant information bottleneck for domain generalization,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 7, 2022, pp. 7399–7407.
  * [55] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [56] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [57] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _International Conference on Learning Representations (ICLR)_ , 2015.
  * [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [59] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5542–5550.
  * [60] C. Fang, Y. Xu, and D. N. Rockmore, “Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 1657–1664.
  * [61] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, “Deep hashing network for unsupervised domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5018–5027.
  * [62] S. Beery, G. Van Horn, and P. Perona, “Recognition in terra incognita,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 456–473.
  * [63] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * [64] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, “Distributionally robust neural networks,” in _International Conference on Learning Representations_ , 2019.
  * [65] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” _arXiv preprint arXiv:1907.02893_ , 2019.
  * [66] V. Vapnik, _Statistical Learning Theory_. New York: Wiley, 1998.
  * [67] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, 2018.
  * [68] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, “Out-of-distribution generalization via risk extrapolation (rex),” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5815–5826.
  * [69] G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott, “Domain generalization by marginal transfer learning,” _Journal of machine learning research_ , vol. 22, no. 2, pp. 1–55, 2021.
  * [70] H. Nam, H. Lee, J. Park, W. Yoon, and D. Yoo, “Reducing domain gap via style-agnostic networks,” _arXiv preprint arXiv:1910.11645_ , vol. 2, no. 7, p. 8, 2019.
  * [71] Z. Huang, H. Wang, E. P. Xing, and D. Huang, “Self-challenging improves cross-domain generalization,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 124–140.
  * [72] K. Ahuja, E. Caballero, D. Zhang, J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish, “Invariance principle meets information bottleneck for out-of-distribution generalization,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 3438–3450, 2021.
  * [73] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [74] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [75] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” _Machine learning_ , vol. 79, pp. 151–175, 2010.
  * [76] L. van der Maaten and G. E. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, pp. 2579–2605, 2008. [Online]. Available: https://api.semanticscholar.org/CorpusID:5855042