WGMwgm QEqe EPep PMSpms BECbec DEde

[]

1]organization=University of Victoria, addressline=800 Finnerty Road,
city=Victoria, postcode=V8P 5C2, state=BC, country=Canada

[]

[1]

[cor1]Corresponding author

#  Vision Transformers in Domain Adaptation and Generalization: A Study of
Robustness

Shadi Alijani<EMAIL_ADDRESS>[  Jamil Fayyad<EMAIL_ADDRESS>Homayoun
Najjaran<EMAIL_ADDRESS>
###### Abstract

Deep learning models are often evaluated in scenarios where the data
distribution is different from those used in the training and validation
phases. The discrepancy presents a challenge for accurately predicting the
performance of models once deployed on the target distribution. Domain
adaptation and generalization are widely recognized as effective strategies
for addressing such shifts, thereby ensuring reliable performance. The recent
promising results in applying vision transformers in computer vision tasks,
coupled with advancements in self-attention mechanisms, have demonstrated
their significant potential for robustness and generalization in handling
distribution shifts. Motivated by the increased interest from the research
community, our paper investigates the deployment of vision transformers in
domain adaptation and domain generalization scenarios. For domain adaptation
methods, we categorize research into feature-level, instance-level, model-
level adaptations, and hybrid approaches, along with other categorizations
with respect to diverse strategies for enhancing domain adaptation. Similarly,
for domain generalization, we categorize research into multi-domain learning,
meta-learning, regularization techniques, and data augmentation strategies. We
further classify diverse strategies in research, underscoring the various
approaches researchers have taken to address distribution shifts by
integrating vision transformers. The inclusion of comprehensive tables
summarizing these categories is a distinct feature of our work, offering
valuable insights for researchers. These findings highlight the versatility of
vision transformers in managing distribution shifts, crucial for real-world
applications, especially in critical safety and decision-making scenarios.

###### keywords:

Vision Transformers Domain Adaptation Domain Generalization Distribution
Shifts

##  1 Introduction

Convolutional Neural Networks (CNNs) are a cornerstone of computer vision
algorithms, largely owing to their proficiency in managing spatial
relationships and maintaining invariance to input translations. Their
widespread success in object recognition tasks can be attributed to
advantageous inductive biases, such as translation equivalence, which enable
them to effectively identify and process visual patterns. The use of
convolutions in neural networks for vision-related tasks, a methodology
initiated with the early implementations [1, 2], and was greatly popularized
by the work of Krizhevsky et al. [3], has led to the scenario where nearly
most vision networks rely on convolutional layers [4, 5, 6, 7, 8]. These
convolutional layers have been further improved with innovations such as
residual connections [5]. Extensive use has led to detailed empirical [9] and
analytical evaluations of convolutional networks [10, 11].

Recent advancements, however, have shown the potential of transformers
regarding to their self-attention mechanisms that find the global features of
the data which provide a more holistic view of the data, reduce inductive
bias, and exhibit a high degree of scalability and flexibility. These factors
collectively enhance the model’s ability to generalize better during testing.
After the tremendous success in Natural Language Processing (NLP) tasks [12,
13, 14], transformers are now being actively integrated into computer vision
tasks [15, 16, 17, 18, 19]. Attention-based architectures known as
transformers, gaining traction in both pure vision and vision-language tasks
[20, 21, 22, 23, 24, 25]. Vision Transformers (ViT) [16], stands out as a key
development in this area, applying a self-attention-based mechanism to
sequences of image patches. It achieves competitive performance on the
challenging ImageNet classification task [26], compared to CNNs. Researchers
discovered that existing CNN architectures exhibit limited generalization
capabilities when confronted with distribution shift scenarios [27, 28].
Subsequent research, as seen in works like [29, 30], has further expanded the
capabilities of transformers, demonstrating impressive performance across
various visual benchmarks. This includes applications in different benchmarks
including COCO (Common Objects in Context) dataset in object detection and
instance segmentation [31], as well as ADE20K dataset for semantic
segmentation [32].

As ViTs gain popularity, it becomes crucial to examine the characteristics of
the representations they learn [33]. This is important in areas like
autonomous driving [34, 35], robotics [36], and healthcare [37], where the
trustworthy and reliability of these systems are crucial. Recent studies delve
into evaluating ViTs’ robustness, focusing not just on standard metrics like
accuracy and computational cost, but also on their intrinsic impact on model
robustness and generalization, especially in handling Distribution Shifts. In
conventional training and testing scenarios, it is assumed that data are
independent and identically distributed (IID). However, this assumption often
doesn’t reflect real-world scenarios. Therefore, exploring the potential of
ViTs as the modern vision networks, to adapt to target domains, and generalize
and perform well on unseen data, becomes a crucial aspect of machine learning
models [38].

Recognizing the unique capabilities of ViTs in modern vision tasks highlights
the need to assess their performance across varied conditions. In traditional
deep learning training and testing scenarios, there is a common assumption
that the data are independent and identically distributed (IID). However, any
shift in data distribution or domain after training can reduce testing
performance [39, 40, 41]. Such IID assumptions often fall short in real-world
scenarios, where distribution shifts are prevalent. Thus, the ability of deep
learning models to generalize and retain performance across different test
domains is crucial for determining their effectiveness [38, 42]. Exploring the
adaptability of ViTs necessitates revisiting Domain Adaptation (DA) and Domain
Generalization (DG), which are fundamental strategies in machine learning
aimed at addressing the challenges posed by distribution shifts between
training and testing data, especially when these shifts are pronounced [43].
Although DA and DG have been traditionally used to overcome such challenges,
applying these strategies within the advanced framework of ViTs offers a new
approach to examine how these innovative models manage and excel distribution
shifts. DA, which provides access to the target domain, and DG, where the
target domain remains unseen, represent two approaches to this issue. DA seeks
to minimize the discrepancy between specific source and target domains, while
DG strives to create a model that remains effective across various unseen
domains by utilizing the diversity of multiple source domains during training.
This often includes the development of domain-agnostic features that are
effective across various domains [44, 45].

Building on this foundation, research efforts have been directed at enhancing
ViTs’ generalization capabilities through various methodologies, delving into
both DA and DG strategies, and integrating ViTs into the broader deep learning
framework. Comparative analyses of ViTs and high-performing CNNs reveal
distinct advantages attributable to ViTs. A key differentiation is the dynamic
nature of weight computation in ViTs through the self-attention mechanism,
contrasting with the static weights learned by CNNs during training. This
attribute provides ViTs with a more flexible and adaptable ability [46]. ViTs
employ multi-head self-attention to intricately parse and interpret contextual
information within images, thereby excelling in scenarios involving
occlusions, domain variations, and perturbations. They demonstrate remarkable
robustness, effectively maintaining accuracy despite image modifications.
Furthermore, ViTs exhibit a reduced texture bias compared to CNNs, favoring
shape recognition which aligns more closely with human visual processing. This
proficiency in discerning overall shapes facilitates accurate image
categorization without relying on detailed, pixel-level analysis. ViTs’
ability to merge various features for image classification enhances their
performance across diverse datasets, proving advantageous in both conventional
and few-shot learning settings where the model is trained with only a few
examples [33, 47, 48]. Refer to Figure 1 for the challenges prevalent in
images. These challenges include severe occlusions, adversarial perturbations,
patch permutations, and domain shifts effectively addressed through the
flexibility and dynamic receptive fields of self-attention mechanisms [33].
Unlike CNNs, which primarily focus on texture [49], ViTs concentrate more on
object shape, enhancing their resistance to texture shifts and benefiting
shape recognition, and they are adept at propagating spatial information, an
advantage for tasks such as detection and segmentation [33, 48].

Figure 1: Various factors that affect the robustness of deep learning models
include: (a) displaying the original image, followed by (b) severe occlusions,
(c) adversarial perturbations, (d) patch permutations, and (e) distributional
shifts, such as stylization to remove texture cues.

In our comprehensive review, the first of its kind to integrate ViTs for
Domain Adaptation (DA) and Domain Generalization (DG) scenarios, we have
meticulously examined how ViTs adapt to distribution shifts. This study delves
into the fundamentals, architecture, and key components of ViTs, offering a
unique categorization and analysis of their role in domain adaptation and
generalization. Within the context of DA, we cover feature-level, instance-
level, model-level, and hybrid approaches, alongside a variety of strategies
to enhance domain adaptation through ViTs. For DG, our exploration includes
multi-domain learning, meta-learning approaches, regularization techniques,
data augmentation strategies, and further methods to boost domain
generalization. A significant portion of our review is dedicated to showcasing
the applications of ViTs beyond image recognition, such as semantic
segmentation, action recognition, face analysis, medical imaging, and other
emerging fields. This broad spectrum of applications highlights the
versatility and potential of ViTs across the vast landscape of computer
vision. In our discussion section, we delve into the initial development
challenges associated with ViTs, aiming to equip researchers with insightful
findings that could steer the direction of future investigations in this area.
Furthermore, we outline prospective research paths.

As the field of computer vision advances, especially with the introduction of
ViTs, this survey emerges as a crucial tool for researchers and practitioners
alike. We aspire that our findings will stimulate further investigation and
innovation in leveraging ViTs for Domain Adaptation and Generalization,
thereby overcoming current hurdles and paving new paths in this ever-evolving
research domain.

The structure of this paper is as follows: Section 2 introduces the
fundamentals and architecture of ViTs. Section 3 assesses the capacity of ViTs
to manage distribution shifts, including DA and DG. Section 4 explores various
applications of ViTs in computer vision beyond image recognition, particularly
their adaptability to distribution shifts. Section 5 wraps up with a
comprehensive discussion and conclusion, also suggesting future research
directions.

##  2 Vision Transformers: Fundamentals and Architecture

The transformer model, initially applied in the field of natural language
processing (NLP) for machine translation tasks [12], consists of an encoder
and a decoder. Both the encoder and the decoder are composed of multiple
transformer blocks, each having an identical architecture. Figure 2
illustrates the basic configuration of ViTs. The encoder is responsible for
generating encodings of the input. In contrast, the decoder leverages the
contextual information embedded within these encodings to generate the output
sequence. Each transformer block within the model encompasses several
components: a multi-head attention layer, a feed-forward neural network,
residual connections, and layer normalization.

Figure 2: (a): An image is divided into fixed-size patches, each is embedded
linearly, and position embeddings are added. The sequence of vectors produced
is then fed into a standard Transformer encoder. For classification purposes,
an additional learnable classification token is incorporated into the
sequence. (b): The Transformer’s architecture is characterized by the use of
stacked self-attention and point-wise, fully connected layers within both its
encoder and decoder components, as depicted in the left and right sections of
the figure, respectively.

###  2.1 Overview of the Vision Transformers Architecture

The advancements in basic transformer models are largely due to their two main
components. The first is the self-attention mechanism, which excels in
capturing long-range dependencies among sequence elements. This surpasses the
limitations of traditional recurrent models in encoding such relationships.
The second key component is the transformer encoder layers. These layers are
pivotal in hierarchical representation learning within transformer models, as
they integrate self-attention with feed-forward networks. This integration
enables effective feature extraction and information propagation throughout
the model [18, 50, 16].

Self-attention mechanism: The self-attention mechanism assesses the importance
or relevance of each patch in a sequence in relation to others. For instance,
in language processing, it can identify words that are likely to co-occur in a
sentence. As a fundamental part of transformers, self-attention captures the
interactions among all elements in a sequence, which is especially beneficial
for tasks that involve structured predictions [18]. A self-attention layer
updates each sequence element by aggregating information from the entire input
sequence.

Let’s denote a sequence of $N$ entities $(x_{1},x_{2},\dots,x_{n})$ by
$\mathbf{X}\in{R}^{n\times d}$, where $d$ is the embedding dimension for each
entity. The goal of self-attention is to capture the relationships among all
the entities by encoding each entity based on the overall contextual
information. To achieve this, it employs three learnable weight matrices:
Queries $\left(\mathbf{W}^{Q}\in{R}^{d\times d_{q}}\right)$, Keys
$\left(\mathbf{W}^{K}\in{R}^{d\times d_{k}}\right)$, and Values
$\left(\mathbf{W}^{V}\in{R}^{d\times d_{v}}\right)$, where $d_{q}$=$d_{k}$. By
projecting the input sequence $X$ onto these weight matrices, it obtains
$Q=X\cdot W^{Q}$, $K=X\cdot W^{K}$, and $V=X\cdot W^{V}$. The self-attention
layer outputs $Z\in{R}^{n\times{d_{v}}}$ [12], calculated as:

$\mathbf{Z}=\operatorname{softmax}\left(\frac{\mathbf{QK}^{T}}{\sqrt{d_{q}}}\right)\mathbf{\cdot}V$
(1)

To determine the importance or weight of each value in the sequence, a softmax
function is applied. This function assigns weights to the values based on
their relevance or significance within the context of the task or model. In
summary, the self-attention mechanism enables each element in the sequence to
be updated based on its interactions with other elements, incorporating global
contextual information. The self-attention mechanism computes the dot product
between the query and all the keys for a specific entity in the sequence.
These dot products are then normalized using the softmax function, resulting
in attention scores. Each entity in the sequence is then updated as a weighted
sum of all the entities, with the weights determined by the attention scores.
We will delve deeper into scaled dot-product attention and its application
within multi-head attention in the following section. 2.2.

Transformer encoder and decoder layers: The encoder is composed of a sequence
of identical layers, with a total of $N$ layers, where $N$ is specified in
Figure 2. Each layer comprises two principal sub-layers: a multi-head self-
attention mechanism and a position-wise fully connected feed-forward network.
Subsequent to each layer, the architecture employs residual connections [5]
and layer normalization [51]. This configuration stands in contrast to CNNs,
in which feature aggregation and transformation are executed concurrently.
Within the transformer architecture, these operations are distinctly
partitioned: the self-attention sub-layer is tasked with aggregation
exclusively, whereas the feed-forward sub-layer focuses on the transformation
of features.

The decoder is structured similarly, consisting of identical layers. Each
layer within the decoder encompasses three sub-layers. The initial two sub-
layers, specifically the multi-head self-attention and the feed-forward
networks, reflect the architecture of the encoder. The third sub-layer
introduces a novel multi-head attention mechanism that targets the outputs
from the corresponding encoder layer, as depicted in Figure 2-b [18].

###  2.2 Key Components and Building Blocks of Vision Transformers

The subsequent sections will provide in-depth explanations of the key
components and fundamental building blocks of ViTs. These include patch
extraction and embedding, positional encoding, multi-head self-attention, and
feed-forward networks. In patch extraction, an image is divided into smaller
patches, each of which is then transformed into a numerical representation
through an embedding process. Positional encoding is employed to incorporate
spatial information, allowing the model to account for the relative positions
of these patches. The multi-head self-attention mechanism is crucial for
capturing dependencies and contextual relationships within the image. Finally,
the feed-forward networks are responsible for introducing non-linear
transformations, enhancing the model’s ability to process complex visual
information.

Patch extraction and embedding: A pure transformer model can be directly
employed for image classification tasks by operating on sequences of image
patches. This approach adheres closely to the original design of the
transformer. To handle $2D$ images, the input image $X\in\mathbb{R}^{h\times
w\times c}$ is reshaped into a sequence of flattened 2D patches
$X_{p}\in\mathbb{R}^{n\times(p^{2}\times c)}$, where $c$ represents the number
of channels. The original image resolution is denoted as $(h,w)$, while
$(p,p)$ signifies the resolution of each image patch. The effective sequence
length for the transformer is defined as $n=\frac{h\times w}{p^{2}}$. Given
that the transformer employs consistent dimensions across its layers, a
trainable linear projection is applied to map each vectorized patch to the
model dimension $d$. This output is referred to as patch embedding [16, 52].

Positional encoding: To optimize the model’s use of sequence order,
integrating information about the tokens’ relative or absolute positions is
essential. This is accomplished through the addition of positional encoding to
the input embeddings at the foundation of both the encoder and decoder stacks.
These positional encodings, matching the dimensionality $d_{\text{model}}$ of
the embeddings, are merged with the input embeddings. Positional encoding can
be generated through various methods, including both learned and fixed
strategies [53]. The precise technique for embedding positional information is
delineated by the equations that follow:

$\text{PE}(\text{pos},2i)=\sin\left(\frac{\text{pos}}{10000^{(2i/d_{\text{model}})}}\right)$
(2)

$\text{PE}(\text{pos},2i+1)=\cos\left(\frac{\text{pos}}{10000^{(2i/d_{\text{model}})}}\right)$
(3)

In the given equations, $pos$ represents the position of a word within a
sentence, and $i$ refers to the current dimension of the positional encoding.
In this manner, the positional encoding in the transformer model assigns a
sinusoidal value to each element. This enables the model to learn relative
positional relationships and generalize to longer sequences during inference.
In addition to the fixed positional encoding employed in the original
transformer, other models have explored the use of learned positional encoding
[53] and relative positional encoding [54, 55, 16].

Multi-head self-attention: The multi-head attention mechanism enables the
model to capture multiple complex relationships among different elements in
the sequence. It achieves this by utilizing several self-attention blocks,
each with its own set of weight matrices. The outputs of these blocks are
combined and projected onto a weight matrix to obtain a comprehensive
representation of the input sequence. The original transformer model employs h
= 8 blocks. Each block has its own distinct set of learnable weight matrices,
denoted as ${W^{Q_{i}},W^{K_{i}},W^{V_{i}}}$ for $i=0,1,...,h-1$. When given
an input X, the output of the h self-attention blocks in the multi-head
attention is concatenated into a single matrix
$\left(\mathbf{[Z_{0},Z_{1},\dots,Z_{h}-1]}\in{R}^{n\times{h.d_{v}}}\right)$.
This concatenated matrix is then projected on to a weight matrix
$\left(\mathbf{W}\in{R}^{{h.d_{v}}\times d}\right)$ [18]. Refer to Figure 3
for a visual overview of the Scaled Dot-Product Attention and its extension
into Multi-Head Attention, fundamental mechanisms for contextual processing in
transformer architectures. The diagram details the flow from input queries to
the final attention output.

Figure 3: Schematic representation of the Scaled Dot-Product Attention and
Multi-Head Attention mechanisms. The top process combines queries, keys, and
values to compute attention scores, while the bottom shows parallel attention
layers merging in Multi-Head Attention, a core feature of transformer models
for capturing varied contextual cues. The depiction of the attention mechanism
inspired by [18].

Self-attention differs from convolutional operations in that it calculates
filters dynamically rather than relying on static filters. Unlike convolution,
self-attention is invariant to permutations and changes in the number of input
points, allowing it to handle irregular inputs effectively. It has been shown
in research that self-attention, when used with positional encodings, offers
greater flexibility and can effectively capture local features similar to
convolutional models [56, 57]. Further investigations have been conducted to
analyze the relationship between self-attention and convolution operations.
Empirical evidence supports the notion that multi-head self-attention, with
sufficient parameters, serves as a more general operation that can encompass
the expressiveness of convolution. In fact, self-attention possesses the
capability to learn both global and local features, enabling it to adaptively
determine kernel weights and adjust the receptive field, similar to deformable
convolutions. This demonstrates the versatility and effectiveness of self-
attention in capturing diverse aspects of data [58].

Feed-forward networks: In both the encoder and decoder, a feed-forward network
(FFN) follows the self-attention layers. This network incorporates two linear
transformation layers and a nonlinear activation function within them. Denoted
as the function $FFN(X)$, it can be expressed as $FFN(X)=W_{2}S(W_{1}X)$,
where $W_{1}$ and $W_{2}$ are the parameter matrices of the linear
transformation layers, and $S$ represents the chosen nonlinear activation
function, such as GELU [59].

###  2.3 Training process of Vision Transformers

Self-attention-based transformer models have revolutionized machine learning
through their extensive pre-training on large datasets. This pre-training
stage employs a variety of learning approaches, including supervised,
unsupervised, and self-supervised methods. Such methodologies have been
explored in the seminal works of Dosovitskiy et al. [16], Devlin et al. [13],
Li et al. [60], and Lin et al. [61]. The primary goal of this phase is to
acclimate the model to a broad spectrum of data, or to a combination of
different datasets. This strategy aims to establish a foundational
understanding of visual information processing, a concept further elucidated
by Su et al. [62] and Chen et al. [63].

After pre-training, these models undergo fine-tuning with more specialized
datasets, which vary in size. This step is crucial for tailoring the model to
specific applications, such as image classification [47], object detection
[64], and action recognition [47], thereby improving their performance and
accuracy in these tasks.

The value of pre-training is particularly evident in large-scale transformer
models utilized across both language and vision domains. For instance, the
Vision Transformer (ViT) model exhibits a marked decline in performance when
trained exclusively on the ImageNet dataset, as opposed to including pre-
training on the more comprehensive JFT-300M dataset, which boasts over 300
million images [16, 65]. While pre-training on such extensive datasets
significantly boosts model performance, it introduces a practical challenge:
manually labeling vast datasets is both labor-intensive and costly.

This challenge leads researchers to the pivotal role of self-supervised
learning (SSL) in developing scalable and efficient transformer models. SSL
emerges as an effective strategy by using unlabeled data, thereby avoiding the
limitations associated with extensive manual annotation. Through SSL, models
undertake pretext tasks that generate pseudo-labels from the data itself,
fostering a foundational understanding of data patterns and features without
the necessity for explicit labeling [66, 67]. This method not only enhances
the model’s ability to discern crucial features and patterns, pivotal for
downstream tasks with limited labeled data but also maximizes the utility of
the vast volumes of unlabeled data available.

Contrastive learning, a subset of SSL, exemplifies this by focusing on
identifying minor semantic differences in images, which significantly sharpens
the model’s semantic discernment [18]. The transition from traditional pre-
training methods to SSL underscores a paradigm shift in how models are
trained, moving from reliance on extensive, manually labeled datasets to an
innovative use of unlabeled data. This shift not only addresses the
scalability and resource challenges but also enhances the generalizability and
efficiency of transformer networks.

Khan et al. [18] categorize SSL methods based on their pretext tasks into
generative, context-based, and cross-modal approaches. Generative methods
focus on creating images or videos that match the original data distribution,
teaching the model to recognize data patterns. Context-based methods use
spatial or temporal relationships within the data, enhancing contextual
understanding. Cross-modal methods exploit correspondences between different
data types, like image-text or audio-video, for a more comprehensive data
understanding.

Generative approaches, especially those involving masked image modeling, train
models to reconstruct missing or obscured parts of images, thereby refining
their generative skills [63]. Other SSL strategies include image colorization
[68], image super-resolution [69], image in-painting [68], and approaches
using GAN networks [70, 71]. Context-based SSL approaches deal with tasks like
solving image patch jigsaw puzzles [72], classifying masked objects [62],
predicting geometric transformations like rotations [47], and verifying the
chronological sequence of video frames [73]. Finally, cross-modal SSL methods
focus on aligning different modalities, ensuring correspondences between
elements such as text and image [74], audio and video [75], or RGB and flow
information [76].

###  2.4 Advantages of Vision Transformers compared to CNNs backbones

The advent of ViTs represents a significant innovation in the field of image
processing, offering substantial benefits over traditional CNNs like ResNet.
These advantages are succinctly demonstrated which underscores the key
distinctions and superiorities of ViTs:

Performance Improvement: ViTs have been effectively adapted for a broad range
of vision recognition tasks, demonstrating significant enhancements over CNNs.
These advancements are particularly pronounced in tasks such as classification
on the ImageNet dataset [16], object detection [64], and semantic segmentation
[29], areas where ViTs have outperformed established benchmarks. Remarkably,
ViTs have also shown the capability to achieve competitive results with
architectures that are smaller in size, highlighting their efficiency and
scalability [77]. Furthermore, the overall improvements brought by ViTs in the
vision domain are supported by comprehensive analyses and comparisons [18].

Exploiting Long-Range Dependencies, the Power of Attention Mechanisms in ViTs:
The attention mechanism within ViTs effectively captures long-range
dependencies in the input data. This modeling of inter-token relationships
facilitates a more comprehensive global context, a significant step beyond the
local processing capabilities of CNNs [78, 17, 79]. Additionally, the
attention mechanism provides insight into the focus areas of the model during
input processing, acting as a built-in saliency map [80].

Flexibility and Extensibility: ViTs have proven to be highly versatile,
serving as a backbone that surpasses previous benchmarks with their dynamic
inference capabilities. They are particularly adept at handling unordered and
unstructured point sets, making them suitable for a broader range of
applications [81, 82].

Enhanced Text-Visual Integration with ViTs: The ability of ViTs to integrate
text and visual data facilitates an unparalleled understanding of the
dependencies between different tasks, effectively harnessing the synergy
between diverse data types [21, 83]. Although CNNs can be adapted for text-
visual fusion—potentially with the assistance of Recurrent Neural Networks
(RNNs), ViTs excel in this area. This superiority stems from ViTs’ inherent
design, which naturally accommodates the parallel processing of complex,
multimodal datasets. Unlike CNNs, which may require additional mechanisms or
complex architectures to achieve similar integrations, ViTs directly leverage
their attention mechanisms to dynamically weigh the importance of different
data elements.

End-to-End Training: The architecture of ViTs facilitates end-to-end training
for tasks such as object detection, streamlining the training process by
obviating the need for complex post-processing steps [64].

In essence, ViTs offer a powerful modeling approach that is well-suited to
extracting meaningful information from vast and varied input data. These
visualizations depict how each network type processes and identifies areas of
focus within the image. The attention maps from the ViT show distinct patterns
indicating specific regions the model attends to when making predictions,
while the feature maps from the ResNet indicate more dispersed and
convolutionally derived features throughout the image.

##  3 Vision Transformers in Domain Adaptation and Domain Generalization

ViTs have shown promising performance in computer vision tasks. In this
section, we focus on exploring the potential of adapting ViT to DA and DG
scenarios to mitigate the distribution shifts.

Recent studies have demonstrated that leveraging ViTs backbones as the feature
extractors offers superior capability in managing distribution shifts compared
to conventional CNN architectures [84, 85, 86]. This superiority is relevant
for practical applications where adapting to varied data distributions is
important. These works have led researchers to develop multi-modal approaches
in DA and DG scenarios, integrating diverse strategies to further enhance the
adaptability and generalization capabilities of ViTs. These approaches often
involve the strategic selection of different models and integrating them and
carefully chosen loss functions, aimed at regularizing the training of multi-
modals designs [42, 85, 86, 87, 88].

In the following sections of our paper, we aim to provide an in-depth analysis
of these methodologies and their implications in the field of computer vision.
This will include detailed discussions on the specific techniques and
innovations that have enhanced the performance of ViTs in regard to
distribution shifts. Figure 4 illustrates the categorization of the research
we have reviewed for this paper, highlighting the respective approaches within
DA and DG methods. In the upcoming sections, we aim to provide a more
comprehensive explanation of the methods employed.

Figure 4: Adapting ViTs to distribution shifts involves categorizing the
studies based on methods and strategies developed to enhance ViTs’ robustness
and generalization in the face of varying distributions.

###  3.1 Vision Transformers in Domain Adaptation

DA is a critical area of research within machine learning that aims to improve
model performance on a target domain by leveraging knowledge from a source
domain, especially when the data distribution differs between these domains.
This discrepancy, known as a distribution shift, poses significant challenges
to the adapting of models across varied application scenarios. DA techniques
are designed to mitigate these challenges by adapting models to perform well
on data that was not seen during training, thereby enhancing model robustness
and generalizability [89].

Utilizing ViTs to address distribution shifts within the framework of DA
approaches., offer novel methods to model robustness and generalizability in
diverse application scenarios. In the majority of studies exploring the
application of ViTs to address the challenges of distribution shifts within DA
strategies, the primary emphasis has been on unsupervised domain adaptation
(UDA). DA strategies in the context of ViTs can be broadly classified into
several categories, each contributing uniquely to the model’s adaptability to
different target domains. Our categorization are based on feature-level
adaptation, instance-level adaptation, model-level adaptation, and hybrid
approaches. In each of these categories, we further elaborate on the
adaptation level, providing insights into their architecture, and efficacy in
DA scenarios.

####  3.1.1 Feature-Level Adaptation

Feature-Level adaptation encompasses techniques where ViTs are adapted at the
feature extraction level. It often involves aligning feature distributions
between source and target domains.
Researchers proposed a Domain-Oriented Transformer (DOT) to address the
challenges faced by conventional UDA techniques for domain discrepancies.
Traditional methods often encounter difficulties when attempting to align
domains, which can compromise the discriminability of the target domain when
classifiers are biased towards the source data. To overcome these limitations,
DOT employs feature alignment across two distinct spaces, each specifically
designed for one of the domains. It leverages separate classification tokens
and classifiers for each domain. This approach ensures the preservation of
domain-specific discriminability while effectively capturing both domain-
invariant and domain-specific information. It achieves this through a
combination of contrastive-based alignment and source-guided pseudo-label. The
novel DOT method is introduced in [87].

TRANS-DA [90], focuses on generating pseudo-labels with reduced noise and
retraining the model using new images composed of patches from both source and
target domains. This approach includes a cross-domain alignment loss for
better matching centroids of labeled and pseudo-labeled patches, aiming to
improve domain adaptation. It falls into the feature-level adaptation
category, as it focuses on refining feature representations and aligning them
across domains. Another study shifts focus to integrating transformers with
CNN-backbones, proposing Domain-Transformer for UDA. This approach, distinct
from existing methods that rely heavily on local interactions among image
patches, introduces a plug-and-play domain-level attention mechanism. This
mechanism emphasizes transferable features by ensuring local semantic
consistency across domains, leveraging domain-level attention and manifold
regularization [91].

Spectral UDA (SUDA) [92], is an innovative UDA technique in spectral space.
SUDA introduces a Spectrum Transformer (ST) for mitigating inter-domain
discrepancies and a multi-view spectral learning approach for learning diverse
target representations. The paper’s approach emphasizes feature-level
adaptation, focusing on learning domain-invariant spectral features
efficiently and effectively across various visual tasks such as image
classification, segmentation, and object detection. In [93] the Semantic Aware
Message Broadcasting (SAMB) is introduced to enhance feature alignment in UDA.
This approach challenges the effectiveness of using just one global class
token in ViTs. It suggests adding group tokens instead. These tokens focus on
broadcasting messages to different semantic regions, thereby enriching domain
alignment features. Additionally, the study explores the impact of
adversarial-based feature alignment and pseudo-label based self-training on
UDA, proposing a two-stage training strategy that enhances the adaptation
capability of the ViT [93].

Gao et al. in [94] addresses the Test Time Adaptation (TTA) challenge for
adapting to target data and avoiding performance degradation due to
distribution shifts. By introducing Data-efficient Prompt Tuning (DePT), the
approach combines visual prompts in ViTs with source-initialized prompt fine-
tuning. This fine-tuning, paired with a memory bank-based online pseudo-
labeling and hierarchical self-supervised regularization, enables efficient
model adjustment to the target domain, even with minimal data. DePT’s
adaptability extends to online or multi-source TTA settings [94]. Furthermore,
CTTA [95] proposes a unique approach to continual TTA using visual domain
prompts. It presents a lightweight, image-level adaptation strategy, where
visual prompts are dynamically added to input images, adapting them to the
source domain model. This approach mitigates error accumulation and
catastrophic forgetting by focusing on input modification rather than model
tuning, a significant shift from traditional model-dependent methods. This
method can be classified as a feature-level adaptation, as it primarily
focuses on adjusting the input image features for domain adaptation without
altering the underlying model architecture.

####  3.1.2 Instance-Level Adaptation

In this section, we shift our focus to techniques that adjust the importance
of individual data points or modify how they are selected during training.
These methods give priority to specific instances that more closely reflect
the characteristics of the target domain, an approach known as instance-level
adaptation. This involves selecting or weighing certain data points more
heavily during the training process to ensure the model learns features that
are relevant to the new domain.

One notable instance of instance-level adaptation is addressed in the study by
[96], which deals with the challenge of Source Free Open Set Domain Adaptation
(SF-OSDA). This scenario involves adapting a pre-trained model, initially
trained on an inaccessible source dataset, to an unlabeled target dataset that
includes open set samples, or data points that do not belong to any class seen
during training. The primary technique involves leveraging a self-supervised
ViT, which learns directly from the target domain to distill knowledge. A
crucial element of their method is a unique style-based data augmentation
technique, designed to enhance the training of the ViT within the target
domain by providing a richer, more contextually diverse set of training data.
This leads to the creation of embeddings with rich contextual information.

The model uses these information-rich embeddings to cluster target images
based on semantic similarities and assigns them weak pseudo-labels with
associated uncertainty levels. To improve the accuracy of these pseudo-labels,
the researchers introduce a metric called Cluster Relative Maximum Logit Score
(CRMLS). This measure adjusts the confidence levels of the pseudo-labels,
making them more reliable. Additionally, the approach calculates weighted
class prototypes within this enriched embedding space, facilitating the
effective adaptation of the source model to the target domain, thus
exemplifying an application of instance-level adaptation techniques.

####  3.1.3 Model-Level Adaptation

Model-level adaptation includes the development of specialized ViT
architectures or layers, designed to enhance adaptability to domain shifts.
Key examples include attention mechanisms that focus on domain-invariant
features. This segment of the analysis highlights various strategies employed
in the adaptation of the entire model architecture of ViTs, signifying a key
aspect of the study’s in-depth review.

Zhang et al. in [38] primarily focus on enhancing the out-of-distribution
generalization of ViTs. It delves into techniques like adversarial learning,
information theory, and self-supervised learning to improve model robustness
against distribution shifts. The study is categorized into model-level
adaptation, as it enhances the general model architecture and training process
of ViTs to achieve better performance across varied distributions. Yang et al
[97] introduce TransDA, a novel framework for source-free domain adaptation
(SFDA), which integrates a transformer with a CNN to enhance focus on
important object regions. Diverging from traditional SFDA approaches that
primarily align cross-domain distributions, TransDA capitalizes on the initial
influence of pre-trained source models on target outputs. By embedding the
transformer as an attention module in the CNN, the model gains improved
generalization capabilities for target domains. Additionally, the framework
employs self-supervised knowledge distillation using target pseudo-labels to
refine the transformer’s attention towards object regions. This approach
effectively addresses the limitations of CNNs in handling significant domain
shifts, which often lead to over-fitting and a lack of focus on relevant
objects, therefore offering a more robust solution for domain adaptation
challenges. In addressing the intricacies of model-level adaptation in DA, a
series of innovative approaches emerge, each offering unique solutions to
prevalent challenges. [98] highlights the use of BeiT, a pre-trained
transformer model, for UDA. The core idea is leveraging BeiT’s powerful
feature extraction capabilities, initially trained on source datasets, and
then adapting them to target datasets. This approach, which significantly
outperforms existing methods in the ViSDA Challenge, primarily focuses on
model-level adaptation. It utilizes the self-attention mechanism inherent in
transformers to adapt to new, out-of-distribution target datasets,
demonstrating a marked improvement in domain adaptation tasks. TFC [99] aims
to bridge this gap by demonstrating the potential of combining convolutional
operations and transformer mechanisms for adversarial UDA through a hybrid
network structure termed transformer fused convolution (TFC). By seamlessly
integrating local and global features, TFC enhances the representation
capacity for UDA and improves the differentiation between foreground and
background elements. Additionally, to bolster TFC’s resilience, an uncertainty
penalty loss is introduced, leading to the consistent assignment of lower
scores to incorrect classes.

####  3.1.4 Hybrid Approaches

Hybrid approaches combine features, instance, and model-based methods for more
robust adaptation. CDTRANS [84] introduces a hybrid domain adaptation approach
via a triple-branch transformer that combines feature-level and model-level
adaptations. It incorporates a novel cross-attention module along with self-
attention mechanisms within its architecture. The design includes separate
branches for source and target data processing and a third for aligning
features from both domains. This setup enables simultaneous learning of
domain-specific and domain-invariant representations, showing resilience
against label noise. Additionally, the paper proposes a two-way center-aware
labeling algorithm for the target domain, utilizing a cross-domain similarity
matrix to enhance pseudo-label accuracy and mitigate noise impact. This
approach effectively merges feature acquisition with alignment, showcasing a
sophisticated method for handling domain adaptation challenges. TVT
(Transferable Vision Transformer) [85] explores the use of ViT in UDA. It
examines ViT’s transferability compared to CNNs and proposes the TVT
framework, which includes a Transferability Adaptation Module (TAM) and a
Discriminative Clustering Module (DCM). TVT integrates various domain
adaptation approaches. It employs model-level adaptations by leveraging ViTs,
and optimizing them for UDA. Simultaneously, it involves feature-level
adaptation through the TAM, which enhances feature representations for better
alignment between source and target domains. Furthermore, instance-level
strategies are incorporated via the Discriminative Clustering Module (DCM),
focusing on the diversification and discrimination of patch-level features.
This multifaceted approach, combining model, feature, and instance-level
adaptations, exemplifies a hybrid strategy in domain adaptation.

SSRT [86] enhances domain adaptation by integrating a ViT backbone with a
self-refinement strategy using perturbed target domain data. The approach
includes a safe training mechanism that adaptively adjusts learning
configurations to avoid model collapse, especially in scenarios with large
domain gaps. This novel solution showcases both model-level adaptations, by
employing ViTs, and feature-level adaptations, through its unique self-
refinement method. BCAT [100] presents a novel framework which introduces a
bidirectional cross-attention mechanism to enhance the transferability of
ViTs. This mechanism focuses on blending source and target domain features to
minimize domain discrepancy effectively. The BCAT model combines this with
self-attention in a unique quadruple transformer block structure to focus on
both intra and inter-domain features. It has a novel transformer architecture
with a bidirectional cross-attention mechanism for model-level adaptation, and
integration and alignment of features from different domains for the instance-
level adaptation.

PMTrans [88] employs a PatchMix transformer to bridge source and target
domains via an intermediate domain, enhancing domain alignment. It
conceptualizes UDA as a min-max cross-entropy game involving three entities:
feature extractor, classifier, and PatchMix module. This unique approach,
leveraging a mix of patches from both domains, leads to significant
performance gains on benchmark datasets. The paper aligns with hybrid
adaptation, combining model-level innovations (PatchMix Transformer) and
feature-level strategies (patch mixing for domain bridging). UniAM [101], for
Universal DA (UniDA) leverages ViTs and introduces a Compressive Attention
Matching (CAM) approach to address the UniDA problem. This method focuses on
the discriminability of attention across different classes, utilizing both
feature and attention information. UniAM stands out by its ability to
effectively handle attention mismatches and enhance common feature alignment.
The paper fits into the hybrid category of domain adaptation, combining
feature-level strategies with model-level adaptations through the use of ViT
and attention mechanisms.

CoNMix [102] presents a novel framework for source-free DA, adept at tackling
both single and multi-target DA challenges in scenarios where labeled source
data is unavailable during target adaptation. This framework employs a Vision
ViT as its backbone and introduces a distinctive strategy that integrates
consistency with two advanced techniques: Nuclear-Norm Maximization and MixUp
knowledge distillation. Nuclear-Norm Maximization is a regularization
technique that encourages the model to learn a low-rank representation of
data, promoting generalization by reducing complexity. MixUp knowledge
distillation, on the other hand, leverages a data augmentation method that
combines inputs and labels in a weighted manner to create synthetic training
examples, enhancing the model’s ability to generalize across domains. The
framework demonstrates state-of-the-art results across various domain
adaptation settings, showcasing its effectiveness in scenarios with privacy-
related restrictions on data sharing. This paper aligns with a hybrid
adaptation approach, incorporating both model-level and feature-level
(consistency and pseudo-label refinement methods) strategies.

[103]introduces the Win-Win Transformer (WinTR) framework. This framework
effectively leverages dual classification tokens in a transformer to
separately explore domain-specific knowledge for each domain while also
interchanging cross-domain knowledge. It incorporates domain-specific
classifiers for each token, emphasizing the preservation of domain-specific
information and facilitating knowledge transfer. This approach exhibits
significant performance improvements in UDA tasks. The paper exemplifies a
hybrid approach, combining model-level adaptations with its transformer
structure and feature-level strategies through domain-specific learning and
knowledge transfer mechanisms.

Table LABEL:tab:Design_highlights_ViT provides a comprehensive summary of the
main categories in adapting ViT for DA: feature level adaptation, instance
level adaptation, model level adaptation, and hybrid approaches. It details
the various methods used, design highlights, and different loss functions
employed during training. Additionally, the table references the publication
details for each representative study, including the journals or conferences
where they were published.

Table 1: Representative Works of ViTs for DA |  |  |  |
---|---|---|---|---
Category |  Method |  Design Highlights |  Loss Functions |  Publication
Feature-level Adaptation |  DOT [87] |  Domain-Oriented Transformer with dual classifiers utilizes individual tokens for source/target domain adaptation and domain-specific learning, enhanced by source-guided pseudo-label refinement |  Cross-entropy, Contrastive, Representation Difference |  ICMR 2022
|  SUDA [92] |  Enhancing visual task generalization, leveraging Spectrum Transformer for domain alignment and multi-view learning to optimize mutual information |  Supervised, Discrepancy, Unsupervised Similarity |  CVPR 2022
Model-level Adaptation |  GE-ViT [38] |  Leveraging ViTs’ inductive biases towards shapes and structures, combined with adversarial and self-supervised learning techniques |  Cross-Entropy, Entropy, Adversarial |  CVPR 2022
|  TFC [99] |  Integrates CNN and ViTs for both capturing of local and global features, achieving distinct foreground-background separation, further refined by an uncertainty-based accuracy enhancement |  Cross-Entropy, Adversarial, Uncertainty Penalization |  IEEE-TCSS 2022
Hybrid Approaches |  CDTRANS [84] |  Triple-branch Transformer with shared weights aligns source/target data, using self/cross-attention and center-aware labeling for enhanced pseudo-label accuracy and noise distillation |  Cross-entropy, Distillation |  ICLR 2022
|  TVT [85] |  Enhances feature learning via adversarial adaptation, using transferability and clustering modules for robust knowledge transfer, and fine-tunes attention for precise domain discrimination |  Cross-Entropy, Adversarial, Patch-Level Discriminator |  WACV2023
Hybrid Approaches |  SSRT [86] |  Utilizes adversarial adaptation and enhances model accuracy by refining through perturbed data feedback, incorporating a robust training approach to minimize KL divergence |  Cross-Entropy, Self-Refinement, Domain Adversarial |  CVPR2022
|  PMTrans [88] |  Employs a module based on transformer architecture for domain representation, leveraging game theory for patch sampling, and combines source/target data to optimize label accuracy with attention-driven adjustments |  Cross-Entropy, Semi-Supervised Mixup |  CVPR 2023
|  SMTDA [102] |  Enhances domain adaptation by maximizing nuclear-norm consistency and distilling knowledge via MixUp, refining pseudo labels for broadened generalization across multiple targets |  Cross-Entropy, Nuclear-Norm, Knowledge Distillation |  WACV 2023
|  UniAM [101] |  Utilizes universal and compressive attention for precise alignment and class separation, achieving categorization without prior label knowledge |  Cross-Entropy, Adversarial, Source-Target Contrastive |  ICCV 2023

####  3.1.5 Diverse Strategies for Enhancing Domain Adaptation

Incorporating ViTs into DA techniques involves multi-modal and varied methods.
We have meticulously reviewed these recent advancements, systematically
categorizing the diverse strategies with its unique approach employed. In
Table LABEL:tab:ViT-DA, we provide a comprehensive overview of the methods
utilized in each investigated study, thereby facilitating an easier comparison
and understanding for the reader. To offer further insight, for each study, we
delve into the various strategies that have been developed. Each strategy
showcases a unique approach and possesses distinct strengths, reflecting the
diverse nature of the field. Following is an explanation of these methods for
a deeper insight: In the evolving landscape of domain adaptation, Adversarial
Learning (ADL), for instance, harnesses adversarial networks to minimize
discrepancies between different domains, creating a competitive scenario where
models continually improve their domain invariance and adaptability. In
contrast, Cross-DA (CRD) tackles the challenge of transferring knowledge from
a source domain to a target domain, effectively handling the variances in data
distributions.

Adding to the diversity, Using Visual Prompts (VisProp) leverages visual cues,
enriching the learning process, especially in vision-based tasks. This method
brings a novel perspective, guiding models through complex visual landscapes.
Meanwhile, Self-Supervised Learning (SSL) takes a different route by
extracting learning signals directly from the data, eliminating the need for
labeled datasets and enabling models to uncover underlying patterns in an
unsupervised manner.

The fusion of different architectural paradigms, as seen in Hybrid Networks
combining ViTs with CNN (ViT+CNN), brings forth the best of both worlds, the
perceptual strengths of CNNs and the relational prowess of transformers.
Knowledge Distillation (KD) enables a smaller, more efficient model to learn
from a larger, more complex one, encapsulating the essence of efficient
learning.

In scenarios where access to the original source data is restricted, Source-
Free DA (SFDA) emerges as a crucial strategy. It relies on the model’s
inherent knowledge and the characteristics of the target domain, showcasing
adaptability in constrained environments. Complementing this, Test-Time
Adaptation (TTA) ensures that models remain flexible and adaptable even during
the inference phase, crucial for dealing with evolving data landscapes.

The adaptation techniques can be further nuanced based on class overlap
scenarios between the source and target domains, leading to Closed-Set,
Partial-Set, and Open-Set Adaptation (CPO). Each addresses a specific kind of
overlap, from complete to none, reflecting the diverse challenges in domain
adaptation. Pseudo Label Refinement(PLR) , on the other hand, enhances the
reliability of labels in unsupervised settings, refining the model-generated
labels for better accuracy.

Lastly, Contrastive Learning (CL), by distinguishing between similar and
dissimilar data points, offers a robust way for models to learn distinctive
features, essential for tasks like classification and clustering. For methods
like game theory, which are sparsely used, we have categorized them under
Aditional Emerging Methods (AEM) to provide a comprehensive overview.

Table 2: Comprehensive Summary of Techniques in ViTs for DA: ADL (Adversarial Learning), CRD (Cross-DA), VisProp (Visual Prompts), SSL (Self-Supervised Learning/Semi-Supervised Learning), ViT+CNN (Hybrid Networks), KD (Knowledge Distillation), SFDA (Source-Free DA), TTA (Test-Time Adaptation), CPO (Closed-Set, Partial-Set, Open-Set Adaptation), PLR (Pseudo Label Refinement), CL (Contrastive Learning), and Additional Emerging Methods (AEM). Study | ADL | CRD | VisProp | SSL | ViT-CNN | KD | SFDA | TTA | CPO | PLR | CL | AEM
---|---|---|---|---|---|---|---|---|---|---|---|---
CDTRANS [84] |  | ✓ |  |  |  |  |  |  |  | ✓ |  |
TransDA [97] |  |  |  | ✓ | ✓ | ✓ | ✓ |  | ✓ |  |  |
GE-ViT[38] | ✓ |  |  | ✓ |  |  |  |  |  |  |  | ✓
TVT[85] | ✓ | ✓ |  | ✓ |  |  |  |  |  |  |  |
BCAT [100] |  | ✓ |  |  |  | ✓ |  |  |  |  |  |
SSRT [86] | ✓ |  |  | ✓ |  |  |  |  |  | ✓ |  |
DoT [87] |  |  |  |  |  |  |  |  |  | ✓ | ✓ |
PMTarns [88] |  | ✓ |  | ✓ |  |  |  |  |  |  |  | ✓
DOT [91] |  | ✓ |  |  | ✓ |  |  |  |  |  |  |
DePT [94] |  |  | ✓ | ✓ |  |  |  | ✓ |  | ✓ |  |
BeiT [98] |  |  |  | ✓ |  |  |  |  |  |  |  |
SUDA [92] | ✓ |  |  | ✓ |  |  |  |  |  |  |  | ✓
SMTDA [102] |  |  |  |  |  | ✓ | ✓ |  |  | ✓ |  |
UniDA [101] |  | ✓ |  | ✓ |  |  |  | ✓ |  |  | ✓ | ✓
WinTR [103] |  | ✓ |  |  |  |  |  |  |  | ✓ | ✓ |

###  3.2 Vision Transformers in Domain Generalization

In our comprehensive review of the existing research, we analyzed how ViTs are
adapted for the DG process. Based on our findings, we have identified four
distinct approaches that encapsulate the common strategies within the
literature. we categorized the approaches into four main categories based on
our analysis of the literature. These categories are: Multi-Domain Learning,
Meta-Learning Approaches, Regularization Techniques, and Data Augmentation
Strategies. In the subsequent section, we will delve into the specifics of the
research within each category.

####  3.2.1 Multi-Domain Learning

This method involves training ViTs across different types of data or domains.
The main goal is to train these models to recognize features that are common
across all these domains. By doing this, the models become better at working
in new and varied environments they haven’t seen before.

INDIGO [104] is a novel method for enhancing DG. INDIGO stands out by
integrating intrinsic modality from large-scale pre-trained vision-language
networks with the visual modality of ViTs. This integration, coupled with the
fusion of multimodal and visual branches, significantly improves the model’s
ability to generalize to new, unseen domains. The effectiveness of INDIGO is
demonstrated through substantial improvements in DG benchmarks like DomainNet
and Office-Home. We will introduce the famous benchmarks in DA and DG in 4.

####  3.2.2 Meta-Learning Approaches

Meta-learning is an approach for training ViTs to adapt rapidly to new domains
with minimal data. By engaging in a variety of learning tasks, ViTs develop
the ability to apply meta-knowledge across different settings, significantly
boosting their adaptability and performance in unseen environments. In the
following paragraphs, we will delve into the details of research focusing on
meta-learning approaches.

[105] introduces the DoPrompt algorithm, a novel approach in the realm of ViTs
for domain generalization. It uniquely incorporates Domain Prompt Learning
(DPL) and Prompt Adapter Learning (PAL), embedding domain-specific knowledge
into prompts for each source domain. These prompts are then integrated through
a prompt adapter for effective target domain prediction.

In [106], the authors present an innovative approach for domain generalization
using ViTs. It leverages a hybrid architecture that combines domain-specific
local experts with transformer-based query-memory decoding. This unique
methodology allows for dynamic decoding of source domain knowledge during
inference, demonstrating enhanced performance and generalization capabilities
on various benchmarks, outperforming existing state-of-the-art methods.

Researchers in [107], propose Common-Specific Visual Prompt Tuning (CSVPT), a
new method integrating domain-common prompts to capture task context and
sample-specific prompts to address data distribution variations, enabled by a
trainable prompt-generating module (PGM). This approach is specifically
tailored for effective adaptation to unknown testing domains, significantly
enhancing out-of-distribution generalization in image classification tasks.

####  3.2.3 Regularization Techniques

Regularization methods are identified as essential for preventing overfitting
and promoting the learning of generalized features.

Researchers introduce a Self-Distillation for ViTs (SDViT) approach [108],
aiming to mitigate overfitting to source domains. This technique utilizes non-
zero entropy supervisory signals in intermediate transformer blocks,
encouraging the model to learn features that are broadly applicable and
generalizable. The modular and plug-and-play nature of this approach
seamlessly integrates into ViTs without adding new parameters or significant
training overhead. This research is aptly classified under Regularization
Techniques in the taxonomy of DG using ViTs, as the self-distillation strategy
aligns with the goal of preventing overfitting and promoting domain-agnostic
generalization.

This study [109] proposes a Cross Attention for DG (CADG) model. The model
uses cross attention to tackle the distribution shifts problem inherent in DG,
extracting stable representations for classification across multiple domains.
Its focus on using cross-attention to align features from different
distributions, a strategy that enhances stability and generalization
capabilities across domains, puts it under the regularization part.

Researchers in [110] center on boosting DG through Intermediate-Block and
Augmentation-Guided Self-Distillation. The proposed method incorporates self-
distillation techniques to boost the robustness and generalization of ViTs,
particularly focusing on improving performance in unseen domains. This
approach has shown promising results on various benchmark datasets, and it has
a commitment to leveraging self-distillation to prevent over-fitting and
foster generalization across varied domains.

Test-Time Adjustment (T3A) [111] proposes an optimization-free method for
adjusting the classifier at test time using pseudo-prototype representations
derived from online unlabeled data. This approach aims to robustify the model
to unknown distribution shifts.

####  3.2.4 Data Augmentation Strategies

Advanced data augmentation techniques, including synthetic data generation,
are employed to broaden the training dataset’s diversity. This strategy is
instrumental in preparing ViTs to better generalize to novel domains. Upcoming
segments will illustrate different data augmentation strategies employed in
ViTs.

The researchers introduce a novel concept known as Token-Level Feature
Stylization (TFS-ViT) [112]. This method transforms token features by blending
normalization statistics from various domains and applies attention-aware
stylization based on the attention maps of class tokens. Aimed at improving
ViTs’ ability to adapt to domain shifts and handle unseen data, this approach
is a prime example of data augmentation strategies in DG using ViTs. TFS-ViT’s
emphasis on feature transformation and utilizing diverse domain data is an
advanced data augmentation technique, aimed at enriching training data variety
for enhanced DG.

[113] explores a unique approach to DG by focusing on spatial relationships
within image features. The proposed hybrid architecture (ConvTran) merges CNNs
and ViTs, targeting both local and global feature dynamics. The methodology is
aimed at learning global feature structures through the spatial interplay of
local elements, with the aim of generalization. In terms of its relation to
data augmentation, the idea of leveraging spatial interplay for categorization
in data augmentation is rooted in a comprehension of how image features
interact spatially allows a model to better adapt to and perform in novel,
previously unseen domains. ConvTran enhances the model’s ability to process
and generalize across different domains by learning and incorporating global
spatial relationships, aligning with strategies aimed at augmenting training
data diversity for better generalization.

The research [43] carefully examines multiple image augmentation techniques to
determine their effectiveness in promoting DG, specifically within the context
of semantic segmentation. This investigation includes experiments utilizing
the DAFormer architecture [46], showcasing the wide-ranging applicability of
these augmentations across various models. It highlights how important it is
to carefully check different ways of changing images. It emphasizes the
importance of evaluating a variety of image augmentation strategies,
considering that carefully selected data augmentations are essential for
improving the generalization abilities of models.

In conclusion, table LABEL:tab:Design_highlights_ViT_DG presents
representative works from recent research that have employed ViTs for DG.
These selected studies highlight the adaptability and potential of ViTs in
improving models’ ability to generalize across various domains.

Table 3: Representative Works of ViTs for DG |  |  |  |
---|---|---|---|---
Category |  Method |  Design Highlights |  Training Strategies |  Publication
Meta-Learning Approaches |  CSVPT[107] |  Boosts OOD generalization with dynamically generated domain-invariant and variant prompts via a trainable module, improving adaptability across datasets |  Cross Entropy |  ACCV 2022
Regularization Techniques |  SDViT[108] |  Reduces overfitting by using self-distillation, entropy-based signals, and a modular approach, aiming for better learning across different domains |  Cross Entropy, KL Divergence |  ACCV 2022
|  T3A[111] |  Enhances DG by updating linear classifiers with online-generated pseudo-prototypes, offering robustness in varying environments without back-propagation |  Cross Entropy, Pseudo Label Refinement |  NeurIPS 2021
|  |  |  |

####  3.2.5 Diverse Strategies for Enhancing Domain Generalization

Building on our discussion from the DA section, we now shift our focus to
exploring the various strategies employed in DG. Similar to DA, DG also
encompasses a spectrum of methods, adapting known DG methods for ViTs, each
tailored to enhance the model’s capability to generalize across unseen
domains. Here, we delve into these diverse techniques, outlining their unique
features and roles in the context of DG. Table LABEL:tab:ViT-DG summarizes the
strategies employed in research to address DG challenge through the
integration of ViT architecture.

Domain Synthesis (DST) creates artificial training domains to enhance the
model’s generalization capability across unseen environments. Self
Distillation (SD) leverages the model’s own outputs to refine and improve its
learning process. Class Guided Feature Learning (CGFL) focuses on extracting
features based on class-specific information to improve classification
accuracy. Adaptive Learning (ADPL) dynamically adjusts the learning strategy
based on the specifics of each domain. ViT-CNN Hybrid Networks combine the
strengths of ViTs and CNN for robust feature extraction. Feature
Augmentation/Feature Learning (FAug) enhances feature sets to improve model
robustness against varying inputs. Prompt-Learning (PL) employs guiding
prompts to direct the learning process, particularly useful in language and
vision tasks. Cross Domain (CRD) learning involves training models across
diverse domains to improve adaptability. Source Domain Knowledge Decoding
(SDKD) decodes and transfers knowledge from the source domain to enhance
generalization. Knowledge Distillation (KD) transfers knowledge from a larger,
complex model to a smaller, more efficient one. Source-Free DA (SFDA) adapts
models to new domains without relying on source domain data, crucial for
privacy-sensitive applications. Multi Modal Learning (MML) uses multiple types
of data inputs, such as visual and textual, to improve learning
comprehensiveness. Test-Time Adaptation (TTA) adjusts the model during
inference to adapt to new environments, ensuring robust performance on unseen
data.

Table 4: Comprehensive Summary of Techniques in ViTs for DG: DST (Domain Synthesis), SD (Self Distillation), CGFL (Class Guided Feature Learning), ADPL (Adaptive Learning), ViT-CNN (Hybrid Networks), FAug (Feature Augmentation/Feature Learning), PL (Prompt-Learning), CRD( Cross Domain), ViT+CNN(Hybrid Networks), SDKD (Source Domain Knowledge Decding), KD (Knowledge Distillation), SFDA (Source-Free DA), MML (Multi Modal Learning), and TTA (Test-Time Adaptation) Study | DST | SD | CGFL | ADPL | ViT-CNN | FAug/FL | PL | CRD | SFKD | MML | TTA
---|---|---|---|---|---|---|---|---|---|---|---
SDViT [108] |  | ✓ | ✓ |  |  |  |  |  |  |  |
TFS-ViT [112] | ✓ |  | ✓ |  |  | ✓ |  |  |  |  |
DoPrompt [105] |  |  |  | ✓ |  |  | ✓ |  |  |  |
ConvTran [113] |  |  |  |  | ✓ |  |  |  |  |  |
D2SDK [106] |  |  |  |  | ✓ |  |  | ✓ | ✓ |  |
INDIGO [104] |  |  |  |  |  |  |  |  |  | ✓ | ✓
CSVPT [107] |  |  |  | ✓ |  |  | ✓ |  |  |  |
CADG [114] |  |  | ✓ |  |  | ✓ |  | ✓ |  |  |
RRLD [110] |  | ✓ |  |  |  | ✓ |  |  |  |  | ✓
T3A+ViT [111] |  |  |  |  |  | ✓ |  |  |  | ✓ |

##  4 Applications Beyond the Image Recognition

Most of the research discussed in section 3 primarily focuses on image
recognition tasks. However, these methods have the potential for broader
application across various domains. A substantial portion of the studies
explores applications extending beyond image recognition to other fields. We
have divided these studies into four distinct categories: semantic
segmentation, which examines the partitioning of images into segments; action
recognition, focusing on identifying and classifying actions within videos;
face analysis, which involves detecting and interpreting facial features and
expressions; and medical imaging, where methods are employed to analyze
interpret medical images. In the upcoming sections, we will first briefly
discuss benchmarking datasets commonly used in the research, providing a
foundation for understanding their methodologies.

Benchmarking Datasets:
In DA and DG approaches, a key focus is on how models perform on datasets with
distribution shifts. Such benchmarks are crucial in determining the robustness
and adaptability of models, against real-world data variation. The methods for
DA/DG, are tested across diverse datasets like VLCS [115], Office-31 [116],
PACS [117], OfficeHome [118], DomainNet [119], and ImageNet-Sketch [120].
These evaluations also include scenarios like synthetic-vs-real [115],
artificial corruptions [121], and diverse data sources [122]. To illustrate
the distribution shift, samples from the PACS dataset are depicted in Figure
5.

Figure 5:  Examples from the PACS dataset [117] demonstrate distribution shift
scenarios. The training set includes images from sketch, cartoon, and art
painting domains, while the testing dataset consists of real images,
highlighting the challenges of distribution shift in PACS dataset.

###  4.1 Semantic Segmentation

In the field of semantic segmentation, a crucial challenge is the limited
generalization of DNNs to unseen domains, exacerbated by the high costs and
effort required for manual annotation in new domains. This challenge
highlights the need for developing new methods and modern visual models to
adapt to new domains without extensive labeling, addressing distribution
shifts effectively. The shift from synthetic to real data is particularly
critical, as it allows for leveraging simulation environments. In this section
we offer an in-depth overview of the latest progress in this research area of
ViTs, focusing on the sustained efforts and the key unresolved issues that
hinder the broader application of ViTs in semantic segmentation across various
domains.

DAFormer [46] stands out as a foundational work in using ViTs for UDA,
presenting groundbreaking contributions at both the method and architecture
levels. This approach significantly improved the state-of-the-art (SOTA)
performance, surpassing ProDA [123], by more than 10% mIoU. The architecture
of DAFormer is based on SegFormer [124], which is utilized as the encoder
architecture. It incorporates two established methods from segmentation DNNs.
DAFormer first introduces skip connections between the encoder and decoder for
improved transfer of low-level knowledge. It then employs an ASPP-like [15]
fusion, processing stacked encoder outputs at various levels with different
dilation rates, aiming to increase the receptive field. At the method level,
DAFormer adapts known UDA methods for CNNs, including self-training with a
teacher-student framework, strong augmentations, and softmax-based confidence
weighting. Additional features include rare class sampling in the source
domain and a feature distance loss to pre-trained ImageNet features. An
interesting observation made in the study is the potential benefit of learning
rate warm-up methods for UDA.

Building directly on the contributions of DAFormer [46], HRDA [125] marks a
substantial progress in the application of ViT models. Its primary
contribution is a scale attention mechanism that processes high and low-
resolution inputs, allocating attention scores to prioritize one over the
other based on class and object scales. This method facilitates better
extraction of contextual information from smaller sections of images and
includes self-training using a sliding window for pseudo-label generation.
While HRDA further enhances DAFormer’s performance, there remains a gap to be
bridged.

TransDA [126]addresses a high-frequency problem identified in ViTs, using the
Swin transformer [29] architecture. It shows that target pseudo labels and
features change more frequently and significantly over iterations compared to
a ResNet-101, suggesting this issue is specific to ViT networks. TransDA’s
solution includes feature and pseudo label smoothing using a momentum network,
combined with self-training and weighted adversarial output adaptation,
similar to CNN’s teacher-student approaches. Zhang et al. in [127] introduce
Trans4PASS+, an advanced model tackling the challenges of panoramic semantic
segmentation. This model addresses image distortions and object deformations
typical in panoramic images, utilizing Deformable Patch Embedding (DPE) and
Deformable MLP (DMLPv2) modules. Additionally, it features a Mutual
Prototypical Adaptation (MPA) strategy for UDA in panoramic segmentation,
enhancing performance in both indoor and outdoor scenarios. The paper also
contributes a new dataset, SynPASS, to facilitate Synthetic-to-Real (SYN2REAL)
adaptation in panoramic imagery.

In the context of these developments, Ding et al. introduce HGFormer [128], a
novel approach for domain generalization in semantic segmentation. HGFormer
groups pixels into part-level masks before assembling them into whole-level
masks. This hierarchical strategy significantly enhances the robustness of
segmentation against domain shifts by combining detailed and broader image
features. HGFormer’s effectiveness is demonstrated through various cross-
domain experimental setups, showcasing its superiority over traditional per-
pixel classification and flat-grouping transformers.

Alongside these innovative approaches, a growing number of studies, such as
ProCST [129], are evaluating ViT networks. ProCST applies hybrid adaptation
with style transfer in the input space, in conjunction with DAFormer [46]and
HRDA[125]. Recently [43] delves into the efficacy of simple image-style
randomization and augmentation techniques, such as blur, noise, and color
jitter, for enhancing the generalization of DNNs in semantic segmentation
tasks. The study is pivotal in its systematic evaluation of these
augmentations, demonstrating that even basic modifications can significantly
improve network performance on unseen domains. Notably, the paper reveals that
combinations of multiple augmentations rival the complexity and effectiveness
of state-of-the-art domain generalization methods. Employing architectures
like ResNet-101 and the ViT DAFormer, the research achieves remarkable
results, with performance on the synthetic-to-real domain shift between
Synthia and Cityscapes datasets reaching up to 44.2% mIoU. Rizzoli et al.
introduce MISFIT [130], a novel framework for multimodal source-free domain
adaptation in semantic segmentation. This method innovatively fuses RGB and
depth data at multiple stages in a ViT architecture. Key features include
input-level depth stylization for domain alignment, cross-modality attention
for mixed feature extraction, and a depth-based entropy minimization strategy
for adaptively weighting regions at different distances. MISFIT, as the first
RGB-D ViT approach for source-free semantic segmentation, demonstrates notable
improvements in robustness and adaptability across varied domains. Various
other works integrate their methods with the DAFormer framework, incrementally
improving performance [131, 132, 133, 134], though not surpassing HRDA.
Notably, CLUDA [133] builds upon HRDA, further improving its performance.

###  4.2 Action Recognition

In the field of surveillance video analysis, a growing area of interest is
domain-adapted action recognition. This involves training action recognition
systems in one environment (the source domain) and applying them in another
with distinct viewpoints and characteristics (the target domain). This
emerging research topic addresses the challenges posed by these environmental
differences [135]. In the context of domain adaptation for action recognition,
while source datasets provide action labels, these labels are not available
for the target dataset. Consequently, evaluating the performance on the target
dataset poses a challenge due to the absence of these labels [136]. In the
field of RGB-based action recognition tasks, transformer-based domain
adaptation methods have demonstrated outstanding. UDAVT [137], a novel
approach in UDA for video action recognition, demonstrates a significant
advancement in handling domain shifts in video data. Central to its design is
the innovative use of a spatio-temporal transformer architecture, which
efficiently captures both spatial and temporal dynamics. The framework is
distinguished by its unique alignment loss term, derived from the information
bottleneck principle, fostering the learning of domain-invariant features.
UDAVT employs a two-phase training process, initially fine-tuning the whole
transformer with source data, followed by the adaptation of the temporal
transformer using the information bottleneck loss, effectively aligning domain
distributions. This approach has shown SOTA performance on challenging UDA
benchmarks like HMDB withUCF and Kinetics with NEC-Drone, outperforming
existing methods and underscoring the potential of transformers in video
analysis. The integration of a queue for recent feature representations
further enhances the method’s effectiveness, making UDAVT a significant
contribution to the field of action recognition in videos.

Lin et al. [138] introduce ViTTA, a method enhancing action recognition models
during test time without retraining. This approach focuses on feature
distribution alignment, dynamically adjusting to match test set statistics
with those of the training set. A key aspect is its applicability to both
convolutional and transformer-based networks. ViTTA also enforces consistency
in predictions over temporally augmented video views, a strategy that
significantly improves performance in scenarios with distribution shifts,
showcasing its effectiveness over previous test-time adaptation techniques. Q.
Yan and Y. Hu’s research [139] introduces a UDA method tailored for skeleton
behavior recognition, addressing the challenge of aligning source and target
datasets in domain adaptation. Their method employs a spatial-temporal
transformer framework with three flows—source, target, and source-target
facilitating effective domain alignment and handling variations in joint
numbers and positions across datasets. Key to this approach is the use of
subsequence encoding and an attention mechanism that emphasizes local joint
relationships, thereby enhancing the representation of skeleton behavior.
Comprehensive testing on various skeleton datasets shows the superiority of
their Spatial-Temporal Transformer-based DA (STT-DA) method, underscoring its
effectiveness in managing the complexities of domain adaptation in skeleton
behavior recognition. The concept of applying transformers for skeleton-based
action recognition in DA is viewed as a promising and potentially impactful
direction in this field of study.[140]

###  4.3 Face Analysis

Face anti-spoofing (FAS) is a crucial aspect of biometric security systems,
addressing the challenge of distinguishing between genuine and fake facial
representations [141]. Recently WACV 2023 research [142]introduced a new
approach for FAS using ViTs. The authors proposed the Domain-invariant ViT
(DiVT) which employs two specific losses to enhance generalizability. The
losses include a concentration loss for learning domain-invariant
representations by aggregating features of real face data, and a separation
loss to differentiate each type of attack across domains. The study highlights
the effectiveness of transformers in capturing long-range dependencies and
globally distributed cues, crucial for FAS tasks. It also addresses the large
model size and computational resource issues commonly associated with
transformer models by adopting a lightweight transformer model, MobileViT. The
proposed approach differs from previous methods by focusing on the core
characteristics of real faces’ features and unifying attack types across
domains, leading to improved performance and efficiency in FAS applications.
In addition, researchers in [141] explored FAS in surveillance contexts, where
image quality varies widely. The paper introduces an Adversarial DG Network
(ADGN), which classifies training data into sub-source domains based on image
quality scores. It then employs adversarial learning for feature extraction
and domain discrimination, achieving quality-invariant features. The approach
also integrates transfer learning to mitigate limited training data issues.
This innovative method proved effective in surveillance FAS, as evidenced by
its performance in the 4th Face Anti-Spoofing Challenge at CVPR 2023.

In addition to the previously discussed work in the domain of FAS, another
significant contribution comes from a study focusing on adaptive transformers
for robust few-shot cross-domain FAS [143]. The study presents a novel
approach by integrating ensemble adapter modules and feature-wise
transformation layers into ViTs, enhancing their adaptability across different
domains with minimal examples. This methodology is especially pertinent in
scenarios where FAS systems encounter diverse and previously unseen
environments. The research demonstrates that this adaptive approach results in
both robust and competitive performance in cross-domain FAS, outperforming
state-of-the-art methods on several benchmark datasets, even when only a few
samples are available from the target domain. This highlights the potential of
adaptive transformers in improving the generalizability and effectiveness of
FAS systems in real-world applications. In the context of FAS under continual
learning, a rehearsal-free method called Domain Continual Learning (DCL) was
proposed. It addressed catastrophic forgetting and unseen domain
generalization using the Dynamic Central Difference Convolutional Adapter
(DCDCA) for ViT models. The Proxy Prototype Contrastive Regularization (PPCR)
was utilized to retain previous domain knowledge without using their data,
resulting in improved generalization and reduced catastrophic forgetting
[144].

###  4.4 Medical Imaging

In the evolving landscape of medical image classification and analysis, ViTs
have emerged as a pivotal technology. Their application is primarily aimed at
overcoming the challenges of domain generalization, thereby boosting the
adaptability of deep learning methods in ever-changing clinical settings and
in the face of unseen environments.

Focusing on the critical area of breast cancer detection, where computer-aided
systems have shown considerable promise, the use of deep learning has been
relatively hampered by a lack of domain generalization. A noteworthy study in
this regard explored this issue within the context of mass detection in
digital mammography. This research, encompassing a multi-center setup, delved
into the analysis of domain shifts and evaluated eight leading detection
methods, including those based on transformer models. The findings were
significant, revealing that the proposed workflow not only reduced domain
shift but also surpassed existing transfer learning techniques in efficacy
[145].

In the realm of skin lesion recognition [146], where deep learning has made
remarkable strides, the issue of overdependence on disease-irrelevant image
artifacts raised concerns about generalization. A groundbreaking study
introduced EPVT, an innovative domain generalization method, leveraging
prompts in ViTs to amalgamate knowledge from various domains. This approach
significantly enhanced the models’ generalization capabilities across diverse
environments [147].

Another challenging area is medical image segmentation, which struggles due to
the inherent variability of medical images. To tackle the limited availability
of training datasets, data-efficient ViTs were proposed. However,
indiscriminate dataset combinations can result in Negative Knowledge Transfer
(NKT). Addressing this, the introduction of MDViT, a multi-domain ViT with
domain adapters, marked a significant advancement. This approach allowed for
the effective utilization of varied data resources while mitigating NKT,
showcasing superior segmentation performance even with an increase in domain
diversity [148]. The robustness against adversarial attacks is a non-
negotiable aspect of deep medical diagnosis systems. A novel CNN-Transformer
hybrid model was introduced to bolster this robustness and enhance
generalization. This model augmented image shape information in high-level
feature spaces, smoothing decision boundaries and thereby improving
performance on standardized datasets like MedMNIST-2D[149]. Liu et al.’s
introduction of the Convolutional Swin-Unet (CS-Unet) represents a notable
advance in medical image semantic segmentation. By integrating the
Convolutional Swin Transformer (CST) block into transformers, they effectively
combined multi-head self-attention with convolutions, providing localized
spatial context and inductive biases essential for delineating organ
boundaries. This model’s efficiency and effectiveness, particularly in its
capability to surpass existing transformer-based methods without extensive
pre-training, underscore the importance of localized spatial modeling in
medical imaging [150]. A significant stride in domain generalization was
achieved with the introduction of BigAug, a deep stacked transformation
approach. This method applies extensive data augmentation, simulating domain
shifts across MRI and ultrasound imaging. BigAug’s application of nine
transformations to each image, validated across diverse segmentation tasks and
challenge datasets, set a new standard, outperforming traditional augmentation
and domain adaptation techniques in unseen domains [151].

The paper by Ayoub et al. [152] brings forth innovative techniques in medical
image segmentation, addressing the challenge of model generalization across
varied clinical environments. Their methodologies significantly enhance the
robustness and applicability of deep learning models in medical imaging,
ensuring their effectiveness in diverse clinical scenarios. Furthermore, Li et
al. introduced DTNet, the UDA method comprising a dispensed residual
transformer block, a multi-scale consistency regularization, and a feature
ranking discriminator. This network significantly improved segmentation
performance in retinal and cardiac segmentation across different sites and
modalities, setting a new benchmark for UDA methods in these applications
[153]. Finally, the use of self-supervised learning with UNETR, incorporating
ViTs into a 3D UNET architecture, addressed inaccuracies caused by out-of-
distribution medical data. This model’s voxel-wise prediction capability
enhances the precision of sample-wise predictions [154].

In reviewing the recent advancements in the application of ViTs for DA and
Generalization in medical imaging, it becomes evident that ViTs are not only
versatile but also increasingly effective in addressing domain-specific
challenges. The studies surveyed indicate a significant shift towards models
that are more adaptable to varying clinical environments, a crucial aspect of
real-world medical applications. However, there is an observable need for
further refinement in these models to ensure even greater accuracy and
robustness, particularly in scenarios involving scarce or highly variable
data. The success of models like EPVT and MDViT in enhancing generalization
capabilities across diverse environments suggests a promising direction toward
more domain-agnostic approaches. Nevertheless, the balance between model
complexity and interpretability remains a key area for future exploration. As
the field moves forward, there’s potential for integrating more advanced self-
supervised learning techniques and exploring hybrid models that combine the
strengths of both CNNs and ViTs. This could lead to a new generation of
medical imaging tools that are not only more efficient in handling domain
shifts but also more accessible and reliable for clinicians in varied
healthcare settings.

To conclude, this chapter has showcased the extensive utility of ViTs beyond
image recognition tasks, highlighting their significant impact in areas such
as semantic segmentation, action recognition, face analysis, and medical
imaging. Beyond these fields, ViTs have proven to be highly adaptable and
effective in sectors like precision agriculture and autonomous driving [155,
156]. These researches highlight that the potential of ViTs extends far beyond
the initially discussed applications. Their adaptability to different
environments and challenges showcases a growing research interest in diverse
fields. Future research could explore even more innovative applications,
leveraging ViTs’ unique ability to handle complex visual tasks and
distribution shifts across different industries.

##  5 Conclusions and Future Research Directions

This comprehensive review looks at how modern vision networks, ViTs, are used
in DA and DG methods to handle distribution shifts. Reviewing the research,
we’ve observed that the transformer’s self-attention mechanisms play a pivotal
role in generalizing and adapting to new, distribution-shifted samples, as
evidenced by reviewing the experiment results of research in the known shifted
datasets including ImageNet-A, ImageNet-C, and Stylized ImageNet. With respect
to distribution shifts’ handling methods, for DA methods, we’ve organized the
research into four categories: feature-level, instance-level, model-level
adaptations, and hybrid approaches. Additionally, we’ve introduced a new
category for studies that combine different strategies with ViTs to tackle
distribution shifts. These hybrid networks show the use of various strategies
alongside ViTs, leading us to categorize the papers based on these combined
approaches. A similar methodological framework was applied to DG, wherein
papers are classified based on multi-domain learning, meta-learning,
regularization techniques, and data augmentation strategies. These two
distinct methods of categorization, by considering various approaches, provide
deeper insights and a broader perspective. Finally, we extend our review to
applications beyond image recognition, showcasing the versatility of these
methods in various domains, and highlighting the potential of ViTs in a broad
range of applications beyond the image recognition tasks.

In our discussion and exploration of various research works, it becomes
evident that we are at the nascent stage of developing these modern vision
networks. Researchers are increasingly focusing on the characteristics of ViTs
across diverse deep learning scenarios. While ViTs face challenges stemming
from various factors, they are on a trajectory of improvement, much like CNNs
were in their early stages. There is a noticeable rise in publications
dedicated to modern vision models, emphasizing the advancements being made. As
highlighted earlier, particularly due to the attention mechanism inherent in
ViTs, their proficiency in handling distribution shifts is undeniable, marking
an advantage over CNNs in this aspect. In line with the focus of this study,
this section intends to outline potential research areas in the field of
tackling distribution shifts. The recommendations take into account the
properties of ViTs, as well as DA and DG approaches for managing distribution
shifts, and how they can be effectively integrated. ViTs have demonstrated the
capacity to surpass previous SOTA methods in certain contexts. However, their
superiority is not consistently overwhelming, partly due to their reliance on
manually designed architectures which may impede their adaptability, as
explored by [50]

###  5.1 Data Requirements and Computational Intensity

Upon investigating the deployment of ViTs within the realms of DA and DG
methods, our research has identified a range of challenges that merit closer
examination. ViTs have emerged as a pivotal enhancement in computer vision
capabilities, promising significant advances across various applications.
However, their adoption in real-world scenarios is fraught by considerable
challenges, primarily due to their extensive data requirements. The necessity
for large-scale datasets, such as JFT-300M, for effective training,
underscores this challenge, as [16] have pointed out. This substantial
reliance on voluminous, pre-trained models, emphasized by [17], necessitates
access to high-quality data, a critical aspect particularly when training ViTs
from scratch on more constrained datasets, a scenario highlighted by [29].
This dependency presents a significant barrier, wherein the effectiveness of
ViTs, upon adaptation through transfer learning and fine-tuning, is closely
linked to the quality of pre-existing object detectors. This necessitates a
meticulous application of these methodologies, as evidenced by the works of
[21] and [157]. In addition, a big challenge is when the models need to learn
from a very small number of examples, a method known as few-shot learning.
Situations that require high safety standards or have very limited data, show
how hard it can be to adjust ViTs under different conditions. Recently, new
methods like source-free domain adaptation and test time adaptation have shown
promise in making models more generalized. Even though researchers made
progress in overcoming biases from the source data, they still have a lot to
learn about how to manage uncertainty when making these adjustments. This area
presents significant opportunities for further research. Finally, the inherent
self-attention mechanisms of ViTs, especially in models with a vast number of
trainable parameters, introduce a layer of complexity that demands substantial
computational resources. This further complicates their deployment in
practical scenarios, as discussed by [61].

###  5.2 Necessity of New Benchmarks

With respect to handling distribution shift approaches, they have presented
their own set of challenges. The recent VisDA-2021 dataset competition [158],
where transformers underpinned the winning solutions, indicates their efficacy
in managing robustness against distribution shifts. This observation aligns
with findings by [28], asserting transformers’ superior generalization
capabilities on target domain samples. While the advancements in performance
over conventional baseline backbones, such as CNNs, are commendable, the quest
for perfection is ongoing and remains substantially unfulfilled. This gap
underscores the necessity for new benchmarks aimed at propelling research on
real-world distribution shift approaches further. The limited number of
datasets currently employed in DA and DG intensifies the challenge of
validating new approaches, highlighting a need for real-world, application-
specific datasets. This review, although broad in scope, reveals a prevailing
bias towards classification tasks, even when exploring applications beyond
image recognition. In the domain of medical imaging, for instance, this bias
persists, underscoring the importance of extending the focus of ViTs to
encompass a wider array of tasks beyond mere classification tasks.

###  5.3 Pre and Post Domain Adaptation Approaches

In the context of DA, the utilization of pre-domain adaptation (Pre-DA) and
post-domain adaptation (Post-DA) strategies plays a pivotal role in enhancing
model performance. Pre-DA focuses on preparing models before they are exposed
to new domains, aiming to address and bridge domain discrepancies beforehand.
Meanwhile, Post-DA strategies are applied after exposure to the new domain
[159], with the goal of mitigating accuracy declines. The significance of
integrating both Pre-DA and Post-DA approaches has been underscored in recent
studies, suggesting that a comprehensive exploration of these strategies could
substantially improve the adaptability and effectiveness of models in
unfamiliar domains [160]. Another significant challenge is the lack of
effective comparison metrics for certain DA and DG scenarios. The common use
of absolute mean Average Precision (mAP) for object detection tasks does not
fully capture the subtleties of evaluation metrics, where relative
improvements post-DA might be more indicative of success. This highlights a
need for robust comparison metrics capable of accommodating the variability
inherent in models trained under diverse conditions [158].

###  5.4 Uncertainty-Aware Vision Transformers

In our analysis of ViTs and their proficiency in navigating distribution
shifts, we’ve highlighted their potential to enhance model generalization
through various techniques. A particularly promising, yet under explored
approach is integrating uncertainty quantification methods with ViTs. This
integration enables models to provide predictions along with confidence
levels, making decision-making more transparent. The presence of
uncertainties, amplified by distribution shifts, is not merely an additional
challenge but a crucial aspect of real-world environments’ unpredictability.
Employing uncertainty-aware ViTs to detect and improve model generalizability
presents a significant research opportunity. Future studies should delve into
how uncertainties influence the adaptation and generalization capabilities of
ViTs, emphasizing the integration of uncertainty quantification methods. Such
investigative efforts are crucial for gaining a thorough understanding of how
modern vision networks can exploit uncertainties to enhance the field of
domain adaptation and generalization.

This review stands as the first to comprehensively gather most of the recent
work on utilizing ViTs to address distribution shifts within the framework of
DA and DG approaches. The increasing number of publications on this topic
showcases a growing interest, underscoring the rapid evolution of the field
and the keenness among researchers to explore further the capabilities of
ViTs. As researchers, we see a promising future in using ViTs to address
distribution shifts. This survey aims to offer a perspective on the evolution
of ViTs, guiding future research toward the creation of more robust and
versatile deep learning models.

## References

  * [1] K. Fukushima, A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, Biol, Cybern 36 (1980) 193–202.
  * [2] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  * [3] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * [4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * [5] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [6] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  * [7] Y.-L. Hsieh, M. Cheng, D.-C. Juan, W. Wei, W.-L. Hsu, C.-J. Hsieh, On the robustness of self-attentive models, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 1520–1529.
  * [8] M. Tan, Q. Le, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, PMLR, 2019, pp. 6105–6114.
  * [9] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, arXiv preprint arXiv:1312.6199 (2013).
  * [10] R. Girshick, F. Iandola, T. Darrell, J. Malik, Deformable part models are convolutional neural networks, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, pp. 437–446.
  * [11] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al., Relational inductive biases, deep learning, and graph networks, arXiv preprint arXiv:1806.01261 (2018).
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [13] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  * [14] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901.
  * [15] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 801–818.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020).
  * [17] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou, Training data-efficient image transformers & distillation through attention, in: International conference on machine learning, PMLR, 2021, pp. 10347–10357.
  * [18] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, M. Shah, Transformers in vision: A survey, ACM computing surveys (CSUR) 54 (10s) (2022) 1–41.
  * [19] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, P. Luo, Adaptformer: Adapting vision transformers for scalable visual recognition, Advances in Neural Information Processing Systems 35 (2022) 16664–16678.
  * [20] C. Sun, A. Myers, C. Vondrick, K. Murphy, C. Schmid, Videobert: A joint model for video and language representation learning, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7464–7473.
  * [21] J. Lu, D. Batra, D. Parikh, S. Lee, Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks, Advances in neural information processing systems 32 (2019).
  * [22] H. Tan, M. Bansal, Lxmert: Learning cross-modality encoder representations from transformers, arXiv preprint arXiv:1908.07490 (2019).
  * [23] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, J. Liu, Uniter: Learning universal image-text representations (2019).
  * [24] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, S. Lee, 12-in-1: Multi-task vision and language representation learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10437–10446.
  * [25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763.
  * [26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (2015) 211–252.
  * [27] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al., The many faces of robustness: A critical analysis of out-of-distribution generalization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8340–8349.
  * [28] Y. Bai, J. Mei, A. L. Yuille, C. Xie, Are transformers more robust than cnns?, Advances in neural information processing systems 34 (2021) 26831–26843.
  * [29] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022.
  * [30] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, L. Shao, Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 568–578.
  * [31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp. 740–755.
  * [32] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, A. Torralba, Scene parsing through ade20k dataset, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 633–641.
  * [33] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, M.-H. Yang, Intriguing properties of vision transformers, Advances in Neural Information Processing Systems 34 (2021) 23296–23308.
  * [34] D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Glaeser, F. Timm, W. Wiesbeck, K. Dietmayer, Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges, IEEE Transactions on Intelligent Transportation Systems 22 (3) (2020) 1341–1360.
  * [35] J. Fayyad, M. A. Jaradat, D. Gruyer, H. Najjaran, Deep learning sensor fusion for autonomous vehicle perception and localization: A review, Sensors 20 (15) (2020) 4220.
  * [36] B. Dhillon, A. Fashandi, K. Liu, Robot systems reliability and safety: A review, Journal of quality in maintenance engineering 8 (3) (2002) 170–212.
  * [37] E. R. Ranschaert, S. Morozov, P. R. Algra, Artificial intelligence in medical imaging: opportunities, applications and risks, Springer, 2019.
  * [38] C. Zhang, M. Zhang, S. Zhang, D. Jin, Q. Zhou, Z. Cai, H. Zhao, X. Liu, Z. Liu, Delving deep into the generalization of vision transformers under distribution shifts, in: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2022, pp. 7277–7286.
  * [39] V. M. Patel, R. Gopalan, R. Li, R. Chellappa, Visual domain adaptation: A survey of recent advances, IEEE signal processing magazine 32 (3) (2015) 53–69.
  * [40] J. Fayyad, Out-of-distribution detection using inter-level features of deep neural networks, Ph.D. thesis, University of British Columbia (2023).
  * [41] J. Fayyad, K. Gupta, N. Mahdian, D. Gruyer, H. Najjaran, Exploiting classifier inter-level features for efficient out-of-distribution detection, Image and Vision Computing 142 (2024) 104897.
  * [42] S. Angarano, M. Martini, F. Salvetti, V. Mazzia, M. Chiaberge, Back-to-bones: Rediscovering the role of backbones in domain generalization, arXiv preprint arXiv:2209.01121 (2022).
  * [43] M. Schwonberg, F. El Bouazati, N. M. Schmidt, H. Gottschalk, Augmentation-based domain generalization for semantic segmentation, in: 2023 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2023, pp. 1–8.
  * [44] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, P. Yu, Generalizing to unseen domains: A survey on domain generalization, IEEE Transactions on Knowledge and Data Engineering (2022).
  * [45] G. Wilson, D. J. Cook, A survey of unsupervised deep domain adaptation, ACM Transactions on Intelligent Systems and Technology (TIST) 11 (5) (2020) 1–46.
  * [46] L. Hoyer, D. Dai, L. Van Gool, Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9924–9935.
  * [47] S. Gidaris, P. Singh, N. Komodakis, Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv:1803.07728 (2018).
  * [48] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, A. Dosovitskiy, Do vision transformers see like convolutional neural networks?, Advances in Neural Information Processing Systems 34 (2021) 12116–12128.
  * [49] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, W. Brendel, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, arXiv preprint arXiv:1811.12231 (2018).
  * [50] T. Lin, Y. Wang, X. Liu, X. Qiu, A survey of transformers, AI Open (2022).
  * [51] J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450 (2016).
  * [52] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al., A survey on vision transformer, IEEE transactions on pattern analysis and machine intelligence 45 (1) (2022) 87–110.
  * [53] J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. N. Dauphin, Convolutional sequence to sequence learning, in: International conference on machine learning, PMLR, 2017, pp. 1243–1252.
  * [54] P. Shaw, J. Uszkoreit, A. Vaswani, Self-attention with relative position representations, arXiv preprint arXiv:1803.02155 (2018).
  * [55] J. Devlin, M.-W. Chang, K. Lee, Google, kt, language, ai: Bert: pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
  * [56] J. Pérez, J. Marinković, P. Barceló, On the turing completeness of modern neural network architectures, arXiv preprint arXiv:1901.03429 (2019).
  * [57] J.-B. Cordonnier, A. Loukas, M. Jaggi, On the relationship between self-attention and convolutional layers, arXiv preprint arXiv:1911.03584 (2019).
  * [58] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 764–773.
  * [59] D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08415 (2016).
  * [60] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al., Oscar: Object-semantics aligned pre-training for vision-language tasks, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, Springer, 2020, pp. 121–137.
  * [61] K. Lin, L. Wang, Z. Liu, End-to-end human pose and mesh reconstruction with transformers, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 1954–1963.
  * [62] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, J. Dai, Vl-bert: Pre-training of generic visual-linguistic representations, arXiv preprint arXiv:1908.08530 (2019).
  * [63] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, J. Liu, Uniter: Universal image-text representation learning, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX, Springer, 2020, pp. 104–120.
  * [64] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European conference on computer vision, Springer, 2020, pp. 213–229.
  * [65] A. Gupta, C. Sun, A. Shrivastava, S. Singh, Revisiting the unreasonable effectiveness of data, URL: https://ai. googleblog. com/2017/07/revisiting-unreasonable-effectiveness. html [retrieved 20 May 2022] (2017).
  * [66] L. Jing, Y. Tian, Self-supervised visual feature learning with deep neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence 43 (11) (2020) 4037–4058.
  * [67] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, J. Tang, Self-supervised learning: Generative or contrastive, IEEE Transactions on Knowledge and Data Engineering 35 (1) (2021) 857–876.
  * [68] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A. A. Efros, Context encoders: Feature learning by inpainting, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536–2544.
  * [69] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al., Photo-realistic single image super-resolution using a generative adversarial network, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  * [70] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014).
  * [71] S. Alijani, J. Tanha, L. Mohammadkhanli, An ensemble of deep learning algorithms for popularity prediction of flickr images, Multimedia Tools and Applications 81 (3) (2022) 3253–3274.
  * [72] U. Ahsan, R. Madhok, I. Essa, Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition, in: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2019, pp. 179–189.
  * [73] H.-Y. Lee, J.-B. Huang, M. Singh, M.-H. Yang, Unsupervised representation learning by sorting sequences, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 667–676.
  * [74] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, Visualbert: A simple and performant baseline for vision and language, arXiv preprint arXiv:1908.03557 (2019).
  * [75] B. Korbar, D. Tran, L. Torresani, Cooperative learning of audio and video models from self-supervised synchronization, Advances in Neural Information Processing Systems 31 (2018).
  * [76] N. Sayed, B. Brattoli, B. Ommer, Cross and learn: Cross-modal self-supervision, in: Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, Springer, 2019, pp. 228–243.
  * [77] R. Ranftl, A. Bochkovskiy, V. Koltun, Vision transformers for dense prediction, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 12179–12188.
  * [78] R. Shao, Z. Shi, J. Yi, P.-Y. Chen, C.-J. Hsieh, On the adversarial robustness of vision transformers, arXiv preprint arXiv:2103.15670 (2021).
  * [79] C. Matsoukas, J. F. Haslum, M. Söderberg, K. Smith, Is it time to replace cnns with transformers for medical images?, arXiv preprint arXiv:2108.09038 (2021).
  * [80] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, A. Joulin, Emerging properties in self-supervised vision transformers, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660.
  * [81] C. Doersch, A. Gupta, A. Zisserman, Crosstransformers: spatially-aware few-shot transfer, Advances in Neural Information Processing Systems 33 (2020) 21981–21993.
  * [82] H. Zhao, L. Jiang, J. Jia, P. H. Torr, V. Koltun, Point transformer, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16259–16268.
  * [83] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, S. Lazebnik, Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2641–2649.
  * [84] T. Xu, W. Chen, P. Wang, F. Wang, H. Li, R. Jin, Cdtrans: Cross-domain transformer for unsupervised domain adaptation, arXiv preprint arXiv:2109.06165 (2021).
  * [85] J. Yang, J. Liu, N. Xu, J. Huang, Tvt: Transferable vision transformer for unsupervised domain adaptation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 520–530.
  * [86] T. Sun, C. Lu, T. Zhang, H. Ling, Safe self-refinement for transformer-based domain adaptation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 7191–7200.
  * [87] W. Ma, J. Zhang, S. Li, C. H. Liu, Y. Wang, W. Li, Making the best of both worlds: A domain-oriented transformer for unsupervised domain adaptation, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5620–5629.
  * [88] J. Zhu, H. Bai, L. Wang, Patch-mix transformer for unsupervised domain adaptation: A game perspective, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3561–3571.
  * [89] M. Wang, W. Deng, Deep visual domain adaptation: A survey, Neurocomputing 312 (2018) 135–153.
  * [90] Y. Ye, S. Fu, J. Chen, Learning cross-domain representations by vision transformer for unsupervised domain adaptation, Neural Computing and Applications (2023) 1–14.
  * [91] R. Chuan-Xian, Z. Yi-Ming, L. You-Wei, L. Meng-Xue, Towards unsupervised domain adaptation via domain-transformer, arXiv preprint arXiv:2202.13777 (2022).
  * [92] J. Zhang, J. Huang, Z. Tian, S. Lu, Spectral unsupervised domain adaptation for visual recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9829–9840.
  * [93] X. Li, C. Lan, G. Wei, Z. Chen, Semantic-aware message broadcasting for efficient unsupervised domain adaptation, arXiv preprint arXiv:2212.02739 (2022).
  * [94] Y. Gao, X. Shi, Y. Zhu, H. Wang, Z. Tang, X. Zhou, M. Li, D. N. Metaxas, Visual prompt tuning for test-time domain adaptation, arXiv preprint arXiv:2210.04831 (2022).
  * [95] Y. Gan, Y. Bai, Y. Lou, X. Ma, R. Zhang, N. Shi, L. Luo, Decorate the newcomers: Visual domain prompt for continual test time adaptation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 7595–7603.
  * [96] G. Vray, D. Tomar, B. Bozorgtabar, J.-P. Thiran, Source-free open-set domain adaptation for histopathological images via distilling self-supervised vision transformer, arXiv preprint arXiv:2307.04596 (2023).
  * [97] G. Yang, H. Tang, Z. Zhong, M. Ding, L. Shao, N. Sebe, E. Ricci, Transformer-based source-free domain adaptation, arXiv preprint arXiv:2105.14138 (2021).
  * [98] B. U. Tayyab, N. Chua, Pre-training transformers for domain adaptation, arXiv preprint arXiv:2112.09965 (2021).
  * [99] M. Wang, J. Chen, Y. Wang, Z. Gong, K. Wu, V. C. Leung, Tfc: Transformer fused convolution for adversarial domain adaptation, IEEE Transactions on Computational Social Systems (2022).
  * [100] X. Wang, P. Guo, Y. Zhang, Domain adaptation via bidirectional cross-attention transformer, arXiv preprint arXiv:2201.05887 (2022).
  * [101] D. Zhu, Y. Li, J. Yuan, Z. Li, Y. Shao, K. Kuang, C. Wu, Universal domain adaptation via compressive attention matching, arXiv preprint arXiv:2304.11862 (2023).
  * [102] V. Kumar, R. Lal, H. Patil, A. Chakraborty, Conmix for source-free single and multi-target domain adaptation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 4178–4188.
  * [103] W. Ma, J. Zhang, S. Li, C. H. Liu, Y. Wang, W. Li, Exploiting both domain-specific and invariant knowledge via a win-win transformer for unsupervised domain adaptation, arXiv preprint arXiv:2111.12941 (2021).
  * [104] P. Mangla, S. Chandhok, M. Aggarwal, V. N. Balasubramanian, B. Krishnamurthy, Indigo: Intrinsic multimodality for domain generalization, arXiv preprint arXiv:2206.05912 (2022).
  * [105] Z. Zheng, X. Yue, K. Wang, Y. You, Prompt vision transformer for domain generalization, arXiv preprint arXiv:2208.08914 (2022).
  * [106] C. Kang, K. Nandakumar, Dynamically decoding source domain knowledge for domain generalization, arXiv preprint arXiv:2110.03027 (2021).
  * [107] A. Li, L. Zhuang, S. Fan, S. Wang, Learning common and specific visual prompts for domain generalization, in: Proceedings of the Asian Conference on Computer Vision, 2022, pp. 4260–4275.
  * [108] M. Sultana, M. Naseer, M. H. Khan, S. Khan, F. S. Khan, Self-distilled vision transformer for domain generalization, in: Proceedings of the Asian Conference on Computer Vision, 2022, pp. 3068–3085.
  * [109] Z. Liu, Y. Xu, Y. Xu, Q. Qian, H. Li, R. Jin, X. Ji, A. B. Chan, An empirical study on distribution shift robustness from the perspective of pre-training and data augmentation, arXiv preprint arXiv:2205.12753 (2022).
  * [110] A. Singh, S. Jayavelu, Robust representation learning with self-distillation for domain generalization, arXiv preprint arXiv:2302.06874 (2023).
  * [111] Y. Iwasawa, Y. Matsuo, Test-time classifier adjustment module for model-agnostic domain generalization, Advances in Neural Information Processing Systems 34 (2021) 2427–2440.
  * [112] M. Noori, M. Cheraghalikhani, A. Bahri, G. A. V. Hakim, D. Osowiechi, I. B. Ayed, C. Desrosiers, Tfs-vit: Token-level feature stylization for domain generalization, arXiv preprint arXiv:2303.15698 (2023).
  * [113] C. Kang, K. Nandakumar, Discovering spatial relationships by transformers for domain generalization, arXiv preprint arXiv:2108.10046 (2021).
  * [114] C. Dai, Y. Lin, F. Li, X. Li, D. Xie, Cadg: A model based on cross attention for domain generalization, arXiv preprint arXiv:2203.17067 (2022).
  * [115] C. Fang, Y. Xu, D. N. Rockmore, Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias, in: Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 1657–1664.
  * [116] K. Saenko, B. Kulis, M. Fritz, T. Darrell, Adapting visual category models to new domains, in: Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, Springer, 2010, pp. 213–226.
  * [117] D. Li, Y. Yang, Y.-Z. Song, T. M. Hospedales, Deeper, broader and artier domain generalization, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5542–5550.
  * [118] H. Venkateswara, J. Eusebio, S. Chakraborty, S. Panchanathan, Deep hashing network for unsupervised domain adaptation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5018–5027.
  * [119] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, B. Wang, Moment matching for multi-source domain adaptation, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406–1415.
  * [120] H. Wang, Z. He, Z. C. Lipton, E. P. Xing, Learning robust representations by projecting superficial statistics out, arXiv preprint arXiv:1903.06256 (2019).
  * [121] D. Hendrycks, T. Dietterich, Benchmarking neural network robustness to common corruptions and perturbations, arXiv preprint arXiv:1903.12261 (2019).
  * [122] S.-A. Rebuffi, H. Bilen, A. Vedaldi, Learning multiple visual domains with residual adapters, Advances in neural information processing systems 30 (2017).
  * [123] P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, F. Wen, Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12414–12424.
  * [124] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, P. Luo, Segformer: Simple and efficient design for semantic segmentation with transformers, Advances in Neural Information Processing Systems 34 (2021) 12077–12090.
  * [125] L. Hoyer, D. Dai, L. Van Gool, Hrda: Context-aware high-resolution domain-adaptive semantic segmentation, in: European Conference on Computer Vision, Springer, 2022, pp. 372–391.
  * [126] R. Chen, Y. Rong, S. Guo, J. Han, F. Sun, T. Xu, W. Huang, Smoothing matters: Momentum transformer for domain adaptive semantic segmentation, arXiv preprint arXiv:2203.07988 (2022).
  * [127] J. Zhang, K. Yang, H. Shi, S. Reiß, K. Peng, C. Ma, H. Fu, P. H. Torr, K. Wang, R. Stiefelhagen, Behind every domain there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation, arXiv preprint arXiv:2207.11860 (2022).
  * [128] J. Ding, N. Xue, G.-S. Xia, B. Schiele, D. Dai, Hgformer: Hierarchical grouping transformer for domain generalized semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15413–15423.
  * [129] S. Ettedgui, S. Abu-Hussein, R. Giryes, Procst: boosting semantic segmentation using progressive cyclic style-transfer, arXiv preprint arXiv:2204.11891 (2022).
  * [130] G. Rizzoli, D. Shenaj, P. Zanuttigh, Source-free domain adaptation for rgb-d semantic segmentation with vision transformers, arXiv preprint arXiv:2305.14269 (2023).
  * [131] Q. Zhou, Z. Feng, Q. Gu, J. Pang, G. Cheng, X. Lu, J. Shi, L. Ma, Context-aware mixup for domain adaptive semantic segmentation, IEEE Transactions on Circuits and Systems for Video Technology 33 (2) (2022) 804–817.