Neha<EMAIL_ADDRESS>Sam<EMAIL_ADDRESS>Jeremy
<EMAIL_ADDRESS>Bayan<EMAIL_ADDRESS>Soheil<EMAIL_ADDRESS>University of Maryland
College Park, MD, USA
CapitalOne Research
Adapting SSL Representations using DDM

# Adapting Self-Supervised Representations to Multi-Domain Setups

###### Abstract

Current state-of-the-art self-supervised approaches, are effective when
trained on individual domains but show limited generalization on unseen
domains. We observe that these models poorly generalize even when trained on a
mixture of domains, making them unsuitable to be deployed under diverse real-
world setups. We therefore propose a general-purpose, lightweight Domain
Disentanglement Module (DDM) that can be plugged into any self-supervised
encoder to effectively perform representation learning on multiple, diverse
domains with or without shared classes. During pre-training according to a
self-supervised loss, DDM enforces a disentanglement in the representation
space by splitting it into a domain-variant and a domain-invariant portion.
When domain labels are not available, DDM uses a robust clustering approach to
discover pseudo-domains. We show that pre-training with DDM can show up to
$3.5\%$ improvement in linear probing accuracy on state-of-the-art self-
supervised models including SimCLR, MoCo, BYOL, DINO, SimSiam and Barlow Twins
on multi-domain benchmarks including PACS, DomainNet and WILDS. Models trained
with DDM show significantly improved generalization ($7.4\%$) to unseen
domains compared to baselines. Therefore, DDM can efficiently adapt self-
supervised encoders to provide high-quality, generalizable representations for
diverse multi-domain data.

Figure 1: Framework of our proposed Domain Disentanglement Module: In our
proposed DDM framework, the representation space ($\mathbf{h}$) of any given
self-supervised encoder is split into two portions, a domain prefix
($\mathbf{h}^{d}$) and a domain-invariant ($\mathbf{h}^{p}$) portion. Along
with the self-supervised loss ($L_{\textit{ssl}}$), $\mathbf{h}^{d}$ is
trained to be distinguishable across domains ($L_{\textit{d\\_var}}$) and
$\mathbf{h}^{p}$ is trained to be invariant to any domain information
($L_{\textit{d\\_invar}}$). DDM also supports scenarios when domain labels are
not available using robust clustering, an iterative process that reduces
outlier noise.

##  1 Introduction

Self-supervised learning [Chen et al.(2020a)Chen, Kornblith, Norouzi, and
Hinton, Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin,
He et al.(2020)He, Fan, Wu, Xie, and Girshick, Grill et al.(2020)Grill, Strub,
Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo, Gheshlaghi
Azar, Piot, kavukcuoglu, Munos, and Valko, Chen and He(2021), Caron et
al.(2018b)Caron, Bojanowski, Joulin, and Douze, Khosla et al.(2020)Khosla,
Teterwak, Wang, Sarna, Tian, Isola, Maschinot, Liu, and Krishnan, Caron et
al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin] has
become a popular paradigm for unsupervised representation learning as it shows
impressive results on downstream tasks. However, we find that current self-
supervised models when trained on a single-domain show very poor
generalizability to domain shifts. This can hinder their deployment in large
scale real-world settings where data almost always comes from multiple diverse
domains. We illustrate this issue in Figure 2, where we show that popular
self-supervised models, SimCLR [Chen et al.(2020a)Chen, Kornblith, Norouzi,
and Hinton], MoCo [He et al.(2020)He, Fan, Wu, Xie, and Girshick] and BYOL
[Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya,
Doersch, Avila Pires, Guo, Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and
Valko], trained on individual domains of PACS [Li et al.(2017)Li, Yang, Song,
and Hospedales] significantly under-perform on unseen domains. This means that
a different self-supervised model needs to be trained for every new domain,
which can add significant computational overheads given that training these
models often require large batch sizes and a large number of training epochs
[Chen et al.(2020a)Chen, Kornblith, Norouzi, and Hinton, He et al.(2020)He,
Fan, Wu, Xie, and Girshick, Wu et al.(2018)Wu, Xiong, Yu, and Lin].

One potential solution for self-supervised learning on multi-domain datasets
is to train the models on the union of all input domains. We illustrate this
in Figure 2 were we plot the multi-domain training results for each baseline
on a mixture of PACS Photo, Sketch and Cartoon. We observe that this solution
may show improved performance on the training domains, however they do not
match the single-domain baselines in all cases. Moreover, they show poor
generalization to unseen domains (PACS Painting). In Section 3, we study the
representation space closely under multi-domain regimes to find that they can
under perform compared to single-domain regimes because domain-related and
content-related information overlap in the representation space, affecting
their quality for instance classification.

To tackle these issues, we propose a Domain-Dientanglement Module (DDM), that
can be plugged in to any self-supervised model during multi-domain training.
With DDM, we enforce a disentanglement in the representation space where a
domain prefix is trained to be distinguishable across domains and the
remaining portion is trained to be domain-invariant to produce better
structured representations. This is achieved by minimizing the Wasserstein
Distance [Arjovsky et al.(2017)Arjovsky, Chintala, and Bottou] between the
known and predicted domain label distributions. We also extend DDM to more
realistic, entirely unsupervised multi-domain setups where domain labels are
unknown. In such scenarios, we present a robust clustering approach that
iteratively reduces outlier noise and detects pseudo-domain-labels that are
used in DDM.

By pre-training with DDM, we show that we can improve the generalization
capability of various state-of-the-art self-supervised baselines including
SimCLR [Chen et al.(2020a)Chen, Kornblith, Norouzi, and Hinton], MoCo [He et
al.(2020)He, Fan, Wu, Xie, and Girshick], BYOL [Grill et al.(2020)Grill,
Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo,
Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and Valko], DINO [Caron et
al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin],
SimSiam [Chen and He(2021)] and Barlow Twins [Zbontar et al.(2021)Zbontar,
Jing, Misra, LeCun, and Deny]. We perform extensive experiments on
generalization benchmarks including PACS [Li et al.(2017)Li, Yang, Song, and
Hospedales], DomainNet [Peng et al.(2019)Peng, Bai, Xia, Huang, Saenko, and
Wang] and WILDS [Koh et al.(2021)Koh, Sagawa, Marklund, Xie, Zhang,
Balsubramani, Hu, Yasunaga, Phillips, Gao, Lee, David, Stavness, Guo,
Earnshaw, Haque, Beery, Leskovec, Kundaje, Pierson, Levine, Finn, and Liang].
Upon linear probing on unseen domains, we observe an improvement of $6.1\%$ on
PACS, $7.4\%$ on DomainNet and $5.9\%$ on WILDS using DDM. In summary, we
propose a lightweight module called DDM which can be simply attached to any
self-supervised encoder to enable training over multiple diverse domains to
produce well-structured, generalizable representations (See Figure 1).

Figure 2: Self-supervised baselines under single and multi-domain setups: We
plot 3 SOTA self-supervised baselines, SimCLR, BYOL and MoCo, trained
individually on PACS Photo, Sketch and Cartoon and on their mixture. We
observe that both single-domain and mult-domain training generalizes poorly to
unseen domains on all baselines. These baselines when pre-trained with DDM
(our method), outperforms even single-domain baselines and shows significantly
improved generalization to the unseen domains.

##  2 Related Work

Building on the success of unsupervised learning techniques [Bojanowski and
Joulin(2017), Dosovitskiy et al.(2014)Dosovitskiy, Springenberg, Riedmiller,
and Brox, YM. et al.(2020)YM., C., and A., Bautista et al.(2016)Bautista,
Sanakoyeu, Tikhoncheva, and Ommer, Caron et al.(2018a)Caron, Bojanowski,
Joulin, and Douze, Caron et al.(2019)Caron, Bojanowski, Mairal, and Joulin,
Huang et al.(2019)Huang, Dong, Gong, and Zhu], self-supervised models have
shown unprecedented capabilities when used in a range of downstream tasks.
Among a number of self-supervised baselines, we focus on SimCLR [Chen et
al.(2020a)Chen, Kornblith, Norouzi, and Hinton], MoCo [He et al.(2020)He, Fan,
Wu, Xie, and Girshick], BYOL [Grill et al.(2020)Grill, Strub, Altché, Tallec,
Richemond, Buchatskaya, Doersch, Avila Pires, Guo, Gheshlaghi Azar, Piot,
kavukcuoglu, Munos, and Valko], DINO [Caron et al.(2021)Caron, Touvron, Misra,
Jégou, Mairal, Bojanowski, and Joulin], SimSiam [Chen and He(2021)] and Barlow
Twins [Zbontar et al.(2021)Zbontar, Jing, Misra, LeCun, and Deny]. These are
joint-embedding self-supervised learning methods, which involve taking two
augmented views of the same input and ensuring their representations are close
using the same encoder or two encoders sharing the same weights.

Extending these self-supervised methods to multiple diverse domains, other
than ImageNet [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause,
Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, and Fei-Fei], is a
relatively less explored topic [Wallace and Hariharan(2020)]. Existing
approaches [Kim et al.(2021)Kim, Saito, Oh, Plummer, Sclaroff, and Saenko, Li
et al.(2021)Li, Shen, Wang, Zhu, Reed, Zhang, Li, Keutzer, and Zhao, Kim et
al.(2020)Kim, Saito, Oh, Plummer, Sclaroff, and Saenko] use pre-trained
encoders and assume few source labels for unsupervised domain adaption and
domain generalization. [Sun and Li(2022)] uses available class information and
novelty discovery to learn new samples in the wild. These works do not
consider fully unsupervised multi-domain setups, where even domain label
information is unavailable. [Feng et al.(2019)Feng, Xu, and Tao] assumes
domain labels and uses mutual information to encode common invariant
information and domain-specific information for each image. [Yang et
al.(2022)Yang, Chen, Wang, Tang, Zhu, Bai, Zhao, and Ouyang] uses multiple
domain-specific decoders to reconstruct images according to their domains such
that the encoder is domain-invariant. This method may not be scalable and is
contingent upon the number of available domains. [Zhang et al.(2022)Zhang,
Zhou, Xu, Cui, Shen, and Liu] proposes a contrastive method that selects
negatives across domains to train invariant representations. Our method
reports better numbers on the PACS dataset compared to these baselines. Our
method also does not assume domain labels and can be flexibly applied on any
self-supervised setup.

In our paper, we focus on a general multi-domain setup with diverse related or
unrelated domains, with and without shared classes, and evaluate on individual
domain-specific tasks. We make it possible to efficiently pre-train a single
encoder on any existing state-of-the-art self-supervised setup, over multiple
domains, to significantly improve their generalizability.

Figure 3: Visualizing Colored-CIFAR representations: We prepare Colored-CIFAR,
multi-domain version of CIFAR-10 [Krizhevsky et al.(a)Krizhevsky, Nair, and
Hinton] where the images are randomly colored red or green. We visualize the
top activating features of the class-averaged representations of both domains.
In the SimCLR baseline, we observe a clear difference in feature distribution
within the same classes, across domains. DDM enables representations to have a
shared domain-specific prefix while the remaining portion is domain-invariant
and almost identical across classes. This structure significantly improves
linear evaluation performance (See Figure 4).

##  3 Self-Supervised Models under Multi-domain Setups

We observed in Figure 2, that state-of-the-art self-supervised learning
methods like SimCLR [Chen et al.(2020a)Chen, Kornblith, Norouzi, and Hinton],
MoCo [Chen et al.(2020b)Chen, Fan, Girshick, and He] and BYOL [Grill et
al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila
Pires, Guo, Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and Valko] show low
transfer performance on unseen domains on both single-domain and multi-domain
regimes. In this section, we take a closer look at the learned representation
space under these regimes to explain this behavior.

We first define some notations. Let us consider a self-supervised model with a
base encoder $f(.)$. We apply data transformations and pass the input samples,
$\mathbf{x}_{i}\in\mathbb{R}^{n}$, through the base encoder to get self-
supervised representations denoted by
$f(\mathbf{x}_{i})=\mathbf{h}_{i}\in\mathbb{R}^{r}$ where $r$ is the size of
the representation space.

Let us take the example of SimCLR [Chen et al.(2020a)Chen, Kornblith, Norouzi,
and Hinton] trained on CIFAR-10 [Krizhevsky et al.(a)Krizhevsky, Nair, and
Hinton] dataset. In the first t-SNE [van der Maaten and Hinton(2008)] plot in
Figure 4(a), we observe that the representations are naturally clustered based
on their classes, which allows us to achieve a top-1 accuracy of $90.18$ after
linear probing. Let us now define a multi-domain version of CIFAR-10 called
Colored-CIFAR where, each sample is randomly colored either red or green as
shown in the first panel of Figure 3. In this dataset, the domains refer to
the colors of the image, while the labels are of the objects. When SimCLR is
trained on Colored-CIFAR, there is a significant drop in top-1 accuracy
($78.52$). We observe that the representation space is divided into two large
clusters, corresponding to the domains (red or green) as shown in 4(b). We
attribute the loss in accuracy to this significant change in representation
structure.

We now study the SimCLR representation space of Colored-CIFAR to further
understand and explain multi-domain behavior. In Figure 3, in the second
panel, we show a heatmap of the domain-wise averaged representations of each
class in CIFAR-10. Each column corresponds to specific feature indices of the
class-averaged representations. The darker the column, the higher the
magnitude of the feature. For fair comparison, we L2 normalize every feature.
For ease of visualization, we display only the subset of feature indices
(called most activating features) that are strongly deviated from the mean in
at least one row. The remaining features show low activation across the board
and are omitted from visualization [Jing et al.(2021)Jing, Vincent, LeCun, and
Tian, Kalibhat et al.(2022)Kalibhat, Narang, Firooz, Sanjabi, and Feizi]. Top
activating features correspond to important physical attributes discovered
from the training data [Kalibhat et al.(2022)Kalibhat, Narang, Firooz,
Sanjabi, and Feizi, Singla and Feizi(2021)]. Two images of a car, one in each
domain, would share all physical attributes except for the color. An ideal
self-supervised encoder is expected to encode all physical attributes
independent of any domain shift.

However, in multi-domain SimCLR, we observe that there is almost no overlap
between the most activating features of each class between the red and green
domains. This suggests that the domain information (color) and instance
information (actual content of the image) are somewhat interleaved in these
representations, causing different sets of features to be strongly activated
for the same class based on the domain. In single-domain SimCLR on CIFAR-10
(no colors), the representations only encode content information, which
results in linearly separable representations by class. In multi-domain SimCLR
on Colored-CIFAR, a combination of both domain and content information is
encoded in every representation which directly affects linear classification
performance. Therefore, to achieve comparable performance to single-domain
setups, we propose to disentangle domain information from representations by
plugging in a general-purpose a Domain-Disentanglement Module (DDM) for Self-
Supervised Models which is discussed in the next section.

|  |  |
---|---|---|---
(a) Accuracy: 90.18 | (b) Accuracy: 78.52 | (c) Accuracy: 87.06 (DDM) | (d) Accuracy: 87.06 (DDM)
Figure 4: SimCLR Representation t-SNE before and after DDM: CIFAR-10
representations are naturally clustered by class, however, Colored-CIFAR
representations are clustered by domain which leads to a significant reduction
in classification performance. When SimCLR is trained with DDM, the prefix
alone has domain-distinguishable representations, while the remaining portion
of the representation is domain-invariant, clustered by class. This structure
notably improves the classification performance.

##  4 Domain-Disentanglement Module for Self-Supervised Representations

As described in the previous section, self-supervised models in their current
state, are not trained to learn content and domain information independently.
We hypothesize that disentangling domain information from the learned
representations can improve the performance of existing state-of-the-art SSL
models in multi-domain setups. We therefore propose a general-purpose Domain-
Disentanglement Module (DDM) that can be simply attached at any SSL encoder
during its pre-training. In this work focus on joint-embedding (involving two
transformed views) self-supervised encoders [Chen et al.(2020a)Chen,
Kornblith, Norouzi, and Hinton, Caron et al.(2020)Caron, Misra, Mairal, Goyal,
Bojanowski, and Joulin, Chen et al.(2020b)Chen, Fan, Girshick, and He, Grill
et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch,
Avila Pires, Guo, Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and Valko, Caron
et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin,
Bardes et al.(2021)Bardes, Ponce, and LeCun, Zbontar et al.(2021)Zbontar,
Jing, Misra, LeCun, and Deny, Chen and He(2021)] and not masked image models
[He et al.(2022)He, Chen, Xie, Li, Dollar, and Girshick].

Recall that for a given sample $\mathbf{x}_{i}$, its representation is denoted
by $f(\mathbf{x}_{i})=\mathbf{h}_{i}\in\mathbb{R}^{r}$. Let $y_{i}$ denote the
domain of the $i^{th}$ representation. We allocate the first $k$ features of
the representation as the domain prefix, $\mathbf{h}_{i,0..k}$, denoted by
$\mathbf{h}_{i}^{d}$ for ease of notation. The remaining portion of the
representation $\mathbf{h}_{i,k..r}$ is denoted by $\mathbf{h}_{i}^{p}$. We
call $\mathbf{h}_{i}^{d}$ as the domain-variant portion and
$\mathbf{h}_{i}^{p}$ as the domain-invariant portion. We train the domain
prefix of the $i^{th}$ sample according to the following contrastive
optimization,
$L_{i_{\textit{d\\_var}}}=\log\frac{\sum_{j=1}^{2N}\mathbbm{1}_{j\neq
i}\mathbbm{1}_{y_{i}=y_{j}}sim(\mathbf{h}_{i}^{d},\mathbf{h}_{j}^{d})}{\sum_{j=1}^{2N}\mathbbm{1}_{y_{i}\neq
y_{j}}sim(\mathbf{h}_{i}^{d},\mathbf{h}_{j}^{d})}$.

where $sim(a,b)=\exp\left({\frac{1}{\tau}\frac{a^{T}b}{\|a\|\|b\|}}\right)$.
This loss maximizes the similarity of the domain prefixes within each domain
and minimizes the similarity of domain prefixes across domains.
$\mathbf{h}_{i}^{p}$ is learned according to any self-supervised loss like
SimCLR, MoCo, DINO etc., denoted by $L_{i_{\textit{ssl}}}$. Splitting the
representation in this manner helps us control each portion independently.
$L_{\textit{ssl}}$ ensures that all content information is encoded in a self-
supervised manner such that representations can be utilized for downstream
tasks. $L_{\textit{d\\_var}}$ ensures that the domain prefixes across samples
of different domains are distinguishable.

We next ensure that $\mathbf{h}_{i}^{p}$ does not contain any domain-related
information (domain-invariance constraint). In other words, it should not be
possible to predict the the domain label $y_{i}$ from the representation
$\mathbf{h}_{i}^{p}$. To achieve this, we pass each $\mathbf{h}_{i}^{p}$
through a domain discriminator $D(.)$ and minimize the Wasserstein distance
(using the dual form as proposed in [Arjovsky et al.(2017)Arjovsky, Chintala,
and Bottou]),
$L_{i_{\textit{d\\_invar}}}=D(\mathbf{h}_{i}^{p},y_{i})-D(\mathbf{h}_{i}^{p},y_{rand})$,
where $y_{rand}\sim\mathbbm{P}(y)$, i.e., randomly drawn from the distribution
of domain labels. The final optimization for the encoder ($f(.)$) and the
discriminator ($D(.)$) is,

$\displaystyle\max_{f}\sum_{i=1}^{2N}\left[\lambda
L_{i_{\textit{ssl}}}+L_{i_{\textit{d\\_var}}}+L_{i_{\textit{d\\_invar}}}\right]$
(1)

where $\lambda$ is a tunable hyperparameter. We optimize both the encoder
$f(.)$ and the discriminator $D$ using alternating gradient descent ascent. We
train $D(.)$ using gradient penalty to improve its stability as proposed in
[Gulrajani et al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and Courville].
This formulation is similar to [Shen et al.(2017)Shen, Qu, Zhang, and Yu,
Kattakinda et al.(2022)Kattakinda, Levine, and Feizi], except that we use
Wasserstein Distance to disentangle domain information from the remaining
portion of the representation space. In summary, our module DDM consists of
splitting the representation space into two parts and applying two additional
loss terms, $L_{\textit{d\\_var}}$ and $L_{\textit{d\\_invar}}$. Note that,
DDM can be plugged in while training any existing state-of-the-art self-
supervised model.

In Figure 3, in the last panel, we show the representation space of SimCLR
trained on Colored-CIFAR using DDM. We observe that among the most activating
features, the first few features (which are part of the domain prefix) are
equivalent for all classes within a domain and clearly distinguishable between
both domains. The remaining portion of the representation is completely
invariant to any domain information as each class shows very similar feature
distribution in both red and green domains. In the t-SNE plots (Figure 4(c)
and (d)), we observe that the domain prefix is separable by domain whereas the
domain-invariant portion shows natural class clusters with overlapping red and
green images. This update in structure leads to a significant improvement in
top-1 accuracy from 78.52 to 87.06.

###  4.1 Experimental Setup

We use ViT-S [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit,
and Houlsby] as the base encoder ($f(.)$) for all of our experiments. Our
domain discriminator ($D(.)$) is an MLP with LeakyReLU activations. The
representations are $384$-dimensional with a $24$-dimensional domain prefix.
We train the encoder according to various self-supervised baselines including
SimCLR [Chen et al.(2020a)Chen, Kornblith, Norouzi, and Hinton], MoCo [He et
al.(2020)He, Fan, Wu, Xie, and Girshick], BYOL [Grill et al.(2020)Grill,
Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo,
Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and Valko], DINO [Caron et
al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin],
SimSiam [Chen and He(2021)] and Barlow Twins [Zbontar et al.(2021)Zbontar,
Jing, Misra, LeCun, and Deny]. We use the same optimization and scheduling for
the encoder as the respective papers. While training with DDM, we use the Adam
optimizer for the domain discriminator with a learning rate of $0.005$ and
cosine-annealing scheduling and $\lambda=0.5$. We experiment with PACS [Li et
al.(2017)Li, Yang, Song, and Hospedales], DomainNet [Peng et al.(2019)Peng,
Bai, Xia, Huang, Saenko, and Wang] and the WILDS [Koh et al.(2021)Koh, Sagawa,
Marklund, Xie, Zhang, Balsubramani, Hu, Yasunaga, Phillips, Gao, Lee, David,
Stavness, Guo, Earnshaw, Haque, Beery, Leskovec, Kundaje, Pierson, Levine,
Finn, and Liang] multi-domain benchmarks. We use Nvidia GeForce RTX A4000 GPUs
for pre-training. We evaluate representations using the linear evaluation
protocol [Kolesnikov et al.(2019)Kolesnikov, Zhai, and Beyer, Bachman et
al.(2019)Bachman, Hjelm, and Buchwalter, van den Oord et al.(2019)van den
Oord, Li, and Vinyals] where we train a linear classifier on top of frozen
representations and compute the top-1 accuracy over the training and unseen
domains.

###  4.2 Self-Supervised Baselines Trained with DDM

In Figure 2, we observed that self-supervised baselines (SimCLR, BYOL and
MoCo), when trained on a single domain or multiple domains, generalize poorly
to unseen domains. These baselines, when pre-trained with DDM, show improved
performance on the training domains (PACS Photo, Sketch and Cartoon) as well
as significantly improved generalization to the unseen domain (PACS Art
Painting). Pre-training on multiple domains with DDM outperforms every self-
supervised baseline as shown in Table 1 with a maximum of $2.6\%$ improvement
on average top-1 accuracy on SimSiam. We also tabulate our results on
DomainNet using Painting, Real and Sketch as training domains and Clipart,
Infograph and Quickdraw as the unseen domains in Table 2. We observe that pre-
training with DDM improves upon each self-supervised baseline with a maximum
of $3.5\%$ improvement on average top-1 accuracy on BYOL. DDM generalizes
significantly better than its baselines showing a $6.1\%$ (SimSiam) increase
in PACS (Painting) and a $7.4\%$ (DINO) in DomainNet (Clipart).

To further evaluate the generalization of self-supervised baselines with DDM,
we utilize the WILDS benchmark [Koh et al.(2021)Koh, Sagawa, Marklund, Xie,
Zhang, Balsubramani, Hu, Yasunaga, Phillips, Gao, Lee, David, Stavness, Guo,
Earnshaw, Haque, Beery, Leskovec, Kundaje, Pierson, Levine, Finn, and Liang].
In this benchmark, pre-train on iWildCam (200K samples, 182 classes, 323
domains), Camelyon17 (456K samples, 2 classes, 5 domains), FMoW (141K samples,
62 classes, 80 domains) and RxRx1 (125K samples, 1139 classes, 51 domains). We
summarize our results in Table 3. On each benchmark, we observe that DDM
outperforms the baselines on the unseen validation set. The accuracy in rxrx1
is low since it is a very hard classification task as it contains 1139 classes
and 51 domains. We observe a $5.9\%$ increase linear classification accuracy
on iWildCam on SimCLR

Table 1: SSL baselines trained on PACS (Photo, Sketch and Cartoon) with DDM

Model | Top-1 Accuracy (Baseline / with DDM)
---|---
Photo | Sketch | Cartoon | Painting (Unseen) | Average
SimCLR | 97.54 / 98.28 |  98.12 / 97.04 | 98.03 / 99.24 | 87.59 / 89.42 | 95.32 / 96.00
MoCo |  93.59 / 93.19 | 92.71 / 94.36 | 91.63 / 92.98 | 77.34 / 78.51 | 88.81 / 89.76
BYOL | 78.08 / 81.61 | 76.55 / 78.24 | 75.55 / 75.58 | 58.10 / 62.67 | 72.07 / 74.53
DINO | 93.67 / 95.25 | 94.33 / 96.42 | 79.44 / 81.77 | 72.12 / 74.43 | 85.89 / 86.97
SimSiam | 83.68 / 84.71 | 80.97 / 85.44 |  93.75 / 92.59 | 57.98 / 64.09 | 79.09 / 81.71
Barlow Twins |  85.09 / 83.94 | 85.44 / 88.07 | 92.0 / 92.83 | 59.01 / 62.67 | 80.39 / 81.89

Table 2: SSL baselines trained on DomainNet (Painting, Real and Sketch) with
DDM

Model | Top-1 Accuracy (Baseline / with DDM)
---|---
Painting | Real | Sketch | Clipart (Unseen) | Infograph (Unseen) | Quickdraw (Unseen) | Average
SimCLR | 74.49 / 75.99 | 79.31 / 82.02 | 85.86 / 86.26 | 68.60 / 70.48 | 34.75 / 39.25 | 22.98 / 24.38 | 60.99 / 63.06
MoCo | 70.20 / 73.08 |  89.79 / 86.37 | 86.66 / 88.15 | 65.10 / 68.91 | 34.56 / 34.75 | 19.89 / 22.12 | 61.03 / 62.23
BYOL | 56.87 / 59.82 | 77.60 / 79.67 | 71.43 / 75.21 | 50.67 / 55.86 | 27.4 / 30.68 | 19.33 / 22.85 | 50.55 / 54.02
DINO |  79.53 / 79.11 | 86.46 / 86.88 | 75.8 / 76.50 | 66.32 / 73.76 | 30.83 / 32.12 | 27.71 / 29.08 | 61.11 / 62.90
SimSiam | 77.55 / 78.78 | 82.02 / 85.88 | 86.52 / 88.38 | 67.43 / 71.53 | 27.03 / 30.56 | 22.29 / 25.67 | 60.47 / 63.47
Barlow Twins | 56.78 / 61.18 | 79.06 / 80.16 | 71.56 / 73.90 | 60.40 / 64.33 | 26.11 / 28.82 | 18.67 / 21.70 | 52.09 / 55.01

Table 3: SSL baselines trained on WILDS with DDM

| Top-1 Accuracy (Baseline / with DDM)
---|---
Model | iWildCam | Camelyon17 | FMoW | RxRx1
SimCLR | 66.01 / 71.87 | 95.19 / 95.68 | 38.94 / 41.23 | 8.43 / 11.20
MoCo | 67.05 / 69.12 | 91.45 / 93.47 | 40.04 / 40.23 | 5.67 / 5.93
BYOL | 71.69 / 74.88 | 95.15 / 96.38 | 38.74 / 39.78 | 4.39 / 6.20
DINO | 64.55 / 68.07 | 94.38 / 95.38 | 33.57 / 34.52 | 7.32 / 7.66
SimSiam | 60.45 / 61.16 | 88.37 / 89.16 | 39.27 / 40.05 | 6.39 / 7.26
Barlow Twins | 63.17 / 63.84 | 96.38 / 97.62 | 44.40 / 47.46 | 5.79 / 6.65

Figure 5: DDM with clustering: When domain labels are not available, we
perform DDM with clustering to identify pseudo-domain-labels. In the above
plots we show the t-SNE of the SimCLR representations trained on "Cartoon" and
"Sketch" domains in PACS. We observe that DDM with robust clustering produces
a better separation between domains.

##  5 DDM without Domain Labels

Most real-world multi-domain datasets are unlabelled (i.e., domain label
information is not available). In this section, we develop an extension of DDM
for such setups by identifying pseudo domain labels via a clustering approach
in the representation space. As it is common in clustering, we assume the
number of domains (denoted by $M$) is known. Depending on the multi-domain
setup, we can also approximate the number of domains by studying any available
meta-data like data sources, geo-location, quality, etc. We can also estimate
the number of domains empirically through clustering and visualization.

Domain labels are required in both DDM losses ($L_{\textit{d\\_var}}$,
$L_{\textit{d\\_invar}}$) as described in the previous section. Let us
consider a fully unlabelled setup, with no domain labels while the number of
domains $M$ is known. We first warm up our self-supervised encoder $f(.)$
treating it as a single-domain setup for a few iterations to get somewhat
distinguishable representations by domain. We next cluster the representations
into $M$ clusters using K-Means clustering [Hartigan and Wong(1979)]. Using
the cluster assignments as pseudo-domain-labels ($y$), we continue training
the encoder $f(.)$ along with a discriminator using the DDM optimization, to
learn domain-disentangled representations.

In practice, clustering does not discover $100\%$ accurate domain labels,
especially for datasets that are distributionally similar. We therefore use a
robust clustering approach coupled with DDM to prevent outlier clustering
noise from affecting the pseudo-domain-labels. Suppose we discover $M$
clusters with centroids $\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{M}$,
before assigning pseudo-domain-labels to each sample, we first determine if
they are outliers or not. If so, we ignore these samples in the next stages of
training to prevent assigning a noisy label to them. We say a representation
is not an outlier if it is significantly closer to one of the clustering
centroids compared to another. Concretely, $\mathbf{h}_{i}$ is not an outlier
if

$\displaystyle\max\left\\{\frac{\|\mathbf{h}_{i}-\mathbf{c}_{m}\|^{2}}{\|\mathbf{h}_{i}-\mathbf{c}_{n}\|^{2}}\
:1\leq m\leq M,1\leq n\leq M\right\\}>1+\epsilon$ (2)

where $\epsilon\geq 0$ is defined as the outlier threshold. When $\epsilon$ is
high, it means that the given sample is very close to its respective centroid.
When $\epsilon$ approaches $0$, it indicates that the sample is almost
equidistant from at least two centroids and therefore, may not be reliably
assigned one pseudo-label. We ignore such samples going forward in training.
When we perform clustering for the first time, we start with $\epsilon=1$. We
repeat the clustering at regular intervals of training on the representations
$\mathbf{h}$ to get improved cluster centroids. Each time we repeat
clustering, we decay the value of $\epsilon$ exponentially such that it
approaches $0$. By the end of training, all samples will contribute to the
training of the self-supervised encoder with DDM. In Figure 5, we illustrate
the difference between regular clustering and robust clustering with MDSSL
trained on the PACS dataset [Li et al.(2017)Li, Yang, Song, and Hospedales]
("Cartoon" and "Sketch" domains). We observe that robust clustering helps in
identifying more accurate and distinguishable clusters.

To evaluate DDM with robust clustering, we combine CIFAR-10 [Krizhevsky et
al.(a)Krizhevsky, Nair, and Hinton], CIFAR-100 [Krizhevsky et
al.(b)Krizhevsky, Nair, and Hinton] and STL-10 [Coates et al.()Coates, Lee,
and Ng] to form a multi-domain dataset. The constituent datasets are
distributionally similar with several shared classes (CIFAR-10 and STL-10
share 9 out of 10 classes). With this setup, we try to simulate a real-world
scenario where data arises from various domains however the actual domains are
undefined. We therefore apply DDM with robust-clustering to identify pseudo-
domain-labels. We then evaluate the pre-trained representations by linear
probing the validation portion of each constituent dataset. We include Tiny-
ImageNet [Le and Yang(2015)] as an unseen domain to test generalization.

In Table 4, we tabulate the results on this prepared multi-domain dataset on
various self-supervised baselines with and without DDM and robust clustering.
We observe an improvement in the average top-1 accuracy across all baselines
with $1.7\%$ improvement in MoCo. DDM shows improved generalization on Tiny-
ImageNet with a $2.9\%$ increase in DINO.

Table 4: SSL baselines trained on a mixture of CIFAR-10, STL-10 and CIFAR-100
using DDM and robust clustering

Model | Top-1 Accuracy (Baseline / with DDM and robust clustering)
---|---
CIFAR-10 | STL-10 | CIFAR-100 | Tiny-ImageNet (Unseen) | Average
SimCLR | 89.43 / 90.03 | 79.77 / 81.01 | 63.33 / 64.90 | 49.58 / 51.22 | 70.53 / 71.79
MoCo |  90.80 / 90.69 | 80.02 / 81.60 | 61.57 / 64.28 | 37.16 / 39.55 | 67.38 / 69.03
BYOL | 88.31 / 89.68 | 75.07 / 75.72 | 64.82 / 65.56 | 50.04 / 51.10 | 69.56 / 70.52
DINO | 90.61 / 92.96 |  84.7 / 82.35 | 62.63 / 63.57 | 49.52 / 52.46 | 71.87 / 72.84
SimSiam | 87.02 / 87.38 | 72.15 / 73.78 |  62.08 / 61.90 | 33.11 / 34.78 | 63.59 / 64.46
Barlow Twins | 88.31 / 89.01 | 75.59 / 76.11 | 65.03 / 66.89 | 40.27 / 41.31 | 67.30 / 68.33

##  6 Conclusion

We proposed a Domain Disentanglement Module (DDM) for self-supervised encoders
that provide better structured representations, domain-invariant
representations that can be used for diverse multi-domain tasks. DDM also
supports training over setups where domain labels are not available by using a
robust clustering technique that reduces outlier noise. With DDM, we prevent
the need for having to train multiple single-domain encoders and instead
leverage a single encoder to perform comparably on multiple domains. The
benefit of invariant representations is better generalization which we show on
various benchmarks including PACS, DomainNet and WILDS.

## References

  * [Arjovsky et al.(2017)Arjovsky, Chintala, and Bottou] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan, 2017.
  * [Bachman et al.(2019)Bachman, Hjelm, and Buchwalter] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/ddf354219aac374f1d40b7e760ee5bb7-Paper.pdf.
  * [Bardes et al.(2021)Bardes, Ponce, and LeCun] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning, 2021.
  * [Bautista et al.(2016)Bautista, Sanakoyeu, Tikhoncheva, and Ommer] Miguel A Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Bjorn Ommer.  Cliquecnn: Deep unsupervised exemplar learning.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper/2016/file/65fc52ed8f88c81323a418ca94cec2ed-Paper.pdf.
  * [Bojanowski and Joulin(2017)] Piotr Bojanowski and Armand Joulin.  Unsupervised learning by predicting noise.  In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 517–526. PMLR, 06–11 Aug 2017.  URL http://proceedings.mlr.press/v70/bojanowski17a.html.
  * [Caron et al.(2018a)Caron, Bojanowski, Joulin, and Douze] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018a.
  * [Caron et al.(2018b)Caron, Bojanowski, Joulin, and Douze] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  _Lecture Notes in Computer Science_ , page 139–156, 2018b.  ISSN 1611-3349.  10.1007/978-3-030-01264-9_9.  URL http://dx.doi.org/10.1007/978-3-030-01264-9_9.
  * [Caron et al.(2019)Caron, Bojanowski, Mairal, and Joulin] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin.  Unsupervised pre-training of image features on non-curated data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 9912–9924. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf.
  * [Caron et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9650–9660, October 2021.
  * [Chen et al.(2020a)Chen, Kornblith, Norouzi, and Hinton] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1597–1607. PMLR, 13–18 Jul 2020a.  URL http://proceedings.mlr.press/v119/chen20j.html.
  * [Chen and He(2021)] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15750–15758, June 2021.
  * [Chen et al.(2020b)Chen, Fan, Girshick, and He] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning, 2020b.
  * [Coates et al.()Coates, Lee, and Ng] Adam Coates, Honglak Lee, and Andrew Y. Ng.  Stanford stl-10 image dataset.  URL https://cs.stanford.edu/~acoates/stl10/.
  * [Dosovitskiy et al.(2014)Dosovitskiy, Springenberg, Riedmiller, and Brox] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with convolutional neural networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc., 2014.  URL https://proceedings.neurips.cc/paper/2014/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [Feng et al.(2019)Feng, Xu, and Tao] Zeyu Feng, Chang Xu, and Dacheng Tao.  Self-supervised representation learning from multi-domain data.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3244–3254, 2019.
  * [Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo, Gheshlaghi Azar, Piot, kavukcuoglu, Munos, and Valko] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 21271–21284. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf.
  * [Gulrajani et al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and Courville] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of wasserstein gans, 2017.
  * [Hartigan and Wong(1979)] J. A. Hartigan and M. A. Wong.  A k-means clustering algorithm.  _JSTOR: Applied Statistics_ , 28(1):100–108, 1979\.
  * [He et al.(2020)He, Fan, Wu, Xie, and Girshick] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9726–9735, 2020.  10.1109/CVPR42600.2020.00975.
  * [He et al.(2022)He, Chen, Xie, Li, Dollar, and Girshick] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick.  Masked autoencoders are scalable vision learners.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun 2022.  10.1109/cvpr52688.2022.01553.  URL http://dx.doi.org/10.1109/CVPR52688.2022.01553.
  * [Huang et al.(2019)Huang, Dong, Gong, and Zhu] Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu.  Unsupervised deep learning by neighbourhood discovery.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2849–2858. PMLR, 09–15 Jun 2019.  URL http://proceedings.mlr.press/v97/huang19b.html.
  * [Jing et al.(2021)Jing, Vincent, LeCun, and Tian] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  _ArXiv_ , abs/2110.09348, 2021.
  * [Kalibhat et al.(2022)Kalibhat, Narang, Firooz, Sanjabi, and Feizi] Neha Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi.  Measuring self-supervised representation quality for downstream classification using discriminative features, 2022.
  * [Kattakinda et al.(2022)Kattakinda, Levine, and Feizi] Priyatham Kattakinda, Alexander Levine, and Soheil Feizi.  Invariant learning via diffusion dreamed distribution shifts, 2022.
  * [Khosla et al.(2020)Khosla, Teterwak, Wang, Sarna, Tian, Isola, Maschinot, Liu, and Krishnan] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning, 2020.
  * [Kim et al.(2020)Kim, Saito, Oh, Plummer, Sclaroff, and Saenko] Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A. Plummer, Stan Sclaroff, and Kate Saenko.  Cross-domain self-supervised learning for domain adaptation with few source labels, 2020.
  * [Kim et al.(2021)Kim, Saito, Oh, Plummer, Sclaroff, and Saenko] Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A. Plummer, Stan Sclaroff, and Kate Saenko.  Cds: Cross-domain self-supervised pre-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9123–9132, October 2021.
  * [Koh et al.(2021)Koh, Sagawa, Marklund, Xie, Zhang, Balsubramani, Hu, Yasunaga, Phillips, Gao, Lee, David, Stavness, Guo, Earnshaw, Haque, Beery, Leskovec, Kundaje, Pierson, Levine, Finn, and Liang] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.  Wilds: A benchmark of in-the-wild distribution shifts, 2021.
  * [Kolesnikov et al.(2019)Kolesnikov, Zhai, and Beyer] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1920–1929, 2019.  10.1109/CVPR.2019.00202.
  * [Krizhevsky et al.(a)Krizhevsky, Nair, and Hinton] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  a.  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * [Krizhevsky et al.(b)Krizhevsky, Nair, and Hinton] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-100 (canadian institute for advanced research).  b.  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * [Le and Yang(2015)] Ya Le and X. Yang.  Tiny imagenet visual recognition challenge.  2015\.
  * [Li et al.(2021)Li, Shen, Wang, Zhu, Reed, Zhang, Li, Keutzer, and Zhao] Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Colorado J. Reed, Jun Zhang, Dongsheng Li, Kurt Keutzer, and Han Zhao.  Invariant information bottleneck for domain generalization, 2021.
  * [Li et al.(2017)Li, Yang, Song, and Hospedales] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales.  Deeper, broader and artier domain generalization.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.  10.1109/iccv.2017.591.  URL http://dx.doi.org/10.1109/ICCV.2017.591.
  * [Peng et al.(2019)Peng, Bai, Xia, Huang, Saenko, and Wang] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1406–1415, 2019.
  * [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, and Fei-Fei] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  10.1007/s11263-015-0816-y.
  * [Shen et al.(2017)Shen, Qu, Zhang, and Yu] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu.  Wasserstein distance guided representation learning for domain adaptation, 2017.
  * [Singla and Feizi(2021)] Sahil Singla and Soheil Feizi.  Salient imagenet: How to discover spurious features in deep learning?, 2021.
  * [Sun and Li(2022)] Yiyou Sun and Yixuan Li.  Opencon: Open-world contrastive learning, 2022.
  * [van den Oord et al.(2019)van den Oord, Li, and Vinyals] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2019.
  * [van der Maaten and Hinton(2008)] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.  URL http://jmlr.org/papers/v9/vandermaaten08a.html.
  * [Wallace and Hariharan(2020)] Bram Wallace and Bharath Hariharan.  Extending and analyzing self-supervised learning across domains.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision – ECCV 2020_ , pages 717–734, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-58574-7.
  * [Wu et al.(2018)Wu, Xiong, Yu, and Lin] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Yang et al.(2022)Yang, Chen, Wang, Tang, Zhu, Bai, Zhao, and Ouyang] Haiyang Yang, Meilin Chen, Yizhou Wang, Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, and Wanli Ouyang.  Domain invariant masked autoencoders for self-supervised learning from multi-domains, 2022.
  * [YM. et al.(2020)YM., C., and A.] Asano YM., Rupprecht C., and Vedaldi A.  Self-labelling via simultaneous clustering and representation learning.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=Hyx-jyBFPr.
  * [Zbontar et al.(2021)Zbontar, Jing, Misra, LeCun, and Deny] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction, 2021\.
  * [Zhang et al.(2022)Zhang, Zhou, Xu, Cui, Shen, and Liu] Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu.  Towards unsupervised domain generalization.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun 2022.  10.1109/cvpr52688.2022.00486.  URL http://dx.doi.org/10.1109/CVPR52688.2022.00486.