# Augmentation-aware Self-supervised Learning with Conditioned Projector

Marcin Przewięźlikowski1,2,3 Mateusz Pyla1,2,3 Bartosz Zieliński1,3
Bartłomiej Twardowski3,4 Jacek Tabor1 Marek Śmieja1
1 Jagiellonian University, Faculty of Mathematics and Computer Science
2 Jagiellonian University, Doctoral School of Exact and Natural Sciences
3 IDEAS NCBR 4 Universitat Autònoma Barcelona Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning (SSL) is a powerful technique for learning robust
representations from unlabeled data. By learning to remain invariant to
applied data augmentations, methods such as SimCLR and MoCo are able to reach
quality on par with supervised approaches. However, this invariance may be
harmful to solving some downstream tasks which depend on traits affected by
augmentations used during pretraining, such as color. In this paper, we
propose to foster sensitivity to such characteristics in the representation
space by modifying the projector network, a common component of self-
supervised architectures. Specifically, we supplement the projector with
information about augmentations applied to images. In order for the projector
to take advantage of this auxiliary conditioning when solving the SSL task,
the feature extractor learns to preserve the augmentation information in its
representations. Our approach, coined Conditional Augmentation-aware Self-
supervised Learning (CASSLE), is directly applicable to typical joint-
embedding SSL methods regardless of their objective functions. Moreover, it
does not require major changes in the network architecture or prior knowledge
of downstream tasks. In addition to an analysis of sensitivity towards
different data augmentations, we conduct a series of experiments, which show
that CASSLE improves over various SSL methods, reaching state-of-the-art
performance in multiple downstream tasks.111We share our codebase at
https://github.com/gmum/CASSLE.

##  1 Introduction

Artificial neural networks have proven to be a successful family of models in
several domains, including, but not limited to, computer vision [33], natural
language processing [11], and solving problems at the human level with
reinforcement learning [45]. This success is attributed largely to their
ability to learn useful feature representations [29] without additional effort
for input signals preparation. However, training large deep learning models
requires extensive amounts of data, which can be costly to prepare, especially
when human annotation is needed [3, 36].

High-quality image representations can be acquired without relying on
explicitly labeled data by utilizing self-supervised learning (SSL). A self-
supervised model is trained once on a large dataset without labels and then
transferred to different downstream tasks. Initially, self-supervised methods
addressed well-defined pretext tasks, such as predicting rotation [28] or
determining patch position [22]. Recent studies in SSL proposed contrastive
methods of learning representations that remain invariant when subjected to
various data augmentations [32, 15, 18] leading to impressive results that
have greatly diminished the disparity with representations learned in a
supervised way [13].

Nevertheless, contrastive methods may perform poorly when a particular
downstream task relies on features affected by augmentation [69]. For example,
color jittering can result in a representation space invariant to color
shifts, which would be detrimental to the task of flower classification (see
Figure 1). Without prior knowledge of possible downstream tasks, this effect
is hard to mitigate in contrastive learning [62, 69]. Solutions for retaining
information about used data augmentations in the feature extractor
representation include forcing it explicitly with a modified training scheme
[69, 40, 70], or by preparing a feature extractor to be adapted to a specific
downstream task, e.g., with hypernetworks [14]. However, these approaches
often involve significant modifications either to the contrastive model
architecture [69], training procedure [40, 70], or costly training of
additional models [14].

Figure 1: In the traditional self-supervised setting, contrastive loss
minimization pulls the representations of augmented image views closer in the
latent space of the projector (left). This may also reduce the distance
between their feature extractor representations (right). Thus, the
representation becomes invariant to augmentation-induced perturbations, which
may hinder the performance on downstream tasks. In contrast, the self-
supervised objective of CASSLE draws together joint representations of images
and their augmentations in the projector space (bottom row). By conditioning
the projector with augmentation information, image representations retain more
sensitivity to perturbations in the feature extractor space. This proves to be
beneficial when solving downstream tasks.

In this work, we propose a new method called Conditional Augmentation-aware
Self-supervised Learning (CASSLE) that mitigates augmentation invariance of
representation without neither major changes in network architecture or
modifications to the self-supervised training objective. We propose to use the
augmentation information during the SSL training as additional conditioning
for the projector network. This encourages the feature extractor network to
retain information about augmented image features in its representation.
CASSLE can be applied to any joint-embedding SSL method regardless of its
objective [17, 15, 18, 72, 19]. The outcome is a general-purpose,
augmentation-aware encoder that can be directly used for any downstream task.
CASSLE presents improved results in comparison to other augmentation-aware SSL
methods, improving transferability to downstream tasks where invariance of the
model representation for specific data changes could be harmful.

The main contributions of our work are threefold:

  * •

We propose a simple yet effective method for Conditional Augmentation-aware
Self-supervised Learning (CASSLE). Using our conditioned projector enables
preserving more information about augmentations in representations than in
existing methods.

  * •

CASSLE is a general modification that can be directly applied to existing
joint-embedding SSL approaches without introducing additional objectives and
major changes in the network architecture.

  * •

In a series of experiments we demonstrate that CASSLE reaches state-of-the-art
performance with different SSL methods for robust representation learning and
improves upon the performance of previous augmentation-aware approaches.
Furthermore, our analysis indicates that CASSLE learns representations with
increased augmentation sensitivity compared to other approaches.

##  2 Related work

#### Self-supervised learning

(SSL) is a paradigm of learning representations from unlabeled data that can
later be used for downstream tasks defined by human annotations [2, 4].
Despite learning artificial _pretext tasks_ , instead of data-defined ones,
SSL models have achieved tremendous success in a plethora of domains [21, 67,
60, 7]. This includes computer vision, where a variety of pretext tasks has
been proposed [22, 74, 48, 28]. However, arguably the most prominent and
successful SSL technique to emerge in recent years is the training of joint-
embedding models for augmentation invariance [6, 64], defined by objectives
such as contrastive InfoNCE loss [32, 15, 17], self-distillation [30, 15, 49]
or Canonical Correlation Analysis [12, 72, 5]. Those objectives are often
collectively referred to as _contrastive objectives_ [63, 4]. A common
component of joint-embedding architectures is the _projector network_ , which
maps representations of the feature extractor into the space where the
contrastive objective is imposed [15, 17]. The usefulness of the projector has
been explained through the lens of transfer learning, where it is often better
to transfer intermediate network representations, to reduce the biases from
the pretraining task [71, 9]. The projector also helps to mitigate the noisy
data augmentations and enforces some degree of pairwise independence of image
features [4, 44].

#### Augmentation invariance of self-supervised models

is a natural consequence of training them with contrastive objectives, as SSL
methods are prone to suppressing features that are not useful for optimizing
the contrastive objectives [16, 57]. While a common set of augmentations
demonstrated to typically work well on natural images in SSL has been
established in the literature [32, 15, 18, 12, 75], the optimal choice of
augmentations varies between specific tasks [62, 24]. [69] find that
augmentation invariance can hinder the performance of the model on downstream
tasks that require attention to precisely those traits that it had been
previously trained to be invariant to. On the other hand, [73] observe that
the objective of predicting augmentation parameters can be in itself a useful
pretext task for SSL. Those works inspired several techniques of retaining
augmentation-specific information in joint-embedding models, such as
projectors sensitive to different augmentation types [69, 24], adding an
objective of explicit prediction of augmentation parameters [40], , as well as
task-specific pretraining [55, 65]. The above approaches produce general-
purpose feature extractors that can be transferred to downstream tasks without
further tuning of their parameters. However, they often involve complex
modifications either to the SSL model architecture [69], training procedure
[40, 70], or simply tedious task-specific pretraining [65]. Another line of
work proposes to train Hypernetworks [31] which produce feature extractors
invariant to chosen subsets of augmentations – a more elastic, but
considerably harder to train approach [14]. Several works have proposed
fostering the equivariance of representations to data transformations using
augmentation information. [70] modulate the contrastive objective with
augmentation strength, [8] use augmentation information as a signal for
equivariance regularization in the supervised setting, whereas [27] extend the
VicReg [5] objective with a predictor whose parameters are generated from
augmentation information by a hypernetwork [31]. Compared to those works, we
do not make any modifications to the objective functions of the extended
approaches and evaluate CASSLE on a wide range of joint-embedding approaches
and conditioning techniques. Following [69, 40], we produce a general-purpose
feature extractor and utilize augmentation information similarly to [73, 40,
14]. Contrary to the above methods, we inject the information about the
applied augmentations directly into the projector and make no modification
either to the contrastive objective or the feature extractor.

##  3 Method

In this section, we present our approach, Conditional Augmentation-aware Self-
supervised learning (CASSLE). Section 3.1 provides background on joint-
embedding self-supervised methods and their limitations. Section 3.2 explains
the essence of CASSLE and how it leverages augmentation information to improve
the quality of learned representations. Section 3.3 details the practical
implementation of CASSLE’s conditioning mechanism.

Figure 2: Overview of CASSLE. We extend the typical self-supervised learning
approaches by incorporating the information of augmentations applied to images
into the projector network. In CASSLE, the SSL objective is thus imposed on
joint representations of images and the augmentations that had been applied to
them. This way, CASSLE enables the feature extractor to be more aware of
augmentations than the methods that do not condition the projector network.

###  3.1 Preliminaries

A typical contrastive framework used in self-supervised learning consists of
an augmentation function $t_{\omega}$ and two networks: feature extractor $f$
and projector $\pi$. Let
$\mathbf{v}_{1}=t_{\mathbf{\omega}_{1}}(\mathbf{x}),\mathbf{v}_{2}=t_{\mathbf{\omega}_{2}}(\mathbf{x})$
be two augmentations of a sample $\mathbf{x}\sim X$ parameterized by
$\mathbf{\omega}_{1},\mathbf{\omega}_{2}\sim\Omega$. The feature extractor
maps them into the embedding space, which is the representation used in
downstream tasks. To make the representation invariant to data augmentations,
$\mathbf{e}_{1}=f(\mathbf{v}_{1})$ is forced to be similar to
$\mathbf{e}_{2}=f(\mathbf{v}_{2})$222While contrastive objectives such as
InfoNCE [64] regularize the representation using negative pairs, we omit them
from our notation for the sake of brevity.. However, instead of imposing
similarity constraints directly on the embedding space of $f$, a projector
$\pi$ transforms the embeddings into target space where the contrastive loss
$\mathcal{L}$ is applied. This trick, known as _Guillotine Regularization_ ,
helps the feature extractor to better generalize to downstream tasks, due to
$f$ not being directly affected by $\mathcal{L}$ [71, 15, 17, 9].

Minimizing $\mathcal{L}(\pi(\mathbf{e}_{1}),\pi(\mathbf{e}_{2}))$ directly
leads to reducing the distance between embeddings $\pi(\mathbf{e}_{1})$ and
$\pi(\mathbf{e}_{2})$. However, $\mathcal{L}$ still indirectly encourages
intermediate network representations (including the output of the feature
extractor $f$) to also conform to the contrastive objective to some extent. As
a result, the feature extractor tends to erase the information about
augmentation from its output representation. This behavior may however be
detrimental for certain downstream tasks (see Figures 1 and 4), which rely on
features affected by augmentations. For instance, learning invariance to color
jittering through standard contrastive methods may lead to degraded
performance on the downstream task of flower recognition, which is not a
color-invariant task [62, 69]. Therefore, the success of typical SSL
approaches depends critically on a careful choice of augmentations used for
model pretraining [15, 62].

###  3.2 CASSLE

To overcome the above limitations of SSL, we facilitate the feature extractor
to encode the information about augmentations in its output representation. In
consequence, the obtained representation will be more informative for
downstream tasks that depend on features modified by augmentations.

CASSLE achieves this goal by conditioning the projector $\pi$ on the
parameters of augmentations used to perturb the input image. Specifically, we
modify $\pi$ so that apart from embedding $\mathbf{e}$, it also receives
augmentation information $\omega$ and projects their joint representation into
the space where the objective $\mathcal{L}$ is imposed. We do not alter the
$\mathcal{L}$ itself; instead, training relies on minimizing the contrastive
loss $\mathcal{L}$ between $\pi(\mathbf{e}_{1}|\mathbf{\omega}_{1})$ and
$\pi(\mathbf{e}_{2}|\mathbf{\omega}_{2})$. Thus, $\pi$ learns to draw
$\mathbf{e}_{1}$ and $\mathbf{e}_{2}$ together in its representation space _on
condition of_ $\mathbf{\omega}_{1}$ _and_ $\mathbf{\omega}_{2}$. We visualize
the architecture of CASSLE in Figure 2.

We provide a rationale for why CASSLE preserves information about augmented
features in the representation space. Since augmentation information vectors
$\omega$ do not carry any information about source images $\mathbf{x}$, their
usefulness during pretraining could be explained only by using knowledge of
transformations $t_{\omega}$ that had been applied to $\mathbf{x}$ to form
views $\mathbf{v}$. However, for such knowledge to be acted upon, features
affected by $t_{\omega}$ must be preserved in the feature extractor
representation $f(\mathbf{v})$.

Let us assume the opposite, that $\omega$ is not useful for CASSLE to solve
the task defined by $\mathcal{L}$. If this is the case, then for any
$\omega_{3}\sim\Omega$ the following would hold:

$p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))=p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{3})).$
(1)

$p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))$ can be
understood as conditional probability that _$\pi(\mathbf{e}_{1}|\omega_{1})$
is a representation of an image $\mathbf{x}$ transformed by $t_{\omega_{1}}$,
given the knowledge that $\pi(\mathbf{e}_{2}|\omega_{2})$ is a representation
of $\mathbf{x}$ transformed by $t_{\omega_{2}}$_. Equation 1 implies that
replacing the knowledge of $t_{\omega_{2}}$ with any other randomly sampled
$t_{\omega_{3}}$ does not affect the inference process of CASSLE.

Figure 3: Similarities of CASSLE projector representations when conditioned
with augmentation information from either their respective images (green), or
randomly sampled (red). Solid lines denote the mean values of similarities.
Conditioning the CASSLE projector with wrong augmentation information
decreases its ability to draw image pairs together, indicating that it indeed
relies on augmentation information to perform its task.

To demonstrate that this is not the case, we measure the similarity of
representations of 5000 positive image pairs from the ImageNet-100 test set,
where one representation is always constructed using true information about
applied augmentations, and the second is constructed using either true or
randomly sampled augmentation information. It is evident from Figure 3 that
the similarity of embeddings decreases when false augmentation parameters
($\omega_{3}$ instead of $\omega_{2}$) are supplied to the projector, i.e:

$\begin{split}\mathbb{E}_{x\sim\mathbb{X},\\{\omega_{1},\omega_{2},\omega_{3}\\}\sim\Omega}[\\\
sim\left(\pi(\mathbf{e}_{1}|\omega_{1}),\pi(\mathbf{e}_{2}|\omega_{2})\right)-\\\
sim\left(\pi(\mathbf{e}_{1}|\omega_{1}),\pi(\mathbf{e}_{2}|\omega_{3})\right)\\\
]>0.\end{split}$ (2)

Recall that in contrastive SSL, the similarity of embeddings corresponds to
their probability density. This is because the InfoNCE loss is formulated as
cross-entropy, where the activation function is defined as similarity between
respective image embeddings, and class labels are replaced with the indices of
corresponding positive embedding pairs [64, 32, 15]. Hence, minimizing
$\mathcal{L}$ leads to:

$sim\left(\pi(\mathbf{e}_{1}|\omega_{1}),\pi(\mathbf{e}_{2}|\omega_{2})\right)\quad\propto\quad\frac{p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))}{p(\pi(\mathbf{e}_{1}|\omega_{1}))}.$
(3)

It follows from 2 that, in practice

$\mathbb{E}_{x\sim\mathbb{X},\\{\omega_{1},\omega_{2},\omega_{3}\\}\sim\Omega}\left[\frac{p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))-p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{3}))}{p(\pi(\mathbf{e}_{1}|\omega_{1}))}\right]>0$
(4)

and, since $p(\pi(\mathbf{e}_{1}|\omega_{1}))>0$,

$\mathbb{E}_{x\sim\mathbb{X},\\{\omega_{1},\omega_{2},\omega_{3}\\}\sim\Omega}\left[p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))-p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{3}))\right]>0.$
(5)

Moreover, we measure whether
$p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{2}))>p(\pi(\mathbf{e}_{1}|\omega_{1})|\pi(\mathbf{e}_{2}|\omega_{3}))$
for each of the considered image view pairs and find it to be true in 92% of
the considered cases.

In CASSLE, _the conditional probability of matching a positive pair of image
representations increases when the correct augmentation information is known_
, which implies that information pertaining to augmented features is indeed
preserved in the representation of it’s feature extractor.

CASSLE can be applied to a variety of joint-embedding SSL methods, as the only
practical modification it makes is changing the projector network to utilize
the additional input $\omega$, describing the augmentations. We do not modify
any other aspects of the self-supervised approaches, such as objective
functions, which is appealing from a practical perspective. Last but not
least, the architecture of the feature extractor in CASSLE is not affected by
the introduced augmentation conditioning, as we only modify the input to the
projector, which is discarded after the pretraining. Just like in vanilla SSL
techniques, the feature extractor can be directly used in downstream tasks.

###  3.3 Practical implementation of the conditioning mechanism

In this work, we focus on a set of augmentations used commonly in the
literature [15, 17, 18], listed below along with descriptions of their
respective parameters $\omega^{aug}$:

  * •

random cropping – $\mathbf{\omega}^{c}\in[0,1]^{4}$ describes the normalized
coordinates of cropped image center and cropping sizes.

  * •

color jittering – $\mathbf{\omega}^{j}\in[0,1]^{4}$ describes the normalized
intensities of brightness, contrast, saturation, and hue adjustment.

  * •

Gaussian blurring – $\mathbf{\omega}^{b}\in[0,1]$ is the standard deviation of
the Gaussian filter used during the blurring operation.

  * •

random horizontal flipping – $\mathbf{\omega}^{f}\in\\{0,1\\}$ indicates
whether the image has been flipped.

  * •

random grayscaling – $\mathbf{\omega}^{g}\in\\{0,1\\}$ indicates whether the
image has been reduced to grayscale.

To enhance the projector’s awareness of the color changes in the augmented
images, we additionally enrich $\mathbf{\omega}$ with information about color
difference – $\mathbf{\omega}^{d}\in[0,1]^{3}$, which is computed as the
difference between the mean values of color channels of the image before and
after the color jittering operation. We empirically demonstrate that inclusion
of $\mathbf{\omega}^{d}$ in $\mathbf{\omega}$ improves the performance of
CASSLE (see Section 4.3).

We construct augmentation information $\mathbf{\omega}\in\Omega$ by
concatenating vectors $\mathbf{\omega}^{aug}$ describing the parameters of
each augmentation type [40]. We consider four methods of injecting $\omega$
into $\pi$: joining $\omega$ and $\mathbf{e}$ through (i) concatenation,
modulating $\mathbf{e}$ with $\omega$ through element-wise (ii) addition or
(iii) multiplication, or (iv) using $\omega$ as an input to a hypernetwork
[31] which generates the parameters of $\pi$. Apart from concatenation, all of
those methods require transforming $\omega$ into _augmentation embeddings_
$\mathbf{g}\in\mathcal{G}$ of shapes required by the conditioning operation.
For example, when modulating $\mathbf{e}$ with $\mathbf{g}$, dimensions of
$\mathbf{e}$ and $\mathbf{g}$ must be equal. For this purpose, we precede the
projector with additional _Augmentation encoder_
$\gamma:\Omega\rightarrow\mathcal{G}$. For the architecture of $\gamma$ we
choose the Multilayer Perceptron. An additional advantage of $\gamma$ is that
it allows for learning a representation of $\omega$ which is more expressive
for processing by $\pi$, which is typically a shallow network. In practice, we
find that conditioning $\pi$ through the concatenation of $\mathbf{e}$ and
$\mathbf{g}$ yields the best-performing representation (see Section 4.3).

##  4 Experiments

In Section 4.1, we evaluate CASSLE’s performance on downstream tasks such as
classification, regression, and object detection333We compare CASSLE to a
number of recently proposed methods and report their performance from the
literature [69, 40, 14], given that the code for [69] and [14] was not made
available at the time of writing. As for the results of baseline SSL models
and AugSelf [40], we report their results from the literature except when our
runs of those methods yielded results different by at least 2 pp. We mark such
cases with ${\dagger}$.. In Section 4.2, we analyze the representations formed
by CASSLE. Finally, we discuss the choice of hyperparameters of CASSLE in
Section 4.3. In all experiments, unless specified otherwise, we utilize the
ResNet-50 architecture [33] and conduct the self-supervised pretraining on
ImageNet-100 - a 100-class subset of the ILSVRC dataset [58] used commonly in
the literature [62, 69, 40, 14]. We use the standard set of augmentations
including horizontal flipping, random cropping, grayscaling, color jittering
and Gaussian blurring [32, 40, 30]. For consistency in terms of
hyperparameters, we follow [40] for MoCo-v2 , and [14] for SimCLR. We include
comparisons on other SSL frameworks [32, 30, 18, 72], extended analysis of
trained representations, as well as additional details of training and
evaluation, in the supplementary material.

###  4.1 Evaluation on downstream tasks

Table 1: Linear evaluation on downstream classification and regression tasks.
CASSLE consistently improves representations formed by vanilla SSL approaches
and performs better or comparably to other techniques of increasing
sensitivity to augmentations [69, 40, 14].

Method | C10 | C100 | Food | MIT | Pets | Flowers | Caltech | Cars | FGVCA | DTD | SUN | CUB | 300W
---|---|---|---|---|---|---|---|---|---|---|---|---|---
SimCLR [15]
Vanilla |  84.41† | 61.40 |  57.48† |  63.10† |  71.60† |  83.37† | 79.67† |  35.14† |  40.03† | 64.90 |  46.92† |  30.98† |  88.59†
AugSelf [40]† | 84.45 | 62.67 | 59.96 | 63.21 | 70.61 | 85.77 | 77.78 | 37.38 | 42.86 | 65.53 | 49.18 | 34.24 | 88.27
AI [14] | 83.90 | 63.10 | – | – | 69.50 | 68.30 | 74.20 | – | – | 53.70 | – | 38.60 | 88.00
CASSLE | 86.31 | 64.36 | 60.67 | 63.96 | 72.33 | 85.22 | 79.62 | 39.86 | 43.10 | 65.96 | 48.91 | 33.21 | 88.88
MoCo-v2 [32, 17]
Vanilla | 84.60 | 61.60 | 59.67 | 61.64 | 70.08 | 82.43 | 77.25 | 33.86 | 41.21 | 64.47 | 46.50 | 32.20 | $88.77^{\dagger}$
AugSelf [40] | 85.26 | 63.90 | 60.78 | 63.36 | 73.46 | 85.70 | 78.93 | 37.35 | 39.47 | 66.22 | 48.52 | 37.00 | $89.49^{\dagger}$
AI [14] | 81.30 | 64.60 | – | – | 74.00 | 81.30 | 78.90 | – | – | 68.80 | – | 41.40 | 90.00
LooC [69] | – | – | – | – | – | – | – | – | – | – | 39.60 | – | –
IFM [57]† | 83.36 | 60.22 | 59.86 | 60.60 | 72.99 | 85.73 | 78.77 | 36.54 | 41.05 | 62.34 | 47.48 | 35.90 | 88.92
CASSLE | 86.32 | 65.29 | 61.93 | 63.86 | 72.86 | 86.51 | 79.63 | 38.82 | 42.03 | 66.54 | 49.25 | 36.22 | 88.93
MoCo-v3 [19] with ViT-Small [23] pretrained on the full ImageNet dataset.
Vanilla† | 83.17 | 62.40 | 56.15 | 53.28 | 62.29 | 81.48 | 69.63 | 28.63 | 32.84 | 57.18 | 42.16 | 35.00 | 87.42
AugSelf [40]† | 84.25 | 64.12 | 58.28 | 56.12 | 63.93 | 83.13 | 72.45 | 29.64 | 32.54 | 60.27 | 43.22 | 37.16 | 87.85
CASSLE | 85.13 | 64.67 | 57.30 | 55.90 | 63.88 | 82.42 | 73.53 | 30.92 | 35.91 | 58.24 | 43.37 | 36.09 | 88.53

We begin the experimental analysis by addressing the most fundamental question
– how does CASSLE impact the ability of models to generalize? In order to
answer it, we evaluate models pretrained via CASSLE and other self-supervised
techniques on a variety of downstream visual tasks, such as classification,
regression, and object detection.

#### Linear evaluation

We evaluate the performance of pretrained networks on the downstream tasks of
classification on a wide array of datasets: CIFAR10/100 (C10/100) [39],
Food101 (Food) [10], MIT67 (MIT) [54], Oxford-IIIT Pets (Pets) [51], Oxford
Flowers-102 (Flowers) [47], Caltech101 (Caltech) [26], Stanford Cars (Cars)
[38], FGVC-Aircraft (FGVCA) [43], Describable Textures (DTD) [20], SUN-397
(SUN) [68], as well as regression on the 300 Faces In-the-Wild (300W) dataset
[59]. We follow the linear evaluation protocol [37, 15, 40], described in
detail in the supplementary material. We evaluate multiple self-supervised
methods extended with CASSLE, as well as other recently proposed extensions
which increase sensitivity to augmentations [40, 69, 14] or prevent feature
suppression in SSL [57]. We report the full results in Tables 1 and 7. We find
that in the vast majority of cases, CASSLE improves the performance of vanilla
joint-embedding methods , as well as other other SSL extensions that foster
augmentation sensitivity [40, 14].

Table 2: Average Precision of object detection on VOC dataset [25, 42]. CASSLE extension of MoCo-v2 and SimCLR outperforms the vanilla approaches and AugSelf extension by a small margin. Method | MoCo-v2 | SimCLR
---|---|---
Vanilla | 45.12 | 44.74
AugSelf [40] | 45.20 | 44.50
CASSLE | 45.90 | 45.60

#### Object detection

We next evaluate the pretrained networks on a more challenging task of object
detection on the VOC 2007 dataset [25]. We follow the training scheme of [32,
17], except that we only train the object detector modules and keep the
feature extractor parameters fixed during training for detection to better
compare the pretrained representations. We report the Average Precision (AP)
[42] of models pretrained through MoCo-v2 and SimCLR [15] with AugSelf [40]
and CASSLE extensions in Table 2. The compared approaches yield similar
results, with CASSLE representation slightly surpassing the vanilla methods
and AugSelf.

###  4.2 Analysis of representations formed by CASSLE

We investigate the awareness of augmentation-induced data perturbations in the
intermediate and final representations of pretrained networks. As a proxy
metric for measuring this, we choose the InfoNCE loss [64, 15]. The value of
InfoNCE is high if embeddings of pairs of augmented images are less similar to
one another than to embeddings of unrelated images, and low if positive pairs
of embeddings are on average separated correctly, and thus, the given
representation is invariant to augmentations. We report the mean InfoNCE loss
values under different augmentation types at subsequent stages of ResNet-50
and projectors of MoCo-v2, AugSelf [40] and CASSLE in Figure 4.

In all networks, the augmentation awareness decreases gradually throughout the
feature extractor and projector stages. In CASSLE, we observe a much softer
decline in the feature extractor stages and a sharper one in the projector.
Representations of CASSLE feature extractor are on average more difficult to
match together than those of vanilla MoCo-v2 and AugSelf [40]. This implies
that the CASSLE feature extractor is indeed more sensitive to augmentations
than its counterparts. On the other hand, representations of all projectors,
including CASSLE, are similarly separable. This suggests that the conditioning
mechanism helps CASSLE projector to better amortize the augmentation-induced
differences between the feature extractor embeddings.

The above observations indicate that in the vanilla and (to a slightly lesser
extent) AugSelf approaches, both the projector and the intermediate
representations are enforced to be augmentation-invariant. On the other hand,
in CASSLE, the task of augmentation invariance is solved to a larger degree by
the projector, and to a smaller degree by the feature extractor, allowing it
to be more augmentation-aware. As shown in Section 4.1, this sensitivity does
not prevent the CASSLE feature extractor from achieving similar or better
performance than its counterparts when transferred to downstream tasks.

Figure 4: A comparison of InfoNCE loss measured on different kinds of
augmentations at subsequent stages of the ResNet-50 and projectors pretrained
by vanilla, AugSelf [40] and CASSLE variants of MoCo-v2. Feature extractor
representation of CASSLE yields higher InfoNCE values which suggests that it
is more susceptible to augmentations.

###  4.3 Ablation Study

In this section, we examine the impact of different hyperparameters of CASSLE.
We compare different variants of MoCo-v2+CASSLE on the same classification and
regression tasks as in Section 4.1. We rank the models from best to worst
performance on each task and report the average ranks in Table 3. We provide
the detailed results in the supplementary material.

#### Augmentation information contents

– We compare conditioning the projector with different subsets of augmentation
information. The average best representation is trained with conditioning on
all possible augmentation information. Moreover, using the additional color
difference ($\omega^{d}$) information additionally improves the results,
indicating that it is indeed useful to consider not only augmentation
parameters but also information about its effects.

#### Impact of utilizing color difference information

– We verify that the improved performance of CASSLE does not stem solely from
using augmentation information that has not been considered in prior works. We
compare a variant of AugSelf which learns to predict color difference values
$(\omega^{d})$ in addition to augmentation information used by [40], i.e.
cropping ($\omega^{c}$) and color jittering ($\omega^{j}$), as well as a
variant of CASSLE conditioned on all augmentation information _except_
$\omega^{d}$ (i.e. $\omega^{\\{c,j,b,f,g\\}}$). We find that while including
$\omega^{d}$ improves the performance of AugSelf and CASSLE, both variants of
CASSLE achieve better results than both variants of AugSelf.

#### Method of conditioning the projector

– We compare conditioning the projector through (i) concatenation, element-
wise (ii) addition or (iii) multiplication, or (iv) using $\pi$ as an input to
a hypernetwork [31] which generates the parameters of $\pi$. Conditioning
through concatenation and addition yields on average the strongest performance
on downstream tasks. We choose to utilize the concatenation method in our
experiments, as it requires a slightly smaller Augmentation encoder.

#### Size of the Augmentation encoder

– While CASSLE is robust to the size of the $\gamma$ MLP, using the depth and
hidden size of 6 and 64, respectively, yields the strongest downstream
performance. In particular, the variant of CASSLE that utilizes the
Augmentation encoder performs better than the variant that concatenates
$\mathbf{e}$ to raw augmentation embeddings $\mathbf{\omega}$. Given such an
architecture of the Augmentation encoder, our computation overhead is
negligible as we increase the overall number of parameters by around $0.1\%$.
We refer to the supplementary material for the detailed ablation of the
Augmentation encoder’s hidden size.

Table 3: Ablation study of CASSLE parameters. CASSLE performs best when
conditioned on all available augmentation information, by concatenating or
adding the augmentation and image embeddings.

Augmentation | Average
---|---
information | rank $\downarrow$
$\omega^{\\{c\\}}$ | 4.54
$\omega^{\\{c,j\\}}$ | 4.54
$\omega^{\\{c,j,d\\}}$ | 3.08
$\omega^{\\{c,j,b,f\\}}$ | 2.85
$\omega^{\\{c,j,b,f,g\\}}$ | 3.54
$\omega^{\\{c,j,b,f,g,d\\}}$ | 2.54
(a) Augmentation information used for conditioning

Method | Average
---|---
| rank $\downarrow$
A – $\omega^{\\{c,j\\}}$ | 3.07
C – $\omega^{\\{c,j,b,f,g\\}}$ | 2.64
A – $\omega^{\\{c,j,d\\}}$ | 2.71
C – $\omega^{\\{c,j,b,f,g,d\\}}$ | 1.57
(b) Impact of including color difference information ($\omega^{d}$) on AugSelf
and CASSLE

Conditioning | Average
---|---
method | rank $\downarrow$
Concatenation | 1.92
Addition | 1.92
Multiplication | 2.69
Hypernetwork | 3.46
(c) Method of conditioning the projector

Number | Average
---|---
of layers | rank $\downarrow$
0 | 3.38
2 | 3.08
4 | 2.38
6 | 2.15
8 | 4.00
(d) Augmentation encoder depth (0 denotes concatenating raw $\omega$ to
$\mathbf{e}$)

##  5 Conclusion

In this paper, we propose a novel method for augmentation-aware self-
supervised learning that retains information about data augmentations in the
representation space. To accomplish this, we introduce the concept of the
conditioned projector, which receives augmentation information while
processing the representation vector. Our solution necessitates only small
architectural changes and no additional auxiliary loss components. Therefore,
the training concentrates on contrastive loss, which enhances overall
performance.

We compare our solution with existing augmentation-aware SSL methods and
demonstrate its superior performance on downstream tasks, particularly when
augmentation invariance leads to the loss of vital information. Moreover, we
show that it obtains representations more sensitive to augmentations than the
baseline methods.

Overall, our method offers a straightforward and efficient approach to
retaining information about data augmentations in the representation space. It
can be directly applied to SSL methods, contributing to the further
advancement of augmentation-aware self-supervised learning.

#### Reproducibility statement

The codebase for CASSLE is available at https://github.com/gmum/CASSLE. We
detail the hyperparameters for training runs in the supplementary material.
All datasets used for training and evaluation are publicly available.

## References

  * [1] Kumar Krishna Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Aaron Richards.  $\alpha$-req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [2] Saleh Albelwi.  Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging.  Entropy, 24(4), 2022.
  * [3] Haoping Bai, Meng Cao, Ping Huang, and Jiulong Shan.  Self-supervised semi-supervised learning for data labeling and quality evaluation.  In NeurIPS Workshop, 2021.
  * [4] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum.  A cookbook of self-supervised learning, 2023.
  * [5] Adrien Bardes, Jean Ponce, and Yann LeCun.  VICReg: Variance-invariance-covariance regularization for self-supervised learning.  In International Conference on Learning Representations, 2022.
  * [6] Suzanna Becker and Geoffrey Hinton.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  Nature, 355:161–3, 02 1992.
  * [7] Javad Zolfaghari Bengar, Joost van de Weijer, Bartlomiej Twardowski, and Bogdan Raducanu.  Reducing label effort: Self-supervised meets active learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1631–1639, October 2021.
  * [8] Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, and Dilip Krishnan.  Steerable equivariant representation learning, 2023.
  * [9] Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent.  Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning.  Transactions on Machine Learning Research, 2023.
  * [10] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 – mining discriminative components with random forests.  In European Conference on Computer Vision, 2014.
  * [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  volume 33, pages 1877–1901, 2020.
  * [12] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912–9924. Curran Associates, Inc., 2020.
  * [13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [14] Ruchika Chavhan, Jan Stuehmer, Calum Heggan, Mehrdad Yaghoobi, and Timothy Hospedales.  Amortised invariance learning for contrastive self-supervision.  In The Eleventh International Conference on Learning Representations, 2023.
  * [15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR, 13–18 Jul 2020\.
  * [16] Ting Chen, Calvin Luo, and Lala Li.  Intriguing properties of contrastive losses.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [17] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning, 2020.
  * [18] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750–15758, June 2021.
  * [19] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640–9649, October 2021.
  * [20] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding, June 2019.
  * [22] Carl Doersch, Abhinav Gupta, and Alexei A. Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [24] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales.  Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks, 2022.
  * [25] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results.  http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  * [26] Li Fei-Fei, R. Fergus, and P. Perona.  One-shot learning of object categories.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594–611, 2006.
  * [27] Quentin Garrido, Laurent Najman, and Yann Lecun.  Self-supervised learning of split invariant equivariant representations.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10975–10996. PMLR, 23–29 Jul 2023.
  * [28] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In International Conference on Learning Representations, 2018.
  * [29] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [30] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271–21284. Curran Associates, Inc., 2020.
  * [31] David Ha, Andrew M. Dai, and Quoc V. Le.  Hypernetworks.  In International Conference on Learning Representations, 2017.
  * [32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, June 2016.
  * [34] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In International Conference on Learning Representations, 2019.
  * [35] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In International Conference on Learning Representations, 2022.
  * [36] Tae Soo Kim, Geonwoon Jang, Sanghyup Lee, and Thijs Kooi.  Did you get what you paid for? rethinking annotation cost of deep learning based computer aided detection in chest radiographs, 2022.
  * [37] Simon Kornblith, Jonathon Shlens, and Quoc V. Le.  Do better imagenet models transfer better?, June 2019.
  * [38] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [39] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * [40] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin.  Improving transferability of representations via augmentation-aware self-supervision.  34:17710–17722, 2021.
  * [41] Xuhong LI, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2825–2834. PMLR, 10–15 Jul 2018\.
  * [42] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  CoRR, abs/1405.0312, 2014.
  * [43] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft, 2013.
  * [44] Grégoire Mialon, Randall Balestriero, and Yann LeCun.  Variance covariance regularization enforces pairwise independence in self-supervised representations, 2023.
  * [45] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [46] Sharada P. Mohanty, David P. Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection, 2016.
  * [47] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [48] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 69–84, Cham, 2016. Springer International Publishing.
  * [49] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * [50] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning, 2018\.
  * [51] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In IEEE Conference on Computer Vision and Pattern Recognition, 2012\.
  * [52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [53] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [54] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 413–420, 2009.
  * [55] Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, and David K Duvenaud.  Meta-learning to improve pre-training.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23231–23244. Curran Associates, Inc., 2021.
  * [56] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [57] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra.  Can contrastive learning avoid shortcut solutions?  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4974–4986. Curran Associates, Inc., 2021.
  * [58] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge, 09 2014.
  * [59] C. Sagonas, E. Antonakos, Tzimiropoulos G, S. Zafeiriou, and M. Pantic.  300 faces in-the-wild challenge: Database and results.  In Image and Vision Computing (IMAVIS), Special Issue on Facial Landmark Localisation, 2016.
  * [60] Madeline C. Schiappa, Yogesh S. Rawat, and Mubarak Shah.  Self-supervised learning for videos: A survey.  ACM Computing Surveys, dec 2022.
  * [61] Yisheng Song, Ting Wang, Puyu Cai, Subrota K. Mondal, and Jyoti Prakash Sahoo.  A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities.  volume 55, New York, NY, USA, jul 2023. Association for Computing Machinery.
  * [62] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  33:6827–6839, 2020.
  * [63] Yuandong Tian.  Understanding deep contrastive learning via coordinate-wise optimization.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [64] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2019.
  * [65] Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirrmeister, Samuel Müller, and Frank Hutter.  On the importance of hyperparameters and data augmentation for self-supervised learning.  In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022.
  * [66] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [67] Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen.  Mixing up contrastive learning: Self-supervised representation learning for time series.  Pattern Recognition Letters, 155:54–61, mar 2022.
  * [68] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485–3492, June 2010.
  * [69] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell.  What should not be contrastive in contrastive learning.  In International Conference on Learning Representations, 2021.
  * [70] Yuyang Xie, Jianhong Wen, Kin Wai Lau, Yasar Abbas Ur Rehman, and Jiajun Shen.  What should be equivariant in self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4111–4120, June 2022.
  * [71] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [72] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12310–12320. PMLR, 18–24 Jul 2021.
  * [73] Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo.  Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [74] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 649–666, Cham, 2016. Springer International Publishing.
  * [75] Simone Zini, Alex Gomez-Villa, Marco Buzzelli, Bartłomiej Twardowski, Andrew D. Bagdanov, and Joost van de weijer.  Planckian jitter: countering the color-crippling effects of color jitter on self-supervised training.  In The Eleventh International Conference on Learning Representations, 2023.

##  Appendix A Pretraining details

###  A.1 Datasets

We use ImageNet-100, a 100-class subset of ImageNet [58, 62], to pretrain the
standard ResNet-50 [33] architecture of self-supervised methods: MoCo-v2 [17],
SimCLR [15], Barlow Twins [72], and SimSiam [18], as well as for common in the
literature on augmentation-aware self-supervised learning [62, 69, 40, 14].
For MoCo-v3 [19], we pretrain the ViT-Small [23] model on the full ImageNet
dataset [58].

###  A.2 Hyperparameters

We follow the pretraining procedures from corresponding papers, described in
[40] for MoCo-v2 and SimSiam, [14] for SimCLR, [72] for Barlow Twins, and [19]
for MoCo-v3. We train variants of all SSL approaches on 2 and 8 NVidia A100
GPUs, except for SimCLR and MoCo-v3 which use larger batch sizes and therefore
require 8 such GPUs. Synchronized batch normalization is employed for
distributed training [18]. In Table 4, we present the training hyperparameters
which are not related specifically to CASSLE, but rather joint-embedding
approaches in general [32, 17, 15, 72, 18, 19].

Table 4: Hyperparameters of self-supervised methods used with CASSLE

SSL method | Architecture | Number of | Batch | Weight | Learning rate | Training
---|---|---|---|---|---|---
|  | epochs | size | decay | Base | Schedule | time
MoCo-v1 [17] | ResNet-50 | 500 | 256 | $10^{-4}$ | 0.03 | Cosine decay | 32h
MoCo-v2 [17] | ResNet-50 | 500 | 256 | $10^{-4}$ | 0.03 | Cosine decay | 34h
SimCLR [15] | ResNet-50 | 300 | 1024 | $10^{-4}$ | 0.05 | Cosine decay | 10h
Barlow Twins [72] | ResNet-50 | 500 | 256 | $10^{-4}$ | 0.05 | Cosine decay with warmup | 36h
SimSiam [18] | ResNet-50 | 500 | 256 | $10^{-4}$ | 0.05 | Cosine decay | 40h
MoCo-v3 [19] | ViT-small | 300 | 1024 | 0.1 | $1.5\cdot 10^{-4}$ | Cosine decay with warmup | 23h

SSL method | Projector | Predictor
---|---|---
| Depth | Hidden size | Out size | Final BatchNorm | Depth | Hidden size | Out size | Final BatchNorm
MoCo-v2 | 2 | 2048 | 128 | No | No
SimCLR | 2 | 2048 | 128 | No | No
Barlow Twins | 3 | 8192 | 8192 | Yes, without affine transform | No
SimSiam | 3 | 2048 | 2048 | Yes | 2 | 512 | 2048 | No
MoCo-v3 | 3 | 4096 | 256 | Yes, without affine transform | 2 | 4096 | 256 | Yes, without affine transform

###  A.3 Augmentations

For self-supervised pretraining, we use a set of augmentations adopted
commonly in the literature [32, 15, 17, 18, 72, 40]. We denote them below:

  * •

random cropping – We sample the cropping scale randomly from [0.2, 1.0].
Afterward, we resize the cropped images to the size of $224\times 224$.

  * •

color jittering – We apply this operation with a probability of $0.8$. We
sample the intensities of brightness, contrast, saturation, and hue and their
maximal values are $0.4$, $0.4$, $0.4$, and $0.1$, respectively.

  * •

Gaussian blurring – We apply this operation with a probability of $0.5$. We
sample the standard deviation from $[0.1,2.0]$ and set the kernel size to
$23\times 23$.

  * •

random horizontal flipping – We apply this operation with a probability of
$0.5$.

  * •

random grayscaling – We apply this operation with a probability of $0.2$.

##  Appendix B Evaluation protocol

###  B.1 Datasets

We outline the datasets and their respective evaluation metrics used for
downstream tasks that we evaluate CASSLE on in Table 5.

Table 5: Information about datasets and their metrics used for downstream
evaluation of CASSLE.

Downstream task | Dataset | Metric
---|---|---
Linear evaluation | CIFAR10 [39] | Top-1 accuracy
CIFAR100 [39] | Top-1 accuracy
Food101 [10] | Top-1 accuracy
MIT67 [54] | Top-1 accuracy
Oxford-IIIT Pets [51] | Mean per-class accuracy
Oxford Flowers-102 [47] | Mean per-class accuracy
Caltech101 [26] | Mean per-class accuracy
Stanford Cars [38] | Top-1 accuracy
FGVC-Aircraft [43] | Mean Per-class accuracy
Describable Textures (split 1) [20] | Top-1 accuracy
SUN397 (split 1) [68] | Top-1 accuracy
Caltech-UCSD Birds [66] | Top-1 accuracy
300 Faces In-the-Wild [59] | R2
Object detection | VOC2007 [25] | Average Precision [42]
Few-shot classification | Few-Shot CIFAR100 [50] | Average accuracy
Caltech-UCSD Birds [66] | Average accuracy
Plant Disease [46] | Average accuracy
Oxford Flowers-102 [47] | Mean per-class accuracy

#### Linear evaluation

We follow the linear evaluation protocol of [15, 30, 37, 40]. Namely, we
center-crop and resize the images from the downstream dataset to the size of
$224\times 224$, pass them through the pretrained feature extractor, and
obtain the embeddings from the final feature extractor stage. The only
exception from this is the CUB dataset [66], where, following [40], for the
training images besides the center crop of the image, we also crop the image
at its corners and do the same for the horizontal flip of the image (this is
known as TenCrop
operation444https://pytorch.org/vision/main/generated/torchvision.transforms.TenCrop.html).

Having gathered the image features, we minimize the $l_{2}$-regularized cross-
entropy objective using L-BFGS on the features of the training images. We
select the regularization parameter from between $[10^{-6},10^{5}]$ using the
validation features. Finally, we train the linear classifier on training and
validation features with the selected $l_{2}$ parameter and report the final
performance metric (see Table 5) on the test dataset. We set the maximum
number of iterations in L-BFGS as 5000 and use the model trained on training
data as initialization for training the final model.

We note that the above linear evaluation procedure is effectively equivalent
to training the final layer of a network on non-augmented data, while keeping
the remainder of the parameters unchanged.

#### Object detection

We closely follow the evaluation protocol of [32]. We train the Faster-RCNN
[56] model with the pretrained backbone. Contrary to [32], we do not tune the
backbone parameters, in order to better observe the effect of different
pretraining methods. We report the Average Precision [42] measured on the VOC
test2007 set [25].

#### Sensitivity to augmentations

We consider image pairs, where one image is the (center-cropped) original and
the second one is augmented by the given augmentation. For each image pair, we
extract image features at four stages of the pretrained ResNet-50 backbone
[33], as well as the final representation of the projector network. We next
calculate cosine similarities between the features of augmented and non-
augmented images in the given mini-batch (of size 256). We report the value of
the InfoNCE loss [64] calculated on such similarity matrices.

#### Dependency of CASSLE projector on conditioning

Similarly to the above experiment, we compare the projector features of
augmented and non-augmented image pairs. When computing the features of the
augmented image, we supply the projector with the augmentation embedding
computed from augmentation parameters corresponding to either this image (true
augmentation information) or another, randomly chosen image from the same
mini-batch (random augmentation information). We then compute cosine
similarities between the original image features and features of the augmented
image computed with true/random augmentation information.

##  Appendix C Additional results and analysis

###  C.1 Evaluation

#### Rotation prediction

In order to understand whether CASSLE learns to generalize to other types of
augmentation, which were not used during pretraining, we inspect the
prediction of applied augmentation through the following experiment. We train
a linear classifier on top of the pretrained model to indicate whether the
image was rotated by $0$, $90$, $180$, or $270$ degrees. We formulate the
problem as classification due to its cyclic nature and test the model on the
datasets used for linear evaluation in Section 4.1 of the main manuscript. We
present the results of vanilla, AugSelf [40] and CASSLE variants of self-
supervised methods in Table 6. Apart from a few exceptions, CASSLE and AugSelf
extensions allow in general for better prediction of rotation than the vanilla
SSL methods. Moreover, in the case of SimCLR, CASSLE representation predicts
the rotations the most accurately on a vast majority of datasets. This occurs
despite the fact, that neither of the methods was trained using rotated images
and thus, never explicitly learned the _concept_ of rotation. This suggests
that CASSLE learns representations which are sensitive to a broader set of
perturbations than those whose information had been used in CASSLE’s
pretraining.

Table 6: Linear evaluation on downstream task: predicting the rotation ($0$,
$90$, $180$ or $270$ degrees)

Method | C10 | C100 | Food | Pets | MIT | Flowers | Caltech | Cars | FGVCA | STL10 | SUN
---|---|---|---|---|---|---|---|---|---|---|---
|  SimCLR [15] |
Vanilla | 67.93 | 56.88 | 68.76 | 65.07 | 70.40 | 24.15 | 56.81 | 87.12 | 98.02 | 73.90 | 67.94
AugSelf [40] | 75.63 | 63.08 | 75.01 | 57.99 | 16.54 | 33.99 | 39.03 | 76.11 | 96.64 | 67.88 | 76.60
CASSLE | 77.07 | 70.20 | 79.55 | 70.97 | 68.77 | 33.08 | 64.97 | 95.14 | 99.46 | 72.69 | 78.16
|  MoCo-v2 [32, 17] |
Vanilla | 67.96 | 56.96 | 75.70 | 71.27 | 67.13 | 58.30 | 87.33 | 58.35 | 93.20 | 95.44 | 68.33
AugSelf [40] | 74.57 | 65.87 | 73.03 | 77.01 | 79.75 | 52.81 | 58.53 | 93.07 | 98.26 | 77.35 | 83.66
CASSLE | 73.21 | 69.91 | 77.31 | 76.04 | 76.40 | 45.11 | 61.06 | 90.49 | 98.56 | 63.65 | 76.01
|  SimSiam [18] |
Vanilla | 68.11 | 64.81 | 77.35 | 61.79 | 79.10 | 46.33 | 64.76 | 80.50 | 97.48 | 55.38 | 74.49
AugSelf [40] | 77.00 | 72.91 | 73.03 | 59.18 | 78.28 | 49.11 | 60.86 | 97.21 | 99.46 | 76.68 | 83.14
CASSLE | 72.00 | 71.38 | 78.01 | 85.30 | 66.80 | 37.65 | 61.64 | 92.04 | 98.65 | 63.69 | 75.86
|  Barlow Twins [72] |
Vanilla | 73.67 | 64.87 | 72.85 | 83.02 | 70.97 | 42.56 | 57.71 | 83.68 | 97.93 | 65.70 | 76.87
AugSelf [40] | 72.79 | 65.91 | 74.93 | 77.76 | 50.30 | 30.93 | 44.31 | 85.85 | 98.35 | 69.68 | 77.19
CASSLE | 74.97 | 65.15 | 76.55 | 77.05 | 86.49 | 43.16 | 53.60 | 92.92 | 99.16 | 55.24 | 74.97

#### Linear evaluation – additional SSL frameworks

We evaluate CASSLE on additional joint-embedding SSL frameworks [32, 30, 18,
72], following the same linear evaluation protocol as in Section 4.1, and
present the results in Table 7. In all cases, CASSLE improves the performance
of the respective Vanilla approaches. In the case of MoCo-v1 and Barlow Twins,
it also achieves stronger performance than AugSelf [40], whereas in the case
of self-distillation methods [30, 18], the performances are comparable. We
suspect that the lower performance of CASSLE compared to AugSelf with those
methods stems from their asymmetric architecture, i.e., distilling the
representations between the projector and an additional predictor network.
Observe that CASSLE and AugSelf also perform comparably on MoCo-v3, which also
utilizes an additional predictor (see Table 1). Perhaps most curiously, CASSLE
lends a major performance boost to MoCo-v1 [32]. This is surprising, as the
original MoCo-v1 architecture does not utilize a projector network. As such,
in this particular case, CASSLE joins the image and augmentation embeddings
through element-wise addition. Nevertheless, this simple modulation improves
the results by a large margin.

Table 7: Linear evaluation on downstream classification and regression tasks.
CASSLE consistently improves representations formed by vanilla SSL approaches
and performs better or comparably to other techniques of increasing
sensitivity to augmentations [69, 40, 14].

Method | C10 | C100 | Food | MIT | Pets | Flowers | Caltech | Cars | FGVCA | DTD | SUN | CUB | 300W
---|---|---|---|---|---|---|---|---|---|---|---|---|---
MoCo-v1 [32] (no projector, $\mathbf{e}$ and $\omega$ joined through addition)
Vanilla† | 58.82 | 28.09 | 25.90 | 31.04 | 47.25 | 33.29 | 44.41 | 5.00 | 10.98 | 36.86 | 19.00 | 9.16 | 88.05
AugSelf | 64.94 | 37.01 | 32.84 | 33.13 | 45.95 | 38.59 | 45.15 | 8.33 | 15.14 | 40.37 | 20.48 | 11.27 | 88.12
CASSLE | 80.53 | 53.55 | 52.11 | 51.94 | 57.58 | 60.56 | 60.33 | 18.68 | 28.68 | 53.94 | 36.71 | 18.88 | 88.21
BYOL [30]
Vanilla | 83.48 | 60.17 | 57.79 | 62.84 | 75.70 | 82.47 | 78.80 | 38.38 | 38.16 | 63.46 | 46.97 | 34.12 | 88.81
AugSelf [40] | 85.55 | 65.03 | 62.59 | 64.33 | 77.48 | 87.48 | 81.13 | 41.06 | 43.39 | 64.63 | 49.31 | 40.90 | 89.28
CASSLE | 84.20 | 62.00 | 57.35 | 60.97 | 73.29 | 85.44 | 81.52 | 40.18 | 42.11 | 65.59 | 47.63 | 33.36 | 93.03
SimSiam [18]
Vanilla | 86.89 | 66.33 | 61.48 | 65.75 | 74.69 | 88.06 | 84.13 | 48.20 | 48.63 | 65.11 | 50.60 | 38.40 | 89.01
AugSelf [40] | 88.80 | 70.27 | 65.63 | 67.76 | 76.34 | 90.70 | 85.30 | 47.52 | 49.76 | 67.29 | 52.28 | 45.30 | 92.84
CASSLE | 87.38 | 67.36 | 63.27 | 66.84 | 75.02 | 88.95 | 84.86 | 48.51 | 49.35 | 66.81 | 51.62 | 39.47 | 89.37
Barlow Twins [72]
Vanilla† | 85.90 | 66.10 | 59.41 | 61.72 | 72.30 | 87.13 | 81.95 | 41.54 | 44.40 | 65.85 | 49.18 | 35.02 | 89.04
AugSelf [40]† | 87.28 | 66.98 | 60.52 | 63.96 | 72.11 | 86.68 | 81.73 | 39.88 | 44.23 | 65.21 | 47.71 | 37.02 | 88.88
CASSLE | 87.03 | 67.27 | 62.19 | 65.08 | 72.75 | 87.99 | 82.56 | 41.68 | 46.63 | 66.31 | 50.09 | 38.25 | 89.52

#### Few-shot classification

We next benchmark various forms of pretraining on a number of few-shot
learning tasks: Few-Shot CIFAR100 (FC100) [50], Caltech-UCSD Birds (CUB) [66],
Plant Disease [46]. For the above datasets, we report the performance on 5-way
1-shot and 5-way 5-shot tasks. Moreover, we also report the performance on the
5-shot and 10-shot versions of the Flowers-102 dataset (Flowers) [47].
Following [40], our few-shot models are trained through logistic regression on
representations of support images, without fine-tuning. We follow the
evaluation protocol of [40]. For the task of 5/10-shot learning of the Flowers
dataset [47], we select the subset of the training set with 5/10 examples per
class, respectively, and utilize the linear evaluation protocol presented
above. For the remaining datasets used for few-shot evaluation, we extract the
representations of support and query images with the feature extractor and
train a logistic regression model555We use the Logistic Regression
implementation from scikit-learn [53]., which we evaluate on the query set. We
report the average accuracy from 2000 $N$-way $K$-shot tasks [61] in Table 8.
In general, CASSLE improves the performance of the vanilla joint-embedding
approaches, but to a lesser extent than AugSelf [40] or AI [14]. The notable
exception is Barlow Twins [72], where CASSLE yields the strongest performance
boost.

Table 8: Few-shot classification accuracy averaged over 2000 tasks on Few-Shot
CIFAR100 (FC100) [50], Caltech-UCSD Birds (CUB) [66], Plant Disease [46], and
accuracy on 5-shot and 10-shot versions of Flowers-102 dataset (Flowers) [47].
(N, K) denotes N-way K-shot tasks. The best performance in each group is
bolded.

Method | FC100 | CUB | Plant Disease | Flowers
---|---|---|---|---
| (5,1) | (5,5) | (5,1) | (5,5) | (5,1) | (5,5) | (102, 5) | (102,10)
|  MoCo v2 [32, 17]
Vanilla | 31.67 | 43.88 | 41.67 | 56.92 | 65.73 | 84.98 | $66.83^{\dagger}$ | $78.80^{\dagger}$
AugSelf [40] | 35.02 | 48.77 | 44.17 | 57.35 | 71.80 | 87.81 | $\mathbf{72.95}^{\dagger}$ | $\mathbf{82.12}^{\dagger}$
AI [14] | 37.40 | 48.40 | 45.00 | 58.00 | 72.60 | 89.10 | – | –
LooC [69] | – | – | – | – | – | – | 70.90 | 80.80
CASSLE | 35.15 | 49.30 | 42.98 | 55.82 | 70.74 | 87.12 | 72.36 | 82.07
|  SimCLR [15]
Vanilla † | 34.60 | 47.21 | 42.27 | 54.72 | 71.12 | 87.49 | 70.73 | 80.72
AugSelf [40]† | 36.63 | 49.78 | 42.88 | 56.96 | 74.09 | 89.72 | 73.86 | 82.91
CASSLE | 34.82 | 48.54 | 41.97 | 55.84 | 72.17 | 88.17 | 73.23 | 83.34
|  Barlow Twins [72]
Vanilla † | 38.08 | 53.93 | 43.20 | 61.95 | 70.98 | 91.06 | 75.28 | 83.43
AugSelf [40]† | 37.48 | 54.16 | 42.50 | 61.86 | 71.28 | 91.16 | 74.49 | 83.62
CASSLE | 38.58 | 54.03 | 43.22 | 62.65 | 73.47 | 92.17 | 76.62 | 85.50
|  SimSiam [18]
Vanilla | 36.19 | 50.36 | 45.56 | 62.48 | 75.72 | 89.94 | $75.53^{\dagger}$ | 85.90
AugSelf [40] | 39.37 | 55.27 | 48.08 | 66.27 | 77.93 | 91.52 |  $\textbf{79.67}^{\dagger}$ | 88.30
CASSLE | 36.18 | 50.16 | 44.93 | 60.07 | 76.43 | 90.82 | 77.96 | 86.64

#### Fine-tuning

Table 9: Results of fine-tuning different variants of MoCo-v2 with L2SP protocol [41]. Method | Caltech-101 | CIFAR-10 | CUB-200 | Food-101 | MIT-67 | Pets
---|---|---|---|---|---|---
| MoCo-v2
Vanilla | 89.80 | 94.65 | 56.31 | 74.36 | 61.79 | 82.97
AugSelf | 89.16 | 94.36 | 56.67 | 74.94 | 63.73 | 83.27
CASSLE | 89.27 | 94.73 | 55.93 | 74.07 | 61.19 | 82.86
| Barlow Twins
Vanilla | 86.54 | 91.67 | 48.02 | 55.68 | 63.43 | 77.98
AugSelf | 85.82 | 91.94 | 48.23 | 56.35 | 62.91 | 77.84
CASSLE | 86.01 | 92.03 | 54.15 | 59.83 | 62.54 | 78.71
| SimCLR
Vanilla | 90.67 | 94.75 | 54.55 | 75.36 | 63.51 | 83.65
AugSelf | 90.38 | 94.76 | 56.36 | 75.86 | 62.91 | 84.25
CASSLE | 90.47 | 95.19 | 55.65 | 76.78 | 63.88 | 83.51

In addition to linear evaluation, we also evaluate the models pretrained with
MoCo-v2 under the L2SP fine-tuning paradigm [41] on Caltech-101 [26], CIFAR-10
[39], CUB-200 [66], Food-101 [10], MIT-67 [54], and Pets [51] datasets. We
follow the hyperparameters of Li. et. al. [41], i.e. $\beta=0.01$,
$\alpha\in\\{0.001,0.01,0.1,1\\}$, and learning rate
$\in\\{0.005,0.01,0.02\\}$. We report our results in Table 9. We find that
while fine-tuned networks differ in performance by a lesser margin than during
linear evaluation, CASSLE-pretrained networks perform favorably in 8 out of 18
cases, more than either vanilla MoCo-v2 or AugSelf.

#### Robustness under perturbations

Table 10: Evaluation of variants of MoCo-v2 on perturbed ImageNet-100 images. Method | Brightness | Frost | Fog | Snow
---|---|---|---|---
Vanilla | 85.30 | 53.70 | 56.92 | 31.78
AugSelf | 83.64 | 51.98 | 53.08 | 33.80
CASSLE | 86.10 | 50.54 | 54.22 | 34.66

We next verify the influence of increased sensitivity to augmentations on the
robustness of MoCo-v2-pretrained models to perturbations. Following the
experimental setup of [40], we train the pretrained networks for
classification of ImageNet-100 and evaluate them on weather-corrupted images
(fog, frost, snow, and brightness) [34] from the validation set. We report the
results in Table 10. We find that the network pretrained with CASSLE achieves
the best results when dealing with images perturbed by brightness and snow,
whereas vanilla MoCo-v2 performs best on images perturbed by fog and frost.

#### Image retrieval

Finally, we evaluate the pretrained models on the task of image retrieval. We
gather the features of images from the Cars and Flowers test sets and for a
given query image, select four images closest to it in terms of the cosine
distance of final feature extractor representations. We compare the images
retrieved by MoCo-v2, AugSelf [40] and CASSLE in Figure 5. CASSLE selects
pictures of cars that are the most consistent in terms of color. In the case
of flowers, the nearest neighbor retrieved by the vanilla model is a different
species than that of the first query image, whereas both CASSLE and AugSelf
select the first two nearest neighbors from the same species but then retrieve
images of flowers with similar shapes, but different colors. This again
indicates greater reliability of features learned by CASSLE. For subsequent
queries, CASSLE and AugSelf retrieve in general more consistently looking
images, in particular in terms of color scheme. However, for some distinct
images, such as the last presented image of flowers, all methods have
retrieved the correct results for the given query.

(a) Cars

(b) Flowers

Figure 5: Image retrieval examples for Cars and Flowers datasets.

###  C.2 Extended analysis of representations formed by CASSLE

#### Analysis of dimensional collapse of representations

Augmentation-invariant SSL models are known to suffer from dimensional
collapse of their representations, due to the usage of strong augmentations
and implicit regularization [35]. Here, we analyze the effect of CASSLE on
this phenomenon. To this end, we compute the embeddings of the ImageNet-100
test dataset with pretrained feature extractors and calculate the empirical
eigenspectra of the embeddings produced by each encoder. We next analyze the
power-law decay rates ($\alpha$) of eigenspectra [1]. Intuitively, smaller
values of $\alpha$ suggest a dense encoding, while larger values of $\alpha$
suggest more sparse encoding and rapid decay. We report the calculated
coefficients in Table 11. In all analyzed SSL methods, except Barlow Twins,
the addition of augmentation information by CASSLE leads to a representation
whose eigenspectrum decays more rapidly, thus suggesting a lower rank of the
effective embedding subspace and as such, greater dimensional collapse.
However, as seen in the main text, this does not necessarily lead to degraded
performance – indeed, performance on downstream tasks has been previously
correlated with how close $\alpha$ is to 1 [1].

Table 11: Power-law eigenspectrum decay rates ($\alpha$) [1] of embeddings of ImageNet-100 test set produced by differently pretrained feature extractors. Method | MoCo-v2 | SimCLR | Barlow Twins | SimSiam
---|---|---|---|---
| [32, 17] | [15] | [72] | [18]
Vanilla | 0.6013 | 0.6150 | 0.5387 | 0.6365
AugSelf [40] | 0.6146 | 0.6251 | 0.5201 | 0.6262
CASSLE | 0.6342 | 0.6908 | 0.5196 | 0.6962

We illustrate this further with a Principal Component Analysis of features
extracted by different variants of joint-embedding methods illustrated in
Figure 6. We extract embeddings of the test ImageNet-100 images and calculate
the ratio of variance explained by principal components of embeddings. We find
that, in general, representations learned with CASSLE require fewer principal
components to explain larger amounts of variance. For example, for MoCo-v2,
90% of variance is explained by 905, 872, and 736 principal components of
representations learned by the vanilla approach, AugSelf [40], and CASSLE,
respectively. This again suggests that representations learned by CASSLE have
lower effective rank compared to other approaches.

Figure 6: Cumulated explained variance ratio of representations learned by
self-supervised models trained in their original forms, as well as with
AugSelf [40] and CASSLE extensions. We additionally denote the number of
principal components which that explain 90% of the variance of each
representation. Models pretrained with CASSLE explain the largest amount of
variance with the smallest amount of components, suggesting a larger
dimensionality collapse of their representations.

#### Analysis of the contrastive learning procedure

We next compare the training of MoCo-v2 [32, 17] with and without CASSLE or
AugSelf [40] extensions, and plot the contrastive loss values measured
throughout training in the left part of Figure 7, and on the right, the values
of losses relative to the baseline MoCo-v2. CASSLE minimizes the contrastive
objective faster than the other two variants, in particular early in the
training procedure. This suggests that augmentation information provides
helpful conditioning for a model not yet fully trained to align augmented
image pairs and thus, CASSLE learns to depend on this information.

Figure 7: Absolute (left) and relative to Baseline (right) values of
contrastive losses of Baseline, AugSelf [40], and CASSLE MoCo-v2 variants,
measured during pretraining. CASSLE minimizes the contrastive objective faster
than Baseline and AugSelf, in particular early in the training procedure.

###  C.3 Expanded ablation study results

In order to optimally select the hyperparameters of CASSLE, we train different
variants of MoCo-v2+CASSLE and evaluate them on the same classification and
regression tasks as in Section 4.1 of the main paper. We rank the models from
best to worst performance on each task and report the full results and average
ranks in Table 12.

We recall the notation used for Augmentation information contents –
$\omega^{\\{x,y\\}}$ denotes including parameters of augmentations $\\{x,y\\}$
in augmentation information vector $\omega$. For example, $\omega^{\\{c,j\\}}$
denotes $\omega$ containing cropping and color jittering parameters.

Table 12: Ablation study of CASSLE parameters. The results are computed with
MoCo-v2+CASSLE. It is best to condition CASSLE on all available augmentation
information. CASSLE yields the best results when implemented by concatenating
or adding the augmentation and image embeddings together.

Parameter | C10 | C100 | Food | MIT | Pets | Flowers | Caltech | Cars | FGVCA | DTD | SUN | CUB | 300W | Avg. rank $\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Augmentation information contents |  |
$\omega^{\\{c\\}}$ | 84.89 | 62.95 | 59.74 | 63.96 | 72.26 | 83.55 | 79.66 | 38.78 | 42.03 | 65.11 | 48.44 | 33.86 | 89.17 | 4.54
$\omega^{\\{c,j\\}}$ | 85.56 | 64.26 | 60.35 | 62.61 | 71.97 | 84.73 | 79.86 | 38.13 | 42.17 | 66.28 | 48.01 | 34.24 | 88.76 | 4.54
$\omega^{\\{c,j,d\\}}$ | 85.87 | 63.91 | 61.07 | 63.51 | 72.71 | 86.53 | 79.51 | 38.27 | 42.53 | 66.70 | 49.27 | 35.76 | 89.11 | 3.08
$\omega^{\\{c,j,b,f\\}}$ | 86.16 | 64.51 | 60.80 | 63.81 | 72.83 | 84.66 | 79.90 | 38.93 | 43.02 | 66.12 | 48.96 | 34.40 | 88.69 | 2.85
$\omega^{\\{c,j,b,f,g\\}}$ | 85.85 | 64.14 | 61.24 | 63.73 | 72.88 | 84.50 | 79.93 | 38.23 | 41.28 | 65.27 | 48.90 | 34.47 | 88.78 | 3.54
$\omega^{\\{c,j,b,f,g,d\\}}$ | 86.99 | 65.28 | 61.83 | 63.51 | 73.22 | 86.55 | 79.87 | 37.97 | 41.70 | 67.18 | 48.85 | 36.92 | 89.03 | 2.54
Method of conditioning the projector |  |
Concatenation | 86.99 | 65.28 | 61.83 | 63.51 | 73.22 | 86.55 | 79.87 | 37.97 | 41.70 | 67.18 | 48.85 | 36.92 | 89.03 | 1.92
Addition | 86.45 | 65.40 | 63.00 | 65.15 | 71.34 | 86.91 | 79.79 | 37.83 | 42.18 | 66.17 | 49.28 | 37.42 | 88.87 | 1.92
Multiplication | 86.72 | 66.70 | 60.65 | 60.97 | 64.60 | 85.17 | 80.09 | 33.54 | 41.56 | 63.99 | 47.63 | 32.15 | 89.48 | 2.69
Hypernetwork | 84.70 | 63.55 | 60.62 | 64.10 | 67.16 | 82.76 | 78.47 | 33.39 | 39.85 | 66.44 | 47.43 | 30.48 | 89.11 | 3.46
Depth of the Augmentation encoder (0 denotes concatenating raw $\mathbf{\omega}$ to $\mathbf{e}$) |  |
0 | 86.53 | 65.99 | 62.54 | 61.72 | 69.04 | 85.46 | 80.74 | 36.44 | 41.91 | 65.64 | 48.55 | 33.88 | 92.72 | 3.38
2 | 86.57 | 64.80 | 61.85 | 62.68 | 72.79 | 86.31 | 79.93 | 37.85 | 42.87 | 66.32 | 49.23 | 35.39 | 88.86 | 3.08
4 | 86.32 | 65.16 | 61.98 | 64.93 | 72.64 | 86.49 | 79.75 | 38.64 | 41.46 | 66.91 | 49.71 | 36.56 | 89.27 | 2.38
6 | 86.99 | 65.28 | 61.83 | 63.51 | 73.22 | 86.55 | 79.87 | 37.97 | 41.70 | 67.18 | 48.85 | 36.92 | 89.03 | 2.15
8 | 85.47 | 64.88 | 61.49 | 63.13 | 72.41 | 85.65 | 78.22 | 37.82 | 40.71 | 66.70 | 49.21 | 36.09 | 88.90 | 4.00
Hidden size of the Augmentation encoder |  |
16 | 84.51 | 63.40 | 61.38 | 62.31 | 71.61 | 85.40 | 78.96 | 37.53 | 41.84 | 66.54 | 48.68 | 35.54 | 88.78 | 5.54
32 | 85.43 | 63.94 | 61.93 | 64.18 | 72.05 | 85.67 | 79.80 | 37.86 | 41.52 | 67.13 | 48.71 | 35.64 | 88.67 | 4.38
64 | 86.99 | 65.28 | 61.83 | 63.51 | 73.22 | 86.55 | 79.87 | 37.97 | 41.70 | 67.18 | 48.85 | 36.92 | 89.03 | 2.54
128 | 86.24 | 64.66 | 61.95 | 64.25 | 72.12 | 86.72 | 79.56 | 37.79 | 42.71 | 67.61 | 49.52 | 36.87 | 88.99 | 2.62
256 | 86.23 | 65.63 | 61.77 | 61.79 | 72.03 | 85.69 | 80.38 | 37.86 | 40.94 | 67.07 | 49.59 | 37.00 | 89.37 | 3.31
512 | 85.77 | 66.05 | 62.21 | 64.55 | 72.45 | 86.38 | 80.06 | 37.91 | 41.94 | 66.22 | 49.35 | 36.24 | 88.87 | 2.69
Impact of utilizing color difference information during pretraining |  |
AugSelf $\omega^{\\{c,j\\}}$ | 85.26 | 63.90 | 60.78 | 63.36 | 73.46 | 85.70 | 78.93 | 37.35 | 39.47 | 66.22 | 48.52 | 37.00 | 89.49 | 3.07
CASSLE $\omega^{\\{c,j,b,f,g\\}}$ | 85.85 | 64.14 | 61.24 | 63.73 | 72.88 | 84.50 | 79.93 | 38.23 | 41.28 | 65.27 | 48.90 | 34.47 | 88.78 | 2.64
AugSelf $\omega^{\\{c,j,d\\}}$ | 84.95 | 64.06 | 61.53 | 63.06 | 73.52 | 86.25 | 77.38 | 36.00 | 42.54 | 66.33 | 48.65 | 37.40 | 88.36 | 2.71
CASSLE $\omega^{\\{c,j,b,f,g,d\\}}$ | 86.32 | 65.29 | 61.93 | 63.86 | 72.86 | 86.51 | 79.63 | 38.82 | 42.03 | 66.54 | 49.25 | 36.22 | 88.93 | 1.57

##  Appendix D Implementation details

We implement CASSLE in the PyTorch framework [52], building upon the codebase
of [40]. The anonymized version of our code is available at
https://anonymous.4open.science/r/CASSLE-037C/README.md.