# Addressing Sample Inefficiency in Multi-View Representation Learning

Kumar Krishna Agrawal1 Arna Ghosh2††footnotemark:  Adam Oberman2,3 Blake
Richards2,3,4,5

1 UC Berkeley 2 Mila - Quebec AI Institute & McGill University
3 Canada CIFAR AI Chair 4 CIFAR Learning in Machines & Brains
5 Montreal Neurological Institute equal contribution

###### Abstract

Non-contrastive self-supervised learning (NC-SSL) methods like BarlowTwins and
VICReg have shown great promise for label-free representation learning in
computer vision. Despite the apparent simplicity of these techniques,
researchers must rely on several empirical heuristics to achieve competitive
performance, most notably using high-dimensional projector heads and two
augmentations of the same image. In this work, we provide theoretical insights
on the implicit bias of the BarlowTwins and VICReg loss that can explain these
heuristics and guide the development of more principled recommendations. Our
first insight is that the orthogonality of the features is more critical than
projector dimensionality for learning good representations. Based on this, we
empirically demonstrate that low-dimensional projector heads are sufficient
with appropriate regularization, contrary to the existing heuristic. Our
second theoretical insight suggests that using multiple data augmentations
better represents the desiderata of the SSL objective. Based on this, we
demonstrate that leveraging more augmentations per sample improves
representation quality and trainability. In particular, it improves
optimization convergence, leading to better features emerging earlier in the
training. Remarkably, we demonstrate that we can reduce the pretraining
dataset size by up to 4x while maintaining accuracy and improving convergence
simply by using more data augmentations. Combining these insights, we present
practical pretraining recommendations that improve wall-clock time by 2x and
improve performance on CIFAR-10/STL-10 datasets using a ResNet-50 backbone.
Thus, this work provides a theoretical insight into NC-SSL and produces
practical recommendations for enhancing its sample and compute efficiency.

##  1 Introduction

Unsupervised representation learning, i.e., learning features without human-
annotated labels, is critical for progress in computer vision. Modern
approaches, grouped under the self-supervised learning (SSL) umbrella, build
on the core insight that similar images should map to nearby points in the
learned feature space. Current SSL methods can be broadly categorized into
contrastive and non-contrastive algorithms. While both categories aim to learn
the desired features using “positive” samples, which refer to different
augmentations of the same image, they diverge in using “negative” samples.
Contrastive methods use augmentations obtained from entirely different images
as negative samples to avoid the trivial solution of mapping all samples to
the same point in the feature space (i.e., representational collapse). But,
this necessitates an elaborate sampling scheme and huge batch sizes. Non-
contrastive methods, on the other hand, eliminate the need for negative
samples altogether and instead rely on regularizing the feature space to avoid
representational collapse.

A prominent subgroup among non-contrastive SSL methods is the family of
Canonical Correlation Analysis (CCA) algorithms, which includes BarlowTwins
[32] and VICReg [5]. These methods aim to enforce orthogonality among the
learned features in addition to learning to map similar images to nearby
points in feature space and have been shown to achieve competitive performance
on benchmark computer vision datasets. These methods have become the preferred
strategy for representation learning in several domains due to the lack of
need for negative samples and their simple formulation. However, despite the
apparent simplicity of their loss functions, the behavior of this family of
algorithms is not well understood. Therefore, researchers often use
empirically driven heuristics to design successful applications, such as using
(i) a high-dimensional projector head or (ii) two augmentations per image.
Although these heuristics help in practice, their theoretical underpinnings
are unclear.

Figure 1:  Design of existing SSL algorithms relies on heuristics. (A) We
investigate the theoretical underpinnings of (i) the number of augmentations
and (ii) the dimensionality of the projector $g_{\phi}$. (B) We show that a
generalized NC-SSL loss with multiple augmentations, & low-dimensional
projectors outperforms existing heuristics while using $\sim 4\times$ smaller
pretraining dataset.

Alongside relying on heuristics and researchers’ intuition for design,
existing SSL algorithms are extremely data-hungry. In particular, state-of-
the-art algorithms often rely on large-scale datasets [26] or data engines
[23] to achieve good representations. While this strategy works exceptionally
well in natural-image settings, its application is limited in other critical
domains, such as medical imaging, where the number of samples is scarce.

With these challenges in mind, the primary focus of this work is making
progress toward establishing theoretical foundations underlying the family of
non-contrastive SSL algorithms (NC-SSL) with an eye toward sample efficiency.
In particular, we analyze the BarlowTwins and VICReg losses and show that they
implicitly learn the data similarity kernel defined by the chosen
augmentations. We show that learning the data similarity kernel is helped by
greater orthogonality in the projector outputs and more data augmentations. As
such, increasing the orthogonality of the projector output eliminates the
requirement for a high-dimensional projector head, and increasing the number
of data augmentations decreases the number of unique samples required. Our
theoretical analysis establishes a principled grounding for the role of
multiple augmentations and the sufficiency of low-dimensional projectors,
outlining a framework for improving the sample efficiency of NC-SSL while
maintaining representation quality.

We empirically verify our theoretical insights using the popular ResNet-50
backbone on benchmark datasets, CIFAR-10 and STL-10. Strikingly, we show that
our multi-augmentation approach can learn good features even with a quarter of
the samples in the pretraining dataset. As such, this suggests that SSL
training can be done with smaller datasets and opens exciting questions in the
design of performance-enhancing transformations. In summary, our core
contributions are:

  * •

Eigenfunction interpretation: We demonstrate that the loss functions of the
CCA family of non-contrastive SSL algorithms are equivalent to the objective
of learning eigenfunctions of the augmentation-defined data kernel.

  * •

Role of heuristics: We provide a mechanistic explanation for the role of
projector dimensionality and the number of data augmentations and empirically
demonstrate that low-dimensional projector heads are sufficient and that using
more augmentations leads to learning better representations.

  * •

Data efficient NC-SSL: Leveraging the convergence benefits of the multi-
augmentation framework, we demonstrate that we can learn good features with
significantly smaller datasets (up to 25%) without harming downstream
performance.

##  2 Related Work

Self-Supervised Pretraining requires significant compute resources, and with
the lack of a unified theoretical framework, most practitioners rely on
empirical heuristics. The SSL cookbook [3] provides a comprehensive summary of
several widely adopted practices. While recent advances in SSL theory explore
learning dynamics in linear (or shallow) models [30, 29], with a focus on
understanding dimensionality collapse [20, 15], the theoretical underpinnings
of most “recipes” essential for good feature learning, are missing.

Contrastive SSL has received more theoretical attention, owing to its
connection with metric learning and noise contrastive estimation [22, 4, 19].
In particular, [18] provides a theoretical framework for the SimCLR loss from
an augmentation graph perspective, which leads to practical recommendations.
Subsequently, [14] establish a duality between contrastive and non-contrastive
learning objectives, further bridging the gap between theory and practice.

Non-contrastive SSL algorithms have comparatively scarce theoretical
foundations. In prior work [1, 14] find that with modified learning
objectives, low-dimensional projectors are sufficient for good downstream
performance. Similarly, previous works have demonstrated notable performance
boosts when using a multi-patch framework in contrastive [10] and non-
contrastive SSL. However, the theoretical basis for the benefits and trade-
offs of either low-dimensional projectors or multiple augmentations is
unclear.

Deep Learning theory has made significant strides in understanding the
optimization landscape and dynamics of supervised learning [2]. In concurrent
works, [33, 7] explore the interplay between the inductive bias of data
augmentations, architectures, and generalization. In recent work, [28] uses a
more straightforward formulation of the BarlowTwins loss and investigates the
learning dynamics in linearized models for the case when the invariance and
orthogonalization losses have equal penalties.

##  3 Preliminaries

We start by formally defining the unsupervised representation learning problem
for computer vision. In particular, we assume access to a dataset
$\mathcal{D}=\\{{x_{1},x_{2},...,x_{n}}\\}$ with $x_{i}\in\mathbb{R}^{p}$
consisting of unlabeled instances (often natural images), and the objective is
to learn a $d$-dimensional representation ($d<p$) that is useful across
multiple downstream applications. We focus on learning the parameters of a
deep neural network $f_{\theta}\in\mathcal{F}_{\Theta}$, using the multi-view
invariance SSL framework, wherein multiple views of an example are used to
optimize pretraining loss function,
$\mathcal{L}_{pretrain}(f_{\theta},\mathcal{D})$

Non-Contrastive Self-Supervised Learning (NC-SSL) algorithms impose invariance
to data-augmentations, which are used to define multiple views of the same
image while imposing specific regularization on the geometry of the learned
feature space. More generally, both families can be thought of as decomposing
$\mathcal{L}_{pretrain}$ into two terms (i) $\mathcal{L}_{invariance}$: to
learn invariance to data augmentations and (ii) $\mathcal{L}_{collapse}$ to
prevent collapsing the feature space to some trivial solution with no
discriminative power.

$\mathcal{L}_{pretrain}:=\mathcal{L}_{invariance}+\beta\mathcal{L}_{collapse}$
(1)

where $\beta$ denotes a hyperparameter that controls the importance of the
collapse-preventing term relative to the invariance term.

This formulation separates instance-level attributes invariant to
augmentations, highlighting the semantic information of the instance. The
ideal feature space is less sensitive to varying attributes and more sensitive
to semantic ones, facilitating generalization to new examples. Understanding
the interplay between pretraining loss and preserved attributes is critical
for time and compute efficiency.

Data Augmentation graph was introduced by [18] to analyze contrastive losses,
like SimCLR [8]. Briefly, we define a graph $\mathcal{G(X,W)}$ with the vertex
set ($\mathcal{X},\rho_{X}$) comprising the result of all possible data
augmentations from each sample in a dataset (could be infinite when continuous
augmentation functions are used) and $\mathcal{W}$ denoting the adjacency
matrix. Let $x_{0}$ be an image in $\mathcal{X}$, and let $z=M(x_{0})$ be a
random data augmentation of the image, $x_{0}$. We define the probability
density of reaching $z$ from $x_{0}$ via a choice of mapping $M$:

$\displaystyle p(z\mid x_{0})=\mathbb{P}(z=M(x_{0})),$ (2)

Since mapping is not generally invertible (e.g., crops), observe that
$p(x_{0}\mid z)\not=p(z\mid x_{0})$. Using this definition, we now define the
strength of the edge between nodes $x$ and $z$ of the augmentation graph as
the joint probability of generating augmentations $x,z$ from the same image
$x_{0}\sim\rho_{X}$. Formally,

$w_{xz}:=\mathbb{E}_{x_{0}\sim\rho_{X}}\left[p(x\mid x_{0})p(z\mid
x_{0})\right]$ (3)

It is worth noting that the magnitude of $w_{xz}$ captures the relative
similarity between $x$ and $z$. A higher value of $w_{xz}$ indicates that both
patches are more likely to come from the same image and, thereby, more
similar.

Figure 2:  Schematic of augmentation graph. (A) Augmentations from each image
span a region in the image space, which could overlap with the augmentation
span of other images. (B) An augmentation graph schematic that uses
probabilities to characterize the interactions among augmentation spans of
different instances.

[18] showed that optimizing a functionally equivalent form of the SimCLR loss
termed the spectral contrastive loss ($\mathcal{L}_{c}$, essentially learns
features whose covariance structure matches the adjacency matrix of the
augmentation graph.

$\mathcal{L}_{c}\propto\|ZZ^{T}-\bar{\mathcal{W}}\|^{2}_{F}$ (4)

where $Z$ denotes the output of the neural network, $\bar{\mathcal{W}}$
denotes the degree-normalized adjacency matrix and $\|.\|_{F}$ denotes the
Frobenius norm operator. This perspective implies that the features learned by
a contrastive SSL framework would align with the top eigenvectors of
$\bar{\mathcal{W}}$. As observed by [18], all rotations of $Z$ that don’t
change its span define an equivalence class of solutions to the above
optimization problem and make no difference for the downstream generalization
of a linear probe. Based on this insight, we define a notion of equivalence
among learned feature spaces.

######  Definition 3.1.

Let $F(x)=(f_{1}(x),\dots f_{d}(x))$ be a $d$-dimensional feature vector (a
vector of functions). Define the subspace

$V=V(F)=\\{h:X\to\mathbb{R}\mid h(x)=w\cdot F(x),\quad w\in\mathbb{R}^{d}\\}$
(5)

to be the span of the components of $F$. Given an $n$-dimensional feature
vector, $G(x)=(g_{1}(x),\dots,g_{n}(x))$ we say the features $G$ and $F$ are
equivalent, if $V(F)=V(G)$.

##  4 Data augmentation kernel perspective of non-contrastive SSL

Following the previous section, we present an augmentation kernel perspective
of BarlowTwins and VICReg losses. Specifically, we show that these losses are
equivalent to the optimization problem of learning eigenfunctions of the
augmentation-defined data covariance kernel. Subsequently, we argue that using
a high-dimensional projector yields better overlap with the top eigenvectors
of the data augmentation kernel at initialization compared to a low-
dimensional projector. Therefore, our analysis suggests using a stronger
orthogonalization constraint during optimization for lower-dimensional
projectors to ensure that features learned are equivalent to those learned
with high-dimensional projectors. Furthermore, we argue that using more
augmentations improves our estimate of the augmentation-defined data
covariance kernel, thereby aiding the eigenfunction optimization problem.
Therefore, our analysis suggests using an averaging operator with more data
augmentations to estimate the true augmentation kernel better.

###  4.1 Features in terms of data augmentation kernels

We will define two notions of the data augmentation kernel. Given two images,
$x,z$, the first kernel, which we call the forward data augmentation
covariance kernel, is given by

$k^{DAF}(x,z)=\mathbb{E}_{x_{0}\sim\rho_{X}}\left[\frac{p(x\mid
x_{0})}{\rho(x)}\frac{p(z\mid x_{0})}{\rho(z)}\right]$ (6)

This covariance kernel measures the similarity between $x,z$ in terms of how
likely they are to be reached from $x_{0}$, weighted by the distribution of
$x_{0}$. Note that this is the edge strength between nodes $x,z$ in the
augmentation graph. We can also define a (backward) data augmentation
covariance kernel that reverses the roles of ($x$,$z$) and $x_{0}$:

$k^{DAB}(x,z)=\mathbb{E}_{x_{0}\sim\rho_{X}}\left[\frac{p(x_{0}\mid
x)}{\rho(x_{0})}\frac{p(x_{0}\mid z)}{\rho(x_{0})}\right]$ (7)

SSL aims to learn features that preserve the covariance kernel structure
(imposed by this choice of mapping $M$) [11]. Therefore, we want to define a
loss which determines _vector features_ , $F:X\to\mathbb{R}^{d}$, which factor
a data augmentation kernel $k^{DA}(x,z)=F(x)^{\top}F(z)$. Doing this directly
is prohibitively data-intensive at scale since it involves a search over data-
augmented images. However, since the covariance kernels are PSD, they define a
Reproducing Kernel Hilbert space (RKHS). This allows us to apply Mercer’s
theorem to find vector features as in [12, 13, 24].

The construction of features using Mercer’s theorem goes as follows. Given a
PSD data augmentation kernel, $k^{DA}$, define the $T_{k}$ operator, which
takes a function $f$ and returns its convolution with the data augmentation
kernel.

$T_{k}f(x)=\mathbb{E}_{z\sim\rho_{X}}[k(z,x)f(z)]$ (8)

We will also make use of the following operator,

${T}_{M}f(x)=\mathbb{E}_{x_{0}\sim
M(x)}\left[f(x_{0})\right]=\sum_{x_{0}}\left[p(x_{0}\mid x)f(x_{0})\right]$
(9)

which averages the values of the function, $f$, over the augmented images
$x_{0}=M(x)$ of the data, $x$.

Since the operator $T_{k}$ is compact and positive, it has a spectral
decomposition consisting of eigenfunctions $\phi_{i}$ and corresponding
eigenvalues $\lambda_{i}$. Using these eigenpairs, we can define the (infinite
sequence of square summable) spectral features, $G:X\to\ell_{2}$, (where
$\ell_{2}$ represents square summable sequences), by

$G(x)=(\sqrt{\lambda_{1}}\phi_{1}(x),\dots,\sqrt{\lambda_{d}}\phi_{d}(x),\dots)$
(10)

Then, Mercer’s theorem gives

$k^{DA}(x,z)=G(x)\cdot G(z)$ (Mercer)

and ensures that the inner product is finite. These are the desired features
that factor in the kernel. However, computing the eigenfunctions of $T_{k}$ is
costly. Instead, we propose an alternative using the more efficient operator
$T_{M}$. According to Definition 3.1, both operators lead to equivalent
features.

######  Theorem 4.1.

Let $G(x)$ be the infinite Mercer features of the backward data augmentation
covariance kernels, $k^{DAB}$. Let $F(x)=(f_{1}(x),f_{2}(x),\dots,f_{k}(x))$
be the features given by minimizing the following data augmentation invariance
loss

$\displaystyle
L(F)=\sum_{i=1}^{N_{k}}\|T_{M}f_{i}-f_{i}\|^{2}_{L^{2}(\rho_{X})},\quad\text{subject
to }\quad(f_{i},f_{j})_{\rho_{X}}=\delta_{ij}$ (11)

which includes the orthogonality constraint. Then, $V(F)\subset V(G)$ ,
$V(F)\to V(G)$ as $N_{k}\to\infty$.

The idea of the proof uses the fact that, as linear operators,
$T_{k^{DAB}}=T_{M}^{\top}T_{M}$ and that $T_{k^{DAF}}=T_{M}T_{M}^{\top}$. Then
we use the spectral theory of compact operators, which is analog of the
Singular Value Decomposition in Hilbert Space, to show that eigenfunctions of
$T_{M}^{\top}T_{M}$ operator are the same as those obtained from optimizing
$L(F)$. A similar result can be obtained using $k^{DAF}$ and $T_{M}^{\top}$.

Note that $L(F)$ is the constrained optimization formulation of the
BarlowTwins loss. Furthermore, $L(F)$ with the additional constraint that
$(f_{i},f_{i})\geq\gamma$ $\forall i\in\\{1,2\dots N_{k}\\}$ is the
constrained optimization formulation of the VICReg loss.

###  4.2 Corollary 1: Low-dimensional projectors are sufficient

While BarlowTwins and VICReg frameworks have advocated using high-dimensional
projectors to facilitate good feature learning on Imagenet, our kernel
perspective challenges this notion. Since the intrinsic dimensionality of
Imagenet is estimated to be $\sim 40$ [25], it is not unreasonable to expect
that the span of desired features would be of similar dimensionality. It is,
thus, intriguing that these frameworks mandate the use of an $\sim 8192-d$
projector head to capture the intricacies of the corresponding data
augmentation kernel. This discrepancy can be explained by observing the
learning dynamics of a linearized model under the BarlowTwins loss
optimization [28]. These dynamics reveal that initializing the projection
weight matrix in alignment with the eigenfunctions of the data kernel retains
this alignment throughout the learning process. Notably, a high-dimensional
projector is more likely to have a greater initialization span than its low-
dimensional counterpart, increasing the likelihood of overlap with the
relevant eigenfunctions. We hypothesize that it is possible to rectify this
issue by using a stronger orthogonalization constraint for low-dimensional
projectors, thereby rendering them sufficient for good feature learning.

###  4.3 Corollary 2: Multiple augmentations improve optimization

Theorem 4.1 implies that the invariance loss optimization would ideally entail
using the $T_{M}$ operator, thereby requiring many augmentations for each
sample $x$. Using only two augmentations per sample yields a noisy estimate of
$T_{M}$, yielding spurious eigenpairs [31] (see Appendix). These spurious
eigenpairs add stochasticity to the learning dynamics and hinder the alignment
of the learned features with the eigenfunctions of the data kernel [28]. We
hypothesize that improving this estimation error by increasing the number of
augmentations could alleviate this issue and improve the speed and quality of
feature learning.

Increasing the number of augmentations (say $m$) in BarlowTwins and VICReg
comes with added compute costs. A straightforward approach would involve
calculating the invariance loss for every pair of augmentations, resulting in
$\mathcal{O}(m^{2})$ operations. However, Theorem 4.1 proposes an alternative
method that uses the sample estimate of $T_{M}$, thereby requiring only
$\mathcal{O}(m)$ operations. Both these strategies are functionally equivalent
(see Appendix), but the latter is computationally more efficient. In summary,
Theorem 4.1 establishes a mechanistic role for the number of data
augmentations, paving the way for a computationally efficient multi-
augmentation framework:

$\widehat{L}(F)=\mathbb{E}_{x\sim\rho_{X}}\left[\sum_{i=1}^{N_{k}}\sum_{j=1}^{m}\|\overline{f_{i}(x)}-f_{i}(x_{j})\|^{2}_{L^{2}(\rho_{X})}\right],\quad\text{subject
to }\quad(f_{i},f_{j})_{\rho_{X}}=\delta_{ij}$ (12)

where $\overline{f_{i}(x)}=\frac{1}{m}\sum_{j=1}^{m}f_{i}(x_{j})$ is the
sample estimate of $T_{M}f_{i}(x)$.

##  5 Experiments

In our experiments, we seek to serve two purposes: (i) provide empirical
support for our theoretical insights and (ii) present practical primitives for
designing efficient self-supervised learning routines. In summary, with
extensive experiments across learning algorithms (BarlowTwins, VICReg) and
training datasets (CIFAR-10/STL-10), we establish that

  * •

low-dimensional projectors as sufficient for learning good representations.

  * •

multi-augmentation improves downstream accuracy and convergence.

  * •

multi-Augmentation improves sample efficiency in SSL pretraining, i.e.,
recovering similar performance with significantly fewer unlabelled samples.

Experiment Setup: We evaluate the effectiveness of different pretraining
approaches for non-contrastive SSL algorithms using image classification as
the downstream task. Across all experiments, we use linear probing with
Resnet-50 as the feature encoder backbone. On CIFAR-10, all models are
pretrained for 100 epochs, while STL-10 models are pretrained for 50 epochs.
All runs are averaged over 3 seeds; error bars indicate standard deviation.
Other details related to optimizers, learning rate, etc., are presented in the
Appendix.

###  5.1 Sufficiency of Low-dimensional projectors

Figure 3:  Low-dimensional projectors are sufficient for good feature
learning. We demonstrate that using a higher orthogonality constraint
($\lambda$ for D, F and $\lambda_{eff}=\frac{1}{d\lambda}$ for E) for lower
projector dimensionality can achieve similar performance over a wide range of
projector dimensions ($d$).

Existing works recommend using high-dimensional MLPs as projectors (e.g.,
d=8192 for Imagenet in [32, 5]), and show significant degradation in
performance for a fixed redundancy coefficient ($\lambda$). To reproduce this
result, we run a grid search to find the optimal coefficient
$(\lambda^{*}_{8192})$ for $d=8192$ and show that performance progressively
degrades for lower $d$ if the same coefficient $\lambda^{*}_{8192}$ is reused
for $d\in\\{64,128,256,512,1024,2048,4096,8192\\}$.

Our insights in Section 4.2 suggest low-dimensional projectors should recover
similar performance with appropriate orthogonalization. To test this, we find
the best $\lambda$ by performing a grid search independently for each
$d\in\\{64,128,256,512,1024,2048,4096,8192\\}$. Low-dimensional projectors are
sufficient, as illustrated in Figure 3. Strikingly, we also observe that the
optimal $\lambda_{d}\propto 1/d$ is in alignment with our theoretical
insights.

Recommendataions: Start with low-dimensional projector, using
$\lambda=\mathcal{O}(\frac{1}{d})$, and sweep over
$(pdim=d,\lambda=\mathcal{O}\left(\frac{1}{d}\right))$ if needed.

###  5.2 Multiple Augmentations Improve Performance and Convergence

Although some SSL pretraining approaches, like SWaV, incorporate more than two
views, the most widely used heuristic in non-contrastive SSL algorithms
involves using two views jointly encoded by a shared backbone. In line with
this observation, our baselines for examining the role of multiple
augmentations use two views for computing the cross-correlation matrix.

Figure 4:  Using multiple augmentations improves representation learning
performance and convergence. (A-C) Across BarlowTwins and VICReg for CIFAR-10
and STL-10 pretraining, using 4 augmentations instead of 2 helps improve
performance. (D-F) Although the 4-augmentations take longer for each epoch,
its performance still trumps the 2-augmentation version of the algorithm at
the same wall clock time.

To understand the role of multiple augmentations in pretraining in light of
the augmentation-kernel interpretation, we propose Equation 12, which
generalizes Barlow-Twins and VICReg to the multi-augmentation setting. In
particular, for $\\#augs\in\\{2,4,8\\}$, we pretrain Resnet-50 with the
generalized NC-SSL loss for 100 epochs on CIFAR-10 and 50-epochs for STL-10.
Building on the insight from the previous section, we use a 256-dimensional
projector head for all experiments.

In Figure 4, we track the downstream performance of the pre-trained models
across training epochs, i.e., we extract features from intermediate
checkpoints and train a linear classifier on top of the features. Here, we use
the linear evaluation protocol as outlined by [6]. Figure 4(A-C) shows that
pretraining with multiple augmentations outperforms the 2-augmentation
baseline. Furthermore, we observe that the four-augmentation pre-trained
models converge faster (both in terms of the number of epochs and wall-clock
time) than their two-augmentation counterparts (see Figure 4(D-F)).

Recommendatation: Using multiple augmentations ( $>2$) with the generalized
NC-SSL loss is likely to improve convergence as well as downstream accuracy.

###  5.3 Sample Efficient Multi-View Learning

Data Augmentation can be viewed as a form of data inflation, where the number
of training samples is increased by $k$ (for $k$ augmentations). In this
section, we examine the role of multi-augmentation in improving sample
efficiency. In particular, we are interested in understanding if the same
performance can be achieved with a fraction of the pretraining dataset.

Figure 5:  Multi-augmentation improves sample efficiency, recovering similar
performance with significantly fewer unique samples in the pretraining
dataset. Across BarlowTwins and VICReg pretraining on CIFAR-10 and STL-10, for
the same effective dataset size ($\\#augs\times\\#unique\\_samples$), using
more patches improves performance at the same epoch (A-C) or wall clock time
(D-F). However, a tradeoff exists wherein doing more data augmentations fails
to improve performance in the scarce data regime.

To examine the relation between the number of augmentations and sample
efficiency, we fixed the effective size of the inflated dataset. This is
achieved by varying the fraction of the unique samples in the pretraining
dataset depending on the number of augmentations $k\in\\{2,4,8\\}$, e.g., we
use 1/2 the dataset for 4 views. We then evaluate the performance of the pre-
trained models on the downstream task, where the linear classifier is trained
on the same set of labeled samples. Strikingly, Figure 5 shows that using
multiple augmentations can achieve similar (sometimes even better) performance
with lesser pretraining samples, thereby indicating that more data
augmentations can be used to compensate for smaller pretraining datasets.

Recommendation: In a low-data regime, using diverse & multiple augmentations
can be as effective as acquiring more unique samples.

##  6 Discussion

Summary: Our work presents a fresh theoretical analysis that sheds light on
the implicit bias of non-contrastive SSL algorithms. We use these insights to
unravel the impact of crucial design heuristics and offer practical
recommendations that improve convergence while maintaining accuracy (on
CIFAR-10/STL-10). We also show that the multi-augmentation framework can be
used to learn good features from fewer unique samples in the pretraining
dataset simply by improving the estimation of the data augmentation kernel.

Pareto Optimal SSL In the context of sample efficiency, training a model using
two augmentations with different fractions of the dataset leads to a natural
Pareto frontier, i.e., training on the full dataset achieves the best error
but takes the most time (Baseline (2-Aug)). Our extensive experiments
demonstrate that using more than two augmentations improves the overall Pareto
frontier, i.e., achieves better convergence while maintaining accuracy (Multi-
Aug). Strikingly, as shown in Figure 6, we observe that we can either use a
larger pretraining dataset or more augmentations for a target error level.
Therefore, the number of augmentations can be used as a knob to control the
sample efficiency of the pretraining routine.

Figure 6:  Using $>2$ augmentations with a fraction of the dataset improves
overall Pareto frontier, speeding runtime up to $\sim 2\times$.

Open Questions: Looking ahead, it would be exciting to extend this analysis to
other categories of SSL algorithms, such as Masked AutoEncoders (MAE).
Furthermore, our insights provide opportunities to explore sample-efficient
methods that rely on less data, which is particularly important in critical
domains such as medical imaging, where data is often scarce and expensive.

Limitations Our algorithm relies on multiple views of the same image to
improve the estimation of the data-augmentation kernel. Although this approach
does add some extra computational overhead, it significantly speeds up the
learning process. We can explore the possibility of making the current design
more computationally efficient to improve it further.

### Acknowledgements

The authors would like to thank Arnab Kumar Mondal for insightful discussions
that helped shape the project’s scope and Colleen Gillon for her aesthetic
contribution to the figures. This research was generously supported by Vanier
Canada Graduate Scholarship (A.G.); NSERC (Discovery Grant: RGPIN-2020-05105;
Discovery Accelerator Supplement: RGPAS-2020-00031; Arthur B. McDonald
Fellowship: 566355-2022) and CIFAR Learning in Machines and Brains Program
(B.A.R.); Canada CIFAR AI Chair program (A.O. & B.A.R.). The authors also
acknowledge the material support of NVIDIA in the form of computational
resources.

## References

  * AMGR [22] Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards.  $\alpha$-req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay.  Advances in Neural Information Processing Systems, 35:17626–17638, 2022.
  * ASS [20] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky.  High-dimensional dynamics of generalization error in neural networks.  Neural Networks, 132:428–446, 2020.
  * BIS+ [23] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al.  A cookbook of self-supervised learning.  arXiv preprint arXiv:2304.12210, 2023.
  * BL [22] Randall Balestriero and Yann LeCun.  Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.  In NeurIPS, 2022.
  * BPL [21] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * CBLL [22] Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun.  Intra-instance vicreg: Bag of self-supervised image patch embedding.  arXiv preprint arXiv:2206.08954, 2022.
  * CKB+ [23] Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti.  The ssl interplay: Augmentations, inductive bias, and generalization.  In International Conference on Machine Learning, pages 3252–3298. PMLR, 2023.
  * CKNH [20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * CNL [11] Adam Coates, Andrew Y Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In International Conference on Artificial Intelligence and Statistics, pages 215–223, 2011.
  * DAT+ [21] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588–9597, 2021.
  * DEHL [22] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang.  Improving self-supervised learning by characterizing idealized representations.  Advances in Neural Information Processing Systems, 35:11279–11296, 2022.
  * [12] Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, and Jun Zhu.  Neural eigenfunctions are structured representation learners.  arXiv preprint arXiv:2210.12637, 2022.
  * [13] Zhijie Deng, Jiaxin Shi, and Jun Zhu.  Neuralef: Deconstructing kernels by deep neural networks.  In International Conference on Machine Learning, pages 4976–4992. PMLR, 2022.
  * GCB+ [22] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun.  On the duality between contrastive and non-contrastive self-supervised learning.  arXiv preprint arXiv:2206.02574, 2022.
  * GMAR [22] Arna Ghosh, Arnab Kumar Mondal, Kumar Krishna Agrawal, and Blake Richards.  Investigating power laws in deep representation learning.  arXiv preprint arXiv:2202.05808, 2022.
  * Goh [90] I Gohberg.  Classes of linear operator theory.  Advances and Applications, 49, 1990.
  * HK [12] Lajos Horváth and Piotr Kokoszka.  Inference for functional data with applications, volume 200.  Springer Science & Business Media, 2012.
  * HWGM [21] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma.  Provable guarantees for self-supervised deep learning with spectral contrastive loss.  Advances in Neural Information Processing Systems, 34:5000–5011, 2021.
  * JHM [23] Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison.  Contrastive learning can find an optimal basis for approximately view-invariant functions.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * JVLT [21] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  arXiv preprint arXiv:2110.09348, 2021.
  * KH [09] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical Report, 2009.
  * LPSG [21] Yazhe Li, Roman Pogodin, Danica J Sutherland, and Arthur Gretton.  Self-supervised learning with kernel dependence maximization.  Advances in Neural Information Processing Systems, 34:15543–15556, 2021.
  * ODM+ [23] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * PPA+ [18] David Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, and Kimberly L Stachenfeld.  Spectral inference networks: Unifying deep and spectral learning.  arXiv preprint arXiv:1806.02215, 2018.
  * PZA+ [20] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.  The intrinsic dimension of images and its impact on learning.  In International Conference on Learning Representations, 2020.
  * RDS+ [15] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * SJK+ [22] Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang.  Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation.  arXiv preprint arXiv:2204.00570, 2022.
  * SKZ+ [23] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht.  On the stepwise nature of self-supervised learning.  arXiv preprint arXiv:2303.15438, 2023.
  * TCG [21] Yuandong Tian, Xinlei Chen, and Surya Ganguli.  Understanding self-supervised learning dynamics without contrastive pairs.  In International Conference on Machine Learning, pages 10268–10278. PMLR, 2021.
  * TYCG [20] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli.  Understanding self-supervised learning with dual deep networks.  arXiv preprint arXiv:2010.00578, 2020.
  * Ver [10] Roman Vershynin.  Introduction to the non-asymptotic analysis of random matrices.  arXiv preprint arXiv:1011.3027, 2010.
  * ZJM+ [21] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  arXiv preprint arXiv:2103.03230, 2021.
  * ZLR+ [23] Runtian Zhai, Bingbin Liu, Andrej Risteski, Zico Kolter, and Pradeep Ravikumar.  Understanding augmentation-based self-supervised representation learning via rkhs approximation.  arXiv preprint arXiv:2306.00788, 2023.

##  Appendix A Hilbert Space of functions

###  A.1 Functions and inner product space

######  Definition A.1.

Given $X,\rho_{X}$, and $f,g:X\to\mathbb{R}$, define the ${L^{2}(\rho_{X})}$
inner product and norm, respectively,

$(f,g)_{{\rho_{X}}}=\int
f(x)g(x)d\rho_{X}(x),\quad\|f\|^{2}_{\rho_{X}}=(f,f)_{\rho_{X}}$ (13)

Define

$L^{2}(\rho,X)=\left\\{f:X\to\mathbb{R}\
\mid\|f\|^{2}_{\rho_{X}}<\infty\right\\}$

to be the (equivalence class) of functions with finite ${\rho_{X}}$ norm.

###  A.2 Spectral theory

In this section we quote the relevant (abstract) Hilbert Space theory.

######  Definition A.2 (Spectral Operator).

Given orthogonal functions, $\Phi=(\phi_{i})_{i\in I}$ in ${L^{2}(\rho_{X})}$,
and non-negative $\Lambda=(\lambda_{i})_{i\in I}$, with
$\|\Lambda||_{2}^{2}=\sum_{i\in I}\lambda_{i}^{2}<\infty$. Call
$(\Phi,\Lambda)$ a spectral pair and define the corresponding spectral
operator by

${T}_{\Phi,\Lambda}(h)=\sum_{j=1}^{\infty}\lambda_{j}\left(h,\phi_{j}\right)\phi_{j},$
(14)

######  Theorem A.3 (Spectral Decomposition).

Suppose $H$ is a Hilbert space. A symmetric positive-definite Hilbert-Schmidt
operator ${T}:\mathbb{H}\to\mathbb{H}$ admits the spectral decomposition
equation 14 with orthonormal $\phi_{j}$ which are the eigenfunctions of ${T}$,
i.e. ${T}\left(\phi_{j}\right)=\lambda_{j}\phi_{j}$. The $\phi_{j}$ can be
extended to a basis by adding a complete orthonormal system in the orthogonal
complement of the subspace spanned by the original $\phi_{j}$.

######  Remark A.4.

The $\phi_{j}$ in equation 14 can thus be assumed to form a basis, but some
$\lambda_{j}$ may be zero.

From [17]. Theorem proved in [16]. Denote by $\mathcal{L}$ the space of
bounded (continuous) linear operators on $\mathbb{H}$ with the norm

$\|{T}\|_{\mathcal{L}}=\sup\\{\|{T}(x)\|\mid\|x\|\leq 1\\}.$

######  Definition A.5 (Compact Operators).

An operator ${T}\in\mathcal{L}$ is said to be compact if there exist two
orthonormal bases $\left\\{g_{j}\right\\}$ and $\left\\{f_{j}\right\\}$, and a
real sequence $\left\\{\lambda_{j}\right\\}$ converging to zero, such that

${T}(h)=\sum_{j=1}^{\infty}\lambda_{j}(h,g_{j})f_{j},\quad h\in\mathbb{H},$
(Compact)

The $\lambda_{j}$ may be assumed positive. The existence of representation
equation Compact is equivalent to the condition: ${T}$ maps every bounded set
into a compact set. Compact operators are also called completely continuous
operators. Representation equation Compact is called the singular value
decomposition.

######  Definition A.6 (Hilbert-Schmidt Operators).

A compact operator admitting representation equation Compact is said to be a
Hilbert-Schmidt operator if $\sum_{j=1}^{\infty}\lambda_{j}^{2}<\infty$. The
space $\mathcal{S}$ of Hilbert-Schmidt operators is a separable Hilbert space
with the scalar product

$\left\langle{T}_{1},{T}_{2}\right\rangle_{\mathcal{S}}=\sum_{i=1}^{\infty}\left({T}_{1}\left(f_{i}\right),{T}_{2}\left(f_{i}\right)\right),$
(15)

where $\left\\{f_{i}\right\\}$ is an arbitrary orthonormal basis. Note the
value of equation 15 is independent of the basis. The corresponding norm is

$\|{T}\|_{\mathcal{S}}^{2}=\sum_{j\geq 1}\lambda_{j}^{2}$ (HS)

One can show that

$\|{T}\|_{\mathcal{L}}\leq\|{T}\|_{\mathcal{S}}$

######  Definition A.7.

An operator ${T}\in\mathcal{L}$ is said to be symmetric if

$\langle{T}(f),g\rangle=\langle f,{T}(g)\rangle,\quad f,g\in\mathbb{H},$

and positive-definite if

$\langle{T}(f),f\rangle\geq 0,\quad f\in\mathbb{H}.$

(An operator with the last property is sometimes called positive semidefinite,
and the term positive-definite is used when the inequality is strict.)

##  Appendix B Data augmentation kernel perspective of non-contrastive SSL

######  Theorem B.1.

Let $G(x)$ be the infinite Mercer features of the backward data augmentation
covariance kernels, $k^{DAB}$. Let $F(x)=(f_{1}(x),f_{2}(x),\dots,f_{k}(x))$
be the features given by minimizing the following data augmentation invariance
loss

$\displaystyle
L(F)=\sum_{i=1}^{N_{k}}\|T_{M}f_{i}-f_{i}\|^{2}_{L^{2}(\rho_{X})},\quad\text{subject
to }\quad(f_{i},f_{j})_{\rho_{X}}=\delta_{ij}$ (16)

which includes the orthogonality constraint. Then, $V(F)\subset V(G)$ ,
$V(F)\to V(G)$ as $N_{k}\to\infty$.

The idea of the proof uses the fact that, as linear operators,
$T_{k^{DAB}}=T_{M}^{\top}T_{M}$ and that $T_{k^{DAF}}=T_{M}T_{M}^{\top}$. Then
we use spectral theory of compact operators, which is analogue of the Singular
Value Decomposition in Hilbert Space, to show that eigenfunctions of
$T_{M}^{\top}T_{M}$ operator are the same as those obtained from optimizing
$L(F)$. A similar result can be obtained using $k^{DAF}$ and $T_{M}^{\top}$.

Note that $L(F)$ is the constrained optimization formulation of the
BarlowTwins loss. Furthermore, $L(F)$ with the additional constraint that
$(f_{i},f_{i})\geq\gamma$ $\forall i\in\\{1,2\dots N_{k}\\}$ is the
constrained optimization formulation of the VICReg loss.

###  B.1 Proof of theorem 3.2

We show we can factor the linear operator, leading to a practical algorithm.
Here, we show that we can capture the backward data augmentation kernel with
the forward data augmentation averaging operator

######  Lemma B.2.

Using the definitions above, and with $k$ in equation 8 given by $k^{DAB}$,

$T_{k}=T_{M}^{\top}T_{M}$

###### Proof.

First, define the non-negative definite bilinear form

$B^{VAR}(f,g)=({T}_{M}f,{T}_{M}g)_{\rho_{X}}$ (17)

Given the backwards data augmentation covariance kernel, $k^{DAB}$, define

$B^{DAB}(f,g)=(T_{k}f,g)_{\rho_{X}}$

We claim, that

$B^{VAR}=B^{DA,B}$ (18)

This follows from the following calculation,

$\displaystyle B^{DA,B}(f,g)$ $\displaystyle=(T_{k}f,g)_{\rho_{X}}$ (19)
$\displaystyle=\mathbb{E}_{x}[T_{k}f(x),g(x)]=\mathbb{E}_{x}\mathbb{E}_{z}[k_{DA,B}(z,x)f(z)g(x)]$
(20) $\displaystyle=\mathbb{E}_{x}\mathbb{E}_{z}\mathbb{E}_{x_{0}}[p(x\mid
x_{0})p(z\mid x_{0})f(z)g(x)]$ (21)
$\displaystyle=\mathbb{E}_{x_{0}}\left[\mathbb{E}_{x}[p(x\mid
x_{0})g(x)],\mathbb{E}_{z}[p(z\mid
x_{0})f(z)],\right]=\mathbb{E}_{x_{0}}{T}_{M}f(x_{0}){T}_{M}g(x_{0})$ (22)
$\displaystyle=({T}_{M}f,{T}_{M}g)_{\rho_{X}}=B^{VAR}(f,g)$ (23)

∎

For implementations, it is more natural to consider _invariance_ to data
augmentations.

######  Theorem B.3 (equivalent eigenfunctions).

Assume that ${T}_{M}$ is a compact operator. Define the invariance bilinear
form

$B^{INV}(f,g)=(T_{M}f-f,T_{M}g-g)$ (24)

Then $B^{INV}$, $B^{VAR}$ share the same set of eigenfunctions. Moreover,
these are the same as the eigenfunctions of $B^{DA,B}$. In particular, for any
eigenfunction $f_{j}$ of $B^{VAR}$, with eigenvalue $\lambda_{j}$, then
$f_{j}$ is also and eigenfunction of $B^{INV}$, with the corresponding
eigenvalue given by $(\sqrt{\lambda_{j}}-1)^{2}$.

###### Proof.

Define ${T}_{MM}$ by,

${T}_{MM}f={T}_{M}^{\top}{T}_{M}f$ (25)

Define

${T}_{MS}=(T_{M}-I)^{\top}(T_{M}-I)$ (26)

Note, by the assumption of compactness, ${T}_{M}$ has the Singular Value
Decomposition, (see the Hilbert Space section for equation SVD),

${T}_{M}(h)=\sum_{j=1}^{\infty}\lambda_{j}(h,g_{j})f_{j}$ (SVD)

Let $f_{j}$ be any right eigenvector of $T_{M}$, with eigenvalue $\mu_{j}$.
Then $f_{j}$ is also a right eigenvector $T_{M}-I$, with eigenvalue
$\mu_{j}-1$. So we see that $T_{MM}$ has $f_{j}$ as an eigenvector, with
eigenvalue $\lambda_{j}=\mu_{j}^{2}$ and $T_{MS}$ has $f_{j}$ as an
eigenvector, with eigenvalue $(\sqrt{\lambda_{j}}-1)^{2}$. Finally, the fact
that there are no other eigenfunctions also follows from equation SVD.

The final part follows from the previous lemma. ∎

##  Appendix C Multi-Augmentation Learning

###  C.1 Augmentation graph

We use the population augmentation graph formulation introduced in [18].
Briefly, we define a graph $\mathcal{G(X,W)}$, where the vertex set
$\mathcal{X}$ comprises of all augmentations from the dataset (could be
infinite when continuous augmentation functions are used) and $\mathcal{W}$
denotes the adjacency matrix with edge weights as defined below:

$w_{xx^{\prime}}:=\mathbb{E}_{\bar{x}\sim\mathcal{P}_{\bar{X}}}\left[\mathcal{A}(x|\bar{x})\mathcal{A}(x^{\prime}|\bar{x})\right]$
(27)

, i.e. the joint probability of generating ‘patches’ $x,x^{\prime}$ from the
same image $\bar{x}$. Here $\mathcal{A}$ defines the set of augmentation
functions used in the SSL pipeline. It is worth noting that the magnitude of
$w_{xx^{\prime}}$ captures the relative similarity between $x$ and
$x^{\prime}$. A higher value of $w_{xx^{\prime}}$ indicates that it is more
likely that both patches came from the same image, and thereby are more
similar. The marginal likelihood of each patch $x$ can also be derived from
this formulation:

$w_{x}=\mathbb{E}_{x^{\prime}\sim\mathcal{X}}\left[w_{xx^{\prime}}\right]$
(28)

###  C.2 Contrastive and non-contrastive losses suffer from the same issues

We will now show that using multiple patches for the
$\mathcal{L}_{invariance}$ is pertinent to both the contrastive and non-
contrastive SSL. Following [18], we use the spectral contrastive loss
formulation and incorporate the augmentation graph relations:

$\displaystyle\mathcal{L}_{c}$
$\displaystyle=-\mathbb{E}_{x,x^{+}}\left[f(x)^{T}f(x^{+})\right]+\beta\mathbb{E}_{x,x^{\prime}}\left[\left(f(x)^{T}f(x^{\prime})\right)^{2}\right]$
$\displaystyle\mathcal{L}_{c}$
$\displaystyle\propto\|ZZ^{T}-D^{-\frac{1}{2}}\mathcal{W}D^{-\frac{1}{2}}\|^{2}_{F}=\|ZZ^{T}-\bar{\mathcal{W}}\|^{2}_{F}$
(29)

where $z:=\sqrt{w_{x}}f(x)$, $D$ is a $N\times N$ diagonal matrix with entries
$\\{w_{x}\\}$ and
$\bar{\mathcal{W}}=D^{-\frac{1}{2}}\mathcal{W}D^{-\frac{1}{2}}$.

We extend the duality results between contrastive and non-contrastive SSL
loss, established by [14], to demonstrate how eq. 29 can be decomposed into
the invariance and collapse-preventing loss terms.

$\displaystyle\|ZZ^{T}-\bar{\mathcal{W}}\|^{2}_{F}$
$\displaystyle=\|Z^{T}Z-I_{d}\|^{2}_{F}+2Tr\left[Z^{T}(I_{N}-\bar{\mathcal{W}})Z\right]+\kappa$
(30)
$\displaystyle=\|Z^{T}Z-I_{d}\|^{2}_{F}+2\sum_{i}\sum_{x}(1-\bar{w}_{x})z_{i}^{2}-2\sum_{i}\sum_{x,x^{\prime}}\bar{w}_{xx^{\prime}}z_{i}z_{i}^{\prime}+\kappa$
(31)

where $\kappa$ is some constant independent of $Z$. The first term in eq. 30
is the covariance regularization term in non-contrastive losses like
BarlowTwins (implicit) or VIC-Reg (explicit), and the second term in eq. 31 is
the variance regularization. Simplifying the third term in eq. 31 gives us:

$\displaystyle\sum_{i}\sum_{x,x^{\prime}}\bar{w}_{xx^{\prime}}z_{i}z_{i}^{\prime}$
$\displaystyle=\sum_{i}\sum_{x,x^{\prime}}w_{xx^{\prime}}f(x)_{i}f(x^{\prime})_{i}=\sum_{i}\sum_{x,x^{\prime}}\mathbb{E}_{\bar{x}\sim\mathcal{P}_{\bar{X}}}\left[\mathcal{A}(x|\bar{x})\mathcal{A}(x^{\prime}|\bar{x})f(x)_{i}f(x^{\prime})_{i}\right]$
$\displaystyle=\sum_{i}\mathbb{E}_{\bar{x}\sim\mathcal{P}_{\bar{X}}}\left[\sum_{x}\mathcal{A}(x|\bar{x})(f(x)_{i}\overline{f(x)}_{i}-f(x)_{i}^{2})\right]$
$\displaystyle=\mathbb{E}_{\bar{x}\sim\mathcal{P}_{\bar{X}}}\left[\sum_{x}\mathcal{A}(x|\bar{x})\left(f(x)^{T}\overline{f(x)}-\|f(x)\|^{2}\right)\right]$
(32)

This term encourages $f(x)$ to be similar to $\overline{f(x)}$, i.e. the mean
representation across all augmentations of $\bar{x}$, thereby requiring to
“sufficiently” sample $A(.|\bar{x})$. Given that both the contrastive and non-
contrastive losses rely on learning invariance properties from data
augmentations, we believe that our multi-patch proposal would improve the
probability density estimation of $A(.|\bar{x})$ and yield better performance
with few training epochs.

###  C.3 Explaining training dynamics in low patch sampling regime

We now turn to a simple form of the augmentation graph to understand how using
low number of augmentations affects the evolution of $ZZ^{T}$. Minimizing eq.
29 implies that the spectral decomposition of $Z$ would align with the top
eigenvectors (and values) of $\overline{\mathcal{W}}$. We will demonstrate
that in the low sampling regime (using few augmentations), the eigenvectors of
the sampled augmentation graph $\tilde{\mathcal{W}}$ may not align with those
of $\overline{\mathcal{W}}$.

Augmentation graph setup. We define an augmentation graph with only two
instances from two different classes, similar to the one presented in [27].
Let us denote the four instances as $\bar{x}_{i}$ for $i\in{1,2,3,4}$, where
$\bar{x}_{1},\bar{x}_{2}$ belong to class 1 (i.e. $y_{1},y_{2}=1$) and
$\bar{x}_{3},\bar{x}_{4}$ belong to class 2 (i.e. $y_{3},y_{4}=4$). Let us
further assume that $\bar{x}_{1},\bar{x}_{3}$ have the highest pixel-level
similarity among $(\bar{x}_{1},\bar{x}_{i})\forall i\in{2,3,4}$, thereby
making it more likely to have similar patches (see Figure 2 for illustration).
We denote this relationship among input examples using $\mathcal{G}$ to
indicate (pixel-wise) global similarity groups. So,
$\mathcal{G}_{1},\mathcal{G}_{3}=1$ and $\mathcal{G}_{2},\mathcal{G}_{4}=2$.
We can use the following probabilistic formulation to model our augmentation
functions (see Figure 2B):

$\displaystyle A(x_{j}|\bar{x}_{i})=\begin{cases}\rho^{\prime}&\text{if
$j=i$}\\\ \mu^{\prime}&\text{if $j\neq i$ and $y_{j}=y_{i}$ and
$\mathcal{G}_{j}\neq\mathcal{G}_{i}$}\\\ \nu^{\prime}&\text{if $j\neq i$ and
$y_{j}\neq y_{i}$ and $\mathcal{G}_{j}=\mathcal{G}_{i}$}\\\
\delta^{\prime}&\text{if $j\neq i$ and $y_{j}\neq y_{i}$ and
$\mathcal{G}_{j}\neq\mathcal{G}_{i}$}\end{cases}$ (33)

In our setting, $\rho^{\prime}+\mu^{\prime}+\nu^{\prime}+\delta^{\prime}=1$.
The adjacency matrix of our augmentation graph (as shown in Figure 2C) is as
follows:

$\displaystyle\overline{\mathcal{W}}=\begin{bmatrix}\rho&\mu&\nu&\delta\\\
\mu&\rho&\delta&\nu\\\ \nu&\delta&\rho&\mu\\\ \delta&\nu&\mu&\rho\\\
\end{bmatrix}$ (34)

We defer the relations between
$\rho^{\prime},\mu^{\prime},\nu^{\prime}\delta^{\prime}$ and
$\rho,\mu,\nu,\delta$ to the appendix. The eigenvalues of this matrix are:
($\rho+\mu+\nu+\delta$, $\rho+\mu-\nu-\delta$,$\rho-\mu+\nu-\delta$,
$\rho-\mu-\nu+\delta$). Corresponding eigenvectors are along
$\left[1,1,1,1\right]^{T}$, $\left[1,1,-1,-1\right]^{T}$.
$\left[1,-1,1,-1\right]^{T}$, $\left[1,-1,-1,1\right]^{T}$. Assuming that the
augmentation functions induce semantically-relevant invariance properties that
are relevant for identifying $y_{i}$ from $f(x_{i})$, we can say that
$\rho^{\prime}>max\\{\mu^{\prime},\nu^{\prime}\\}$ and
$min\\{\nu^{\prime},\mu^{\prime}\\}>\delta^{\prime}$. When we have
sufficiently sampled the augmentations, any SSL loss will learn $Z$ such that
its singular values are span the top eigenvectors of the augmentation graph,
and the eigenspectrum of $ZZ^{T}$ would simply be the above eigenvalues. In
practical settings, the augmentation graph would have significantly higher
dimension that the feature/embedding dimension 111Contrastive algorithms use a
large batch size, thereby optimizing a high-dimensional $ZZ^{T}$ whereas non-
contrastive algorithms use a large embedding dimension, thereby optimizing a
high-dimensional $Z^{T}Z$.. Therefore, singular vectors of $Z$ would span the
top eigenvectors of $\overline{\mathcal{W}}$ and the smaller eigenmodes are
not learned. When we have accurately sampled the augmentation graph, $\mu>\nu$
and therefore, the class-information preserving information is preferred over
pixel-level preserving information during learning. But what happens when we
do not sufficiently sample the augmentation space?

Figure 7:  Empirical verification of the subsampling Ansatz.

Ansatz. Based on our empirical experience, we define an ansatz pertaining to
the eigenvalues of a sampled augmentation graph and validate it in tractable
toy settings, such as the one described above. Specifically, we claim that
when the augmentation space is not sufficiently sampled,
$\\{|\mu-\nu|,\delta\\}\to 0$. In other words, we claim that when only few
augmentations per example are used, it is more likely to have an equal
empirical likelihood for augmentations that preserve (pixel-level) global
information and class/context information. Moreover, it is very unlikely to
have augmentations that change both the class and global information. This is
demonstrated in Figure 7.

Consequences of the Ansatz. When only a few augmentations are sampled,
learning can suppress the class information at the cost of preserving the
pixel-level information, thereby leading to an increased smoothness in the
learned feature space.

##  Appendix D Implementation Details

Image Classification Datasets Across all experiments, our settings mainly
follow [6]. In particular, we run Table 1 summarizes our pretraining settings
on Cifar-10 [21], and STL-10 [9]. In Table 1, we outline the corresponding
linear evaluation settings for Resnet-50. Note that we add a linear classifier
layer to the encoder’s features and discard the projection layers for
evaluation.

config | value
---|---
optimizer | Adam
learning rate | 1e-3
batch size | 128
epochs | 100 (CIFAR10), 50 (STL10)
weight-decay | 1e-6
(a)  Pretraining

config | value
---|---
optimizer | Adam
learning rate | 1e-3
batch size | 512
epochs | 200
weight-decay | 1e-6
test-patches | 16
(b)  Linear Evaluation

Table 1:  Experiment Protocol for comparing SSL algorithms

###  D.1 Longer Pretraining

Figure 8:  BarlowTwins pretraining on full CIFAR-10 dataset for 400 epochs. Algorithm | Best accuracy | Best accuracy @ epoch
---|---|---
Barlow-Twins (2-augs) w/ pdim=256 | 92.04 +/- 0.16 | 400
Barlow-Twins (4-augs) w/ pdim=256 | 92.39 +/- 0.17 | 340
Barlow-Twins (8-augs) w/ pdim=256 | 92.64 +/- 0.10 | 140
Table 2:  BarlowTwins pretraining on full CIFAR-10 dataset at 400 epochs (with
early stopping)

###  D.2 SwAV-like augmentations for compute efficient multi-augmentation
framework

Figure 9:  BarlowTwins pretraining on full STL-10 dataset for 100 epochs using
SwAV-like augmentations. Specifically, the 2-augmentations setting uses two
views that are $64\times 64$, whereas the 4 (or 8) augmentation setting uses
additional two (or six) augmentations that are $32\times 32$.

###  D.3 On training with full dataset with 4/8 augmentations

Figure 10:  BarlowTwins pretraining on full CIFAR-10 dataset. Algorithm | #augs=2 | #augs=4 | #augs=8
---|---|---|---
Barlow-Twins w/ pdim=256 | 86.43 +/- 0.72 | 91.73 +/- 0.16 | 92.71 +/- 0.19
Barlow-Twins w/ pdim=8192 | 85.44 +/- 0.54 | 91.40 +/- 0.32 | 92.40 +/- 0.13
Table 3:  BarlowTwins pretraining on full CIFAR-10 dataset at 100 epochs

###  D.4 Empirical results on transfer learning

In this section, we present extended version of results presented in Figure 4,
Figure 5 but pretraining on CIFAR-10 (or STL-10) and evaluating on STL-10 (or
CIFAR-10). These results, coupled with the ones in Figure 4 Figure 5, present
a strong case for the advantage of using the proposed multi-augmentation loss
for better convergence as well as downstream accuracy.

Figure 11:  BarlowTwins pretraining on CIFAR-10, linear evaluation on STL-10
labelled set. Figure 12:  VICReg pretraining on CIFAR-10, linear evaluation on
STL-10 labelled set. Figure 13:  BarlowTwins pretraining on STL-10, linear
evaluation on CIFAR-10 labelled set. Figure 14:  BarlowTwins pretraining on
fraction of CIFAR-10 trainset, linear evaluation on STL-10 labelled set.
Figure 15:  VICReg loss pretraining on fraction of CIFAR-10 trainset, linear
evaluation on STL-10 labelled set. Figure 16:  BarlowTwins loss pretraining on
fraction of STL-10 unlabelled set, linear evaluation on CIFAR-10 train set.