capbstabboxtable[][]

# A simple, efficient and scalable contrastive masked autoencoder for learning
visual representations

Shlok Mishra∗
University of Maryland
<EMAIL_ADDRESS>
&Joshua Robinson
MIT CSAIL
<EMAIL_ADDRESS>
&Huiwen Chang
Google Research
David Jacobs
University of Maryland &Aaron Sarna
Google Research &Aaron Maschinot
Google Research &Dilip Krishnan
Google Research
<EMAIL_ADDRESS>Equal contribution, order chosen randomly. Work done
during an internship at Google.

###### Abstract

We introduce CAN, a simple, efficient and scalable method for self-supervised
learning of visual representations. Our framework is a minimal and
conceptually clean synthesis of (C) contrastive learning, (A) masked
autoencoders, and (N) the noise prediction approach used in diffusion models.
The learning mechanisms are _complementary_ to one another: contrastive
learning shapes the embedding space across a batch of image samples; masked
autoencoders focus on reconstruction of the low-frequency spatial correlations
in a single image sample; and noise prediction encourages the reconstruction
of the high-frequency components of an image. The combined approach results in
a robust, scalable and simple-to-implement algorithm. The training process is
symmetric, with $50\%$ of patches in _both views_ being masked at random,
yielding a considerable efficiency improvement over prior contrastive learning
methods. Extensive empirical studies demonstrate that CAN achieves strong
downstream performance under both linear and finetuning evaluations on
transfer learning and robustness tasks. CAN outperforms MAE and SimCLR when
pre-training on ImageNet, but is especially useful for pre-training on larger
uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN
achieves $75.4\%$ compared to $73.4\%$ for SimCLR and $64.1\%$ for MAE. The
finetuned performance on ImageNet of our ViT-L model is $86.1\%$, compared to
$85.5\%$ for SimCLR, and $85.4\%$ for MAE. The overall FLOPs load of SimCLR is
$70\%$ _higher_ than CAN for ViT-L models111Code will be released..

##  1 Introduction

Self-supervised learning promises continued advances in the state of the art
by enabling the use of increasingly large models and datasets. However,
interest in larger datasets has precipitated an increased reliance on web-
scraped data collection processes, which result in heterogeneous and
“uncurated” datasets (Yu et al., 2022; Radford et al., 2021; Jia et al.,
2021). Extreme image heterogeneity has made scaling vision models to uncurated
datasets a non-trivial challenge (Tian et al., 2021; Cole et al., 2022). There
are two families of self-supervised methods for images which have both proven
highly effective on curated datasets (e.g., ImageNet), and are therefore
natural candidates for scaling to large, uncurated data. First, masked image
models such as the masked autoencoder (MAE) (He et al., 2022) are a nascent
set of methods based on a mask-and-reconstruct training mechanism. This
classical idea (Ballard, 1987) is enjoying a rejuvenation thanks to favourable
efficiency when combined with the vision transformer architecture (Dosovitskiy
et al., 2021b). Second, contrastive learning (van den Oord et al., 2018; Chen
et al., 2020b; He et al., 2020) trains an encoder to distinguish between pairs
of positive samples generated with data augmentations and negative pairs
sampled at random. Both approaches have proven to be very powerful self-
supervised methods.

Contrastive learning and masked autoencoders (MAE) employ very different
learning mechanisms: the former train the encoder to be invariant to
semantics-preserving data variations, while MAEs learn spatial statistical
correlations. Furthermore, MAE methods treat each sample independently in the
loss function, while contrastive methods explicitly look at the relationship
between all samples in the batch, by either reducing or increasing embedding
distance. Given this, we hypothesize that these two approaches are
_complementary_ , extracting different discriminative features for a given
input. If this hypothesis holds, then we expect to see improved performance on
various downstream tasks based on the extracted features. This motivates our
exploration of a combined method.

Figure 1: Left: CAN scales better than SimCLR since it uses masked inputs.
Right: CAN outperforms SimCLR and MAE on ImageNet linear probe evaluation for
ViT-L models, especially when pre-training on uncurated data such at JFT-300M.

Further, inspired by advances in diffusion models (Ho et al., 2020; Song et
al., 2021), we introduce a third loss based on _noise prediction_ during the
masked autoencoder reconstruction. We add Gaussian noise to unmasked input
patches, and train the model to predict the _noise_ added to each patch.
Denoising encourages the encoder to extract higher-frequency information from
the input, while autoencoder reconstructions tend to focus on low-frequency
information (Hou et al., 2017). This additional loss has two purposes: it
improves downstream performance; and it addresses a source of wasted
computation in MAE with a negligible impact on FLOPs: that reconstruction of
unmasked patches is thrown away unused.

Combining these ingredients we present CAN, a minimal fusion of contrastive
learning, masked autoencoders and denoising diffusion training loss. Our
method enjoys stronger performance than its constituent parts do on their own,
especially pronounced benefits on more uncurated datasets such as JFT-300M,
which contains 300 million highly heterogeneous images, often containing
artifacts (e.g., watermarks). For instance, evaluating JFT-trained ViT-L
models using the top-1 accuracy of an ImageNet-trained linear probe, MAE
achieves $64.1\%$ and SimCLR achieves $73.4\%$, while CAN achieves $75.4\%$.
CAN masks $50\%$ of patches in each view, making it significantly more
scalable than prior contrastive methods that use two full image views. Our
contributions are:

  1. 1.

We present CAN, a simple self-supervised learning algorithm with good scaling
properties, making it suitable for training on very large image datasets, such
as the JFT-300M dataset.

  2. 2.

CAN is much more efficient than SimCLR (Figure 1). For instance, SimCLR uses
$70\%$ more FLOPs than CAN with ViT-L models.

  3. 3.

CAN is more robust to distribution shifts than MAE or SimCLR, and performs
better on a wide range of few-shot and linear transfer tasks.

##  2 Related Work

Masked image models with Vision Transformers. The advent of the Vision
Transformer (ViT) (Dosovitskiy et al., 2021b) provoked a focused effort to
develop strong self-supervised learning frameworks that use ViT backbones.
Works such as DINO (Caron et al., 2021) and MoCo-v3 (Chen et al., 2021b)
demonstrated that techniques developed with ConvNet backbones in mind could
also perform competitively using ViTs after proper tuning to suit the new
architecture. ViT-specific methods have emerged since then, particularly
masked image modelling (Bao et al., 2022; Chen et al., 2022; Xie et al.,
2022), which takes inspiration from pre-training methods used in NLP (Devlin
et al., 2018). Notably MAE (He et al., 2022) showed that classical masked
autoencoding approaches could be used to pre-train ViTs _without_ passing
masked tokens through the encoder. This provides a significant efficiency
boost; our method similarly takes advantage of this.

Contrastive learning in computer vision. Self-supervision has received
significant attention in computer vision as it offers a way to extract general
purpose features without supervision. In particular, contrastive learning (van
den Oord et al., 2018; Hénaff et al., 2020; Chen et al., 2020b; He et al.,
2020; Tian et al., 2020; Chuang et al., 2020; Hénaff et al., 2021) has
achieved state of the art performance by enforcing invariance to
augmentations, whilst using negative samples (Robinson et al., 2021a; Ge et
al., 2021) to avoid trivial solutions by spreading the embedding out uniformly
on the sphere (Wang & Isola, 2020). The contrastive pre-training task is
conceptually very different from masked image models such as MAE, which learn
spatial statistical dependencies. Another distinction is that autoencoders
encourage information preservation in latent representations, whilst
contrastive learning could suppress features (Chen et al., 2021a; Robinson et
al., 2021b). This leads us to hypothesize that the two approaches learn
different data features, and may therefore be complementary learning
mechanisms. This motivates us to combine contrastive learning and masked image
modelling so as to develop a reinforced pre-training task that enjoys the
merits of each.

Denoising diffusion models. Denoising autoencoders (DAE) (Vincent et al.,
2010) learn to reconstruct clean data given a noisy input. By learning to map
low-density data regions to high-density regions, DAE learns the shape of the
data manifold. This connection was made precise by Vincent (2011), who showed
that DAEs learn the score-function $s(\mathbf{x})=\nabla_{\mathbf{x}}\log
p(\mathbf{x})$. This key observation underpins the significant recent advances
in generative diffusion models, which use an estimate of the score-function to
generate samples (Ho et al., 2020; Song et al., 2021). The recent success of
DAEs in generative modelling has not yet translated to representation
learning, with some exceptions (Asiedu et al., 2022; Zaidi et al., 2022). In
this work we exploit a denoising autoencoder to eliminate the MAE inefficiency
of reconstructing unmasked patches but never using them.

Concurrent work. Several recent works propose approaches that combine ideas
from masked image modelling and Siamese self-supervised learning. For
instance, Huang et al. (2022) propose a combination of contrastive and masked
reconstruction objectives using one masked view, and one full (unmasked) view.
Other recent works (Tao et al., 2022; Chen et al., 2022; Assran et al., 2022)
use similar asymmetric designs. The key distinction between CAN and concurrent
work is that we strike a different balance between simplicity, efficiency, and
performance: we focus on developing a _simple_ , _efficient_ and symmetric
method: we use _two masked views_ and no momentum encoder. We hope the
simplicity and efficiency of CAN will make it easy to adapt and modify in
future work.

##  3 A simple contrastive masked autoencoder framework

Our approach is a minimal synthesis of contrastive learning, the masked
autoencoder (He et al., 2022), and the denoising loss used in the training of
diffusion models. We focus on simplicity and scalability, aiming to design a
hybrid with as few complex or costly components as possible. We also aim to
minimize _wasted_ computation: in particular, the MAE decoder requires
reconstructions of all patches, but only those of masked patches are used in
the loss. Below, first we detail the basic pipeline of generating views and
passing masked inputs through the encoder and decoder. Then we explain the
three different objectives we use: contrastive, reconstruction, and denoising.
The penultimate section describes the combined objective, and the final
section discusses scalability.

Figure 2: The CAN framework: Two views of an image are generated, $50\%$ of
patches randomly masked in each, and noise is added to patches. An encoder is
trained to solve three tasks: 1) Reconstruction: encoded patches are passed to
a decoder that reconstructs missing patches, 2) Denoise: reconstructs the
noise added to unmasked patches, and 3) Contrast: pooled patches are passed to
a contrastive loss, using in-batch samples as negatives (Chen et al., 2020b).

###  3.1 Overview of method

Given a batch of $n$ images $\\{\mathbf{x}\\}_{i=1}^{n}$, we generate two
views $\mathbf{x}_{i}^{1},\mathbf{x}_{i}^{2}\in\mathbb{R}^{h\times w\times 3}$
of each image without supervision using the same data augmentations as Chen et
al. (2020b). Each image is then split into $T=(h/p)\times(w/p)$ patches of
size $p\times p$:
$\mathbf{x}_{i,\text{patch}}^{1},\mathbf{x}_{i,\text{patch}}^{2}\in\mathbb{R}^{T\times
p\times p\times 3}$ in preparation for input to the ViT encoder. We always
assume that $p$ divides $h$ and $w$. Two masks
$\mathbf{M}_{i}^{1},\mathbf{M}_{i}^{2}\in\\{0,1\\}^{T}$ are independently
generated, with a $1$ in coordinate $t\in\\{1,\ldots T\\}$ indicating that the
$t$th patch is masked. Each patch is left unmasked independently with
probability $m$, conditioned on always having exactly $T^{\prime}=m\cdot T$
patches unmasked, which we assume is an integer. In experiments our default
masking rate is $m=50\%$ unless explicitly stated otherwise. Following He et
al. (2022), only the $T^{\prime}$ _unmasked_ patches are passed to the ViT
encoder, which processes the two views in parallel. Masking a large fraction
of patches from both views make our method much more efficient (see Table 1)
than contrastive methods that use two full views, and recent works that use
one full view and one masked view (Assran et al., 2022; Huang et al., 2022).
Finally, we collect the embeddings of unmasked tokens
$\mathbf{z}^{1}_{i},\mathbf{z}_{i}^{2}\in\mathbb{R}^{T^{\prime}\times d}$ and
reshape into $T\times d$ tensors by adding a learned [M] embedding to
positions corresponding to masked tokens. The result is passed through a
comparatively lightweight ViT decoder to produce outputs
$\hat{\mathbf{x}}^{1}_{i},\hat{\mathbf{x}}^{2}_{i}$ in image space
$\mathbb{R}^{h\times w\times 3}$.

###  3.2 Contrastive learning objective

The embeddings
$\mathbf{z}^{1}_{i},\mathbf{z}_{i}^{2}\in\mathbb{R}^{T^{\prime}\times d}$
returned by the encoder are pooled via a simple mean along the first dimension
to form $d$-dimensional embeddings, which are passed through a lightweight MLP
projection head that maps into a lower dimension space $\mathbb{R}^{r}$,
$r<d$, and normalized to unit length to produce embeddings
$\mathbf{u}^{1}_{i},\mathbf{u}_{i}^{2}\in\mathbb{R}^{r}$ for $i=1,\ldots n$.
For the $i$th batch item we collect the other $2n-2$ samples in-batch
$\mathcal{N}_{i}=\\{\mathbf{u}^{1}_{j},\mathbf{u}^{2}_{j}\\}_{j\neq i}$ to use
as negatives, and compute the InfoNCE loss:

$\mathcal{L}_{\text{InfoNCE}}=\frac{1}{2n}\sum_{v=1,2}\sum_{i=1}^{n}-\log\frac{e^{{\mathbf{u}^{1}_{i}}^{\top}\mathbf{u}^{2}_{i}/\tau}}{e^{{\mathbf{u}^{1}_{i}}^{\top}\mathbf{u}^{2}_{i}/\tau}+\sum_{\mathbf{u}^{-}\in\mathcal{N}_{i}}e^{{\mathbf{u}^{v}_{i}}^{\top}\mathbf{u}^{-}/\tau}}$

where $\tau>0$ is a temperature parameter, which we set to $\tau=0.1$ by
default.

###  3.3 Patch reconstruction objective

The outputs $\hat{\mathbf{x}}^{1}_{i},\hat{\mathbf{x}}^{2}_{i}$,
$i=1,\ldots,n$ of the ViT decoder are trained to reconstruct the missing
patches of each image. Corroborating the findings of He et al. (2022), we find
it best to only compute the reconstruction loss on masked patches:

$\mathcal{L}_{\text{rec}}=\frac{1}{2n}\sum_{v=1,2}\sum_{i=1}^{n}\|\mathbf{M}_{i}^{v}\circ(\mathbf{x}^{v}_{i}-\hat{\mathbf{x}}^{v}_{i})\|_{2}^{2}$

where $\circ$ multiplies all pixels in the $t$th patch of the residual image
$\mathbf{x}^{v}_{i}-\hat{\mathbf{x}}^{v}_{i}$ by
$(\mathbf{M}_{i}^{v})_{t}\in\\{0,1\\}$. Whilst computing the loss only on
masked patches gives better performance, it indicates wasted computation since
the decoder also produces reconstructions for unmasked patches. To avoid waste
we propose an alternative objective specifically for unmasked patches, which
we discuss next.

###  3.4 Denoising objective

Inspired by the significant advances in diffusion modelling using _denoising_
training objectives (Ho et al., 2020; Kingma et al., 2021) and their
equivalent score-based counterparts (Song et al., 2021; Vincent, 2011) we
revisit the suitability of denoising for self-supervised learning. We add
independent isotropic Gaussian noise to each image
$\mathbf{x}_{i}^{v}\leftarrow\mathbf{x}_{i}^{v}+\sigma_{i}^{v}\mathbf{e}^{v}_{i}$
with $\mathbf{e}^{v}_{i}\sim\mathcal{N}(\mathbf{0},I)$ and $\sigma_{i}^{v}$
uniformly sampled from an interval $[0,\sigma_{\text{max}}]$. This noisy input
is masked and passed to the encoder as described in Section 3.1. When passing
encoded patches to the decoder we make a small addition to the method in
Section 3.1 to provide the decoder with information on the noise level
$\sigma_{i}^{v}$ to help it separate noise from the ground truth image. This
is motivated by denoising diffusion methods, which pass both the noisy image
and the noise level as inputs to the denoising model (Ho et al., 2020). We
achieve this by using $\sigma_{i}^{v}$ as a positional encoding in the
decoder, similarly to Vaswani et al. (2017). First we produce a sinusoidal
embedding of $\sigma_{i}^{v}$, which is passed through a light MLP to produce
a (learnable) embedding $\mathbf{p}_{i}^{v}\in\mathbb{R}^{d}$, whose dimension
matches the latent dimension of $\mathbf{z}_{i}^{v}\in\mathbb{R}^{T\times d}$.
We add the result to each embedded token (including missing tokens [M]) to
provide noise-level information:
$(\mathbf{z}_{i}^{v})_{t}\leftarrow(\mathbf{z}_{i}^{v})_{t}+\mathbf{p}_{i}^{v}$
for $t=1\ldots,T$, and pass the result to the decoder producing
$\mathbf{x}^{v}_{i}$. We define our denoising loss function, which is computed
only on unmasked pixels:

$\mathcal{L}_{\text{denoise}}=\frac{1}{2n}\sum_{v=1,2}\sum_{i=1}^{n}\|(1-\mathbf{M}_{i}^{v})\circ(\sigma_{i}^{v}\mathbf{e}^{v}_{i}-\hat{\mathbf{x}}^{v}_{i})\|_{2}^{2}$

Figure 3: Denoising: Both the encoded patches and the noise level $\sigma$ are
passed to the decoder by passing $\sigma$ through an MLP, and adding the
result to each embedded token.

where, $\circ$ is as defined in Section 3.2. Note that this denoising loss is
extremely lightweight, introducing only a very small overhead due to the MLP.
We emphasize that the reconstruction of noise patches comes at zero additional
cost since the decoder produces reconstructions of all patches, both masked
and unmasked, even though only reconstructions of masked patches are used in
$\mathcal{L}_{\text{rec}}$. Finally, it has often been observed in the
diffusion modelling literature that although it is equivalent to train a
denoising model to estimate the noise, or to estimate the clean input itself
(Vincent, 2011), there is a big empirical gap between the two, with noise
prediction faring better. While we do not pursue it further, our testing
corroborates this.

Table 1: Ablating components of the denoising objective. “Full” denotes the entire method as described in Section 3.4 None | +noise | +noise, +loss | Full
---|---|---|---
67.9 | 68.6 | 68.4 | 68.9

Ablation: Table 1 studies the effect of each of the components of the
denoising method. We use ViT-B models trained for 100 epochs on ImageNet, and
consider four settings, each adding in more parts of the method: 1) CAN with
no denoising, 2) adding noise to the input only, 3) adding noise and using the
denoising loss, and 4) the full method with all of the described components,
including using $\sigma_{i}^{v}$ as a positional encoding in the decoder.
Results show that simply adding noise as a data augmentation improves
performance by $0.7\%$, which can be improved to $1\%$ by adding a
reconstruction loss with noise level passed as an argument. The noise level
argument is necessar: the reconstruction loss without noise level argument
performs worse ($68.4\%$) than noise with no reconstruction at all ($68.6\%$).

We emphasize that the improvement from denoising comes at minimal cost to run
time and memory during training, since it uses reconstructions produced by the
decoder, which in the case of MAE are simply thrown away unused. Denoising
prediction encourages the encoder to extract high-frequency features, which we
hypothesize is complementary to reconstruction and contrastive tasks.

###  3.5 The combined objective function

The overall CAN objective trains the encoder and decoder to optimize three
losses combined:

$\mathcal{L}_{\text{CAN}}=\lambda_{\text{InfoNCE}}\mathcal{L}_{\text{InfoNCE}}+\lambda_{\text{rec}}\mathcal{L}_{\text{rec}}+\lambda_{\text{denoise}}\mathcal{L}_{\text{denoise}}$

where
$0\leq\lambda_{\text{InfoNCE}},\lambda_{\text{rec}},\lambda_{\text{denoise}}$,
and $\lambda_{\text{InfoNCE}}+\lambda_{\text{rec}}+\lambda_{\text{denoise}}=1$
weight the objectives. In practice we parameterize the weights by eliminating
one variable using the equality constraint, taking:
$\lambda_{\text{rec}}=(1-\lambda_{\text{InfoNCE}})\cdot\lambda$ and
$\lambda_{\text{denoise}}=(1-\lambda_{\text{InfoNCE}})\cdot(1-\lambda)$ where
$0\leq\lambda\leq 1$. This parameterization makes it easy to control the
relative weighting between the two reconstruction losses
$\mathcal{L}_{\text{rec}},\mathcal{L}_{\text{denoise}}$ on the one hand, and
the contrastive loss $\mathcal{L}_{\text{InfoNCE}}$ on the other. Empirically
we find that performance is very robust to the choice of $\lambda$, and many
choices of $\lambda_{\text{InfoNCE}}$ also work well (see Section 5).

###  3.6 Discussion on Efficiency

The goal of this work is to propose a conceptually minimal combined
contrastive masked autoencoder approach, aiming to find better trade-offs
between simplicity, efficiency, and performance. Consequently, we choose to
omit a number of commonly used self-supervised learning design components. For
instance, we do not use a momentum target network or multiple views (multi-
crop), since they both increase memory requirements and run time. Even without
these commonly used components, our minimal framework achieves very strong
performance compared to prior work, and importantly improves performance over
its contrastive and autoencoder constituent parts. We expect that a wide range
of modifications, such as momentum target networks (He et al., 2020) and
multi-crop (Caron et al., 2020), will improve performance further on top of
the core method.

##  4 Results

###  4.1 Pre-training on uncurated data: JFT-300M

| Architecture | Epochs | IN-1K top-1
---|---|---|---
MoCLR (Tian et al., 2021) | R50 | 5000 | 67.6
BYOL (Grill et al., 2020) | R50 | 5000 | 67.9
DnC (Tian et al., 2021) | R50 | 1000 | 67.9
DnC (Tian et al., 2021) | R50 | 4500 | 70.7
MoCLR (Tian et al., 2021) | R200$\times$2 | 5000 | 74.2
DnC (Tian et al., 2021) | R200$\times$2 | 3000 | 77.3
MAE† (He et al., 2022) | ViT-L | 1600 | 50.5
MAE† (He et al., 2022) | ViT-L | 5000 | 64.1
SimCLR† (Chen et al., 2020b) | ViT-B | 800 | 65.8
SimCLR† (Chen et al., 2020b) | ViT-L | 800 | 72.6
SimCLR† (Chen et al., 2020b) | ViT-L | 1600 | 73.1
SimCLR† (Chen et al., 2020b) | ViT-L | 5000 | 73.4
CAN (ours) | ViT-B | 800 | 67.1
CAN (ours) | ViT-L | 800 | 72.8
CAN (ours) | ViT-L | 1600 | 74.3
CAN (ours) | ViT-L | 3000 | 75.3
CAN (ours) | ViT-L | 5000 | 75.4

Table 2: JFT-300M pre-training: Comparison to the state of the art on ImageNet
linear probe. CAN outperforms all methods except DnC, which uses a complicated
multi-stage training process. Computation is measured as ImageNet-equivalent
epochs. †Our implementation.

A key promise of self-supervised learning is to allow models to be trained on
extremely large scale image datasets collected from the Web. Not only is such
data likely to be _unannotated_ , but also _uncurated_ : images containing
many objects, variable lighting, artifacts (e.g., watermarks) and so on. The
large variation in images found online presents a major challenge to self-
supervised learning, and it is not guaranteed that methods that work well on
curated (and comparatively smaller) datasets such as ImageNet will work
equally well on less curated data. To study how CAN scales to large datasets
we use JFT-300M (Sun et al., 2017), a dataset of around 300 million images.

Setup. Training time is measured in ImageNet-equivalent epochs: 1 epoch equals
$1281167/[\text{batch size}]$ steps, the number of steps in one IN-1K epoch.
Models are evaluated using linear probe and finetuning on IN-1K. All
hyperparameers were tuned on IN-1K, besides learning rate and weight decay
which we cut by a factor of $4$ and $2$ respectively to stabilize training on
JFT-300M. See Appendix C and Section 5 for details.

Results. Figure 1 compares CAN to SimCLR and MAE baselines using ViT-L models.
CAN achieves a much better trade-off between efficiency (measured in FLOPs)
and performance using ViT-L models for all three methods: CAN uses $41\%$
fewer FLOPs than SimCLR and consistently outperforms SimCLR and MAE: for
training ViT-L models for 5000 epochs, CAN achieves an IN-1K linear probe
performance of $75.4\%$, compared to $71.8\%$ for SimCLR and $64.1\%$ for MAE.
The relatively poorer linear probe performance of MAE on JFT-300M highlights
the non-triviality of scaling from IN-1K to larger datasets and suggests that
while MAE is scalable for _model size_ , scalability to larger _datasets_
requires further study. Figure 4 gives the corresponding finetuning results.
CAN performs favourably: for a 5000 epoch pre-training schedule, CAN achieves
an IN-1K linear probe performance of $86.1\%$, compared to $85.5\%$ for SimCLR
and $85.4\%$ for MAE. CAN also enjoys better scaling with training schedule
length than either MAE or SimCLR.

Figure 4: CAN outperforms SimCLR and MAE on ImageNet finetuning evaluation for
ViT-L models.

We also compare CAN to the current state of the art on JFT-300M pre-training
in Table 2. Our best performance, $75.4\%$ with ViT-L outperforms all methods
besides DnC, with $77.3\%$ (Tian et al., 2021) with R200$\times$2\. However we
note that CAN is _considerably_ simpler than DnC, which involves multiple
training steps including training 10 separate “expert” models (each as large
as the final model), and then using MoCLR (an improvement of SimCLR that adds
a momentum encoder and more), and finally using distillation to produce a
single model. Our calculations suggest that training a ViT-L with CAN is about
3$\times$ faster than training the considerably smaller ResNet50 with DnC in
terms of wall clock time (see Appendix B for explanation). CAN on ViT-L
outperforms MoCLR with R200$\times$2 backbone (similar parameter counts),
where we note that MoCLR performs as well or better than BYOL and MoCo-v3 on
IN-1K (Tian et al., 2021).

###  4.2 Pre-training on ImageNet

Method | Pre-training epochs | Encoder | No Additional params. | Masked image | Finetune | Linear probe
---|---|---|---|---|---|---
_from scratch_ † | 100 | ViT-B | ✓ | ✗ | 79.1 | —
MoCo-v3 (Chen et al., 2021b) | 300 | ViT-B | ✗ | ✗ | 83.0 | 76.7
DINO (Caron et al., 2021) | 1600 | ViT-B | ✗ | ✗ | 82.8 | 78.2
CIM (Fang et al., 2022) | 300 | ViT-B | ✗ | ✗ | 83.1 | —
CAE (Chen et al., 2022) | 800 | ViT-B | ✗ | ✗ | 83.8 | 68.6
CAE (Chen et al., 2022) | 1600 | ViT-B | ✗ | ✗ | 83.9 | 70.4
BEiT (Bao et al., 2022) | 800 | ViT-B | ✗ | ✗ | 83.2 | 37.6∗
SimMIM (Xie et al., 2022) | 800 | ViT-B | ✓ | ✗ | 83.8 | 56.7
MAE (He et al., 2022) | 800 | ViT-B | ✓ | ✓ | 83.1 | —
MAE (He et al., 2022) | 1600 | ViT-B | ✓ | ✓ | 83.6 | 68.0
CAN (ours) | 800 | ViT-B | ✓ | ✓ | 83.4 | 74.0
CAN (ours) | 1600 | ViT-B | ✓ | ✓ | 83.6 | 74.8
SimCLR† (Chen et al., 2020b) | 800 | ViT-L | ✓ | ✗ | 83.4 | 73.9
MAE (He et al., 2022) | 800 | ViT-L | ✓ | ✓ | 84.9 | 73.5
MAE† (He et al., 2022) | 800 | ViT-L | ✓ | ✓ | 83.7 | 71.4
CAN (ours) | 800 | ViT-L | ✓ | ✓ | 84.7 | 76.2

Table 3: Pre-training on ImageNet-1K. †Our implementation. *Quoted from Chen
et al. (2022).

Next we evaluate our method using ImageNet (IN-1K) pre-training to verify that
it remains competitive in this setting. Results in Table 3 record the top-1
accuracy on IN-1K classification of finetuned models, and linear probes.
Finetuning CAN achieves $83.6\%$ with ViT-B, outperforming other contrastive
approaches such as MoCo-v3 ($83.0\%$), and is competitive with other state-of-
the-art approaches such as CAE ($83.9\%$). The linear probe performance of CAN
is $74.8\%$ using ViT-B, beating all masked image modelling methods, the best
of which is CAE with $70.4\%$ (Chen et al., 2022). CAN is only outperformed by
MoCo-v3 and DINO, both of which use momentum encoders and two full image
views, and in the case of DINO a further 10 multi-crop views. Note that the
_masked image_ column indicates whether a method uses one or more full image
views as input to the model, and the _no additional parameters_ column
indicates whether a method relies on other parameters besides the main
encoder, e.g., from a pre-trained tokenizer, or a momentum updated target
encoder. We also report results for our MAE implementation, which
approximately matches the original numbers reported by He et al. (2022),
validating our MAE results on JFT-300M.

###  4.3 Few-shot learning

We use linear probes to evaluate suitability of CAN for few-shot learning,
following the protocol of Dosovitskiy et al. (2021a). We use the models pre-
trained on JFT-300M for 5000 epochs whose ImageNet performance is recorded in
Figure 1. Results in Figure 5 for few-shot transfer learning on 9 other
datasets show that the superior performance on IN-1K translates to strong
performance on other tasks. We also note that our 25-shot ViT-L models beat
_full-shot_ both DnC and BYOL ResNet50 models (also trained for 5000 epochs on
JFT-300M) on 6 out of 8 datasets (Tian et al., 2021). See Appendix A for many
additional results for different training schedules and model sizes.

Figure 5: Few-shot: ViT-L models pre-trained on JFT-300M for 5000 epochs are
evaluated on 9 datasets in few-shot setting (10-shot and 25-shot). CAN
outperforms MAE and SimCLR.

###  4.4 Robustness to distribution shift

Finally, we consider the robustness of CAN to distribution shifts. We use
ViT-L backbones trained for 5000 epochs on JFT-300M, which have been finetuned
on IN-1K. Model performance is evaluated on a number of different validation
sets with the same 1000 classes as IN-1K Mao et al. (2022). Figure 6 reports
results on the following 7 validation sets, which cover a large variety of
distribution shifts: original IN-1K (Deng et al., 2009), IN-v2 (Recht et al.,
2019), IN-ReaL (Beyer et al., 2020), IN-Adversarial (Hendrycks et al., 2021b),
IN-Rendition (Hendrycks et al., 2021a), ObjectNet (Barbu et al., 2019). CAN
performs favourably under both JFT-300M and IN-1K pre-training, beating SimCLR
and MAE baselines in nearly all cases. See Appendix A for additional results.

Figure 6: Robustness: Evaluating performance under distribution shifts with
respect to models finetuned on IN-1K. Validation performance of ViT-L models
is reported on 7 different datasets.

##  5 Hyperparameter analysis

Figure 7: CAN and SimCLR with different masking rates. ViT-B models are pre-
trained for 100 epochs on IN-1K (left), and 800 epochs on JFT-300M (right).

Figure 8: ViT-B models pre-trained on IN-1K for 100 epochs. Left: The best
contrastive loss weight is small but non-negative. Middle: A wide range of
$\sigma_{\text{max}}$ values improve over no-noise. Right: Performance is not
sensitive to the denoising loss weight.

We study the different components of CAN to better understand the effect of
the different mechanisms, and to determine optimal parameter configurations.
All ablations use ViT-B models trained for 100 epochs on IN-1K, unless
explicitly said otherwise. We use the best loss weights and noise level in
these experiments for experiments in Section 4.

Complementarity of contrastive and reconstruction losses. A key hypothesis
motivating our work is that contrastive learning and masked autoencoder
reconstruction may not only be compatible training objectives, but are
_complementary_ ones. Table 4 compares the final training value of the
contrastive $\mathcal{L}_{\text{InfoNCE}}$ and reconstruction
$\mathcal{L}_{\text{rec}}$ when jointly trained (i.e., CAN) compared to only
optimizing $\mathcal{L}_{\text{InfoNCE}}$ (SimCLR) or only
$\mathcal{L}_{\text{rec}}$ (MAE). The results support the hypothesis: joint
training achieves a lower loss on _both_ objectives compared to individual
training.

Method | Contrastive loss $\downarrow$ | Reconstruction loss $\downarrow$
---|---|---
SimCLR | 9.157 | —
MAE | — | 0.1658
CAN (ours) | 9.143 | 0.1633

Table 4: Complementary training: All methods use $50\%$ masking for fair
comparison. CAN training achieves _lower_ training loss for both contrastive
and reconstruction than individual training.

Masking rate. Figure 7 reports the behavior of CAN and SimCLR under different
masking rates on IN-1K and JFT-300M pre-training (for JFT-300M we use 800
epochs). The performance of SimCLR decreases as the masking rate increases,
suggesting that masking is not an effective data augmentation. In contrast,
performance of CAN peaks at a non-zero masking rate, but at a much lower rate
than the $75\%$ used by MAE on IN-1K. This occurs since very low masking rates
are preferred by the contrastive part of CAN, but severely damage the
autoencoder part, which can learn trivial solutions. The considerable
efficiency improvement from masking $50\%$ of patches more than compensates
for the small drop in performance for a fixed number of epochs.

Contrastive loss weight. We vary the weighting $\lambda_{\text{InfoNCE}}$ used
to weight the contribution of the contrastive and reconstruction losses.
Recall that larger $\lambda_{\text{InfoNCE}}$ places higher weight on the
contrastive loss. Results in Figure 8 show that the best weight is
$\lambda_{\text{InfoNCE}}=0.03$, which approximately balances the magnitudes
of the two terms (see Table 4).

Denoising loss weight and noise level. We study the noise level interval
$[0,\sigma_{\text{max}}]$ from which to sample input noise, and the weight
$\lambda$ balancing the denoising and reconstruction losses. Results in Fig 8
show that the best maximum noise level is $\sigma_{\text{max}}=0.05$, and that
similar performance is attained for a number of different weights on the
denoising loss.

##  6 Discussion

We present CAN, a simple, efficient and scalable self-supervised method for
visual representation learning. CAN combines ideas from contrastive learning,
masked autoencoding, and diffusion denoising into a single high-performing
method. Extensive empirical results show that CAN scales with minimal changes
to the large uncurated datasets, providing a significant boost over SimCLR and
MAE methods on a wide range of downstream tasks and evaluations, including
linear probes, few-shot, robustness, and finetuning. Our results suggests that
contrasting and reconstruction are complementary principles that can mutually
reinforce one another.

## References

  * Asiedu et al. (2022) Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad Norouzi.  Decoder denoising pretraining for semantic segmentation.  _preprint arXiv:2205.11423_ , 2022.
  * Assran et al. (2022) Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas.  Masked siamese networks for label-efficient learning.  In _preprint arXiv:2204.07141_ , 2022.
  * Ballard (1987) Dana H Ballard.  Modular learning in neural networks.  In _Association for the Advancement of Artificial Intelligence (AAAI)_ , volume 647, pp. 279–284, 1987.
  * Bao et al. (2022) Hangbo Bao, Li Dong, and Furu Wei.  BEiT: BERT pre-training of image transformers.  In _Int. Conf. on Learning Representations (ICLR)_ , 2022.
  * Barbu et al. (2019) Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32, 2019.
  * Beyer et al. (2020) Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord.  Are we done with ImageNet?  In _preprint arXiv:2006.07159_ , 2020.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pp. 9912–9924, 2020.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Int. Conference on Computer Vision (ICCV)_ , pp. 9650–9660, 2021.
  * Chen et al. (2020a) Mark Chen, Alec Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _Int. Conference on Machine Learning (ICML)_ , 2020a.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _Int. Conference on Machine Learning (ICML)_ , pp. 1597–1607. PMLR, 2020b.
  * Chen et al. (2021a) Ting Chen, Calvin Luo, and Lala Li.  Intriguing properties of contrastive losses.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pp. 11834–11845, 2021a.
  * Chen et al. (2022) Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang.  Context autoencoder for self-supervised representation learning.  In _preprint arXiv:2202.03026_ , 2022.
  * Chen et al. (2021b) Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In _Int. Conference on Computer Vision (ICCV)_ , pp. 9640–9649, 2021b.
  * Chuang et al. (2020) Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka.  Debiased contrastive learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pp. 8765–8775, 2020.
  * Cole et al. (2022) Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie.  When does contrastive visual representation learning work?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14755–14764, 2022.
  * Cubuk et al. (2020) Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  RandAugment: Practical automated data augmentation with a reduced search space.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 3008–3017, 2020.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 248–255. Ieee, 2009.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2018.
  * Dosovitskiy et al. (2021a) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Int. Conf. on Learning Representations (ICLR)_ , 2021a.
  * Dosovitskiy et al. (2021b) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Int. Conf. on Learning Representations (ICLR)_ , 2021b.
  * Fang et al. (2022) Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei.  Corrupted image modeling for self-supervised visual pre-training.  _preprint arXiv:2202.03382_ , 2022.
  * Ge et al. (2021) Songwei Ge, Shlok Kumar Mishra, Haohan Wang, Chun-Liang Li, and David Jacobs.  Robust contrastive learning using negative samples with diminished semantics.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume abs/2110.14189, 2021.
  * Goyal et al. (2017) Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch SGD: Training ImageNet in 1 hour.  _preprint arXiv:1706.0267_ , 2017.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 9729–9738, 2020.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 16000–16009, June 2022.
  * Hénaff et al. (2020) Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Eslami S. M. Ali Doersch, Carl Doersch, and Aäron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  In _Int. Conference on Machine Learning (ICML)_ , pp. 4182–4192, 2020.
  * Hénaff et al. (2021) Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aäron Van den Oord, Oriol Vinyals, and João Carreira.  Efficient visual pretraining with contrastive detection.  In _Int. Conference on Computer Vision (ICCV)_ , 2021.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Int. Conference on Computer Vision (ICCV)_ , pp. 8340–8349, 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 15262–15271, 2021b.
  * Herrmann et al. (2022) Charles Herrmann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krishnan, and Deqing Sun.  Pyramid adversarial training improves ViT performance.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 13419–13429, 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pp. 6840–6851, 2020.
  * Hou et al. (2017) Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu.  Deep feature consistent variational autoencoder.  In _2017 IEEE winter conference on applications of computer vision (WACV)_ , pp. 1133–1141. IEEE, 2017.
  * Huang et al. (2022) Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng.  Contrastive masked autoencoders are stronger vision learners.  _arXiv:2207.13532v1_ , 2022.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pp. 4904–4916. PMLR, 2021.
  * Kingma et al. (2021) Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pp. 21696–21707, 2021.
  * Kolesnikov et al. (2020) Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In _ECCV_ , 2020.
  * Loshchilov & Hutter (2017a) Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in Adam.  _preprint arXiv:1711.05101_ , 2017a.
  * Loshchilov & Hutter (2017b) Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic gradient descent with warm restarts.  In _Int. Conf. on Learning Representations (ICLR)_ , 2017b.
  * Mao et al. (2022) Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa.  Discrete representations strengthen vision transformer robustness.  In _Int. Conf. on Learning Representations (ICLR)_ , 2022.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do ImageNet classifiers generalize to ImageNet?  In _Int. Conference on Machine Learning (ICML)_ , pp. 5389–5400. PMLR, 2019.
  * Robinson et al. (2021a) Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka.  Contrastive learning with hard negative samples.  In _Int. Conf. on Learning Representations (ICLR)_ , 2021a.
  * Robinson et al. (2021b) Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra.  Can contrastive learning avoid shortcut solutions?  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pp. 4974–4986, 2021b.
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _Int. Conf. on Learning Representations (ICLR)_ , 2021.
  * Steiner et al. (2021) Andreas Steiner, Alexander Kolesnikov, , Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your ViT? Data, augmentation, and regularization in vision transformers.  In _Transactions on Machine Learning Research (TMLR)_ , 2021.
  * Sun et al. (2017) Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In _Int. Conference on Computer Vision (ICCV)_ , pp. 843–852, 2017\.
  * Tao et al. (2022) Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai.  Siamese image modeling for self-supervised vision representation learning.  In _preprint arXiv:2206.01204_ , 2022.
  * Tian et al. (2020) Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In _Europ. Conference on Computer Vision (ECCV)_ , pp. 776–794, 2020.
  * Tian et al. (2021) Yonglong Tian, Olivier J Henaff, and Aäron van den Oord.  Divide and contrast: Self-supervised learning from uncurated data.  In _Int. Conference on Computer Vision (ICCV)_ , pp. 10063–10074, 2021.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _preprint arXiv:1807.03748_ , 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Vincent (2011) Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7):1661–1674, 2011\.
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal of machine learning research_ , 11(12), 2010.
  * Wang & Isola (2020) Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _Int. Conference on Machine Learning (ICML)_ , pp. 9929–9939. PMLR, 2020.
  * Xie et al. (2022) Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 9653–9663, 2022.
  * You et al. (2017) Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks.  In _ECCV_ , 2017.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  CoCa: Contrastive captioners are image-text foundation models.  _preprint arXiv:2205.01917_ , 2022.
  * Zaidi et al. (2022) Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin.  Pre-training via denoising for molecular property prediction.  _preprint arXiv:2206.00133_ , 2022.
  * Zhong et al. (2020) Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In _Association for the Advancement of Artificial Intelligence (AAAI)_ , 2020.

##  Appendix A Additional Transfer Learning Results

We report additional results for few-shot learning and robustness.

Robustness: Sections 4.4 reports robustness results for ViT-L models pre-
trained on JFT-300M for 5000 epochs, and ViT-L models pre-trained on IN-1K for
800 epochs. In both cases we report the performance of the models after
finetuning on IN-1K.

Here we report the same robustness results for ViT-L models trained on
JFT-300M for 1600 and 800 epochs (Figure 9), and ViT-B models pre-trained for
800 epochs (Figure 10). Figure 10 also compares our ViT-B model to ViT-B
models trained from scratch on ImageNet. We find that our model is
considerably more robust than training with cross-entropy and Mixup from
scratch, and also outperforms PyramidAT (Herrmann et al., 2022), an
adversarial training method that introduces significant overheads compared to
standard cross-entropy training. We emphasize that here there are two
differences in the training: a) the training algorithm itself, and b) the data
seen by the model. Our model sees extra JFT-300M data not seen by the other
two approaches. This means that the methods are not exactly comparable. It is,
however, a realistic setting showing the benefits to robustness of pre-
training on large datasets.

Figure 9: ViT-L models pre-trained on JFT-300M for 800 and 1600 epochs
respectively, evaluated on 7 datasets with distribution shifts from IN-1K.

Figure 10: Top: ViT-B models pre-trained on JFT-300M for 800 epochs, evaluated
on 7 datasets with distribution shifts from IN-1K. Bottom: Comparison of our
JFT-300M pre-trained ViT-B model to training ViT-B from scratch on IN-1K. We
compare to standard supervised cross-entropy training with Mixup, and to
PyramidAT (Herrmann et al., 2022), which uses an adversarial training method.
CAN considerably outperforms supervised training, and beats PyramidAT in 6 out
of 7 cases without requiring adversarial training.

Few shot: Section 4.3 reports $10$\- and $25$-shot results for ViT-L models
pre-trained on JFT-300M for $5000$ epochs. Here we report $1$\- and $5$-shot
results for the same models in Figure 11. We additionally show the full set of
$\\{1,5,10,25\\}$-shot results for ViT-L models pre-trained on JFT-300M for
800 and 1600 epochs (Figures 12 and 13 respectively), ViT-B models pre-trained
on JFT-300M for 800 epochs (Figure 14), and ViT-L models pre-trained on IN-1K
for 800 epochs (Figure 15).

Figure 11: Few shot: ViT-L models pre-trained on JFT-300M for 5000 epochs
evaluated on 9 few-shot learning tasks. Results accompany the 10- and 25-shot
results in Figure 5.

Figure 12: Few shot: ViT-L models pre-trained on JFT-300M for 800 epochs are
evaluated on 9 few-shot learning tasks.

Figure 13: Few shot: ViT-L models pre-trained on JFT-300M for 1600 epochs are
evaluated on 9 few-shot learning tasks.

Figure 14: Few shot: ViT-B models pre-trained on JFT-300M for 800 epochs are
evaluated on 9 few-shot learning tasks.

Figure 15: Few shot: ViT-L models pre-trained on IN-1K for 800 epochs are
evaluated on 9 few-shot learning tasks.

We make a number of observations.

  1. 1.

Across all settings CAN generally performs the best on JFT-300M pre-training.

  2. 2.

The situation is less consistent on IN-1K pre-training. For instance, although
MAE has comparatively poor few-shot performance on IN-1K, it is competitive on
others for 25-shot evaluation: in this setting CAN only beats MAE on 5 out of
9 datasets. However, on 10-shot CAN outperforms MAE and SimCLR in 8 out of 9
cases, showing a subtle picture.

  3. 3.

JFT-300M pre-training often outperforms IN-1K pre-training. Comparing Figures
12 and 15, JFT-300M yields better 25-shot CAN performance on 6 out of 9
datasets.

  4. 4.

Model scale helps. Comparing Figures 14 and 12, ViT-L models perform best in
nearly all cases.

##  Appendix B Runtime of CAN compared to DnC

In the main paper we estimate our method is significantly faster than DnC
(Tian et al., 2021). We determined this approximate comparison from the
following two pieces of information: 1) DnC reports that 3000 ImageNet epochs
takes 29 hours on 512 TPUs for a ResNet-50 model ( 25M parameters), and 2)
3000 ImageNet epochs of CAN take 78 hours on 64 TPUs for a ViT-L model ( 300M
parameters). We assume a linear relationship between number of TPUs and
runtime. Under this assumption, we estimate that CAN would take approximately
10 hours to train with 512 TPUs, compared to the 29 hours reported by Tian et
al. (2021) for a model with 1/10th the number of parameters. We emphasize that
this is far from an exact comparison and is only intended as a very
approximate guide.

##  Appendix C Hyperparameter settings

We list hyperparameters used for CAN pre-training in Table 5 and Table 6. For
preprocessing we closely follow SimCLR Chen et al. (2020b). We use the same
hyperparameters for SimCLR pre-training. For MAE pre-training, we use the same
hyperparameters as listed in He et al. (2022), except for the use of Glorot
uniform initialization instead of LeCun initialization as done in He et al.
(2022). We found that this provided better performance for our JAX-based MAE
implementation. Table 9 lists the hyperparameters for finetuning evaluations.
We use the same set of hyperparameters for each finetuning each pre-training
method, and for both ViT-B and ViT-L model sizes. For linear probing we list
the hyperparameters in Table 10 for which we followed the settings in He et
al. (2022). We use global average pool of the final representation instead of
the cls token.

#### MAE longer training:

MAE pre-training for longer training (5000 epochs) on JFT becomes unstable
after about 500k steps (training loss oscillates); this results in poorer
fine-tuning performance. To overcome this, we decrease the base learning rate
by $75\%$ as shown in Table 8. However our model CAN is more stable and we use
the same hyperparameters across different numbers of epochs.

#### Few shot training:

For few-shot learning we use the same hyperparameters and pipeline as
Dosovitskiy et al. (2021a). We use the same pre-processing as was done in
(Kolesnikov et al., 2020). We use a base learning rate of 0.01 and train for
2500 steps, using an input resolution of $384\times 384$.

#### Hardware details:

We use TPU-v4 for all of our experiments. CAN on ViT-B uses 64 TPUs for a
batch size of 4096. SimCLR, on the other hand, uses 128 TPUs for the same
batch size, and is more compute intensive than CAN.

#### Decoder architecture:

Our decoder architecture is the same as He et al. (2022). We use standard ViT
with a decoder depth of 8 and decoder width of 512. We use 16 heads and 2048
as the dimension of the MLP.

#### Projection head architecture:

We use 2 hidden layers in our projection heads. Each layer has a Fully-
Connected (FC) layer (dim 4096) followed by BatchNorm (momentum=0.9) followed
by ReLU. After these 2 layers we have a FC layer which transforms the features
to 128 dimensions. We apply contrastive learning on top of these 128
dimensional features.

#### JFT-300M specific hyperparameters:

All hyperparameters were determined by training on IN-1K, and directly
transferred with JFT-300M pre-training, with the exception of learning rate
and weight decay, which found needed to be at a lower level for JFT-300M. For
all methods we divided the learning rate by a factor of $4$, and the weight
decay by a factor of $2$, except for MAE where we found that the original
weight decay tuned on ImageNet worked better. Specifically, for CAN and SimCLR
we used following parameter choices: $wd=0.1/2=0.05$ and $lr=1.25\times
10^{-4}/4=3.125\times 10^{-5}$ and for MAE we used $lr=1.5\times
10^{-4}/4=3.75\times 10^{-5}$ , and tried $wd=0.05/2=0.025$, but found that
the original $wd=0.05$ worked better, so kept this value.

###  C.1 CAN and SimCLR hyperparameters

config |  value
---|---
optimizer |  AdamW (Loshchilov & Hutter, 2017a)
base learning rate(ViT-B) |  2.5e-4
base learning rate (ViT-L) |  1.25e-4
weight decay (ViT-B) |  0.05
weight decay (ViT-L) |  0.1
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.95$ (Chen et al., 2020a)
batch size |  4096
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  40
augmentation |  RandomResizedCrop, Color Jittering(strength=1.0), GrayScale(probability=0.2), Gaussian Blurring (probability=0.5)
Table 5: Hyperparameters for CAN pre-training on ImageNet. Note that we use lower learning rate for ViT-L as compared to ViT-B, following Steiner et al. (2021). We use the same hyper-parameters for SimCLR training. config |  value
---|---
optimizer |  AdamW (Loshchilov & Hutter, 2017a)
base learning rate (ViT-B) |  2.5e-4
base learning rate (ViT-L) |  3.125e-5
weight decay |  0.05
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.95$ (Chen et al., 2020a)
batch size |  4096
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  40
augmentation |  RandomResizedCrop, Color Jittering(strength=1.0), GrayScale(probability=0.2), Gaussian Blurring (probability=0.5)
Table 6: Hyperparameters for CAN pre-training on JFT-300M. Note that we use
lower learning rate for ViT-L as compared to ViT-B, following Steiner et al.
(2021). We use the same hyper-parameters for SimCLR pre-training.

###  C.2 MAE hyperparameters

config |  value
---|---
optimizer |  AdamW (Loshchilov & Hutter, 2017a)
base learning rate (ViT-L) |  1.5e-4
weight decay |  0.05
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.95$ (Chen et al., 2020a)
batch size |  4096
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  40
augmentation |  RandomResizedCrop, Color Jittering(strength=1.0), GrayScale(probability=0.2), Gaussian Blurring (probability=0.5)
Table 7: Hyperparameters for MAE pre-training on IN-1K. We follow the choices made by He et al. (2022). config |  value
---|---
optimizer |  AdamW (Loshchilov & Hutter, 2017a)
base learning rate (ViT-L) |  3.75e-5
weight decay |  0.05
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.95$ (Chen et al., 2020a)
batch size |  4096
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  40
augmentation |  RandomResizedCrop, Color Jittering(strength=1.0), GrayScale(probability=0.2), Gaussian Blurring (probability=0.5)
Table 8: Hyperparameters for MAE pre-training on JFT-300M with ViT-L models.
The only difference from the IN-1K configuration is the learning rate, which
we reduced since we found tarining to be unstable.

###  C.3 Funetuning and lienar probe hyperparameters

config |  value
---|---
optimizer |  AdamW (Loshchilov & Hutter, 2017a)
base learning rate |  5e-4
weight decay |  0.005
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.999$ (Chen et al., 2020a)
batch size |  1024
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  5
training epochs |  100
label smoothing |  0.1
drop path |  0.1
layer-wise lr decay |  0.65
augmentation |  RandomResizedCrop, Flip, RandAug(layers=2, magnitude=9) (Cubuk et al., 2020), Random Erase (Zhong et al., 2020)(probability=0.25)
Table 9: Hyperparameters for finetuning CAN pre-trained model on ImageNet. We use the same hyperparameters for ViT-B and ViT-L, for both JFT-300M and ImageNet pre-trainined models. config |  value
---|---
optimizer |  LARS (You et al., 2017)
base learning rate |  0.1
weight decay |  0
optimizer momentum |  0.9
batch size |  16384
learning rate schedule |  cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017) |  10
training epochs |  100
batch norm momentum |  0.9
label smoothing |  0
augmentation |  RandomResizedCrop
Table 10: Hyperparameters for linear probing for CAN pre-trained model on
ImageNet. We use the same hyperparameters for ViT-B and ViT-L, for both
JFT-300M and ImageNet pre-trainined models. Note that these hyperparamters are
same as reported in He et al. (2022).


# Entropic Timescales of Dynamic Heterogeneity in Supercooled Liquid

Vinay Vaibhav<EMAIL_ADDRESS>The Institute of Mathematical Sciences,
CIT Campus, Taramani, Chennai 600113, India  Department of Physics “A.
Pontremoli”, University of Milan, Via Celoria 16, 20133 Milan, Italy  Suman
Dutta<EMAIL_ADDRESS>(Corresponding Author) International Centre for
Theoretical Sciences - Tata Institute of Fundamental Research
Survey No 51, Hessaraghatta Hobli, Shivakote, Bangalore 560089, India

###### Abstract

Non-Gaussian displacement distributions are universal predictors of dynamic
heterogeneity in slowly varying environments. Here, we explore heterogeneous
dynamics in supercooled liquid using molecular dynamics simulations and show
the efficiency of the information-theoretic measure in quantifying dynamic
heterogeneity over the widely used moment-based quantifications of non-
Gaussianity. Our analysis shows that the heterogeneity quantified by the
negentropy is significantly different from the one obtained using the
conventional approach that considers deviation from Gaussianity up to lower-
order moments. Further, we extract the timescales of dynamic heterogeneity
using the two methods and show that the differential changes diverge as the
system experiences strong intermittency near the glass transition.

Introduction.— Fickian theory of diffusion has been unquestionably successful
for more than a century in analyzing particle-level dynamics in soft condensed
matter that appear in different forms and shapes unless the system is
intermittent or has widely separating timescales glotzer1997 ; edigerRev ;
blaaderen2000 ; weitz ; gao ; richertRev ; eric0 ; glotzer2003 ; andersen2005
; shell2005 ; szamel2010 ; garrahan2011 ; kirsten ; berthier2011 ;
berthier_book2011 ; berthierBiroliRev2011 ; manning2013 ; sastryChandanRev ;
szamel2014 ; ploymerHetro2017 ; karmakar2018 ; karmakar2020 ; saroj2023 . In
metastable systems, the fundamental route to diffusion becomes difficult in
the presence of complex energy landscapes, specifically, when a small
magnitude of thermal fluctuation is not enough to supply the energy cost of
achieving diffusion, overcoming the energy barriers berthier_book2011 ; heuer
. For instance, molecular displacements deviate from their usual Gaussian form
blaaderen2000 ; gao  in liquids approaching glass transition pinaki2007 ;
pinaki2008 , showing slow heterogeneous density relaxation ngcage . Such
dramatic slowing down observed generically in a host of systems without any
reproducible thermodynamic transition has remained a surprise even after
decades of research edigerRev ; berthierBiroliRev2011 ; sastryChandanRev .
Dynamic heterogeneity is the observed complex dynamics of particles in such
temporally fluctuating environments in the presence of spatial degrees of
heterogeneity, where both locally fast and slow relaxation processes coexist
simultaneously chandan2010 ; berthier_book2011 ; eric ; karmakar2018 ;
dutta2019 .

Dynamic heterogeneity in the supercooled liquid has been affirmed with
persisting non-Gaussian tails in the displacement distributions, even when the
mean-squared displacement linearly increases with time granick2009 ;
granick2012 ; berthier2022 . Such class of non-Gaussian diffusion has been
explained as an effective dynamics in the presence of a diffusion spectrum
where the dynamics is strongly influenced by the presence of cages and
diffusion is only restored upon cage-breaking, making it distinct from the
Fickian class of liquids granick2009 ; granick2012 . The persistence of
dynamic heterogeneity is, thus, debated for the presence of multiple
timescales of relaxation and various ways of their determination. It has also
remained indecisive whether the onset of diffusion at all occurs in the finite
time when a liquid approaches its glass transition berthier2011 ; burov2020 ;
berthier2022 ; pastore .

Extracting the fundamental timescale of dynamic heterogeneity directly from
the displacement distributions or the self-Van Hove function is advantageous
than obtaining it by other quantities which are either related to its moments
or derivatives. Using the displacement distributions, it is possible to
identify timescales of heterogeneity by finding the maximal non-Gaussianity
using conventional measures that rely on moment-based relationships kobBook .
However, one hindrance of this method is that such moment ratios are primarily
limited to lower-order moments, which raises natural questions: How optimal
are the moment-based predictions of dynamic heterogeneity or its detection by
the conventional techniques? This calls for newer directions in the precise
identification of non-Gaussianity by informative approaches with the data-
driven resources, like the information-theory-based optimizations simplify
challenging multi-scale and inverse problems in bio-informatics shell2016 ;
barbier  or the prediction of dynamics and structures using machine
intelligence is unveiling newer avenues in physics andrea2015 ; deepmind2020 ;
berthier2023 .

Figure 1: [(a)-(b)] Temporal evolution of the Non Gaussian displacement
distributions, $G_{S}(x,t)$ vs. $x$ for two different $T$, (a) $T=0.70$ (b)
$T=0.45$ for different $t$: $t_{1}=928.98$, $t_{2}=9527.62$, $t_{3}=39913.85$,
$t_{4}=68300.79$, $t_{5}=139796.16$ with $x\in[X,Y,Z]$. Long time behavior of
$G_{S}(x,t)$ at large $x$ is Gaussian for $T=0.70$ and exponential for
$T=0.45$. [(c)-(n)] Temporal evolution of the displacement distributions,
$G_{S}(x,t)$ (open symbols) and respective reconstructed equal-time nearest
Gaussians, $G^{G}_{S}(x,t)$ (black solid line) vs. $x$ for $T=0.70$ [(c)-(h)]
and $T=0.45$ [(i)-(n)] for $t=$ $0.60$ [(c),(i)], $21.62$ [(d),(j)], $129.59$
[(e),(k)], $27899.00$ [(f),(l)], $47740.94$ [(g),(m)] and $167210.14$
[(h),(n)]. $G^{G}_{S}(x,t)$ are the optimal Gaussians with identical first two
moments same as $G_{S}(x,t)$.

Here, we explore dynamic heterogeneity in the supercooled liquid above the
glass transition using molecular dynamics simulations in three dimensions. We
examine the spatio-temporal dynamics in terms of the evolving probability
distribution functions of the displacements at single particle-level which are
strongly heterogenous and non-Gaussian till it diffuses at sufficiently long
time. We quantify temporal intermittency in terms of non-Gaussianity extracted
from the molecular displacements using the conventional moment-based
descriptions and a relative entropy based non-Gaussian information that
considers the statistical distance between the time-dependent displacement
distribution and its equal-time nearest Gaussian trained from the original
probability distribution functions. We extract and compare the identified
timescales of optimal heterogeneity obtained from the two methods and show
that they surprisingly differ in estimating microscopic heterogeneity, in
particular when the situation is strongly intermittent at low temperatures.
Further, we show that such difference diverge when approaching the glass
transition while the timescales are similar at relatively high temperatures
within the supercooled regime. We correlate the two quantities and interpret
the deviation.

Model.— We simulate a well studied model glass-forming system, popularly known
as the Kob-Anderson 80:20 (A:B) binary mixture kob1995testing , in three
dimensions at different temperatures $T$ within the supercooled regime. The
particles (each with unit mass) interact via the Lennard-Jones (LJ) pair-
potential with a cutoff at a distance $R_{c}$. Here, we use $N=1000$ particles
at a constant number density ($\rho=1.20$) in a cubic box of length $L=9.41$
and $R_{c}=2.5$. All measurements are done in the LJ reduced unit (see Ref.
vaibhav2020  for model related information). Independent molecular dynamics
trajectories are generated using LAMMPS lammps  under periodic boundary
conditions with integration time step $\Delta t=0.004$ at each of the
different $T\in[0.45,0.70]$, above the mode-coupling temperature
$T_{MCT}\approx 0.435$ of the mixture kob1995testing .

Results.— We track the particle trajectories and compute time-dependent
probability distribution functions of particle displacements (also known as
self-van Hove functions) which have been mapped in one dimension, $x$ in time
interval, $t$,
$G_{s}(x,t)=\frac{1}{N}\sum_{i=1}^{N}\delta(x-(x_{i}(t)-x_{i}(0)))$ vHove  by
averaging over displacement distributions in all spatial dimensions obtained
in different independent trajectories.

We show the time evolution of $G_{S}(x,t)$ for different $t$ in Fig. 1 for two
different values of $T$. At finite temperatures, $G_{S}(x,t)$ spreads with
increasing $t$ due to diffusion which we see here. For $T=0.70$, it develops
non-Gaussian tail for intermediate times that reverts back to Gaussian at long
times, suggesting that the dynamics is intermittent and heterogeneous even at
relatively high temperatures, within the supercooled regime [Fig. 1(a)]. For
$T=0.45$, the heterogeneity is not only stronger, but also its lifetime is
prolonged as the non-Gaussian tail ($G_{S}(x,t)\sim\exp(-x/\lambda(t))$ with
$\lambda(t)\sim\sqrt{t}$) persists for very long time [Fig. 1(b)]. Such
persistent exponential tails have been earlier reported in both simulations
shell2005 ; pinaki2007  and experiments blaaderen2000 ; berthier_book2011 ;
granick2012 .

In order to assess the degree of temporal heterogeneity in microscopic
dynamics, we show the temporal evolution of $G_{S}(x,t)$ for various $t$ for
two different $T$ in Figs. 1(c)-1(n) and compare it with the reconstructed
equal-time nearest Gaussian distributions (see Ref. sumanEPL2020 ),
$G^{G}_{S}(x,t)$ that has first two moments same as $G_{S}(x,t)$. This ensures
that for a purely Gaussian distribution, $G^{G}_{S}(x,t)\rightarrow
G_{S}(x,t)$ and $G^{G}_{S}(x,t)\neq G_{S}(x,t)$ when it is strictly non-
Gaussian. We observe $G_{S}(x,t)\approx\delta(x)$ for $t=0$, but the
calculation of $G^{G}_{S}(x,t)$ will become more meaningful when $G_{S}(x,t)$
has some finite support at finite $t$. For $T=0.70$, we observe that for very
small $t$, the difference between $G_{S}(x,t)$ and $G^{G}_{S}(x,t)$ is not
very much significant [Fig. 1(c)]. The difference enhances with increasing $t$
[Figs. 1(d), 1(e)] where $G_{S}(x,t)$ is prominently non-Gaussian while for
larger $t$, again $G_{S}(x,t)$ reverts back to Gaussian form with
$G^{G}_{S}(x,t)\approx G_{S}(x,t)$ [Figs. 1(f)-1(h)], suggesting the onset of
Fickian diffusion that is characterized by the presence of Gaussian
displacement distribution and linear mean squared displacement granick2012 .

Figure 2: Quantification of dynamic heterogeneity in terms of non-Gaussianity
of $G_{S}(x,t)$ via (a) $\alpha_{2}$ and (b) $\Delta S_{ng}$ vs. $t$ for
different $T$ as marked.

The situation is surprisingly different for $T=0.45$ which we show in Figs.
1(i)-1(n) and compare it with respective $G^{G}_{S}(x,t)$, as earlier. For
small $t$, the distributions are highly spiked, centered at $x=0$. Within the
smaller intervals, $G_{S}(x,t)\approx G^{G}_{S}(x,t)$ [Fig. 1(i)] and smaller
deviations from delta functions are mostly contributed by the lower-order
moments. For larger $t$, as $G_{S}(x,t)$ broadens in $x$, more deviation could
be seen from $G^{G}_{S}(x,t)$ in the form of exponential tail that grows with
a significantly large proportion of displacements appearing beyond the support
of the equal time nearest Gaussians [Figs. 1(j), 1(k)], suggesting that the
heterogeneity is maximally contributed by the higher-order moments. With
increasing $t$ [Figs. 1(l)-1(n)], the difference between $G_{S}(x,t)$ and
$G^{G}_{S}(x,t)$ decreases yet it remains non-Gaussian within our observation
time-window.

Now we attempt to quantify the dynamic heterogeneity in terms of non-
Gaussianity in $G_{S}(x,t)$ using conventional approaches that use lower order
moment-based relationships. One such very simple quantification of non-
Gaussianity that is widely available in the literature is based on the
deviation of kurtosis from the square of second moment, defined as
$\alpha_{2}(t)=\frac{\langle x^{4}(t)\rangle}{3\langle x^{2}(t)\rangle^{2}}-1$
with $<x^{n}(t)>=\int dx\ x^{n}G_{S}(x,t)$, as in Ref. kob1995testing ;
glotzer1997 ; horbach1998 ; blaaderen2000 . We show the dependence of
$\alpha_{2}$ with $t$ for different $T$ in Fig. 2(a). For $T=0.70$,
$\alpha_{2}$ grows with increasing $t$ till a peak is observed. After that,
$\alpha_{2}$ decreases for larger $t$. For smaller temperatures, the peak in
$\alpha_{2}$ shifts to larger $t$ and the height of the peak grows. In all
these cases, $\alpha_{2}$ decreases monotonically after the peak. The
timescale corresponding to the peak is generally identified as the
characteristic timescale of dynamic heterogeneity. However, such
quantification is fundamentally limited to fourth-order moments of
$G_{S}(x,t)$. Figs. 1(c)-1(n) suggest that the higher order moments may be
associated with the larger degrees of heterogeneity for which we explore
another quantification of non-Gaussianity that captures contributions of all
order moments of $G_{S}(x,t)$ at a given time.

Non-Gaussianity in $G_{S}(x,t)$ is now analysed using non-Gaussian information
sumanEPL2020  that we developed following Ref. simon2012 , originally proposed
as Negentropy. It uses the statistical distance between $G_{S}(x,t)$ and
$G^{G}_{S}(x,t)$ to quantify $\Delta
s_{ng}(t)=A~{}D_{KL}(G_{S}(x,t)||G^{G}_{S}(x,t))$ where $A$ is a constant. We
further define, $\Delta S_{ng}(t)(={\Delta s_{ng}(t)}/{A})=-\int
dx~{}G_{S}(x,t)\log_{e}\frac{G^{G}_{S}(x,t)}{G_{S}(x,t)}=S^{G}_{ng}-S_{ng}$
when $G_{S}(x,t)$ has some finite support. Here,
$S_{ng}(t)=-\int~{}dx~{}G_{S}(x,t)\log_{e}G_{S}(x,t)$ and
$S^{G}_{ng}(t)=-\int~{}dx~{}G^{G}_{S}(x,t)\log_{e}G^{G}_{S}(x,t)$ and
$D_{KL}(P||Q)$ is the Kullback-Leibler (KL) divergence kl1951  between two
probability distribution functions, $P(x)$ and $Q(x)$. We show the time
dependence of $\Delta S_{ng}$ for different $T$ in Fig. 2(b). $\Delta S_{ng}$
shows non-monotonic dependence on $t$, similar to $\alpha_{2}$. For all these
cases, $\Delta S_{ng}$ grows up to a peak and then decreases monotonically.
Surprisingly, for every $T$, we observe that $\Delta S_{ng}$ still grows to
higher values when $\alpha_{2}$ has already reached the maximum. This suggests
that the dynamic heterogeneity in $G_{S}(x,t)$ may not be completely captured
in $\alpha_{2}$ and the peak in $\alpha_{2}$ may also not be a true
representation of timescale of the underlying heterogeneity as it
underestimates the contribution of the higher order moments in $G_{S}(x,t)$
because the information in $\alpha_{2}$ is limited up-to the fourth moment of
the displacement distribution.

Figure 3: (a) Correlating the non-Gaussian information, $\Delta S_{ng}$ with
the non-Gaussian parameter, $\alpha_{2}$ for different $T\in[0.45,0.7]$. (b)
Inset: Temperature dependence of the timescales extracted corresponding to the
peak of $\alpha_{2}$, $\tau_{A}$ and $\Delta{S}_{ng}$, $\tau_{S}$. Main Panel:
$\log_{e}\tau_{S}$ vs. $\log_{e}\tau_{A}$. The dashed line shows a linear fit:
$\log_{e}\tau_{S}\sim\beta_{0}+\beta\log_{e}\tau_{A}$. (c) Dependence of
$\Delta S^{P}_{ng}$ on $T$. The dashed line shows $\Delta
S^{P}_{ng}\sim\nu_{0}-(T-T*)^{\gamma}$. Inset: $\log_{e}\tau_{S}$ vs.
$\log_{e}\Delta S^{P}_{ng}$. The dashed line shows
$\log_{e}\tau_{S}\sim\zeta_{0}+\zeta\log_{e}\Delta S^{P}_{ng}$. (d)
$\log_{e}\tau_{S}$ vs. $(T\Delta S^{P}_{ng})^{-1}$. The dashed line shows
$\log_{e}\tau_{S}\sim(T\Delta S^{P}_{ng})^{-\psi}$. See text for the values of
different fitting parameters.

We compare the behavioural differences between $\alpha_{2}$ and $\Delta
S_{ng}$ in Fig. 3. In Fig. 3(a), we statistically correlate $\alpha_{2}$ and
$\Delta S_{ng}$ for the different cases of $T$. In all these cases, they form
loop-like shapes and the area covering the curve is larger for smaller $T$. We
observe that the behavioral difference between $\alpha_{2}$ and $\Delta
S_{ng}$ grows when both have higher values, representing the maximal
heterogeneity. The degree of such deviation significantly increases for
decreasing $T$, suggesting strongly that the dynamic heterogeneity is
underestimated by $\alpha_{2}$ when $G_{S}(x,t)$ is maximally non-Gaussian.
This is consistent with our earlier investigations sumanEPL2020  where we
obtained similar loops in the case of a model supercooled liquid based on
continuous time random walk (CTRW), suggesting that the underlying dynamic
heterogeneity picture is qualitatively similar and the behavior of non-
Gaussianity based quantifications remain universal under such slowly varying
conditions.

We further extract the associated timescales corresponding to the peaks of
$\alpha_{2}$, $\tau_{A}$ and that of $\Delta S_{ng}$, $\tau_{S}$ [see Fig. 2]
and show its dependence on inverse of $T$ in the inset of Fig. 3(b). Both
$\tau_{A}$ and $\tau_{S}$ increase sharply for larger values of $1/T$. In all
these cases, we observe that $\tau_{S}$ has higher values than that of
$\tau_{A}$. We finally correlate $\log_{e}\tau_{A}$ and $\log_{e}\tau_{S}$ in
the main panel of Fig. 3(b). We fit the data of $\log_{e}\tau_{S}$ and
$\log_{e}\tau_{A}$ and obtain
$\log_{e}\tau_{S}\approx\beta_{0}+\beta\log_{e}\tau_{A}$ with
$\beta_{0}\approx-0.379,\beta\approx 1.286$. This suggests that $\tau_{S}$
diverges from $\tau_{A}$ in a power-law ($\tau_{S}\sim\tau_{A}^{\beta}$) and
such difference diverges with decreasing $T$. Further, we extract the height
of the peak in $\Delta S_{ng}$, $\Delta
S^{P}_{ng}(T)(=\max_{t\in[0,\infty]}\Delta S_{ng}(t;T))$ and correlate it with
the corresponding entropic timescale, $\tau_{S}$. In inset of Fig. 3(c), we
show that $\log_{e}\tau_{S}\sim\zeta_{0}+\zeta\log_{e}\Delta S^{P}_{ng}$ where
$\zeta_{0}\approx 9.956,\zeta\approx 2.164$. Also, we observe that $\Delta
S^{P}_{ng}$ increases with decreasing $T$, sharply near $T=0.45$ [Fig. 3(c)],
when $\tau_{S}$ grows sharply as shown in inset of Fig. 3(b). We fit the data
as $\Delta S^{P}_{ng}\sim\nu_{0}-(T-T*)^{\gamma}$ with $T*\approx
0.443,\nu_{0}\approx 0.928,\gamma\approx 0.064$. This suggests that the non-
Gaussian information diverges as $T\rightarrow T*$ for the sharp increase of
$\tau_{S}$ when approaching to the state of dynamic arrest at the glass
transition. This also indicates that the degree of heterogeneity at a given
$T$ is strongly connected with $\Delta S^{P}_{ng}$. Thus, the system explores
a strongly intermittent environment and dynamically heterogeneous states with
larger non-Gaussian information when the divergence of $\Delta S^{P}_{ng}$
represents the absolute entropic distance of the dynamic arrest from the
nearest diffusive route. Hence, the term $T\Delta S^{P}_{ng}$ can be
considered as an energy term associated with the heterogeneity that competes
with the diffusion in overcoming the dynamic heterogeneity at a given $T$. So,
we correlate the entropic timescale, $\tau_{S}$ with $1/(T\Delta S^{P}_{ng})$
and observe that $\tau_{S}$ grows and shows a sharp rise with decreasing
$1/(T\Delta S^{P}_{ng})$ [Fig. 3(d)]. Our data follows
$\log_{e}\tau_{S}\sim\eta_{0}+\eta(T\Delta S^{P}_{ng})^{-\psi}$ with
$\eta_{0}\approx-8.993,\eta\approx 25.691,\psi\approx-0.205$. Using the form
of $\Delta S^{P}_{ng}$, one can obtain $\log_{e}\tau_{S}\sim
T^{-\psi}[1-(T-T*)^{\gamma}]^{-\psi}$, where the divergence of $\tau_{S}$, for
$T\rightarrow T*$, is primarily governed by the term
$[1-(T-T*)^{\gamma}]^{-\psi}$, which qualitatively explains the nature of
growth of $\tau_{S}$ with decreasing $T$, as seen in inset of Fig. 3(b). The
rapid rise of $\tau_{S}$ is due to the presence of strong intermittency, for
which achieving diffusion becomes increasingly difficult overcoming the
dynamic heterogeneity, for the exploration within the rough energy landscapes
at low temperatures.

Discussion.— These results align with our theoretical predictions of the
entropic timescales of dynamic heterogeneity in a model supercooled liquid
sumanEPL2020  where the development of intermittent non-Gaussian tail was
modeled using the Montroll-Weiss CTRW framework, considering complex hopping
of particles within the mobile and immobile regions and jumps from one region
to another langerMukh2008 . The persistence of such vibrations and jumps
within these regions overall controls the nature of intermittency and its
lifetime pinaki2007 ; pinaki2008 . Such intermittent non-Gaussian tails were
inferred as a convolution process of cooperative diffusion, known as the
Brownian yet Non-Gaussian diffusion granick2012 ; sokolov . Supportive
literature slater ; metzler ; jain2016 ; dutta2016 ; karmakar2018 ;
karmakar2020  shows that the diffusion spectrum obtained upon the
deconvolution of the non-Gaussian displacement distribution validates the
physical picture of dynamic heterogeneity which considers the simultaneous
presence of slow and fast regions within the system dutta2019 . Such
heterogeneity exhibits anomalous spatiotemporal fluctuations that have all
order information Lemaitre . Therefore, to estimate the dynamic heterogeneity
effectively, consideration of all order moments is advantageous. Gaussianity
reverts when the Fickian diffusion sets in and the dynamics follow the central
limit theorem, marking the onset of diffusion. However, extraction of this
timescale is difficult due to the scale-free decay of $\alpha_{2}(t)$ pastore
, specifically, when the ambient temperature is close to the Glass transition
and the situation becomes intrinsically non-equilibrium. Our analysis also
affirms that both $\tau_{A}$ and $\tau_{S}$ diverge while the latter diverges
faster than the former in a power law which can be further tested using other
CTRW models pinaki2007 ; pinaki2008  or using Mode-coupling-theory ho . The
sharp increase of $\Delta S^{P}_{ng}$ with decreasing $T$ is due to diverging
dynamic heterogeneity that also leads to a sharp increase in entropic
timescale as the system approaches the state of dynamic arrest. Whether the
non-Gaussian-information has any connection with the configurational entropy
sastryjack  or non-equilibrium free energyinformation00 , needs further
investigations.

Conclusions.— We quantify the timescales of dynamic heterogeneity in
supercooled liquids using the conventional non-Gaussian parameter and the non-
Gaussian information. We show that the entropic timescales are significantly
different and diverging from those obtained using the non-Gaussian parameter.
This difference arises because the moment-based definitions are limited up to
the fourth order, while the information-theoretic quantification takes into
account all order moments. Although several other moment-based definitions are
available szamel2006 ; horbach1998 , it is always challenging to estimate
timescales using them as the computation of higher-order moments can be
increasingly noisy or unreliable without the quality data. On the other hand,
our framework is easy to implement and extracts heterogeneity optimally. This
makes the information-theoretic-framework scientifically robust in quantifying
non-Gaussianity in practical situations, also in a more general context, in
out-of-equilibrium systems in identifying or predicting novel cross-over or
transition where small fluctuations lead to catastrophic changes, or
differentiate phases or states of matter Lemaitre ; loewen .

Acknowledgements.— V.V. gratefully acknowledges funding from the European
Union through Horizon Europe ERC Grant number: 101043968 “Multimech”. S.D.
acknowledges support of the Department of Atomic Energy, Government of India,
under project no RTI4001. We acknowledge HPC facilities at IMSc, ICTS-TIFR and
supporting grants, ICTS/eiosm2018/08, ICTS/ISPCM2023/02. We thank D. Bagchi
and S. Sikdar for sharing data. We thank S. Bose, P. Chaudhuri, C. Dasgupta,
J. Horbach, S. Karmakar, S. K. Nandi, M. S. Shell and A. Zaccone for
insightful discussions.

Research Contribution.— VV: Performed simulations and formal analysis,
Validation, Visualizations. SD: Conceptualization, Methodology, Validation,
Research Coordination, Data Interpretation, Writing.

## References

  * (1) W. Kob, C. Donati, S. J. Plimpton, P. H. Poole, and S. C. Glotzer, Phys. Rev. Lett. 79, 2827 (1997).
  * (2) M. D. Ediger, Annu. Rev. Phys. Chem. 51, 99 (2000).
  * (3) W. K. Kegel and A. van Blaaderen, Science 287, 290 (2000).
  * (4) E. R. Weeks, J. C. Crocker, A. C. Levitt, A. Schofield, and D. A. Weitz, Science 287, 5453 (2000).
  * (5) Y. Gao and M. L. Kilfoil, Phys. Rev. Lett. 99, 078301 (2008).
  * (6) R. Richert, J. Phys.: Condens. Matter 14, R703 (2002).
  * (7) E. Bertin, J.-P. Bouchaud, and F. Lequeux, Phys. Rev. Lett. 95, 015702 (2005).
  * (8) N. Lačević, F. W. Starr, T. B. Schrøder, and S. C. Glotzer, J. Chem. Phys. 119, 7372 (2003).
  * (9) H. C. Andersen, Proc. Natl. Acad. Sci. 102, 6686 (2005).
  * (10) M. S. Shell, P. G. Debenedetti, and F. H. Stillinger, J. Phys.: Condens. Matter 17 S4035 (2005).
  * (11) E. Flenner and G. Szamel, Phys. Rev. Lett. 105, 217801 (2010).
  * (12) J. P. Garrahan, Proc. Natl. Acad. Sci. 108, 4701 (2011).
  * (13) K. Martens, L. Bocquet, and J-L. Barrat, Phys. Rev. Lett. 106, 156001 (2011).
  * (14) L. Berthier, Physics 4, 42 (2011).
  * (15) L. Berthier, G. Biroli, J.-P. Bouchaud, L. Cipeletti, and W. van Saarloos, Oxford University Press, Oxford, (2011).
  * (16) L. Berthier and G. Biroli, Reviews of modern physics, 83(2), 587 (2011).
  * (17) E. M. Schoetz, M. Lanio, J. A. Talbot, and M. L. Manning, Journal of The Royal Society Interface 10, 20130726 (2013).
  * (18) S. Karmakar, C. Dasgupta, and S. Sastry, Annu. Rev. Condens. Matter Phys. 5(1), 255 (2014).
  * (19) E. Flenner and G. Szamel, Phys. Rev. Lett. 112, 097801 (2014).
  * (20) W. Zhang, J. F. Douglas, and F. W. Starr, J. Chem. Phys 146, 203310 (2017).
  * (21) B. P. Bhowmik, I. Tah, and S. Karmakar, Phys. Rev. E 98, 022122 (2018).
  * (22) R. Das, C. Dasgupta, and S. Karmakar, Frontiers in Physics 8, 210 (2020).
  * (23) P. Pareek, M. Adhikari, C. Dasgupta, and S. K. Nandi, arXiv preprint arXiv:2305.18042 (2023).
  * (24) A. Heuer, J. Phys.: Condens. Matter 20, 373101 (2008).
  * (25) P. Chaudhuri, L. Berthier, and W. Kob, Phys. Rev. Lett. 99, 060604 (2007).
  * (26) P. Chaudhuri, Y. Gao, L. Berthier, M. Kilfoil, and W. Kob, J. Phys.: Condens. Matter 20, 244126 (2008).
  * (27) B. Vorselaars, A. V. Lyulin, K. Karatasos, and M. A. J. Michels, Phys. Rev. E 75, 011504 (2007).
  * (28) S. Karmakar, C. Dasgupta, and S. Sastry, Phys. Rev. Lett. 105, 015701 (2010).
  * (29) G. Gradenigo, E. Bertin, and G. Biroli, Phys. Rev. E 93, 060105(R) (2016).
  * (30) S. Dutta, Chem. Phys. 522, 256 (2019).
  * (31) B. Wang, S. M. Anthony, S. C. Bae, and S. Granick, Proc. Natl. Acad. Sci. 106(36), 15160 (2009).
  * (32) B. Wang, J. Kuo, S. C. Bae, and S. Granick, Nat. Mater. 11, 481 (2012).
  * (33) L. Berthier, E. Flenner, and G. Szamel, arXiv preprint arXiv:2210.07119 (2022).
  * (34) E. Barkai and S. Burov, Phys. Rev. Lett. 124(6), 060603 (2020).
  * (35) F. Rusciano, R. Pastore, and F. Greco, Phys. Rev. Lett. 128, 168001 (2022).
  * (36) K. Binder and W. Kob, Glassy materials and disordered solids: An introduction to their statistical mechanics. World Scientific, 2011.
  * (37) M. S. Shell, Coarse-Graining with the Relative Entropy, edited by Rice S. A. Rice and A. R. Dinner (John Wiley) 2016.
  * (38) M. Gabrie, A. Manoel, C. Luneau, J. Barbier, N. Macris, F. Krzakala, and L. Zdeborova, Advances in Neural Information Processing Systems 31, 1826 (2018).
  * (39) E. D. Cubuk, S. S. Schoenholz, J. M. Rieser, B. D. Malone, J. Rottler, D. J. Durian, E. Kaxiras, and A. J. Liu, Phys. Rev. Lett. 114, 108001 (2015).
  * (40) V. Bapst, T. Keck, A. Grabska-Barwińska, C. Donner, E. D. Cubuk, S. S. Schoenholz, A. Obika, A. W. Nelson, T. Back, D. Hassabis, and P. Kohli, Nat. Phys. 16, 448 (2020).
  * (41) G. Jung, G. Biroli, and L. Berthier, Phys. Rev. Lett. 130, 238202 (2023).
  * (42) W. Kob and H. C. Andersen, Phys. Rev. E 51, 4626 (1995).
  * (43) V. Vaibhav, J. Horbach, and P. Chaudhuri, Phys. Rev. E 101(2), 022605 (2020).
  * (44) S. Plimpton, J. Comp. Phys. 117, 1 (1995).
  * (45) L. Van Hove, Phys. Rev. 95 249, (1954).
  * (46) R. Dandekar, S. Bose, and S. Dutta, Europhys. Lett. 131, 18002 (2020).
  * (47) J. Horbach, W. Kob, and K. Binder, Philos. Mag. B 77, 297 (1998).
  * (48) J. S. Ivan, M. S. Kumar, and R. Simon, Quantum Inf. Process. 11 853 (2012).
  * (49) S. Kullback and R. A. Leibler, Ann. Math. Stat. 22, 79 (1951).
  * (50) J. S. Langer and S. Mukhopadhyay, Phys. Rev. E 77, 061505 (2008).
  * (51) A. V. Chechkin, F. Seno, R. Metzler, and I. M. Sokolov, Phys. Rev. X 7, 021002 (2017).
  * (52) S. Dutta and J. Chakrabarti, Europhys. Lett. 116, 38001 (2016).
  * (53) M. V. Chubynsky and G. W. Slater, Phys. Rev. Lett. 113, 098302 (2014).
  * (54) V. Sposini, A. Chechkin, and R. Metzler, J. Phys. A: Math. Theor. 52, 04LT01 (2018).
  * (55) R. Jain and K. L. Sebastian, J. Phys. Chem. B 120, 9215 (2016).
  * (56) E. Lemaitre, I. M Sokolov, R. Metzler, and A. V Chechkin, New. J. Phys. 25, 013010 (2023).
  * (57) S.-H. Chong, Phys. Rev. E 78, 041501 (2008).
  * (58) F. W. Starr, J. F. Douglas, and S. Sastry, J. Chem. Phys. 138, 12A541 (2013).
  * (59) J. M. R. Parrondo, J. M. Horowitz and T. Sagawa, Nat. Phys., 11, 131 (2015)
  * (60) G. Szamel and E. Flenner, Phys. Rev. E 73, 011504 (2006).
  * (61) X. Zheng, B. ten Hagen, A. Kaiser, M. Wu, H. Cui, Z. Silber-Li, and H. Löwen, Phys. Rev. E 88, 032304 (2013).