# Masked Autoencoding Does Not Help Natural Language Supervision at Scale

Floris Weers  Vaishaal Shankar  Angelos Katharopoulos  Yinfei Yang  Tom Gunter
<EMAIL_ADDRESS>
###### Abstract

Self supervision and natural language supervision have emerged as two exciting
ways to train general purpose image encoders which excel at a variety of
downstream tasks. Recent works such as M3AE [31] and SLIP [64] have suggested
that these approaches can be effectively combined, but most notably their
results use small ($<$20M examples) pre-training datasets and don’t
effectively reflect the large-scale regime ($>$100M samples) that is commonly
used for these approaches. Here we investigate whether a similar approach can
be effective when trained with a much larger amount of data. We find that a
combination of two state of the art approaches: masked auto-encoders, MAE [38]
and contrastive language image pre-training, CLIP [69] provides a benefit over
CLIP when trained on a corpus of 11.3M image-text pairs, but little to no
benefit (as evaluated on a suite of common vision tasks) over CLIP when
trained on a large corpus of 1.4B images. Our work provides some much needed
clarity into the effectiveness (or lack thereof) of self supervision for
large-scale image-text training.

##  1 Introduction

Large scale pretraining has become a powerful tool in the arsenal of computer
vision researchers to produce state of the art results across a wider variety
of tasks [99, 89, 40, 96]. However, when pre-training on tens of millions to
billions of images it is difficult to rely on standard supervised methods to
train models, as datasets of this size often lack reliable labels. In the
presence of these massive but largely under-curated datasets, two general
classes of methods to train general purpose image encoders have emerged:

  1. 1.

Self Supervised techniques that learn visual representations from the image
data alone [37, 11]

  2. 2.

Natural Language Supervised methods that utilize paired free-form text data to
learn visual representations [70, 44]

Due to the unique strengths and weaknesses of each approach111Self supervised
methods can learn representations without labels, but natural language
supervision learns better representations. Natural language supervised methods
rely on quality of captions, a recent flurry of work has introduced methods
that combine both forms of supervision [65, 57, 32, 79] to varying degrees of
success. While each of these methods establishes some regime where the
additional supervision helps, none of these “joint-supervision” methods
advance state of the art in any meaningful way. Additionally, to our knowledge
none of these methods have shown comparative results at the scale many large
scale vision models are currently trained at ($>$100M examples) [70, 44, 83,
67, 81, 74, 99]. Furthermore, methods that use both forms of supervision start
with the presumption that the additional supervision is helpful and either
often lack clean ablations or lack evaluations in a “high accuracy”
regime—leading to further confusion regarding whether a combination of these
methods can actually improve the state of the art. To clarify this issue, in
this work, we investigate a simple question:

> Does a combination of self supervision and natural language supervision
> actually lead to higher quality visual representations?

In order to answer this, we first introduce a straightforward baseline
approach that combines standard self supervision and language supervision
techniques. We combine masked auto-encoders (MAE) and contrastive language
image-pretraining (CLIP) to make MAE-CLIP. We then present a careful study of
the performance of MAE, M3AE, CLIP and MAE-CLIP across a wide variety of tasks
in two distinct regimes: a “low-sample” 222We note that what low sample means
has changed substantially over the last few years 11.3 million example regime
and a “high-sample” 1.4 billion example regime. We train self-supervised and
language-supervised methods using the same pre-training datasets under the
assumption that we have no knowledge about downstream tasks. Our experiments
show:

  1. 1.

In the low sample size regime, without changing the final pooling operation in
the network, we observe a large performance improvement, namely $6$% on
ImageNet [18] and 4% on VTAB [106]. However, when we modify the pooling
operation, the improvement substantially decreases to around $1$% on both
ImageNet and VTAB.

  2. 2.

In the high sample size regime, there is virtually no difference in
performance between MAE-CLIP and CLIP across ImageNet, VTAB, and VQA tasks.

We believe our work is the first careful study of this form and contextualizes
recent progress in both self-supervision and natural language supervision.

The rest of the paper is organized as follows: In Section 2, we cover related
work in the areas of self supervision and natural language supervision. In
Section 3, we give an overview of the baseline methods we study, MAE, M3AE,
CLIP and our new baseline MAE-CLIP. Then we present and analyse our small
scale and large scale experimental findings in Sections 4 and 5. Finally, we
discuss potential explanations for our findings and some future work in 6.

##  2 Related Work

Figure 1: MAE-CLIP: CLIP [69] augmented with a generative decoder in the style
of MAE [31].

Our work combines natural language supervision and self-supervision in a
multi-task approach to visual encoding, and so research from these three areas
is relevant.

Natural language supervision for visual encoding covers a variety of
approaches that assume access to datasets of images or videos associated with
text. Some of the most successful use a contrastive pairwise alignment signal
applied to very large batch and dataset sizes [69]. This large batch size
means hard negative pairs can be produced by random sampling, avoiding the
need for a memory bank [12, 13, 39] or momentum distillation [15]. FILIP [97]
further improves zero-shot performance for some tasks by using a more fine-
grained elementwise contrastive loss. Image captioning has also shown promise
as a pre-training task [20, 91], with [20] in particular demonstrating strong
data efficiency. UNITER [16] pursues a similar idea, applying a generative
loss to both image and text modalities. [85, 23, 2, 98, 52, 62, 82, 16, 42,
48] apply the masked patch prediction problem from [21, 51, 38] to a joint
image-text data space. These approaches either use a pre-trained convolutional
neural network (CNN) to generate region of interest (RoI) proposal encodings,
or predict the labelled class of the masked patch instead of the raw pixel
values (or the quantized patch ID). UNIMO-2 [54] attempts to ground image and
text patches for a single data example into a top-$k$ quantized set of
embeddings, before passing the grounded embeddings along with the raw non-
grounded embeddings into a decoder, targeting a loss based on masked language
modeling (MLM) and image-text-matching (ITM).

Self-supervised approaches for visual encoding may be loosely categorized into
those that target consistency constraints between multiple views of the same
scene and those that attempt visual reconstruction on corrupted or
conditioning representations of images. Consistency constraints are often
derived through data-augmentation applied to a single real image [103, 10, 36,
14, 6], but these may also take the form of e.g. a temporal constraint if
video data is available [90], or be applied at an image-patch level by making
a smoothness assumption and selecting neighboring patches as ‘consistent’
whilst treating distant patches (or those sourced from another image) as
negative examples [102]. Forms of denoising autoencoder (DAE) [88] are popular
as a means of self-supervision, and have been investigated at a variety of
scales; iGPT [9] learns to generate images as a row-major flattened sequence
of pixels, whilst [66], [73], and [26] compress images into a short sequence
of discrete codes before then regenerating and scoring at a pixel level. BEiT
[3] first encodes an image into a sequence of discrete codes before masking
and then predicting a subset of the codes. MAE [38] drops the discretization,
instead predicting raw pixel loss for a subset of the encoded patches in a
manner strongly reminiscent of BART [51]. SplitMask [25] applies both a masked
image patch prediction loss and a pairwise contrastive loss, using the masked
example as well as the inverted masked example to form each positive pair.
Meanwhile, M3AE [24] and VL-BEiT [4] both propose a masked patch prediction
problem applied to image and text modalities jointly. Recent concurrent work
shows that while masking does _not_ help generalization, the masking can be
used to speed up training by dropping the masked tokens during the forward and
backward pass [56]. EVA [29] shows that doing masked image modeling to predict
the output _embeddings_ (rather than pixels) of the masked image can be an
effective pretraining task when combined with natural language supervision. We
note for EVA the masked image modeling occurs only on smaller well curated
datasets such as ImageNet, Conceptual Captions and COCO.

Multi-task methods for pre-training visual encoders are a highly active area
of research. CoCA [100] combines the cross-modality contrastive task from [69]
with image captioning in the style of [20], and by pre-training at very large
scale show that the resulting model is more performant than prior art across a
very broad array of downstream visual understanding tasks. SLIP [65] combines
a SimCLR style self supervision loss with the standard CLIP contrastive loss
to train a jointy self-supervised and natural language supervised model,
however while they show large performance gains they are all in a low accuracy
regime (below 40% Top-1 accuracy on ImageNet). Florence [101] employs a
diverse array of tasks and datasets at pre-training time (including object
detection, CLIP-style contrastive modeling, Visual Question Answering (VQA),
and supervised classification). ALBEF [53] builds on a CLIP-style architecture
by incorporating a single-stream encoder that consumes both modalities. It
uses a masked language modeling loss for the text modality and an image-text
matching loss for within-batch hard-negatives, mined according to contrastive
similarity. FLAVA [80] also uses masked-image modeling, and uses both single
and joint modality encoders, with additional single-modality datasets (i.e.
not just image-text pairs) that allow it to generalize to longer text inputs
and visual data which is unlikely to be captioned. X-VLM [104] utilizes image
data which includes object-labelled bounding boxes to extend the image-text
matching and contrastive losses to cover within-image regions as well as
whole-image captions. They also include a masked language modelling objective,
achieving impressive performance on many zero-shot downstream tasks—although
the requirement for object bounding box data may not scale well for large pre-
training datasets. SupMAE [58] adds a supervised labeling task to the MAE
architecture, demonstrating that this results in better data efficiency. MVP
[92], meanwhile, finetunes a pre-trained CLIP backbone using MIM, and
demonstrates that starting with a pre-trained model improves downstream visual
recognition performance. MAE-CLIP is reminiscent of M3AE (with the addition of
a contrastive task), ALBEF (but applying the masked prediction task to both
modalities, dropping the momentum distillation), and most of all FLAVA
(without the single-modality masked-patch-prediction tasks, and with the use
of high ratio random masking following MAE and M3AE).

##  3 Background

###  3.1 Contrastive language-image pre-training

CLIP [69] and ALIGN [45] both demonstrated that contrastive image-language
supervision applied at scale was capable of producing an image encoder that
excels at a range of downstream tasks—often in a zero-shot setting. In our
implementation, we begin by following the design outlined in [69].
Specifically, we use per-modality encoders to produce a single $\ell
2$-normalized dense embedding vector for each image or text input, before
applying a pairwise InfoNCE [66] loss to a large global batch of paired image-
text embeddings, using all non-paired examples as hard negatives. We define
the image-to-text loss $L_{i2t}$ and the text-to-image loss $L_{t2i}$ as:

$L_{i2t}=-\frac{1}{N}\sum_{j}^{N}\log{\frac{\exp(x_{j}^{T}y_{j}/\sigma)}{\sum_{k=1}^{N}\exp(x_{j}^{T}y_{k}/\sigma)}}$
(1)
$L_{t2i}=-\frac{1}{N}\sum_{j}^{N}\log{\frac{\exp(y_{j}^{T}x_{j}/\sigma)}{\sum_{k=1}^{N}\exp(y_{j}^{T}x_{k}/\sigma)}},$
(2)

where $N$ is the global batch size, $x_{j}$ is the normalized embedding of the
image for the $j$-th pair and $y_{k}$ is the normalized embedding of the text
in the $k$-th pair in the batch. $\sigma$ is a learnable temperature
parameter. The total contrastive loss $L_{c}=\frac{1}{2}(L_{t2i}+L_{i2t})$ is
the average of these two losses.

We use transformer encoders for both modalities, however, differently to [69]
we explore several strategies for aggregating the image-patch or text-token
outputs into a single embedding (see Section 4.4). We also eschew the
autoregressive masking strategy for the text encoder, instead allowing full
bi-directional self attention following [45].

####  3.1.1 Pooling

In CLIP [69] the authors use a separate ‘CLS’ token projected through the
network as the overall image representation. On the other hand, ALIGN [45] use
global-average pooling over the encoded visual features. Whilst both
approaches produced good downstream evaluation results, recent work [8, 105,
72], suggested that the choice of pooling strategy can strongly influence the
quality of visual semantic embeddings. In particular, [72] demonstrated that
this effect is present for CLIP-like visual encoders. Noting this, we opt to
investigate three pooling strategies: 1. the default multihead-attention
pooling (MAP), 2. global average pooling (GAP)—both described in [105]—as well
as 3. non-maximal suppression pooling (MAX) as in [8, 72].

###  3.2 Masked Autoencoders

In MAE [38] the authors demonstrate a simple technique for self-supervised
image-encoder pre-training that—to our knowledge—is still considered state-of-
the-art. They use a ViT [22] encoder-decoder architecture and apply it to
heavily masked input images. The input to the encoder consists of the visible,
unmasked image patches, first embedded via a linear projection before additive
positional embeddings are applied, and then the result is fed through the
encoder’s transformer layers. They demonstrate that a very high patch masking
ratio is critical to achieving good performance, and usually retain only
$25\%$ of the image. The decoder consumes the output of the encoder, as well
as a learned ‘masked-patch’ embedding, which is included to represent each
masked token. Positional embeddings are also reapplied, to ensure that spatial
information is passed through the decoder for the masked patches. The output
of the decoder at the masked positions is then measured against ground truth
using a simple mean-squared-error loss. As in [38] we experiment with
predicting both normalized and un-normalized patch values, finding that
predicting the normalized patch value slightly improves the performance of our
MAE implementation.

Masking across modalities: BART [51] applies a similar strategy to text data,
but uses a cross-entropy loss over the masked-token output distribution and a
far lower masking ratio (typically around $15\%$). M3AE [31] extends MAE and
BART to incorporate inputs from both text and image modalities, using per-
modality input and output projections. Otherwise the encoder-decoder
architecture resembles MAE, with the addition of learned modality-indicating
embeddings to each transformer input.

The M3AE loss relies only on the contents of the image-text pairs, and
comprises mean-squared error for the masked image patches ($L_{gen\\_i}$) and
cross-entropy over the vocabulary for the masked text tokens ($L_{gen\\_t}$):

$\displaystyle L_{\text{gen\\_i}}$
$\displaystyle=\frac{1}{N}\sum_{j}^{N}\left(p_{j}-P_{j}\right)^{2}$ (3)
$\displaystyle L_{\text{gen\\_t}}$
$\displaystyle=-\frac{1}{N}\sum_{j}^{N}\log\frac{\exp(t_{n,T_{n}})}{\sum_{c=1}^{C}\exp(t_{n,c})},$
(4)

where $p_{j}$ and $P_{j}$ refer to the predicted and ground truth pixel value
respectively and $t_{n}$ and $T_{n}$ refer to the predicted token distribution
and the ground truth token, with $C$ as the number of unique tokens. For the
image reconstruction loss, we normalize the ground-truth per-patch, following
[38].

###  3.3 MAE-CLIP

As the name suggests, MAE-CLIP attempts to incorporate aspects of MAE/M3AE
into CLIP through the addition of a single dual-modality transformer decoder
(as in M3AE) which consumes the output of the CLIP encoders. We show this in
Figure 1.

####  3.3.1 Components

The model architecture is similar to ALBEF [53] and FLAVA [79], consisting of
three components.

An image encoder: Following ViT [22], we divide the input image into equally-
sized, non-overlapping patches. After applying a linear projection and adding
a 2-D position encoding, we feed the per-patch representations through a
number of transformer layers [87].

A text encoder: Following ALIGN [44], our text encoder is based on BERT [21],
in which text is first tokenized and embedded, and a $1$-D trainable position
encoding is added. Differently to BERT, we use pre-layernorm [95] and
initialize the parameters according to the simplified scheme outlined in [71].

A cross-modality decoder: The decoder receives per-element encoded image and
text representations from the encoders for both masked and un-masked elements.
For the image modality, masked patches are added by replacing their values
with a shared trainable mask token. Positional encodings are once again added
to all elements, following [38]. We also add a per-modality trainable encoding
to allow the decoder to easily distinguish between the two modalities. The
decoder uses the same transformer implementation as the encoders. The output
of the final decoder layer is then projected into a per-patch-modality output
space and the overall loss is computed.

####  3.3.2 Losses

MAE-CLIP employs the losses of both CLIP (Eq. 1, 2) and M3AE (Eq. 3, 4). Our
final loss is a weighted sum of the losses from each task, as follows:

$L=\frac{1}{2}\left(L_{t2i}+L_{i2t}\right)+w_{i}\cdot
L_{\text{gen\\_i}}+w_{t}\cdot L_{\text{gen\\_t}}$ (5)

where $w_{i}$ and $w_{t}$ are scalars used to control the relative weight of
the generative losses. We always provide paired image-text inputs to the
model, leaving it to future work to explore the benefits of also incorporating
single modality inputs and reconstruction losses, at scale.

To avoid severely impacting the CLIP loss via computing it with masked inputs,
we run the encoders twice: once with full unmasked input in order to compute
the CLIP loss, before then making a second pass using only the unmasked input
to compute the M3AE loss. Finally, we compute the weighted sum of the two
losses, and use this to calculate the overall update. 333Alternative, more
memory efficient, strategies exist for computing the overall updates (e.g.
calculate per-loss gradients, or perform round-robin updates), but prior work
[1] has demonstrated that some of these techniques may negatively impact the
overall training result.

##  4 Experiments

In this section, we present our experimental results to study how the addition
of self-supervision affects the representation quality of natural language
supervised models. We study standard masked auto-encoders (MAE), multimoddal
masked auto-encoders (M3AE), contrastive language image pretraining (CLIP) and
our newly introduced baseline method (MAE-CLIP), which combines mask based
self-supervision with contrastive language-image learning.

###  4.1 Experimental Setup

Architecture: Experiments are performed using an image encoder based on the
architecture described as ViT-B-16 in [94], in combination with a $12$-layer,
$512$-wide text encoder with $8$ attention heads. For the decoder, we use
another $12$-layer, $512$-wide transformer with $8$ attention heads, identical
to the text encoder. We also use a byte-pair encoding tokenizer, fit to the
OpenWebText corpus [33]. When training MAE-CLIP, we replace the random masking
by a similarity masking strategy, which makes use of the element-wise CLIP
similarity scores to select the masked element for the reconstruction task.
444We compare the similarity masking strategy with the original random masking
strategy on various tasks, the results show they perform similarly.

Pretraining Datasets: Our analysis is divided into two sections. First, in
sections 4.2 to 4.4, we present our study on the “low-sample” regime using the
combination of CC12M [7] and CC3M [78] excluding all images whose text
contains a [PERSON] tag555Preliminary experiments on CLIP suggest removing
such data leads to better performance.. The final dataset contains 11.3
million comparatively high quality image-text pairs. We refer to this dataset
as _CC_ or “small” scale.

Pretraining configuration: We use a minibatch size of $16,384$ samples and
train using the AdamW optimizer [61] with decoupled weight decay. We use a
base learning rate of $5\times 10^{-4}$ and train using cosine learning rate
decay [60] with an initial linear warmup of $200$ steps. We reduce the
$\beta_{2}$ decay factor to $0.98$, and increase $\epsilon$ to $10^{-6}$,
finding that these changes improved rate of convergence for all models
trained. To increase training convergence speed, we switch from a local (per-
GPU) to a global contrastive loss after $500$ steps. During the global
contrastive loss phase, we set the generative text and image loss weights to
$w_{t}=0.05$ and $w_{i}=0.1$ respectively. For all models which rely on
masking, we use a $75\%$ masking ratio, consistent with [38, 31], as we did
not find an alternative that improved downstream results. In total, we train
our models on _CC_ for $32$ epochs, which corresponds to $\sim 22,000$ steps.
For our M3AE training runs, we use per-modality encoders so as to produce a
baseline that is flop and parameter matched to our MAE-CLIP runs.

Models | Zero-shot | Linear Probing
---|---|---
MAE | – | 33.9
M3AE | – | 52.5
CLIP | 29.7 | 52.6
MAE-CLIP | 33.8 | 58.9
Table 1: ImageNet classification with zero-shot transfer or linear probing after pretraining on the _CC_ dataset (11.3M images). MAE-CLIP significantly improves the classification performance of CLIP in the small scale regime. Table 2: Linear probing accuracy (%) on classification tasks. All models are trained on the _CC_ Dataset (11.3M images). ($\bullet$ VTAB/natural, $\bullet$ VTAB/specialized and $\bullet$ VTAB/structured.) |  $\bullet$ Caltech101 |  $\bullet$ CIFAR-100 |  $\bullet$ DTD |  $\bullet$ Flowers102 |  $\bullet$ Pets |  $\bullet$ SVHN |  $\bullet$ EuroSAT |  $\bullet$ Camelyon |  $\bullet$ Resisc45 |  $\bullet$ Clevr/Closest |  $\bullet$ Clevr/Count |  $\bullet$ DMLab |  $\bullet$ dSprites/Ori |  $\bullet$ dSprites/Loc |  $\bullet$ KITTI/Dist |  $\bullet$ sNORB/Azim |  $\bullet$ sNORB/Elev |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MAE | 75.3 | 56.2 | 58.5 | 70.2 | 37.4 | 71.9 | 96.2 | 82.8 | 84.0 | 65.1 | 60.6 | 42.8 | 37.9 | 79.9 | 36.7 | 37.4 | 64.5 | 62.2
M3AE | 85.6 | 66.6 | 69.4 | 86.5 | 58.4 | 69.2 | 97.4 | 82.9 | 90.7 | 69.4 | 71.2 | 48.2 | 49.8 | 83.3 | 42.6 | 34.3 | 73.4 | 69.3
CLIP | 84.2 | 62.8 | 57.7 | 81.6 | 69.7 | 52.5 | 95.5 | 82.6 | 86.7 | 53.7 | 52.9 | 44.8 | 45.9 | 61.3 | 45.7 | 31.0 | 42.6 | 61.8
MAE-CLIP | 89.8 | 66.2 | 64.8 | 86.5 | 74.5 | 56.7 | 95.4 | 81.6 | 86.9 | 53.6 | 59.6 | 45.9 | 44.6 | 68.6 | 48.0 | 36.1 | 51.0 | 65.3
Model | CLEVR | VQAv2 | GQA
---|---|---|---
MAE | 93.5 | 45.7 | 46.1
M3AE | 97.5 | 55.8 | 50.9
CLIP | 87.5 | 55.6 | 50.0
MAE-CLIP | 96.0 | 58.5 | 52.2
Table 3: VQA finetuning results for models pre-trained on CC Dataset (11.3M
images). We train a new decoder for all methods while keeping the encoders
frozen. MAE-CLIP performs significantly better than either only self-
supervised or language supervised methods by themselves.

###  4.2 Image Classification

We evaluate the quality of the representations learned from pre-training on
_CC_ by measuring the image classification performance, either by zero-shot
transfer, or by training a linear classifier using the predicted visual
features. For zero-shot classification, we use the average of $80$ prompts and
follow the routine prescribed in [70]. To train the linear classifier, we pre-
compute features using the visual encoder666We use no data augmentation here
and run AdamW for $20$ to $80$ epochs with a learning rate of $0.01$.

Table 1 shows the zero-shot and linear probing classification results on the
ImageNet [19] dataset while Table 2 shows our linear probing results on most
($17$) of the VTAB [107] benchmark datasets777We do not include results for
either the diabetic retinopathy or the Sun397 tasks, due to licensing issues.
More details in the supplementary material. Initially, we observe that the
combination of self-supervision and natural language supervision provides a
consistent and substantial improvement over either form of supervision by
itself. This result concurs with previous works such as [32] and [65], which
show that self-supervision aids natural language supervision.

###  4.3 VQA

Subsequently, we compare the performance of MAE-CLIP to the rest of the
baselines on the visual question answering task using three datasets. CLEVR
[46], VQAv2 [35] and GQA [43]. VQA assesses the model’s multimodal reasoning
capabilities as well as its visual grounding. In order to finetune our models
for VQA, we freeze the image and text encoders and either randomly
reinitialize the decoder in models such as MAE-CLIP or M3AE or add a new
identical decoder for CLIP. During finetuning, we use a layer-wise learning
rate decay, following [17, 38, 3]. For CLEVR and VQAv2 we finetune the decoder
for 50 epochs, while for GQA we finetune for 5 epochs on the “train-all” split
and 2 epochs on the “train-balanced” split following the protocol of [55]. In
all cases, we treat the problem as a classification problem of selecting one
answer out of the set of possible answers for each dataset.

In Table 3 we observe that self-supervision combined with natural language
supervision (MAE-CLIP) performs consistently better than either MAE or CLIP by
themselves. Combining information from both modalities is evidently critical
for effectively answering visual questions as the performance of MAE on VQAv2
and GQA is several percentage points lower than the rest of the methods.
Interestingly, methods that employ a form of self-supervision perform
significantly better on CLEVR. We argue that this discrepancy is due to CLEVR
requiring object localization and spatial reasoning which benefits from this
form of self-supervision.

###  4.4 Pooling Analysis

To better understand whether the gap between MAE-CLIP and CLIP we observed in
the previous section is fundamental, we investigate the effect of the pooling
operator on different downstream tasks. Table 4 compares the performance of
CLIP and MAE-CLIP with the standard multi-head attention pooling (MAP), global
average pooling (GAP) and max pooling (MAX) on all image classification tasks
and visual question answering. We note that GAP and MAX pooling perform
substantially better than the standard MAP pooling across ImageNet, VTAB and
VQA tasks. A hypothesis posed by [72] is that the alternative pooling
operators lead to better perceptual grouping in the visual representation. We
study this with a qualitative analysis in Section 6.1.

Model | Pooling | ImageNet | VTAB | VQA
---|---|---|---|---
ZS | LP | LP | FT
CLIP | MAP | 29.7 | 52.6 | 61.8 | 64.4
MAE-CLIP | MAP | 33.8 | 58.9 | 65.3 | 68.9
CLIP | GAP | 29.3 | 59.8 | 70.6 | 65.1
MAE-CLIP | GAP | 33.5 | 62.5 | 71.7 | 69.2
CLIP | MAX | 33.1 | 62.3 | 68.7 | 63.5
MAE-CLIP | MAX | 35.2 | 63.2 | 69.1 | 68.5
Table 4: Results of CLIP and MAE-CLIP with different pooling options trained on CC dataset (11.3M images). We compare the models on ImageNet Zero-Shot (ZS), ImageNet Linear-Probing (LP), VTAB 17-task Linear-Probing average, and VQA 3 tasks Fine-Tuning (FT) average. Table 5: Linear probing accuracy (%) on classification tasks. Models are all trained on our _web-crawled_ dataset (1.4B images). ($\bullet$ VTAB/natural, $\bullet$ VTAB/specialized and $\bullet$ VTAB/structured.) In the large scale pretraining regime, the difference between MAE-CLIP and CLIP is reduced to $<1\%$. * At evaluation time, our M3AE model was $50\%$ trained, so that performance may improve further. |  $\bullet$ Caltech101 |  $\bullet$ CIFAR-100 |  $\bullet$ DTD |  $\bullet$ Flowers102 |  $\bullet$ Pets |  $\bullet$ SVHN |  $\bullet$ EuroSAT |  $\bullet$ Camelyon |  $\bullet$ Resisc45 |  $\bullet$ Clevr/Closest |  $\bullet$ Clevr/Count |  $\bullet$ DMLab |  $\bullet$ dSprites/Ori |  $\bullet$ dSprites/Loc |  $\bullet$ KITTI/Dist |  $\bullet$ sNORB/Azim |  $\bullet$ sNORB/Elev |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
M3AE* | 93.0 | 74.8 | 78.2 | 95.4 | 81.2 | 69.6 | 97.3 | 84.8 | 92.7 | 65.7 | 74.7 | 51.0 | 52.7 | 80.7 | 47.5 | 36.2 | 68.8 | 73.2
CLIP | 94.9 | 78.4 | 80.0 | 97.3 | 86.9 | 59.0 | 94.1 | 82.3 | 92.7 | 45.6 | 62.1 | 46.0 | 46.1 | 53.3 | 50.9 | 20.3 | 35.8 | 66.2
CLIPMAX | 96.1 | 81.0 | 80.9 | 97.3 | 89.9 | 65.7 | 96.0 | 83.2 | 94.1 | 52.8 | 67.8 | 49.9 | 59.5 | 67.6 | 41.2 | 23.4 | 45.8 | 70.1
MAE-CLIPMAX | 95.8 | 79.2 | 81.5 | 96.8 | 88.2 | 62.1 | 95.8 | 81.8 | 93.0 | 52.0 | 66.9 | 49.6 | 53.7 | 72.5 | 53.0 | 32.3 | 45.4 | 70.6
CLIPGAP | 95.8 | 80.5 | 81.6 | 97.6 | 88.7 | 66.0 | 97.0 | 84.4 | 93.3 | 56.7 | 71.4 | 53.3 | 58.0 | 70.1 | 50.6 | 38.3 | 55.1 | 72.9
MAE-CLIPGAP | 95.4 | 79.3 | 82.2 | 97.4 | 88.6 | 72.8 | 96.6 | 84.5 | 93.5 | 57.5 | 73.6 | 52.7 | 57.5 | 71.2 | 51.6 | 45.6 | 55.2 | 73.8

##  5 Experiments at Scale

Models | Zero-shot | Linear Probing
---|---|---
M3AE∗ | – | 69.3
CLIPGAP | 61.8 | 75.9
CLIPMAX | 63.7 | 77.5
MAE-CLIPGAP | 57.4 | 75.7
MAE-CLIPMAX | 60.9 | 76.6
Table 6: ImageNet classification after pretraining on _web-crawled_ dataset
(1.4B images). In the large scale regime, self-supervision does not complement
natural language supervision.

At small-scale, we showed that self-supervision combined with natural language
supervision marginally improves the quality of the learned visual
representations, and that the choice of image-encoder pooling strategy can
heavily influence results. We now investigate whether these conclusions still
hold when training on a much larger, $1.4$B example dataset.

###  5.1 Experimental Setup

We follow the configuration described in Section 4.1, with a few key
differences, noted below.

Pretraining Dataset: We combine a $2.2$B example web-crawled image-text pair
dataset, termed the English-Web-Image-Text dataset (EWIT-$2.2$B), with
LAION-$400$M [75], CC$3$M [78], CC$12$M [7] and an internal high-quality
image-text pair dataset containing approximately $134$M image text pairs which
we term the High Quality Image Text Pairs Dataset (HQITP-134M). We globally
deduplicate this—keyed by image bytes—reducing the number of image-text pairs
significantly, to yield a final $1.4$B examples. Details provided in the
supplementary material. In the rest of the paper we refer to this as the _web-
crawled_ or large-scale dataset.

Pretraining configuration: We increase the learning rate warm-up to $1,000$
steps, and compute a local contrastive loss for the first $10,000$ steps, as
during early training this improves the models rate of convergence. We train
for a total of $480,000$ steps, which corresponds to $6$ full passes through
the dataset. We do not increase the batch size, and use the same learning rate
schedule as for _CC_.

Model variants: We train M3AE as well as both GAP and MAX variants of CLIP and
MAE-CLIP, having noted in the small-scale regime that these are the highest
performing architectures and pre-training strategies. We do not train MAE at
scale due to resource limitations, and because its performance was
significantly lower than the models that utilize natural language
supervision—a finding that is consistent with prior literature [31].

(a) CLIP

(b) MAE-CLIP

(c) CLIPGAP

(d) MAE-CLIPGAP

(e) CLIPMAX

(f) MAE-CLIPMAX

Figure 2: Zero-shot segmentation masks (top row) and GradCAM [76]
visualizations (bottom row) for the prompt “a photo of a dog” for CLIP and
MAE-CLIP. Self-supervision qualitatively improves the visual grounding of the
model, however, the choice of pooling operator has the largest effect, with
max pooling producing significantly better results [72].

###  5.2 Image Classification

Table 15(a) provides encoder performance for both zero-shot and linear-probing
on ImageNet [19]. We note that at scale, MAE-CLIP typically shows worse
performance than CLIP across both pooling strategies and tasks. We hypothesize
that this may be due to the fact that masked-patch-prediction results in model
capacity being lost to the task of predicting semantically irrelevant patches,
meaning that we lose some of the benefits provided by the larger scale dataset
(despite tuning the relative loss weights). If we are correct in this
hypothesis then it is possible that a larger model capacity would help to
alleviate this issue. We leave this to a future investigation. Meanwhile, in
Table 14 we once again explore the encoder performance through linear-probing
for VTAB [107]. We note that GAP provides a consistent improvement over MAX
across both CLIP and MAE-CLIP models, and that M3AE is also a strong
performer—as was the case in the small-scale regime. M3AE provides particular
benefit to the “structured” VTAB tasks, however underperforms (sometimes by a
large margin) on the “natural” tasks. MAE-CLIP ends up in between the
two—doing better than CLIP but worse than M3AE on the structured tasks, and
worse than CLIP but better than M3AE on the natural tasks.

###  5.3 VQA

We follow the same finetuning strategy as before, with results in Table 15(b).
Notably, we do not see a consistent difference across tasks for CLIP vs MAE-
CLIP, with the possible exception of CLEVR [46], where it is clear that M3AE
is by far the best performing approach—and its benefits improve the MAE-CLIP
results on that task. GAP is perhaps a marginally stronger pooling strategy as
measured against MAX, however the difference is smaller than it was in the
small-scale regime.

Model | CLEVR | VQAv2 | GQA
---|---|---|---
M3AE | 96.9 | 59.9 | 53.3
CLIPGAP | 87.8 | 61.8 | 55.0
CLIPMAX | 89.5 | 60.6 | 53.6
MAE-CLIPGAP | 92.8 | 61.9 | 55.3
MAE-CLIPMAX | 93.9 | 61.5 | 53.7
Table 7: VQA finetuning results on _web-crawled_ dataset (1.4B images).
Similar to Table 15(a), for large pretraining datasets, self-supervision is
not improving the downstream VQA performance.

##  6 Analysis and Discussion

While our results present a pessimistic view on combining self-supervision
with natural language supervision, we don’t have a clear reason as to _why_
self-supervision fails at scale. Here we present two hypotheses that may be
explored in future work.

###  6.1 Visual Grounding

Visual grounding measures how well a particular representation or network can
_localize_ objects within an image. It is thought that representations with
better visual grounding will excel at more general purpose tasks. In section
4, we found that pooling operator had a larger effect on performance than the
addition of MAE. We qualitatively study the effect of both pooling and MAE on
the visual grounding of the CLIP image and text encoders. We extract zero-shot
segmentation masks and GradCAM [76] visualizations for the prompt “a photo of
a dog”, shown in Figure 2. Details are provided in the supplementary material.
We observe that, with respect to visual grounding, the pooling operator plays
a much more significant role than self-supervision [72]. In particular,
CLIPMAX correctly segments the dog in the picture while CLIP focuses only in
the background. However, both for CLIP and CLIPMAX we observe only an
incremental improvement of the localization of the dog in the image when self-
supervision is added. Unfortunately, even though the GAP variants of MAE-CLIP
and CLIP achieve the best performance on VTAB, they do not also exhibit the
most visual grounding. We hope that a more thorough future analysis can
elucidate the relationship between self supervision and visual grounding.

###  6.2 Dataset Diversity

Alternatively, it may be that self-supervision and natural language
supervision excel for entirely different parts of the dataset diversity-size
spectrum. Historically, the strongest self-supervised visual-encoder baselines
(when compared to supervised methods) include methods such as: SimCLR [11],
SimCLRV2 [12] and masked auto encoders [37], which all _excel_ on ImageNet—a
somewhat diverse but a relatively “clean” and object-centric dataset with 1000
disjoint classes. On less diverse datasets such as Cifar-10 [50], specialized
self supervised methods such as ReMixMatch [5] have shown excellent
performance in the extremely low data regime. Outside of vision, MAEs have
shown near state-of-the-art performance on datasets such as Audioset [30, 41],
a clean 2M example, 632 class audio/video dataset. Recent works which perform
controlled experiments scaling self supervised methods to massive datasets
[63] show that self-supervised methods achieve numbers roughly 10% worse (as
measured by ImageNet linear probe) as compared to natural language supervised
methods at similar data scale.

On the other hand, natural language supervised models seem to only show
competitive performance on massive and diverse datasets, CLIP [70] was trained
on 400M examples, ALIGN [44] was trained on 1.8B examples and BASIC [67] was
trained on 6.6B examples. Recently [28] performs controlled natural language
supervision at a smaller scale, showing that natural language supervised
models are 5-10% worse (on ImageNet) than their supervised counterparts.

While all of these past results only pose circumstantial and not rigorous
experimental evidence, we believe it poses a fruitful line of future work on
studying the scaling trends of self supervised methods and contrasting them
with other forms of supervision.

##  7 Acknowledgement

We are grateful to many colleagues for helpful advice and guidance throughout
this project. We especially thank Jonathan Shlens, Brandon McKinzie, and Jason
Ramuparam for feedback on paper drafts, Ruoming Pang for useful comments and
suggestions early in the project, Tom Nickson for his training infrastructure
support and Albin Madappally Jose for his assistance in building the training
dataset. We also extend thanks to Dr. Nicola Strisciuglio for his clarifying
input on earlier incarnations of this work, and Dr. Samuel Albanie for several
engaging discussions and helpful pointers to related literature.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a Visual Language Model for Few-Shot Learning.  Technical Report arXiv:2204.14198, arXiv, Apr. 2022.  arXiv:2204.14198 [cs] type: article.
  * [2] Tarik Arici, Mehmet Saygin Seyfioglu, Tal Neiman, Yi Xu, Son Train, Trishul Chilimbi, Belinda Zeng, and Ismail Tutar.  MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling.  arXiv:2109.12178 [cs], Sept. 2021.  arXiv: 2109.12178.
  * [3] Hangbo Bao, Li Dong, and Furu Wei.  BEiT: BERT Pre-Training of Image Transformers.  arXiv:2106.08254 [cs], June 2021.  arXiv: 2106.08254.
  * [4] Hangbo Bao, Wenhui Wang, Li Dong, and Furu Wei.  VL-BEiT: Generative Vision-Language Pretraining.  Technical Report arXiv:2206.01127, arXiv, June 2022.  arXiv:2206.01127 [cs] type: article.
  * [5] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  CoRR, abs/1911.09785, 2019.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  DINO: Emerging Properties in Self-Supervised Vision Transformers.  arXiv:2104.14294 [cs], May 2021.  arXiv: 2104.14294.
  * [7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts.  Technical Report arXiv:2102.08981, arXiv, Mar. 2021.  arXiv:2102.08981 [cs] type: article.
  * [8] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang.  Learning the best pooling strategy for visual semantic embedding.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  iGPT: Generative Pretraining From Pixels.  In Proceedings of the 37th International Conference on Machine Learning, pages 1691–1703. PMLR, Nov. 2020.  ISSN: 2640-3498.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  SimCLR: A Simple Framework for Contrastive Learning of Visual Representations.  In Proceedings of the 37th International Conference on Machine Learning, pages 1597–1607. PMLR, Nov. 2020.  ISSN: 2640-3498.
  * [11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  CoRR, abs/2002.05709, 2020.
  * [12] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  SimCLRv2: Big Self-Supervised Models are Strong Semi-Supervised Learners.  arXiv:2006.10029 [cs, stat], Oct. 2020.  arXiv: 2006.10029.
  * [13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  MoCov2: Improved Baselines with Momentum Contrastive Learning.  arXiv:2003.04297 [cs], Mar. 2020.  arXiv: 2003.04297.
  * [14] Xinlei Chen and Kaiming He.  SimSiam: Exploring Simple Siamese Representation Learning.  arXiv:2011.10566 [cs], Nov. 2020.  arXiv: 2011.10566.
  * [15] Xinlei Chen, Saining Xie, and Kaiming He.  MoCov3: An Empirical Study of Training Self-Supervised Vision Transformers.  arXiv:2104.02057 [cs], Aug. 2021.  arXiv: 2104.02057.
  * [16] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: UNiversal Image-TExt Representation Learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 104–120, Cham, Sept. 2020. Springer International Publishing.
  * [17] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  Technical Report arXiv:2003.10555, arXiv, Mar. 2020.  arXiv:2003.10555 [cs] type: article.
  * [18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, June 2009.  ISSN: 1063-6919.
  * [20] Karan Desai and Justin Johnson.  VirTex: Learning Visual Representations from Textual Annotations.  arXiv:2006.06666 [cs], Sept. 2021.  arXiv: 2006.06666.
  * [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805 [cs], May 2019.  arXiv: 1810.04805.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv:2010.11929 [cs], June 2021.  arXiv: 2010.11929.
  * [23] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng.  METER: An Empirical Study of Training End-to-End Vision-and-Language Transformers.  arXiv:2111.02387 [cs], Nov. 2021.  arXiv: 2111.02387.
  * [24] Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi.  Multi-modal Alignment using Representation Codebook.  arXiv:2203.00048 [cs], Mar. 2022.  arXiv: 2203.00048.
  * [25] Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave.  SplitMask: Are Large-scale Datasets Necessary for Self-Supervised Pre-training?  arXiv:2112.10740 [cs], Dec. 2021.  arXiv: 2112.10740.
  * [26] Patrick Esser, Robin Rombach, and Björn Ommer.  VQGAN: Taming Transformers for High-Resolution Image Synthesis.  arXiv:2012.09841 [cs], June 2021.  arXiv: 2012.09841.
  * [27] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The Pascal Visual Object Classes (VOC) Challenge.  International Journal of Computer Vision, 88(2):303–338, June 2010\.
  * [28] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip), 2022.
  * [29] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale, 2022.
  * [30] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776–780. IEEE, 2017.
  * [31] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel.  M3AE: Multimodal Masked Autoencoders Learn Transferable Representations.  Technical Report arXiv:2205.14204, arXiv, May 2022.  arXiv:2205.14204 [cs] type: article.
  * [32] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel.  Multimodal masked autoencoders learn transferable representations, 2022\.
  * [33] Aaron Gokaslan and Vanya Cohen.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * [34] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.  arXiv:1706.02677 [cs], Apr. 2018.  arXiv: 1706.02677.
  * [35] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  VQA2: Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering.  pages 6904–6913, 2017.
  * [36] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  BYOL: Bootstrap your own latent: A new approach to self-supervised Learning.  arXiv:2006.07733 [cs, stat], Sept. 2020.  arXiv: 2006.07733.
  * [37] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick.  Masked autoencoders are scalable vision learners.  CoRR, abs/2111.06377, 2021.
  * [38] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  MAE: Masked Autoencoders Are Scalable Vision Learners.  arXiv:2111.06377 [cs], Nov. 2021.  arXiv: 2111.06377.
  * [39] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  MoCov1: Momentum Contrast for Unsupervised Visual Representation Learning.  pages 9729–9738, 2020.
  * [40] Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales.  Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9068–9077, June 2022.
  * [41] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer.  Masked autoencoders that listen, 2022.
  * [42] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing Out of the Box: End-to-End Pre-Training for Vision-Language Representation Learning.  pages 12976–12985, Apr. 2021.
  * [43] Drew A. Hudson and Christopher D. Manning.  GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering.  pages 6700–6709, 2019.
  * [44] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  CoRR, abs/2102.05918, 2021.
  * [45] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision.  arXiv:2102.05918 [cs], June 2021.  arXiv: 2102.05918.
  * [46] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick.  CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988–1997, 2017.
  * [47] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection.  https://www.kaggle.com/c/diabetic-retinopathy-detection/data, July 2015.
  * [48] Wonjae Kim, Bokyung Son, and Ildoo Kim.  ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.  arXiv:2102.03334 [cs, stat], June 2021.  arXiv: 2102.03334.
  * [49] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  Technical Report arXiv:1412.6980, arXiv, Jan. 2017.  arXiv:1412.6980 [cs] type: article.
  * [50] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [51] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  arXiv:1910.13461 [cs, stat], Oct. 2019.  arXiv: 1910.13461.
  * [52] Chenliang Li, Ming Yan, Haiyang Xu, Fuli Luo, Wei Wang, Bin Bi, and Songfang Huang.  SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels.  arXiv:2103.07829 [cs], Mar. 2021.  arXiv: 2103.07829.
  * [53] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.  arXiv:2107.07651 [cs], Oct. 2021.  arXiv: 2107.07651.
  * [54] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  UNIMO-2: End-to-End Unified Vision-Language Grounded Learning.  arXiv:2203.09067 [cs], Mar. 2022.  arXiv: 2203.09067.
  * [55] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 121–137, Cham, July 2020. Springer International Publishing.
  * [56] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking, 2022.
  * [57] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  CoRR, abs/2110.05208, 2021.
  * [58] Feng Liang, Yangguang Li, and Diana Marculescu.  SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners.  Technical Report arXiv:2205.14540, arXiv, May 2022.  arXiv:2205.14540 [cs] type: article.
  * [59] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [60] Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic Gradient Descent with Warm Restarts.  arXiv:1608.03983 [cs, math], May 2017.  arXiv: 1608.03983.
  * [61] Ilya Loshchilov and Frank Hutter.  AdamW: Decoupled Weight Decay Regularization.  arXiv:1711.05101 [cs, math], Jan. 2019.  arXiv: 1711.05101.
  * [62] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  arXiv:1908.02265 [cs], Aug. 2019.  arXiv: 1908.02265.
  * [63] Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan.  A simple, efficient and scalable contrastive masked autoencoder for learning visual representations, 2022.
  * [64] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  SLIP: Self-supervision meets Language-Image Pre-training.  arXiv:2112.12750 [cs], Dec. 2021.  arXiv: 2112.12750.
  * [65] Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie.  SLIP: self-supervision meets language-image pre-training.  CoRR, abs/2112.12750, 2021.
  * [66] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation Learning with Contrastive Predictive Coding.  arXiv:1807.03748 [cs, stat], Jan. 2019.  arXiv: 1807.03748.
  * [67] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le.  Combined scaling for zero-shot transfer learning.  CoRR, abs/2111.10050, 2021.
  * [68] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models.  pages 2641–2649, 2015.
  * [69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  CLIP: Learning Transferable Visual Models From Natural Language Supervision.  arXiv:2103.00020 [cs], Feb. 2021.  arXiv: 2103.00020.
  * [70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  CoRR, abs/2103.00020, 2021.
  * [71] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  GPT-2: Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [72] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander T Toshev, and Jonathon Shlens.  Perceptual grouping in vision-language models, 2023.
  * [73] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating Diverse High-Fidelity Images with VQ-VAE-2.  Technical Report arXiv:1906.00446, arXiv, June 2019.  arXiv:1906.00446 [cs, stat] type: article.
  * [74] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
  * [75] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.  arXiv:2111.02114 [cs], Nov. 2021.  arXiv: 2111.02114.
  * [76] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.  International Journal of Computer Vision, 128(2):336–359, Feb. 2020\.  arXiv:1610.02391 [cs].
  * [77] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of ACL, 2018.
  * [78] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  GCC: Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, Melbourne, Australia, 2018. Association for Computational Linguistics.
  * [79] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  FLAVA: A foundational language and vision alignment model.  CoRR, abs/2112.04482, 2021.
  * [80] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  FLAVA: A Foundational Language And Vision Alignment Model.  arXiv:2112.04482 [cs], Feb. 2022.  arXiv: 2112.04482.
  * [81] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross B. Girshick, Piotr Dollár, and Laurens van der Maaten.  Revisiting weakly supervised pre-training of visual perception models.  CoRR, abs/2201.08371, 2022.
  * [82] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  arXiv:1908.08530 [cs], Feb. 2020.  arXiv: 1908.08530.
  * [83] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [84] Yihong Sun, Adam Kortylewski, and Alan Yuille.  Amodal Segmentation through Out-of-Task and Out-of-Distribution Generalization with a Bayesian Model.  Technical Report arXiv:2010.13175, arXiv, July 2022.  arXiv:2010.13175 [cs] type: article.
  * [85] Hao Tan and Mohit Bansal.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  arXiv:1908.07490 [cs], Dec. 2019.  arXiv: 1908.07490.
  * [86] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of machine learning research, 9(11), 2008.
  * [87] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [88] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning - ICML ’08, pages 1096–1103, Helsinki, Finland, 2008. ACM Press.
  * [89] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks, 2022.
  * [90] Xiaolong Wang and Abhinav Gupta.  Unsupervised Learning of Visual Representations using Videos.  Technical Report arXiv:1505.00687, arXiv, Oct. 2015.  arXiv:1505.00687 [cs] type: article.
  * [91] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.  arXiv:2108.10904 [cs], Aug. 2021.  arXiv: 2108.10904.
  * [92] Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian.  MVP: Multimodality-guided Visual Pre-training.  arXiv:2203.05175 [cs], Mar. 2022.  arXiv: 2203.05175.
  * [93] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  * [94] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick.  Early Convolutions Help Transformers See Better.  Technical Report arXiv:2106.14881, arXiv, Oct. 2021.  arXiv:2106.14881 [cs] type: article.
  * [95] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  Pre-layernorm: On Layer Normalization in the Transformer Architecture.  In Proceedings of the 37th International Conference on Machine Learning, pages 10524–10533. PMLR, Nov. 2020.  ISSN: 2640-3498.
  * [96] Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, and Jianfeng Gao.  Focal modulation networks, 2022.
  * [97] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: Fine-grained Interactive Language-Image Pre-Training.  arXiv:2111.07783 [cs], Nov. 2021.  arXiv: 2111.07783.
  * [98] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph.  June 2020.
  * [99] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models, 2022.
  * [100] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  CoCa: Contrastive Captioners are Image-Text Foundation Models.  arXiv:2205.01917 [cs], May 2022.  arXiv: 2205.01917.
  * [101] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.  Florence: A New Foundation Model for Computer Vision.  arXiv:2111.11432 [cs], Nov. 2021.  arXiv: 2111.11432.
  * [102] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin.  Patch-level Representation Learning for Self-supervised Vision Transformers.  Technical Report arXiv:2206.07990, arXiv, June 2022.  arXiv:2206.07990 [cs] type: article.
  * [103] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction.  arXiv:2103.03230 [cs, q-bio], June 2021.  arXiv: 2103.03230.
  * [104] Yan Zeng, Xinsong Zhang, and Hang Li.  X:VLM: Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts.  arXiv:2111.08276 [cs], Feb. 2022.  arXiv: 2111.08276.
  * [105] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12104–12113, June 2022.
  * [106] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  The visual task adaptation benchmark.  CoRR, abs/1910.04867, 2019.
  * [107] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  VTAB: A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark.  arXiv:1910.04867 [cs, stat], Feb. 2020.  arXiv: 1910.04867.
  * [108] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  ADE20K: Semantic Understanding of Scenes through the ADE20K Dataset.  arXiv:1608.05442 [cs], Oct. 2018.  arXiv: 1608.05442.

##  Appendix A Supplementary Overview

In what follows we first provide more exhaustive details on our training and
evaluation datasets in Section B, before then detailing our pre-training
configuration in C. Section D shows the effect of masking CLIP input in the
low-data regime. Sections E, F and G show more results that were used to
compare MAE-CLIP and CLIP. In Section H we explain our VQA evaluation setup,
and in Section I we cover zero-shot segmentation. Finally, Section J gives
background on our choice of “similarity masking” (as described in the main
paper) over random masking, and Section K provides updated results for our
large-scale M3AE [31] run, as at the time of submission we were unable to
provide results for a fully trained baseline.

##  Appendix B Datasets

In this section we describe our pre-training dataset, and also provide
background as to why there are two missing tasks in our VTAB [107] evaluation.

###  B.1 Pre-training Data

We made use of two internal datasets as well as several public datasets to
build our “large-scale” pre-training dataset.

High Quality Image Text Pairs dataset: The High Quality Image Text Pairs
(HQITP-134M) dataset consists of approximately $134$M diverse and high quality
images paired with descriptive captions and titles. Images range in spatial
resolution from 320 to 2048 pixels on the short side. All images are JPEG
format and most are RGB. Each example image is associated with a title, and a
list of several captions. A small fraction ($\ll$ $1\%$) of the examples are
missing both captions and title. We favor the associated captions, and find
that these tokenize to an average length of $20.1$ tokens, although the
shortest caption is only one token and the longest is over $1,000$. This
dataset was licensed to our industrial research lab by a third party for
commercial use.

English Web Image Text dataset: The English-Web-Image-Text-2.2B (EWIT-2.2B)
dataset consists of approximately $2.2$B images paired with one or more
related pieces of text. The data is the result of filtering English-language
web-sourced data, using a combination of the filtering rules described in
ALIGN [45] and CLIP [69]. Images range in spatial resolution from $200$ to
$5000$ pixels on the short side, and $200$ to $8650$ on the long side, with a
maximum aspect ratio of $3$ and a mean of $1.385$. Each image has an average
of $1.341$ pieces of text associated with it, although some have as many as
$179$. We find that the average associated text produces $15.5$ tokens when
tokenized.

Public datasets: As well as our internal datasets, we include Conceptual $12$M
(CC12M) [7], CC3M [77], and LAION-$400$M [75].

Overall pre-training dataset: Our overall training dataset is the result of
combining HQITP-134M, CC12M, CC3M, and LAION-$400$M, before applying global
image-byte-level de-duplication to drop image text pairs where either the
image occurs more than once or the image occurs in one of our test sets. This
results in a final training dataset of just over $1.4$B image-text pairs.

###  B.2 VTAB Evaluation Data

In Sections 4 and 5 in the main paper we evaluate the quality of the learned
visual representations by training a linear classifier on the predicted visual
features of the VTAB datasets [106]. However, we do not include the Diabetic
Retinopathy [47] dataset due to licensing concerns (the original dataset was
provided solely for use in a Kaggle competition), and Sun397 [93], due to a
missing image at the time of preparing the datasets for the VTAB benchmark.
The issue with Sun397 has since then been resolved, and it could be included
in a future iteration of this work.

##  Appendix C Training configuration

Component | Parameter | Value
---|---|---
Image Encoder | Depth | 12
| Width | 768
MLP Heads | 12
Text Encoder | Depth | 12
| Width | 512
MLP Heads | 8
Decoder | Depth | 8
| Width | 512
| MLP Heads | 8
Model | Weight decay | 0.1
| Base LR | 5e-04
| LR Schedule |  Cosine decay [60]
| LR Warmup steps | 200
|  Local contrastive steps [34] | 500
| Batch size | 256
| Optimizer |  AdamW [61]
| Optimizer momentum |  $\beta$1 , $\beta$2 = 0.9, 0.98
| Augmentation | RandomResizedCrop
Table 8: Parameters used for pre-training

Table 8 shows the used hyperparameter setup for pre-training of MAE-CLIP and
all our baselines. Following [69], we use an AdamW optimizer [49, 61], a
linear learning rate warmup over $200$ steps before then decaying to $0$ with
a cosine schedule [60] over the remainder of training. Using warmup steps,
(where only a local contrastive loss is used instead of a global contrastive
loss), helps the model to converge faster at the beginning of training. To
train MAE-CLIP, we simply sum the contrastive and generative losses for the
local contrastive phase of training, but multiply the generative image loss by
$0.05$ and the generative text loss by $0.1$ when computing the global
contrastive loss. This allows us to accommodate for the dramatically reduced
gradient norm of a global contrastive loss with a large batch size, and was
arrived at through hyperparameter search. For the image encoder, initial
empirical experimentation showed that using a trainable or fixed position
encoding does not influence results and we therefore use a fixed 2D position
encoding. For both image and text encoders, we use pre-layer-norm [95] and the
initialization scheme from [71].

We use the same training configuration for the _web-crawled_ dataset, with a
few changes due to the larger number of total steps. We use $10,000$ local
contrastive loss steps, and $1,000$ warmup steps for the cosine learning rate
scheduler.

##  Appendix D Masking influence

In MAE-CLIP, we never mask the input for the contrastive task. However, one
can argue that in the low-data regime, masking is a heavy data augmentation
that improves performance (similar to how e.g. DINO [6] has great performance
in the low-data regime, likely also because of its heavy data augmentations).
We therefore also run an ablation where the input to a contrastive-only
(normal CLIP) model is masked, similar to how input is masked for the
generative task in MAE-CLIP.

Models | Zero-shot | Linear Probing
---|---|---
masked CLIPMAP | 23.0 (29.7) | 48.0 (52.6)
masked CLIPGAP | 23.6 (29.3) | 51.7 (59.8)
MAE-CLIP | 33.8 | 58.9
Table 9: ImageNet classification with zero-shot transfer or linear probing
after pretraining on the _CC_ dataset. CLIP is trained on masked input,
showing unmasked performance between brackets.

##  Appendix E Retrieval results

Table 10 shows zero-shot retrieval accuracy for three different datasets. We
compare COCO [59], FLICKR [68] and COCOAmodal [84]. We chose COCOAmodal as a
third retrieval evaluation set, as adding the masked auto-encoder might have
provided MAE-CLIP a benefit over CLIP on occluded objects or non-object-
centric datasets. We show that even on non-object-centric datasets, CLIP
outperforms MAE-CLIP at scale.

Model | COCO | FLICKR | COCOA
---|---|---|---
I$\rightarrow$T | T$\rightarrow$I | I$\rightarrow$T | T$\rightarrow$I | T1 | T5
CLIPGAP | 51.9 | 36.6 | 78.8 | 62.3 | 24.2 | 46.9
CLIPMAX | 55.3 | 39.0 | 80.5 | 65.3 | 22.7 | 51.6
MAE-CLIPGAP | 53.0 | 37.0 | 77.3 | 62.0 | 20.7 | 39.5
MAE-CLIPMAX | 54.4 | 37.7 | 81.2 | 64.2 | 24.6 | 41.4

Table 10: Zero-shot (retrieval) accuracy (%). All trained on our _web-crawled_
dataset (1.4B images). I$\rightarrow$T: Image to Text, T$\rightarrow$I: Text
to Image.

##  Appendix F VTAB Fine-tuning

|  $\bullet$ Caltech101 |  $\bullet$ CIFAR-100 |  $\bullet$ DTD |  $\bullet$ Flowers102 |  $\bullet$ Pets |  $\bullet$ SVHN |  $\bullet$ EuroSAT |  $\bullet$ Camelyon |  $\bullet$ Resisc45 |  $\bullet$ Clevr/Closest |  $\bullet$ Clevr/Count |  $\bullet$ DMLab |  $\bullet$ dSprites/Ori |  $\bullet$ dSprites/Loc |  $\bullet$ KITTI/Dist |  $\bullet$ sNORB/Azim |  $\bullet$ sNORB/Elev |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MAE-CLIPGAP | 92.6 | 84.2 | 74.4 | 95.1 | 85.7 | 80.2 | 97.0 | 88.0 | 92.9 | 87.1 | 84.3 | 71.0 | 96.4 | 100 | 46.8 | 99.0 | 95.0 | 86.5
MAE-CLIPMAX | 95.9 | 86.3 | 80.3 | 97.5 | 90.4 | 97.3 | 98.9 | 88.8 | 95.6 | 87.1 | 84.1 | 72.7 | 96.1 | 100 | 53.4 | 99.3 | 96.6 | 89.4
CLIPGAP | 96.0 | 87.7 | 81.8 | 98.1 | 90.7 | 97.4 | 98.8 | 85.7 | 95.9 | 87.8 | 80.0 | 72.5 | 96.5 | 100 | 51.0 | 98.7 | 93.6 | 88.9
CLIPMAX | 95.5 | 86.3 | 81.1 | 98.0 | 90.6 | 97.6 | 98.9 | 89.6 | 96.2 | 87.6 | 80.1 | 73.6 | 96.2 | 100 | 52.9 | 99.4 | 95.9 | 89.4
MAE-CLIPGAP _CCxM_ | 91.9 | 80.9 | 70.5 | 91.2 | 82.7 | 96.3 | 98.5 | 88.0 | 95.5 | 79.6 | 67.7 | 66.0 | 96.3 | 100 | 49.6 | 99.7 | 85.9 | 84.7
MAE-CLIPMAX _CCxM_ | 90.8 | 81.6 | 69.0 | 90.3 | 81.0 | 96.8 | 98.4 | 89.5 | 95.3 | 69.0 | 74.3 | 67.4 | 956 | 100 | 48.0 | 98.7 | 83.6 | 84.1
CLIPGAP _CCxM_ | 91.1 | 81.3 | 69.7 | 91.8 | 81.1 | 96.2 | 98.1 | 85.9 | 95.3 | 75.1 | 65.1 | 64.6 | 96.4 | 100 | 48.6 | 99.6 | 78.0 | 83.4
CLIPMAX _CCxM_ | 89.6 | 80.6 | 68.4 | 90.3 | 79.8 | 96.5 | 98.4 | 87.1 | 95.3 | 78.3 | 72.2 | 64.0 | 96.0 | 100 | 47.3 | 99.8 | 76.3 | 83.5
Table 11: Full-finetuning accuracy (%) on classification tasks. Models are all
trained on our _web-crawled_ dataset (1.4B images) or CCxM when specified.
$\bullet$ VTAB/natural, $\bullet$ VTAB/specialized and $\bullet$
VTAB/structured.

As MAE is often used in a full-finetuning setting, we also show that fully-
finetuning follows the linear-probing results (see Table 11). Adding masked
autoencoding still does not outperform a contrastive-only baseline in a large-
scale training. When training on CCxM only, we see MAE-CLIP outperforming CLIP
on average.

##  Appendix G T-SNE analysis

We also visually inspect the generated embeddings using t-SNE [86], see Figure
3. There is no clear difference between the embeddings of CLIP and MAE-CLIP,
at scale (see Figure 3(a) and Figure 3(b)). For CLIP, MAE-CLIP and MAE trained
on CCMxM, we can see that the MAE embeddings (see Figure 3(e)) look more
cluttered than the CLIP and MAE-CLIP embeddings.

(a) CLIP

(b) MAE-CLIP

(c) CLIP _CCxM_

(d) MAE-CLIP _CCxM_

(e) MAE _CCxM_

Figure 3: T-sne visualizations for CLIP and MAE-CLIP, both large-scale
training as CCXM.

##  Appendix H VQA Finetuning

We evaluate on three VQA benchmark datasets CLEVR [46], VQAv2 [35] and GQA
[43]. As mentioned in the main paper, we finetune our models for VQA, by
freezing the image and texts encoders and adding a new decoder. We treat the
problem as a classification problem by calculating the set of possible answers
and treating each as a separate class. Following our pre-training setup, we
concatenate the image and text embeddings, add positional encoding and a
modality specific token before using it as input in the decoder. The `BOS`
token is used as an output token and linearly projected to the possible
classes. Figure 4 depicts our finetuning pipeline

Figure 4: VQA Fine-tuning: decoder classification. Encoders are frozen, the
BOS token of the decoder output is used as the classification token.

##  Appendix I Zero-shot Segmentation

In Section 6 in the main paper, we present results on zero-shot semantic
segmentation in order to evaluate the effects of self-supervision on visual
grounding. Here, we describe our zero-shot semantic segmentation methodology
and present a quantitative evaluation on three datasets. In particular, we use
Pascal VOC [27], ADE20K [108] and COCO [59] with 20, 150 and 133 labels
respectively. To compute segmentation masks, we first extract a feature per
input pixel using bilinear interpolation from the per-patch features.
Subsequently, we classify each pixel by computing the similarity of the
feature to the embedding of the prompt for each class.

Model | Pooling | COCO | ADE20K | Pascal VOC
---|---|---|---|---
CLIP | MAP | 8.4 | 4.6 | 19.1
MAE-CLIP | MAP | 9.1 | 5.7 | 20.6
CLIP | GAP | 7.6 | 4.1 | 19.4
MAE-CLIP | GAP | 8.5 | 5.5 | 20.4
CLIP | MAX | 16.5 | 9.4 | 36.6
MAE-CLIP | MAX | 17.8 | 11.1 | 36.9
Table 12: Zero-shot semantic segmentation results for CLIP and MAE-CLIP after
training on the CC dataset ($11.3$M images). MAE-CLIP consistently improves
upon CLIP for semantic segmentation regardless of the pooling strategy, as
also seen qualitatively in Figure 2 in the main paper.

Table 12 evaluates the performance of CLIP and MAE-CLIP trained on the CC
dataset with respect to mean intersection over union for all three datasets.
We observe that self-supervision consistently improves the performance of CLIP
for zero-shot semantic segmentation. However, as mentioned in Section 6 in the
main paper, the choice of pooling operator has a much larger effect.

###  I.1 Prompts

In this section, we provide the prompts used for our zero-shot semantic
segmentation experiments. Each prompt is made into a sentence by prepending “a
photo of a” or “an” depending on whether the label starts with a vowel. For
COCO [59] we simply use the 133 label names as they are provided by
https://github.com/cocodataset/panopticapi. For Pascal VOC [27] we use

> aeroplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, table,
> dog, horse, motorbike, person, potted plant, sheep, sofa, train, television
> monitor

for classes $1$ to $20$ and

> background, bag, bed, bench, book, building, cabinet, ceiling, cloth,
> computer, cup, door, fence, floor, flower, food, grass, ground, keyboard,
> light, mountain, mouse, curtain, platform, sign, plate, road, rock, shelves,
> sidewalk, sky, snow, bedclothes, track, tree, truck, wall, water, window,
> wood

for the 0-th class. Namely, if a pixel is classified as any of the latter
categories it is considered to be a background pixel. Finally, for ADE20K
[108] we associate several prompots for each of the 150 categories.
Subsequently, we compute the similarity of each pixel with each of the prompts
and select the maximum similarity per category from all the associated
prompts. The per-category prompts are as follows:

1. wall, walls, brick wall, stone wall, interior wall 2. building, buildings, edifice, edifices 3. sky, clouds 4. floor, flooring 5. tree, trees 6. ceiling 7. road, route, street, roads, streets, routes 8. bed, beds 9. windowpane, window, windows 10. grass, grass field 11. cabinet, cabinets, wall mounted cabine 12. sidewalk, pavement 13. person, child, girl, boy, woman, man, people, children, girls, boys, women, men 14. earth, ground 15. door, double door, doors 16. table, tables, tablecloth 17. mountain, mount, mountains 18. plant, flora, plant life, plants, bushes 19. curtain, drape, drapery, mantle, pall 20. chair, chairs 21. car, automobile, cars 22. water 23. painting, picture, paintings, pictures, wallart, framed canvas 24. sofa, couch, sofas, couches 25. shelf, shelves 26. house exterior 27. sea, ocean 28. mirror, mirrors 29. rug, carpet, carpeting 30. field 31. armchair, armchairs 32. seat, seats 33. fence, fencing 34. desk, desks 35. rock, stone, rocks, stones 36. wardrobe, closet, press, wardrobes, closets 37. lamp, lamps 38. bathtub, bathing tub, bath, tub 39. railing, rail 40. cushion, cushions 41. pedestal 42. box, boxes 43. column, pillar 44. signboard, sign, signboards, signs 45. chest of drawers, chest, bureau, dresser 46. counter 47. sand 48. sink 49. skyscraper, skyscrapers 50. fireplace, hearth, open fireplace 51. refrigerator, icebox 52. grandstand, covered stand 53. path 54. stairs, steps 55. runway 56. case, display case, showcase, vitrine 57. pool table, billiard table, snooker table 58. pillow, pillows 59. screen door, shower door 60. stairway, staircase 61. river 62. bridge, span 63. bookcase 64. window screen, door screen 65. coffee table, cocktail table 66. toilet, commode, crapper, potty 67. flower, flowers 68. book, books 69. hill 70. bench, benches 71. countertop, counter top, worktop 72. stove, kitchen stove, kitchen range, kitchen range, cooking stove 73. palm tree, palm trees 74. kitchen island 75. computer, computing machine, computing device, data processor, electronic computer, information processing system 76. swivel chair 77. boat 78. bar 79. arcade machine, arcade machines 80. hovel, hut, hutch, shack, shanty 81. bus, autobus, double-decker, jitney, motorbus, motorcoach, omnibus, passenger vehicle 82. towel 83. light bulb, lightbulb, bulb, incandescent lamp, electric light, electric-light bulb 84. truck, motortruck 85. tower, towers 86. chandelier, pendant, pendent 87. awning, sunshade, sunblind 88. streetlight, street lamp 89. booth, cubicle, stall, kiosk 90. television receiver, television, television set, tv, tv set 91. airplane, aeroplane, airplanes, aeroplanes 92. dirt track 93. apparel, wearing apparel, dress, clothes 94. pole 95. land, soil 96. bannister, banister, balustrade, balusters, handrail 97. escalator, moving staircase, moving stairway 98. ottoman, pouf, pouffe, puff, hassock 99. bottle, bottles, water bottle 100. buffet, sideboard 101. poster, posting, placard, notice, bill, card 102. stage 103. van 104. ship 105. fountain 106. conveyer belt, conveyor belt, conveyer, conveyor, transporter 107. canopy 108. washer, automatic washer, washing machine 109. plaything, toy, toys 110. swimming pool, swimming bath 111. stool, stools 112. barrel, cask, barrels, casks 113. basket, handbasket 114. waterfall, falls 115. tent, collapsible shelter 116. bag, bags, gift bag, paper bag 117. minibike, motorbike 118. cradle 119. oven 120. ball, balls 121. food, solid food 122. step, stair 123. tank, storage tank 124. trade name, brand name, brand, marque 125. microwave, microwave oven 126. plant pots, plant pot, flower pot, flowerpot, planter 127. animal, animate being, dog, cat, horse, cow, sheep, zebra, girraffe, bird 128. bicycle, bike 129. lake 130. dishwasher, dish washer, dishwashing machine 131. projection screen 132. blanket, cover 133. sculpture, sculptures 134. exhaust hood 135. sconce, sconce lamp, sconce light 136. vase, vases 137. traffic light, traffic signal, traffic lights 138. tray, trays 139. ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin 140. ceiling fan, floor fan 141. pier, wharf, wharfage, dock 142. crt screen 143. plate, plates 144. monitor, monitoring device, monitors 145. bulletin board, notice board 146. shower 147. radiator 148. cup, cups, drinking glass, drinking glasses 149. clock 150. flag, flags

##  Appendix J MAE-CLIP masking strategy

In Table 13 we compare random masking to similarity masking for MAE-CLIPMAX,
training on the CC dataset. Our experiments show that both strategies perform
very similarly in all cases with random masking showing a small improvement
for classification tasks while similarity masking an improvement on VQA and
semantic segmentation, namely tasks that benefit from better visual grounding.
All MAE-CLIP experiments in the main paper employ similarity masking. Further
experiments are needed to properly evaluate the effect of the masking strategy
across different scales and pooling methods.

Model | Masking | VQAAvg. | SemSegAvg. | VTABAvg | IN1KLP | INZS
---|---|---|---|---|---|---
MAE-CLIPMAX | similarity | 68.53 | 21.95 | 69.14 | 63.16 | 35.2
MAE-CLIPMAX | random | 68.44 | 21.27 | 69.92 | 63.46 | 35.4
Table 13: Similarity vs random masking for MAE-CLIPMAX trained on the CC
dataset. We show average VQA, zero-shot semantic segmentation (SemSeg), and
VTAB results, as well as ImageNet1K (IN) Top-1 linear probe and zero-shot
scores as measured on the validation set.

##  Appendix K M3AE results

This section provides an updated version of the “large-scale” tables of
results from Section 5 in the main paper. They are presented in Tables 14,
15(a) and 15(b). We include these because our M3AE [31] baseline had not fully
converged at time of submission. All numbers except those associated with M3AE
are identical to the ones presented in the main paper.

Firstly, we report the full linear probing evaluation on the VTAB tasks (Table
14). We observe that M3AE performs on par with MAE-CLIPGAP. Moreover, we note
that M3AE performs consistently worse for all VTAB natural tasks while
outperforming MAE-CLIP on VTAB strucuted tasks. This trend also continues with
the rest of the results, where M3AE performs measurably worse on ImageNet
linear-classification (Table 15(a)) while siginficantly better on CLEVR VQA
(Table 15(b)); the former being a very natural task and the latter being very
structured.

Table 14: Linear probing accuracy (%) on classification tasks. Models are all trained on our _web-crawled_ dataset (1.4B images). ($\bullet$ VTAB/natural, $\bullet$ VTAB/specialized and $\bullet$ VTAB/structured.) In the large scale pretraining regime, the difference between MAE-CLIP and CLIP is reduced to $<1\%$. This table provides the results for a fully trained M3AE compared to Table 5 in the main paper where M3AE had completed 50% of the training steps. |  $\bullet$ Caltech101 |  $\bullet$ CIFAR-100 |  $\bullet$ DTD |  $\bullet$ Flowers102 |  $\bullet$ Pets |  $\bullet$ SVHN |  $\bullet$ EuroSAT |  $\bullet$ Camelyon |  $\bullet$ Resisc45 |  $\bullet$ Clevr/Closest |  $\bullet$ Clevr/Count |  $\bullet$ DMLab |  $\bullet$ dSprites/Ori |  $\bullet$ dSprites/Loc |  $\bullet$ KITTI/Dist |  $\bullet$ sNORB/Azim |  $\bullet$ sNORB/Elev |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
M3AE | 94.0 | 75.6 | 78.8 | 96.1 | 84.6 | 67.8 | 97.2 | 85.3 | 92.4 | 65.4 | 75.9 | 52.0 | 57.4 | 79.2 | 51.1 | 40.6 | 68.3 | 74.2
CLIP | 94.9 | 78.4 | 80.0 | 97.3 | 86.9 | 59.0 | 94.1 | 82.3 | 92.7 | 45.6 | 62.1 | 46.0 | 46.1 | 53.3 | 50.9 | 20.3 | 35.8 | 66.2
CLIPMAX | 96.1 | 81.0 | 80.9 | 97.3 | 89.9 | 65.7 | 96.0 | 83.2 | 94.1 | 52.8 | 67.8 | 49.9 | 59.5 | 67.6 | 41.2 | 23.4 | 45.8 | 70.1
MAE-CLIPMAX | 95.8 | 79.2 | 81.5 | 96.8 | 88.2 | 62.1 | 95.8 | 81.8 | 93.0 | 52.0 | 66.9 | 49.6 | 53.7 | 72.5 | 53.0 | 32.3 | 45.4 | 70.6
CLIPGAP | 95.8 | 80.5 | 81.6 | 97.6 | 88.7 | 66.0 | 97.0 | 84.4 | 93.3 | 56.7 | 71.4 | 53.3 | 58.0 | 70.1 | 50.6 | 38.3 | 55.1 | 72.9
MAE-CLIPGAP | 95.4 | 79.3 | 82.2 | 97.4 | 88.6 | 72.8 | 96.6 | 84.5 | 93.5 | 57.5 | 73.6 | 52.7 | 57.5 | 71.2 | 51.6 | 45.6 | 55.2 | 73.8

Models | Zero-shot | Linear Probing
---|---|---
M3AE | – | 71.5
CLIPGAP | 61.8 | 75.9
CLIPMAX | 63.7 | 77.5
MAE-CLIPGAP | 57.4 | 75.7
MAE-CLIPMAX | 60.9 | 76.6
(a) ImageNet classification

Model | CLEVR | VQAv2 | GQA
---|---|---|---
M3AE | 97.2 | 61.1 | 54.6
CLIPGAP | 87.8 | 61.8 | 55.0
CLIPMAX | 89.5 | 60.6 | 53.6
MAE-CLIPGAP | 92.8 | 61.9 | 55.3
MAE-CLIPMAX | 93.9 | 61.5 | 53.7
(b) VQA finetuning results

Table 15: ImageNet classification and VQA results after pretraining on _web-
crawled_ dataset (1.4B images). In the large scale regime, self-supervision
does not complement natural language supervision and all methods perform
similarly on both tasks. This table provides updated results for M3AE (after
it was fully trained) compared to Tables 6 and 7 in the main paper.