# Mixed Autoencoder for Self-supervised Visual Representation Learning

Kai Chen1∗ Zhili Liu1,2∗ Lanqing Hong2 Hang Xu2 Zhenguo Li2 Dit-Yan Yeung1
1Hong Kong University of Science and Technology 2Huawei Noah’s Ark Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Masked Autoencoder (MAE) has demonstrated superior performance on various
vision tasks via randomly masking image patches and reconstruction. However,
effective data augmentation strategies for MAE still remain open questions,
different from those in contrastive learning that serve as the most important
part. This paper studies the prevailing mixing augmentation for MAE. We first
demonstrate that naïve mixing will in contrast degenerate model performance
due to the increase of mutual information (MI). To address, we propose
homologous recognition, an auxiliary pretext task, not only to alleviate the
MI increasement by explicitly requiring each patch to recognize homologous
patches, but also to perform object-aware self-supervised pre-training for
better downstream dense perception performance. With extensive experiments, we
demonstrate that our proposed Mixed Autoencoder (MixedAE) achieves the state-
of-the-art transfer results among masked image modeling (MIM) augmentations on
different downstream tasks with significant efficiency. Specifically, our
MixedAE outperforms MAE by +0.3% accuracy, +1.7 mIoU and +0.9 AP on
ImageNet-1K, ADE20K and COCO respectively with a standard ViT-Base. Moreover,
MixedAE surpasses iBOT, a strong MIM method combined with instance
discrimination, while accelerating training by 2$\times$. To our best
knowledge, this is the very first work to consider mixing for MIM from the
perspective of pretext task design. Code will be made available.

†† ∗Equal contribution.

##  1 Introduction

Figure 1: Fine-tuning accuracy on ImageNet-1K. Our MixedAE achieves the best
trade-off between pre-training overhead and transfer performance.
Specifically, MixedAE surpasses MAE [27] consistently with only 3% extra
overhead, while outperforms the strong iBOT [63] with only 53.4% of its
computation overhead. See more detailed comparisons in Tab. 1. ID stands for
instance discrimination, while MIM represents masked image modeling.

Self-supervised learning (SSL) has become one of the most popular pre-training
paradigm due to its independence of human annotation. Previous literature
mainly focuses on the handcrafted pretext task design [16, 46, 23] and
instance discrimination [11, 7], while with the development of Vision
Transformer [18], masked image modeling (MIM), deeply motivated by masked
language modeling [15], has started to demonstrate more superior effectiveness
by firstly masking some patches of the input images and then reconstructing
the masked patches from visible ones by predicting certain targets generated
by masked patches. In order to complete reconstruction, the encoder is
expected to generate highly semantic representation which can be better
transferred to downstream tasks [62, 37, 26, 36] for superior performance.

Existing MIM works mainly concentrate on the design of the reconstruction
targets (_e.g_., visual tokenizers [3, 17], pixels [27, 57], graphical
features [53] and instance discrimination [19, 63, 2]) and masking strategies
(_e.g_., random [3, 27], attention-guide [31] and sample-dependent [49]). See
more detailed discussions in Sec. 2. Despite the superior performance, we
observe that the input augmentations for MIM have been seldom explored.
Specifically, adding color jittering, an essential augmentation technique of
contrastive learning [9], with MAE [27] even degrades transfer results,
suggesting that MIM might possess a different preference for data
augmentations, and the effective data augmentation strategies for MIM are
still an open question.

In this paper, we explore the usage of image mixing, a commonly used technique
in both supervised [60, 59] and contrastive learning [48, 58], with MAE [27].
We start by constructing a simple baseline to adopt mixing with MAE directly,
which, different from in supervised and contrastive learning, would instead
ease the reconstruction pretext by increasing the mutual information between
the model input and reconstruction target due to the usage of image mixing
with global self-attention as proved in Sec. 3.1. To address this issue, we
propose homologous recognition, an auxiliary pretext task to enforce each
patch to recognize homologous patches explicitly according to attention
distributions before reconstruction, and build our Mixed Autoencoder network
(MixedAE) in Sec. 3.3. Moreover, we demonstrate that our simple yet effective
method can not only achieve significant performance improvement, but also
conduct object-aware SSL pre-training without any specifically designed
modules for better downstream dense perception results in Sec. 3.4.

Concurrently, MixMIM [38] also considers mixing with MAE, but different from
ours, 1) Purpose: MixMIM uses mixing to recover the 2D structure after random
masking for an efficient implementation to conduct MAE-style pre-training on
hierarchy Vision Transformers [40], while ours utilizes mixing to conduct
object-aware SSL pre-training for better representation learning. 2) Method:
MixMIM uses masked self-attention to only perform attention within patches
from the same images given the mixing masks as input, sharing the exactly same
pretext task with MAE, while ours requires explicit homologous recognition
given mixing masks as target, actively emerging mixing into the pretext
design. 3) Formulation: The mixing ratio $r$ is limited to 0.5 in MixMIM,
which instead can be flexibly selected from $(0,0.5]$ in our formulation. See
more details in Sec. 3.

The main contributions of this work contain three parts:

  1. 1.

We propose the Mixed Autoencoder (MixedAE), a simple yet effective approach to
conduct object-aware pre-training without introducing any specifically
designed modules. With extensive experiments, we demonstrate that MixedAE can
achieve the state-of-the-art transfer performance on various downstream tasks
including image classification, semantic segmentation and object detection,
while maintaining significant efficiency.

  2. 2.

We theoretically demonstrate the underlying design differences between MIM and
previous supervision with mixing (_e.g_., supervised and contrastive
learning).

  3. 3.

To our best knowledge, this is the first work to consider mixing as an
effective data augmentation strategy for MIM from the perspective of pretext
design with a pure autoencoder-based architecture.

Figure 2: Model architecture of Mixed Autoencoder (MixedAE). (a) The input
images are first separated into groups to generate mixed samples
independently, which are further taken as input to the encoder for feature
extraction. (b) The self-attention operations are replaced with our homologous
attention, enforcing each patch to only attend to patches with the highest
attention mass. (c) The encoder features will be “unmixed” and fed into the
decoder for pixel reconstruction. (d) Meanwhile, the homologous contrastive
loss is adopted to verify the sampling accuracy by encouraging features of
homologous patches to be similar, while heterologous ones to be dissimilar.

##  2 Related Work

Self-supervised learning aims at learning a transferable representation
without human annotation. Previous works mainly focus on handcrafted pretext
design [16, 46, 23] and instance discrimination [9, 10, 25]. Mask image
modeling (MIM), inspired by the mask language modeling [15], has achieved
significant performance with superior pre-training efficiency by firstly
masking portion of an image, and then reconstructing the masked part based on
the visible one.

#### Reconstruction target.

BEiT [3] pioneeringly proposes to predict visual tokens generated by a pre-
trained visual tokenizer [47], which is simplified by SimMIM [57] to use pixel
values as the reconstruction target directly. MAE [27] proposes an asymmetric
encoder-decoder architecture for better efficiency. Besides pixels as the
target, MaskFeat [53] utilizes HOG features, while PeCo [17] enhances the BEiT
tokenizer with an additional perceptual loss. Recent works combine the idea of
instance discrimination [9] with MIM. iBOT [63] considers MIM as a self-
distillation process with a Siamese architecture, and data2vec [2] proposes a
unified framework to conduct masked reconstruction pre-training for speech,
images and languages. SplitMask [19] divides an image into two equal
partitions and performs contrastive learning and MIM in the multi-task manner.
In this paper, we build MixedAE based on MAE due to its efficiency and
effectiveness, while the improvement brought by MixedAE is complementary to
more advanced reconstruction targets.

#### Masking strategy.

Instead of random masking [3, 27], AttMask [31] proposes a novel attention-
guided masking strategy by masking according to the attention map of the final
Transformer layer, while ADIOS [49] introduces an adversarial objective
between masking and reconstruction to generate learnable masks for MIM pre-
training. In this paper, we utilize random masking for MixedAE following MAE
due to its simplicity and effectiveness.

#### Input augmentation

instead has been less explored for MIM. Instead of masking, CIM [22] adopts a
small BEiT as the generator to corrupt an image, which is further taken as
input to an enhancer to reconstruct the corrupted patches or distinguish the
corrupted patches from the uncorrupted ones. Concurrently, MixMIM [38]
considers mixing with MAE also, but different from ours, MixMIM uses masked
self-attention to only perform attention within patches from the same images
given the mixing masks as input, which is exactly the same with MAE from the
perspective of pretext design, while ours utilizes mixing as part of the
pretext task actively to conduct object-aware SSL pre-training.

##  3 Method

In this section, we start by adopting mixing in MAE [27] with a simple
baseline in Sec. 3.1, which, as we can prove, will instead ease the
reconstruction pretext task. Then, we propose a novel auxiliary pretext task
to formulate our final MixedAE, which can not only alleviate the ease of
reconstruction, but also achieve object-aware SSL pre-training without
specifically designed modules in Secs. 3.2, 3.3 and 3.4.

###  3.1 Mixing: A Simple Baseline

Given a unlabeled dataset, we randomly sample a clean data batch with size
$B$, which are later divided into non-overlapping patch sequences
$\\{\boldsymbol{x}_{i}\\}_{i=1}^{B}$
$(\boldsymbol{x}_{i}\in\mathcal{R}^{L\times(P^{2}\cdot C)})$ following ViT
[18], where $L$ is the sequence length, $P$ is the patch size, and $C$ is the
image channel dimension.

#### Mixing.

The data batch is further separated into groups
$\\{\\{\boldsymbol{x}_{i}^{j}\\}_{i=1}^{1/r}\\}_{j=1}^{Br}$, and each group
will generate a single mixed image, where $r\in(0,0.5]$ is the mixing ratio
representing the ratio of patches each clean image contributes to within a
single mixed sample. Different from MixMIM [38], $r$ is not restricted to 0.5
in our formulation. Therefore, the mixing process for the $j$-th group can be
represented as,

$\boldsymbol{\hat{x}}^{j}=\sigma_{mix}(\\{\boldsymbol{x}_{i}^{j}\\},\boldsymbol{M}^{j})=\sum_{i=1}^{1/r}\mathbbm{1}(\boldsymbol{M}^{j}=i)\boldsymbol{x}_{i}^{j},$
(1)

where $\mathbbm{1}(\cdot)$ is indicator function and
$\boldsymbol{M}^{j}\in\\{1,2,...,1/r\\}^{L}$ represents a random mixing mask
independently generated for the $j$-th group, which satisfies,

$\sum_{l=1}^{L}\mathbbm{1}(M^{j}_{l}=i)=rL,\ \forall i\in\\{1,2,...,1/r\\}.$
(2)

So, $\boldsymbol{M}^{j}$ determines the source patch in each position of
$\boldsymbol{\hat{x}}^{j}$, while maintaining the mixing ratio for each clean
image equal with $r$ (_i.e_. symmetric mixing). The mixed images
$\boldsymbol{\hat{x}}^{j}$ are further fed into the encoder for feature
extraction, which can be represented as
$\boldsymbol{\hat{z}}^{j}=\textsc{enc}(\boldsymbol{\hat{x}}^{j})$.

#### Unmixing.

Following MAE [27], $\boldsymbol{\hat{z}}^{j}$ is then “unmixed” to recover
the input batch before mixing by inserting a special [MASK] token with
$\boldsymbol{M}^{j}$. For $\forall i\in\\{1,2...,1/r\\}$, we have,

$\boldsymbol{z}^{j}_{i}=\mathbbm{1}(\boldsymbol{M}^{j}=i)\boldsymbol{\hat{z}}^{j}+[1-\mathbbm{1}(\boldsymbol{M}^{j}=i)]\texttt{[MASK]}.$
(3)

The “unmixed” group $\\{\boldsymbol{z}_{i}^{j}\\}_{i=1}^{1/r}$ is then taken
as input to the decoder for pixel reconstruction, as
$\boldsymbol{y}_{i}^{j}=\textsc{dec}(\boldsymbol{z}_{i}^{j})$. Finally, the
reconstruction loss can be formulated as,

$\mathcal{L}_{recon}=\sum_{i=1}^{1/r}\sum_{l=1}^{L}[1-\mathbbm{1}(M^{j}_{l}=i)](\boldsymbol{y}_{i,l}^{j}-\boldsymbol{x}_{i,l}^{j})^{2}.$
(4)

So far, we have built a simple baseline to adopt mixing for MAE, which,
however, performs even worse than MAE, as demonstrated in
LABEL:tab:homo_summary. In the following, we provide a theoretical explanation
to prove that this naïve incorporation will actually ease the reconstruction
pretext task.

#### Mutual information analysis.

Without loss of generality, we take $r=0.5$ as an example. Denote
$\boldsymbol{X}_{1},\boldsymbol{X}_{2}$ as two random variables representing
two input images, and $\boldsymbol{X}_{1}$ is further considered as the
reconstruction target (symmetric for $\boldsymbol{X}_{2}$). Then, we can prove
that the mutual information (MI) between the mixed input
$\sigma_{mix}(\\{\boldsymbol{X}_{1},\boldsymbol{X}_{2}\\},\boldsymbol{M})$ and
the target $\boldsymbol{X}_{1}$ is no smaller than that between the MAE input
$\sigma_{MAE}(\boldsymbol{X}_{1},\boldsymbol{M})$ and $\boldsymbol{X}_{1}$ as
(see proofs in Appendix A),

$I(\sigma_{mix}(\\{\boldsymbol{X}_{1},\boldsymbol{X}_{2}\\},\boldsymbol{M});\boldsymbol{X}_{1})\geq
I(\sigma_{MAE}(\boldsymbol{X}_{1},\boldsymbol{M});\boldsymbol{X}_{1}).$ (5)

Therefore, different from masking, which is introduced to decrease the mutual
information between the model input and the reconstruction target due to the
redundancy of image signals [27], naïve mixing will instead increase the MI,
and thus, ease the reconstruction pretext task. Verification experiment is
conducted in Appendix C.

Also note that the MI increasement brought by mixing is target-invariant,
suggesting that Eq. 5 also holds when the target is semantic labels for
supervised learning or positive samples for contrastive learning, for which MI
increasement is appealing. This might explain why naïve mixing without
auxiliary supervision is beneficial for supervised [60, 59] and contrastive
learning [48, 58], but not for MAE.

(a) Uncertainty of generative modeling.

(b) Different segment embeddings represent different images.

Figure 3: Visualization of segment embeddings. (a) Due to the uncertainty of
generative modeling, green colors of the cucumber and the forest are both
reasonable for patches in the red ellipse. (b) We adopt different segment
embeddings for different images to provide necessary information for
homologous recognition.

###  3.2 Recognition: Homologous Recognition

Another indispensable factor to achieve MI increasement is the usage of global
self-attention in ViT, with which each query patch will inevitably attend to
heterologous patches from other images. Due to the uncertainty of generative
modeling, heterologous patches might provide a shortcut to complete
reconstruction (_e.g_., the green color of cucumbers is a “valuable” cue to
reconstruct the forest behind the fox in Fig. 3(a)). To address, we propose a
novel auxiliary pretext task called homologous recognition to enforce each
query to explicitly recognize and only attend to homologous patches.

#### Homologous attention

recognizes homologous patches on-the-fly by enforcing each query patch to only
attend to key patches with the highest attention mass using a
$\operatorname{TopK}(\cdot)$ sampling operation. Specifically, the homologous
attention can be formulated as,

$A_{HomoAtt}=\operatorname{softmax}(\operatorname{TopK}(\boldsymbol{q}\boldsymbol{k}^{T}/\sqrt{D_{h}})),$
(6)

where $\boldsymbol{q}$ is the query patch, $\boldsymbol{k}$ are the key
patches and $D_{h}$ is the feature dimension. By default, all the self-
attention operations in ViT are replaced with homologous attention except the
very first layer. See comparisons in LABEL:tab:attn_layer.

#### Homologous contrastive

aims at verifying the $\operatorname{TopK}(\cdot)$ sampling accuracy by
encouraging the encoder features of homologous patches to be similar, while
heterologous ones to be dissimilar in the supervised contrastive manner [32].
The homologous contrastive loss can be formulated as,

$\mathcal{L}_{HomoCon}=-\sum_{l=1}^{L}\sum_{l^{+}}\log\frac{exp(cos(\boldsymbol{\hat{z}}^{j}_{l},\boldsymbol{\hat{z}}^{j}_{l^{+}})/\tau)}{\sum_{l^{\prime}\neq
l}^{L}exp(cos(\boldsymbol{\hat{z}}^{j}_{l},\boldsymbol{\hat{z}}^{j}_{l^{\prime}})/\tau)},$
(7)

where $\tau$ is the temperature and $cos(\cdot,\cdot)$ is the cosine
similarity. As demonstrated in Fig. 7, the $\operatorname{TopK}(\cdot)$
sampling accuracy is significantly improved and stabilized with the usage of
the homologous contrastive loss $\mathcal{L}_{HomoCon}$.

#### Segment embedding.

Beside the positional embeddings, we add another segment embedding to the
mixed sequence $\boldsymbol{\hat{x}}^{j}$ following BERT [15] to provide
necessary information to complete homologous recognition, due to the
uncertainty of generative modeling. The segment embedding is shared for
patches from the same image, while different for patches from different
images, as demonstrated in Fig. 3(b).

(a) Compose mixing mode.

(b) Full mixing mode.

Figure 4: Visualization of two mixing modes when $r=0.5$. (a) Each group
generates a single mixed sample for the compose mixing mode, (b) while $1/r$
mixed samples are generated for the full mixing mode to maintain the effective
batch size unchanged.

#### Mixing mode.

For a fair comparison under different training overheads, two mixing modes are
adopted for MixedAE, 1) Compose: each group generates a single mixed sample
following Eq. 1, and the effective encoder batch size is $Br$; 2) Full: each
group generates $1/r$ mixed samples by sampling $\boldsymbol{M}^{j}$ for $1/r$
times independently, and the effective encoder batch size is $B$. An example
is provided in Fig. 4 when $r=0.5$. As shown in Tab. 1, MixedAE achieves the
SoTA performance under different training overheads. If no otherwise
specified, compose mixing is adopted by default.

Figure 5: Visualizations of attention maps on images from ImageNet-1K [14]
(1st-3rd columns), Microsoft COCO [37] (4th-6th columns) and ADE20K [62]
datasets (7th-9th columns). Both MAE and MixedAE are pre-trained on
ImageNet-1K for 300 epochs. Compared with MAE which mainly focuses on the most
discriminative patches, (_e.g_., boundaries (1st, 2nd & 5th) and edges (6th &
8th)), MixedAE discovers foreground object patches more precisely (3rd & 9th)
and completely (4th & 7th). See more attention maps in Appendix E.

###  3.3 Reconstruction: Loss Function

#### Loss function.

We formulate MixedAE in the multi-task learning manner and the final loss
function is a weighted sum of the reconstruction loss $\mathcal{L}_{recon}$
and the homologous contrastive loss $\mathcal{L}_{HomoCon}$ as,

$\mathcal{L}_{MixedAE}=\mathcal{L}_{recon}+\lambda\mathcal{L}_{HomoCon},$ (8)

where the balanced weight $\lambda$ is set to be 0.1 by default.

###  3.4 Discussion: Object-aware Pre-training

With the usage of mixing, we observe that MixedAE can achieve object-aware
self-supervised pre-training without any specifically designed components,
such as K-means [8], selective search [54] and object discovery network [29],
because homologous recognition requires each query patch to recognize all
homologous patches within a mixed image. Due to the single-centric-object
guarantee [8] of ImageNet, that most images are pre-processed to guarantee
only one object in the center part of them, the mixed image can be considered
as a “pseudo” multi-instance image, and given a query patch, the process of
recognizing all patches from the same image within a mixed sample is exactly
recognizing all patches from the same object within a given “pseudo” multi-
instance image. Therefore, the awareness of object existence and completeness
is enhanced in the leant representation of our MixedAE.

In Fig. 5, we visualize the attention maps of MAE and MixedAE by averaging all
attention heads of the last layer, taken the [CLS] token as query and patch
tokens as keys. Compared with MAE which mainly focuses on the most
discriminative patches (_e.g_., boundaries and corners), our MixedAE can
successfully discover the foreground object patches more precisely and
completely, which might also explain why MixedAE improves more significantly
when transferred to dense perception tasks, such as semantic segmentation [62]
and object detection [37], as shown in Tab. 1.

##  4 Experiments

###  4.1 Implementation Details

#### Architecture.

We mainly use the standard ViT-Base [18] as the backbone architecture, and
further provide ViT-Large experiments in Appendix C. The input images are
resized to $224\times 224$, resulting in a total sequence length $L=196$ with
the patch size being $16\times 16$. Following MAE [27], the decoder consists
of 8 Transformer layers with the hidden dimension as 512 by default. For a
fair comparison with BEiT [3], we additionally build a MixedAE in full mixing
mode with a lightweight decoder made up of 2 Transformer layers and the hidden
dimension as 256, as shown in Tab. 1.

The mixing ratio $r$ is set to be 0.25 (_i.e_., corresponding to the 0.75
masking ratio in MAE [27]) by default, and the threshold $K$ in the
$\operatorname{TopK}(\cdot)$ operation is therefore set to be $0.25L$.
Following common practices [11, 7], we adopt a linear projector with the
output dimension as 128 right before the homologous contrastive loss, and the
temperature coefficient $\tau$ is set to be 0.25.

Method | Pre-train | Pre-train† | ImageNet | ADE20K | COCO
---|---|---|---|---|---
Epochs | GPU-days | Top-1 Acc. | mIoU | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
DeiT [51] | 300 | 19.6 | 81.8 | 46.9 | 48.8 | 68.7 | 52.7 | 42.5 | 65.9 | 45.5
MoCov3 [11] | 600†† | 54.8 | 82.8 | 46.8 | 47.2 | 66.9 | 50.8 | 41.1 | 63.6 | 44.1
DINO [7] | 1600†† | 120.5 | 82.8 | 46.9 | 49.5 | 69.1 | 53.6 | 42.9 | 66.0 | 46.3
BEiT [3] | 300 | 32.1 | 82.9 | 44.7 | 39.3 | 57.7 | 42.4 | 34.8 | 55.2 | 36.8
MAE [27] | 300 | 16.4 | 82.7 | 46.1 | 47.2 | 65.8 | 51.3 | 41.1 | 62.9 | 44.4
MixMIM [38] | 300 | 40.2 | 83.2 | - | - | - | - | - | - | -
CIM-RevDet [22] | 300 | 42.7 | 83.1 | - | - | - | - | - | - | -
CIM-ResPix [22] | 300 | 42.7 | 83.3 | - | - | - | - | - | - | -
MixedAE | 300 | 16.9 | 83.1+0.4 | 47.0+0.9 | 47.8+0.6 | 66.6+0.8 | 52.0+0.7 | 41.6+0.5 | 63.6+0.7 | 45.0+0.6
MixedAE-Full∗ | 300 | 30.8 | 83.7+1.0 | 47.4+1.3 | 48.9+1.7 | 67.6+1.8 | 53.3+2.0 | 42.5+1.4 | 64.8+1.9 | 45.9+1.5
MixedAE-Full | 300 | 62.3 | 83.8+1.1 | 48.9+2.8 | 51.0+3.8 | 69.7+3.9 | 55.2+3.9 | 44.1+3.0 | 67.0+4.1 | 47.9+3.5
BEiT [3] | 800 | 85.5 | 83.2 | 45.6 | 40.8 | 59.4 | 44.1 | 36.0 | 56.8 | 38.2
MAE [27] | 800 | 43.7 | 83.3 | 47.2 | 49.4 | 68.1 | 53.9 | 42.9 | 65.5 | 46.6
MixedAE | 800 | 45.0 | 83.5+0.2 | 48.7+1.5 | 50.3+0.9 | 69.1+1.0 | 54.8+0.9 | 43.5+0.6 | 66.2+0.7 | 47.4+0.8
MAE [27] | 1600 | 87.4 | 83.6 | 48.1 | 50.6 | 69.4 | 55.0 | 43.8 | 66.6 | 47.5
iBOT [63] | 1600†† | 172.1 | 83.8 | 49.6 | 51.2 | 70.1 | 55.2 | 44.3 | 67.4 | 48.0
MixedAE | 1600 | 90.1 | 83.9+0.3 | 49.8+1.7 | 51.5+0.9 | 70.2+0.8 | 55.9+0.9 | 44.5+0.7 | 67.5+0.9 | 48.2+0.7
Table 1: Transfer performance comparison between methods pre-trained on
ImageNet-1K. 1) Effectiveness: MixedAE achieves the state-of-the-art
performance under different pre-training epochs and overheads. 2) Efficiency:
MixedAE consistently surpasses the strong iBOT [63] baseline, while only
requiring 53.4% of the pre-training overhead. 3) Object-aware pre-training:
more significant improvements are achieved when transferred to downstream
dense perception tasks (0.3 vs. 1.7 vs. 0.9). ∗: a lightweight decoder is
deployed to maintain similar pre-training overhead with BEiT [3]. †: GPU-days
estimated on Tesla V100 GPUs. ††: effective epochs following iBOT [63].

#### Optimization.

We pre-train MixedAE on the ImageNet-1K [14] training set with the AdamW [42]
optimizer and a cosine learning rate schedule with a linear warm-up of 40
epochs. The batch size is set to be 4096 for the compose mixing, while 1024
for the full mode. The base learning rate is set to be $7.5e^{-5}$, which will
scale linearly with the batch size ($lr=lr_{base}\times bs/256$). Only
standard random cropping and flipping are utilized for data augmentation. The
remaining hyperparameters are all maintained the same with MAE for a fair
comparison (see Appendix B for more details).

###  4.2 Transfer Results on ImageNet-1K

#### Setup.

We consider the fully fine-tuning performance on ImageNet-1K for 100 epochs
and report the Top-1 accuracy. Following MAE [27], we average all the patch
tokens after the final Transformer layer, which is taken as input to a linear
head for classification. See more details in Appendix B.

#### Comparison with MAE.

As shown in Tab. 1, MixedAE obtains consistent improvements over MAE under
different pre-training epochs with only 3% additional overhead. The 300-epoch
pre-trained MixedAE with full mixing acquires even better accuracy than the
1600-epoch pre-trained MAE, demonstrating the efficiency of data utilization.

#### Comparison with other MIM augmentations.

Our MixedAE with the lightweight decoder and the full mixing mode obtains
83.7% Top-1 accuracy, 0.5% and 0.4% higher than MixMIM [38] and CIM [22]
respectively, meanwhile saving at least 23.4% computational overhead,
revealing the simplicity of our MixedAE.

#### Comparison with other SSL approaches.

Our MixedAE obtains consistent improvements under various pre-training epochs
and overheads, and achieves a better trade-off between pre-training overhead
and transfer performance, as shown in Fig. 1. Specifically, MixedAE pre-
trained for 1600 epochs achieves 83.9% accuracy, constructing a new state-of-
the-art result with a pure autoencoder-based framework. Requiring only 53.4%
of the training overhead, MixedAE surpasses the strong iBOT [63],
demonstrating remarkable efficiency. Moreover, the improvement brought by
mixing is orthogonal to the usage of more advanced reconstruction targets [17,
53, 2] and masking strategies [31, 49].

###  4.3 Transfer Results on Downstream Tasks

We further consider three downstream settings to evaluate the learnt
representation, and more details about the different transfer procedures are
included in Appendix B.

#### Semantic segmentation.

We utilize the UperNet [56] to perform semantic segmentation on ADE20K [62]
following BEiT [3]. As reported in Tab. 1, our 800-epoch MixedAE achieves 48.7
mIoU, even surpassing the MAE pre-trained for 1600 epochs by 0.6 mIoU, and our
1600-epoch MixedAE further improves to 49.8 mIoU, outperforming all baseline
methods by a non-trivial margin, which is more significant than the
improvement on ImageNet-1K (1.7 vs. 0.3), thanks to the object-aware pre-
training, as discussed in Sec. 3.4.

#### Object detection and instance segmentation.

We utilize Cascade Mask R-CNN [5, 28] to produce bounding boxes and instance
masks simultaneously on COCO [37]. As demonstrated in Tab. 1, MixedAE
consistently outperforms MAE under different epochs (0.6/0.9/0.9 &
0.5/0.6/0.7). Similarly with ADE20K, more significant improvements are
observed due to the high-quality attention maps learnt by the object-aware
pre-training, as demonstrated in Fig. 5.

Method | Aircraft | Caltech | Cars | C10 | C100 | DTD | Flowers | Food | Pets | SUN | VOC | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
SSL ResNets
MoCov2 [10] | 79.9 | 84.4 | 75.2 | 96.5 | 71.3 | 69.5 | 94.4 | 76.8 | 79.8 | 55.8 | 71.7 | 77.7
SimCLR [9] | 78.7 | 82.9 | 79.8 | 96.2 | 79.1 | 70.2 | 94.3 | 82.2 | 83.2 | 61.1 | 78.2 | 80.5
BYOL [25] | 79.5 | 89.4 | 84.6 | 97.0 | 84.0 | 73.6 | 94.5 | 85.5 | 89.6 | 64.0 | 82.7 | 84.0
SwAV [6] | 83.1 | 89.9 | 86.8 | 96.8 | 84.4 | 75.2 | 95.5 | 87.2 | 89.1 | 66.2 | 84.7 | 85.3
SDR [39] | 82.6 | 89.0 | 87.5 | 97.4 | 84.4 | 75.6 | 97.0 | 86.1 | 89.3 | 66.1 | 85.3 | 85.5
SSL Transformers
MoCov3 [11] | 76.6 | 91.2 | 86.6 | 98.3 | 88.3 | 72.6 | 95.5 | 86.4 | 92.0 | 65.6 | 84.5 | 85.2
DINO [7] | 69.4 | 91.2 | 81.3 | 98.4 | 88.9 | 77.6 | 96.9 | 87.3 | 93.5 | 64.7 | 86.3 | 85.1
BEiT [3] | 66.3 | 80.2 | 78.6 | 96.1 | 80.0 | 69.9 | 92.9 | 83.2 | 85.3 | 57.1 | 76.7 | 78.7
MAE [27] | 78.2 | 91.2 | 88.4 | 97.0 | 82.5 | 75.3 | 96.6 | 84.7 | 92.6 | 65.4 | 86.0 | 85.3
MixedAE | 82.1 | 91.5 | 88.8 | 97.9 | 85.9 | 78.7 | 97.1 | 87.4 | 93.6 | 66.2 | 86.4 | 86.9+1.6

Table 2: Transfer performance comparison on 11 downstream classification
tasks. Our 1600-epoch pre-trained MixedAE achieves consistent improvements
over MAE on all 11 classification datasets with an average accuracy of 86.9%,
surpassing all counterparts.

#### Downstream classification.

Following [20, 39, 61], we study transfer performance on 11 downstream
classification datasets, including both fine-grained (_e.g_., Cars [33]) and
coarse-grained ones (_e.g_., CIFAR100 [34]) in Tab. 2. Our MixedAE achieves
consistent improvement over MAE on all 11 downstream tasks with an average
accuracy of 86.9%, outperforming all counterparts as demonstrated in Tab. 2.

mixing ratio $r$ |  GPU-days |  acc |  mIoU
---|---|---|---
0.25 |  16.9 |  82.4 |  45.0
0.5 |  21.0 |  82.3 |  44.2
(a)

case |  acc |  mIoU
---|---|---
pre-[LN] |  82.4 |  44.5
post-[LN] |  82.5 |  45.3
(b)

case |  acc |  mIoU
---|---|---
all positives separately |  82.5 |  45.3
mean positive |  82.5 |  45.1
(c)

threshold $K$ |  acc |  mIoU
---|---|---
0.125$L$ |  82.6 |  45.1
0.25$L$ |  82.6 |  45.6
0.5$L$ |  82.6 |  45.5
1.0$L$ (no sampling) |  82.5 |  45.3
(d)

layer ID |  acc |  mIoU
---|---|---
none |  82.6 |  45.6
1st |  82.7 |  46.4
1st & 2nd |  82.5 |  44.8
1st & 2nd & 3rd |  82.5 |  45.6
(e)

|  HomoAtt |  HomoCon |  acc |  mIoU
---|---|---|---|---
MixedAE |  |  |  82.4 |  45.0
|  ✓ |  |  82.6 |  44.5
|  |  ✓ |  82.5 |  45.3
|  ✓ |  ✓ |  82.7 |  46.4

(f)

Table 3: MixedAE ablation experiments with ViT-B/16. We report the ImageNet-1K
fine-tuning accuracy (acc) and ADE20K semantic segmentation performance
(mIoU). We explore (a) the mixing ratio $r$, (b,c) the position and positives
for homologous contrastive, (d,e) the threshold $K$ and position for
homologous attention, and (f) a summary of homologous recognition. Default
settings are marked in gray.

(a) Accuracy curve with respect to Transformer layers of our MixedAE pre-
trained for 300 epochs, where the 1st layer adopts global self-attention.

(b) Accuracy curve with respect to pre-training epochs of the very final
Transformer layer in our MixedAE.

Figure 7: Analysis of $\operatorname{TopK}$ sampling accuracy. Without the
homologous contrastive loss for verification (green curve), utilizing the
homologous attention only cannot achieve promising accuracy, dramatically
varying between different layers. However, with the usage of homologous
contrastive loss (orange curve), the sampling accuracy is significantly
improved and stabilized mostly around 70% to 80% throughout the whole pre-
training process, which is essential to achieve remarkable transfer
performance, as demonstrated in LABEL:tab:homo_summary.

###  4.4 Ablation Study

#### Setup.

We conduct 300-epoch pre-training with a base learning rate of $1.5e^{-4}$ for
all ablation studies on MixedAE with compose mixing. By default, we report the
fine-tuning accuracy on ImageNet-1K [14] and mIoU on ADE20K [62]. See more
detailed settings and results in Appendix C.

#### Mixing ratio $r$.

Different from MixMIM [38], the mixing ratio $r$ in our formulation can be
flexibly selected from the range of $(0,0.5]$. As shown in
LABEL:tab:mixing_ratio, $r=0.25$ works better, while requiring less pre-
training overhead (since the effective encoder batch size scales linearly with
$r$, as shown in Eq. 1). Note that $r=0.25$ is also corresponding to the
default 0.75 masking ratio in MAE.

#### Position of homologous contrastive.

We study whether the encoder features before or after the final Layer
Normalization ([LN]) [1] of ViT [18] achieves better performance as input to
the homologous contrastive loss in LABEL:tab:homo_position. The latter
achieves consistent improvements on both ImageNet-1K and ADE20K, suggesting
that the features after [LN] are more suitable for homologous recognition.

#### Positives of homologous contrastive.

In Eq. 7, given a query patch, all homologous patches are considered as
positive samples but taken separately to calculate $\mathcal{L}_{HomoCon}$ in
the supervised contrastive manner [32]. We further ablate to utilize the
average of both the query and all its homologous patches as its positive,
which, however, performs slightly worse than the separate manner (-0.2 mIoU on
ADE20K).

#### Threshold $K$ of homologous attention.

We study the threshold number $K$ in $A_{HomoAtt}$ in LABEL:tab:attn_K, where
all the global self-attention operations in the ViT are replaced with our
homologous attention. Compared with the no sampling baseline, the usage of
homologous attention obtains consistent improvement on ImageNet-1K, while the
best achieves at $K=0.25L$, which is consistent with the mixing ratio $r$ in
LABEL:tab:mixing_ratio specifically.

#### Position of homologous attention.

As shown in Fig. 7(a), homologous attention cannot achieve promising accuracy
in early Transformer layers without sufficient information engagement. Thus,
we further explore to maintain the global self-attention in early layers
unchanged in LABEL:tab:attn_layer, and observe empirically that utilizing
global self-attention in the very first layer only achieves the best
performance.

#### Homologous recognition.

In LABEL:tab:homo_summary, we further compare the effectiveness of different
components of homologous recognition. Without the homologous contrastive loss
for verification, utilizing homologous attention only obtains a significant
drop of 0.5 mIoU on ADE20K. Although achieving improvement, utilizing the
homologous contrastive only still suffers from the ease of reconstruction
brought by MI increasement, as previously discussed in Eq. 5. Finally, the
best performance is achieved when using homologous attention and contrastive
loss simultaneously.

###  4.5 Analysis

#### Effectiveness of $\operatorname{TopK}$ sampling.

To further observe the effectiveness of $\operatorname{TopK}$ sampling in
homologous attention, we visualize the sampling accuracy with respect to
different layers and pre-training epochs in Fig. 7. As demonstrated in Fig.
7(a), the naïve usage of homologous attention only cannot achieve promising
sampling accuracy, which, therefore, suffers from a significant performance
drop in LABEL:tab:homo_summary. Specifically, neither the sampling accuracy of
the first two nor final two layers exceed 60%, and accuracy of the very final
layer even maintains under 40% throughout the whole pre-training process, as
shown in Fig. 7(b) (the green curve).

However, with the usage of the homologous contrastive loss for verification,
the sampling accuracy is significantly improved and stabilized around 70% to
80% for all layers except the first two, as in Fig. 7(a). The sampling
accuracy of the final layer rapidly increases to around 70% when pre-trained
only for 20 epochs, maintaining stable throughout the remaining pre-training,
as in Fig. 7(b) (the orange curve).

#### Comparison with existing MIM methods combined with contrastive.

Although also utilizing a “contrastive loss” with reconstruction, MixedAE
differs from existing MIM works [19, 63] combined with contrastive learning
from two perspectives, 1) Purpose: existing works utilize contrastive loss to
perform instance discrimination simultaneously with MIM, while our homologous
contrastive is only utilized to guarantee the sampling accuracy. Therefore,
homologous contrastive performs more like a regularization term instead of an
individual self-supervision in [19, 63]. To verify, we pre-train a MixedAE
with $\mathcal{L}_{HomoCon}$ only without $\mathcal{L}_{recon}$, which cannot
achieve reasonable performance, as reported in Appendix C. 2) Efficiency:
given a single input, existing works require forward propagation at least
twice for several augmented views to conduct instance discrimination, while
only once for our homologous contrastive, resulting in the significant
efficiency. Specifically, our MixedAE surpasses iBOT [63] with only 53.4% of
its computational overhead.

##  5 Conclusion

This paper explores the usage of image mixing for MAE. Different from in
supervised and contrastive learning, we first theoretically demonstrate naïve
mixing might instead ease the reconstruction pretext task. To address that,
our MixedAE with the proposed homologous recognition as the auxiliary
supervision can not only achieve state-of-the-art performance with a better
trade-off between transfer results and pre-training overhead, but also conduct
object-aware pre-training without any specifically designed modules. We hope
our simple yet effective method can bring researchers’ attention to more
effective data augmentations for MIM.

#### Acknowledgments.

We gratefully acknowledge support of MindSpore, CANN (Compute Architecture for
Neural Networks) and Ascend AI Processor used for this research.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  arXiv preprint arXiv:2202.03555, 2022.
  * [3] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 – mining discriminative components with random forests.  In ECCV, 2014.
  * [5] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: delving into high quality object detection.  In CVPR, 2018.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [8] Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-Yan Yeung.  Multisiam: Self-supervised multi-instance siamese representation learning for autonomous driving.  In ICCV, 2021.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [11] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  arXiv preprint arXiv:2104.02057, 2021.
  * [12] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, 2014.
  * [13] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  In ICLR, 2020.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [16] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In ICCV, 2015.
  * [17] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu.  Peco: Perceptual codebook for bert pre-training of vision transformers.  arXiv preprint arXiv:2111.12710, 2021.
  * [18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [19] Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave.  Are large-scale datasets necessary for self-supervised pre-training?  arXiv preprint arXiv:2112.10740, 2021.
  * [20] Linus Ericsson, Henry Gouk, and Timothy M Hospedales.  How well do self-supervised models transfer?  In CVPR, 2021.
  * [21] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  In ICCV, 2010.
  * [22] Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei.  Corrupted image modeling for self-supervised visual pre-training.  arXiv preprint arXiv:2202.03382, 2022.
  * [23] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [24] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In AISTATS, 2010.
  * [25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, and others.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [26] Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Xiaodan Liang, and Chunjing Xu.  Soda10m: Towards large-scale object detection benchmark for autonomous driving.  arXiv preprint arXiv:2106.11118, 2021.
  * [27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * [28] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [29] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović.  Object discovery and representation networks.  arXiv preprint arXiv:2203.08777, 2022.
  * [30] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.  Deep networks with stochastic depth.  In ECCV, 2016.
  * [31] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis.  What to hide from your students: Attention-guided masked image modeling.  arXiv preprint arXiv:2203.12719, 2022.
  * [32] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  In NeurIPS, 2020.
  * [33] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei.  Collecting a large-scale dataset of fine-grained cars.  In Workshop on Fine-Grained Visual Categorization, 2013.
  * [34] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Master’s thesis, University of Tront, 2009.
  * [35] Fei-Fei Li, Fergus Rob, and Perona Pietro.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In CVPRW, 2004.
  * [36] Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al.  Coda: A real-world road corner case dataset for object detection in autonomous driving.  arXiv preprint arXiv:2203.07724, 2022.
  * [37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [38] Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li.  Mixmim: Mixed and masked image modeling for efficient visual representation learning.  arXiv preprint arXiv:2205.13137, 2022.
  * [39] Zhili Liu, Jianhua Han, Kai Chen, Lanqing Hong, Hang Xu, Chunjing Xu, and Zhenguo Li.  Task-customized self-supervised pre-training with scalable dynamic routing.  In AAAI, 2022.
  * [40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [41] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In ICLR, 2016.
  * [42] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [43] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [44] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
  * [45] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [46] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In CVPR, 2016.
  * [47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In ICML, 2021.
  * [48] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing.  Un-mix: Rethinking image mixtures for unsupervised visual representation learning.  In AAAI, 2022.
  * [49] Yuge Shi, N Siddharth, Philip HS Torr, and Adam R Kosiorek.  Adversarial masking for self-supervised learning.  arXiv preprint arXiv:2201.13100, 2022.
  * [50] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [51] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [53] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  arXiv preprint arXiv:2112.09133, 2021.
  * [54] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning pretraining for detection via object-level contrastive learning.  In NeurIPS, 2021.
  * [55] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In CVPR, 2010.
  * [56] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In ECCV, 2018.
  * [57] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  arXiv preprint arXiv:2111.09886, 2021.
  * [58] Haohang Xu, Xiaopeng Zhang, Hao Li, Lingxi Xie, Wenrui Dai, Hongkai Xiong, and Qi Tian.  Seed the views: Hierarchical semantic alignment for contrastive representation learning.  In PAMI, 2022.
  * [59] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Junsuk Choe, and Youngjoon Yoo.  CutMix: Regularization strategy to train strong classifiers with localizable features.  In ICCV, 2019.
  * [60] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  MixUp: Beyond empirical risk minimization.  In ICLR, 2018.
  * [61] LIU Zhili, Kai Chen, Jianhua Han, HONG Lanqing, Hang Xu, Zhenguo Li, and James Kwok.  Task-customized masked autoencoder via mixture of cluster-conditional experts.  In ICLR, 2023.
  * [62] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  In IJCV, 2019.
  * [63] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  arXiv preprint arXiv:2111.07832, 2021.

## Appendix

##  Appendix A Proof of Eq. 5

In this section, we prove Eq. 5 with the terminologies maintained consistent
with Sec. 3. We start to prove when $r=0.5$ (_i.e_., two clean images within a
single group). Denote $\boldsymbol{X}_{1},\boldsymbol{X}_{2}$ as two random
variables representing two input images, and $\boldsymbol{M}$ as the random
mask, which can be considered as a constant here since it is independently
generated with $\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$. Then, according
to Eq. 1, the mixed input can be represented as,

$\sigma_{mix}(\\{\boldsymbol{X}_{1},\boldsymbol{X}_{2}\\},\boldsymbol{M})=\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{X}_{2},$
(9)

while the MAE input can be represented as,

$\sigma_{MAE}(\boldsymbol{X}_{1},\boldsymbol{M})=\\{\boldsymbol{X}_{1,l}|M_{l}=1\\}=\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{\overrightarrow{0}}.$
(10)

Therefore, given $\boldsymbol{X}_{1}$ as the reconstruction target, we can
represent the mutual information (MI) between the mixed input and the
reconstruction target $\boldsymbol{X}_{1}$ as,

$\displaystyle
I(\sigma_{mix}(\\{\boldsymbol{X}_{1},\boldsymbol{X}_{2}\\},\boldsymbol{M});\boldsymbol{X}_{1})$
$\displaystyle=I(\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{X}_{2};\boldsymbol{X}_{1})$
(11)
$\displaystyle=H(\boldsymbol{X}_{1})-H(\boldsymbol{X}_{1}|\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{X}_{2})$
$\displaystyle=H(\boldsymbol{X}_{1})-H(\boldsymbol{X}_{1}|\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{\overrightarrow{0}}+\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{\overrightarrow{0}}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{X}_{2})$
$\displaystyle=H(\boldsymbol{X}_{1})-H(\boldsymbol{X}_{1}|\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{\overrightarrow{0}},\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{\overrightarrow{0}}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{X}_{2})$
$\displaystyle\geq
H(\boldsymbol{X}_{1})-H(\boldsymbol{X}_{1}|\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{X}_{1}+\mathbbm{1}(\boldsymbol{M}=2)\boldsymbol{\overrightarrow{0}})$
$\displaystyle=I(\sigma_{MAE}(\boldsymbol{X}_{1},\boldsymbol{M});\boldsymbol{X}_{1}),$

where $H(\cdot)$ is the entropy. The conclusion above also holds when
$\boldsymbol{X}_{2}$ is considered as the reconstruction target symmetrically.

Note that although independent in the data space, $\boldsymbol{X}_{1}$ and
$\boldsymbol{X}_{2}$ are not independent in the feature space, because the
global self-attention would introduce inevitable information leakage from
$\boldsymbol{X}_{2}$ to $\boldsymbol{X}_{1}$, which would be enhanced for
“relevant” $\boldsymbol{X}_{2}$ (_e.g_., the green cucumber in Fig. 3(a)),
while restrained for “irrelevant” ones (_e.g_., the blue sky) by assigning
different attention weights. As shown in Fig. 7, the
$\operatorname{TopK}(\cdot)$ sampling accuracy converges to around 80%,
suggesting that the information leakage does exist in practice. Therefore, the
independence is affected, and the equality in Eq. 11 does not hold.

For $r\in(0,0.5)$, there are more than two images within a single group
$\\{\boldsymbol{X}_{1},\boldsymbol{X}_{2},...,\boldsymbol{X}_{1/r}\\}$.
Considering $\boldsymbol{X}_{1}$ as the reconstruction target, we can first
mix all images except $\boldsymbol{X}_{1}$ to generate a pseudo
$\boldsymbol{\hat{X}}_{2}$ as,

$\boldsymbol{\hat{X}}_{2}=\sigma_{mix}(\\{\boldsymbol{\overrightarrow{0}},\boldsymbol{X}_{2},\boldsymbol{X}_{3},...,\boldsymbol{X}_{1/r}\\},\boldsymbol{M})=\mathbbm{1}(\boldsymbol{M}=1)\boldsymbol{\overrightarrow{0}}+\sum_{i=2}^{1/r}\mathbbm{1}(\boldsymbol{M}=i)\boldsymbol{X}_{i},$
(12)

which is then mixed with $\boldsymbol{X}_{1}$ following Eq. 9. Therefore, the
conclusion in Eq. 11 can still be satisfied.

As discussed above, the usage of the global self-attention in ViT [18] is
another indispensable factor to achieve the MI increasement in Eq. 11.
Therefore, in this paper, we propose the homologous attention to replace the
global self-attention together with the homologous contrastive loss as
verification for our MixedAE.

##  Appendix B More Implementation Details

config | value
---|---
optimizer | AdamW [42]
base learning rate | $7.5e^{-5}$
weight decay | 0.05
optimizer momentum | $\beta_{1},\beta_{2}=0.9,0.95$
batch size | 4096 (B), 2048 (L)
learning rate schedule | cosine decay [41]
warmup epochs | 40
augmentation | RandomResizedCrop
reconstruction target | normalized pixels [27]
Table 4: Pre-training settings.

config | value
---|---
optimizer | AdamW
base learning rate |  $5e^{-4}$ (B), $1e^{-3}$ (L)
weight decay | 0.05
optimizer momentum | $\beta_{1},\beta_{2}=0.9,0.999$
batch size | 1024 (B), 512 (L)
learning rate schedule | cosine decay
warmup epochs | 5
training epochs | 100 (B), 50 (L)
augmentation | RandAug (9, 0.5)
label smoothing [50] | 0.1
mixup [60] | 0.8
cutmix [59] | 1.0
drop path [30] | 0.1
Table 5: Fully fine-tuning settings.

#### Pre-training.

The default settings are provided in Tab. 5. We use xavier_uniform [24] to
initialize all Transformer layers, following the official code of ViT [18].
Normalized pixels [27] are utilized as the reconstruction target, and the mask
patch size is set to be $32\times 32$, following [57, 38]. In practice, we
utilize the sine-cosine encodings [52] for segment embeddings, which is added
to the input of each Transformer layer following [40]. If no otherwise
specified, the compose mixing mode is by default adopted to generate a single
mixed sample for each image group.

#### ImageNet classification.

The default settings are provided in Tab. 5. We mainly adopt the fully fine-
tuning transfer setting to fine-tune the parameters of the backbone and the
classification head simultaneously. We utilize the layer-wise learning rate
decay strategy [13] following [3]. In practice, we sweep the decay ratio in
$\\{0.65,0.7,0.75\\}$ following [63, 38].

#### ADE20K semantic segmentation.

We use UperNet [56] as the segmentor following BEiT [3]. The input resolution
is $512\times 512$, and the batch size is set to be 16. The learning rate is
set to be $3e^{-4}$ with the layer-wise learning rate decay ratio as 0.65 for
ViT-Base and 0.75 for ViT-Large. We conduct fine-tuning for 160K iterations,
and evaluate the performance without the multi-scale augmentation.

#### COCO object detection and instance segmentation.

We utilize the Cascade Mask R-CNN [5, 28] following iBOT [63]. Multi-scale
training is adopted with the shorted side randomly resized between 480 and 800
while the longer side no larger than 1333. The batch size is 16, the initial
learning rate is $1e^{-4}$, and the layer-wise learning rate decay ratio is
set to be 0.75. We adopt the standard $1\times$ schedule to train for 12
epochs, and decrease the learning rate by $10\times$ at epoch 9 and 11.

#### Downstream classification.

We mainly follow the setups in [39, 61] to evaluate the transfer performance
on 11 downstream classification datasets, including both the fine-grained
datasets (_e.g_., Aircraft [43], Cars [33], Flowers [44], Food [4], Pets [45]
and SUN397 [55]), and the coarse-grained ones (_e.g_., Caltech101 [35],
CIFAR10 [34], CIFAR100 [34], DTD [12] and VOC2007 [21]). Specifically, we
adopt a linear classification head upon the pre-trained ViT backbone and fully
fine-tune the whole model simultaneously for 5000 iterations. The SGD
optimizer and the cosine learning rate schedule are adopted. We grid search
the optimal learning rate among $\\{1e^{-3},3e^{-3},1e^{-2},3e^{-2}\\}$, and
set the weight decay to be 0.

##  Appendix C More Experiments

#### Scaling property of MixedAE.

We further build MixedAE with the standard ViT-Large [18] as the backbone
architecture, and pre-train for 1600 epochs on ImageNet-1K [14] following the
same optimization recipe with ViT-Base as in Appendix B. As demonstrated in
Tab. 6, MixedAE still outperforms MAE consistently with ViT-Large, especially
on downstream dense perception tasks [62, 37, 26, 36] thanks to the object-
aware pre-training, revealing the scalability of our proposed MixedAE.

Moreover, we further report the performance of the 1500-epoch pre-trained
MixedAE with ViT-Large to maintain similar computational overhead with MAE in
Tab. 6. MixedAE still obtains consistent improvements over MAE, while
outperforming iBOT with a 1.8$\times$ acceleration, suggesting that MixedAE
can achieve a better trade-off between the computational overhead and the
transfer performance regardless of the architecture size.

#### Ablation settings in Tab. 3.

We conduct the ablation of new components based on previous results. Starting
from the naïve baseline in Sec. 3.1, we first ablate the mixing ratio $r$ in
LABEL:tab:mixing_ratio. Accordingly, we fix $r=0.25$ to explore the position
and positives of the homologous contrastive in LABEL:tab:homo_position and
LABEL:tab:homo_positive. Similarly, we ablate homologous attention in
LABEL:tab:attn_K and LABEL:tab:attn_layer based on results of
LABEL:tab:homo_positive, and finally summarize all components in
LABEL:tab:homo_summary.

#### Main cause of performance degradation of naïve mixing.

To verify that the information leakage brought by the mutual information
increasement, as proved in Appendix A, is indeed the main cause of performance
degradation of the naïve mixing baseline introduced in Sec. 3.1 instead of the
optimization difficulty, we further build a mixing baseline by: 1) applying
the masked self-attention to perform self-attention only within homologous
patches to prevent information leakage without homologous recognition (_i.e_.,
neither homologous $\operatorname{TopK}(\cdot)$ attention nor homologous
contrastive loss); 2) adopting exactly the same optimization recipe with MAE.
As demonstrated in LABEL:tab:main_reason, the model achieves 82.6% accuracy
and 45.9 mIoU, comparably with MAE, suggesting that the information leakage is
definitely the culprit here.

#### Functionality of the $\mathcal{L}_{HomoCon}$.

To verify that the homologous contrastive loss performs more like a
regularization term instead of an individual self-supervision as in [63, 19],
we further pre-train a MixedAE with $\mathcal{L}_{HomoCon}$ only following the
settings in Sec. 4.4. As demonstrated in LABEL:tab:loss_function, MixedAE
performs well when the reconstruction loss $\mathcal{L}_{recon}$ is utilized
only or together with the homologous contrastive loss $\mathcal{L}_{HomoCon}$.
However, when adopting $\mathcal{L}_{HomoCon}$ only, MixedAE cannot even
achieve reasonable transfer performance, suggesting that
$\mathcal{L}_{HomoCon}$ cannot work alone without $\mathcal{L}_{recon}$.

#### Necessity of mixing.

To verify to necessity of adopting mixing augmentation in our MixedAE, we
extend MAE with the homologous contrastive $\mathcal{L}_{HomoCon}$ by applying
Eq. 7 to patches across different images in groups of 4 for a fair comparison
with MixedAE, which achieves 81.1% accuracy and 42.9 mIoU as demonstrated in
LABEL:tab:mix_and_segment (2nd & 3rd rows), significantly worse than our
default MixedAE, revealing the necessity of using mixing augmentation.

#### Necessity of segment embeddings.

As shown in LABEL:tab:mix_and_segment (1st & 3rd rows), we build a MixedAE
without segment embeddings and achieve 82.2% accuracy and 44.9 mIoU,
significantly worse than our default MixedAE, suggesting the importance of
adopting segment embeddings to provide necessary information for homologous
recognition.

Method | Pre-train | Pre-train | ImageNet | ADE20K | COCO
---|---|---|---|---|---
Epochs | GPU-days | Top-1 Acc. | mIoU | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
iBOT [63] | 1000∗ | 285.0 | 85.0 | 52.2 | 49.9 | 69.5 | 54.1 | 42.9 | 66.5 | 45.9
MAE [27] | 1600 | 151.1 | 85.9 | 53.6 | 54.0 | 72.6 | 59.0 | 46.3 | 69.6 | 50.4
MixedAE | 1500 | 159.7 | 86.0 | 53.8 | 54.5 | 73.4 | 59.3 | 46.9 | 70.4 | 50.9
MixedAE | 1600 | 170.4 | 86.2+0.3 | 54.0+0.4 | 54.6+0.6 | 73.5+0.9 | 59.4+0.4 | 47.1+0.8 | 70.7+1.1 | 51.0+0.6
Table 6: Transfer performance comparison with ViT-Large [18]. 1) Scalability:
our MixedAE outperforms MAE consistently even with ViT-Large. 2) Efficiency:
MixedAE achieves a better trade-off between the computational overhead and the
transfer performance regardless of the architecture size. ∗: effective epochs
following iBOT [63].

|  Masked SA |  acc |  mIoU
---|---|---|---
MAE |  |  82.7 |  46.1
Naïve |  |  82.4 |  45.0
Mixing |  ✓ |  82.6 |  45.9
(a)

$\mathcal{L}_{recon}$ |  $\mathcal{L}_{HomoCon}$ |  acc |  mIoU
---|---|---|---
✓ |  |  82.4 |  45.0
|  ✓ |  7.8 |  8.3
✓ |  ✓ |  82.7 |  46.4
(b)

Mixing |  SE |  acc |  mIoU
---|---|---|---
✓ |  |  82.2 |  44.9
|  ✓ |  81.1 |  42.9
✓ |  ✓ |  82.7 |  46.4

(c)

Table 7: More MixedAE ablation experiments with ViT-B/16. Default settings are
marked in gray.

##  Appendix D More Analysis

#### Exploration for other augmentations.

As discussed in Sec. 3.1, based on mixing, we observe that common augmentation
strategies will increase mutual information (MI) between the model input and
the reconstruction target, suggesting that data augmentations like random
augmentation and color jittering might not be suitable or require specific
designs for MIM, which will be explored in the future work.

#### Limitations.

Although demonstrating significant performance, we notice that the
$\operatorname{TopK}(\cdot)$ sampling accuracy in homologous attention still
cannot achieve 100% as shown in Fig. 7, for which there exist several
potential reasons accounting: 1) The background patches might be included
during random cropping inevitably, which are difficult for attention-based
methods to recognize. 2) There is still further improvement space for MixedAE.
For example, more strict verification than homologous contrastive loss is an
appealing future work direction.

##  Appendix E More Visualizations

We provide more visualizations of the attention mapes learnt by MAE and
MixedAE on images from ImageNet-1K [14], ADE20K [62] and Microsoft COCO [37]
datasets in Fig. 9. As demonstrated, our MixedAE can generate more reasonable
and discriminative attention maps, revealing the effectiveness of MixedAE to
conduct object-aware pre-training without any specifically designed modules
(_e.g_., K-means [8] and object discovery network [29]).

Figure 9: More visualizations of attention maps from ImageNet-1K (1-3 rows),
ADE20K (4-6 rows) and COCO (7-12 rows).