# Asymmetric Patch Sampling for Contrastive Learning

Chengchao Shen1, Jianzhong Chen1, Shu Wang1, Hulin Kuang1, Jin Liu1, Jianxin
Wang1
1Central South University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Asymmetric appearance between positive pair effectively reduces the risk of
representation degradation in contrastive learning. However, there are still a
mass of appearance similarities between positive pair constructed by the
existing methods, which inhibits the further representation improvement.

In this paper, we propose a novel asymmetric patch sampling strategy for
contrastive learning, to further boost the appearance asymmetry for better
representations. Specifically, dual patch sampling strategies are applied to
the given image, to obtain asymmetric positive pairs. First, sparse patch
sampling is conducted to obtain the first view, which reduces spatial
redundancy of image and allows a more asymmetric view. Second, a selective
patch sampling is proposed to construct another view with large appearance
discrepancy relative to the first one.

Due to the inappreciable appearance similarity between positive pair, the
trained model is encouraged to capture the similarity on semantics, instead of
low-level ones. Experimental results demonstrate that our proposed method
significantly outperforms the existing self-supervised methods on both
ImageNet-1K and CIFAR dataset, e.g., 2.5% finetune accuracy improvement on
CIFAR100. Furthermore, our method achieves state-of-the-art performance on
downstream tasks, object detection and instance segmentation on COCO.
Additionally, compared to other self-supervised methods, our method is more
efficient on both memory and computation during training. The source code is
available at https://github.com/visresearch/aps.

##  1 Introduction

Figure 1: Asymmetric patch sampling strategy to construct positive pairs with
large appearance discrepancy, which effectively reduces the appearance
similarity but without changing image semantics. By contrasting positive pairs
with inappreciable appearance similarity, the trained model is encouraged to
capture the similarity from the perspective of image semantics.

In recent years, massive breakthroughs are achieved in self-
supervised/unsupervised learning field. Based on the difference of pretext
tasks, the popular branches contain contrastive learning (CL) [35, 5, 18, 16,
9, 37, 33] and masked image modeling (MIM) [36, 12, 17, 15, 7].

For contrastive learning task, the trained model is required to discriminate
different views of the same images from other images, named instance
discrimination [35]. To learn semantic representations of images, contrastive
learning methods introduce a series of asymmetric designs, such as data
augmentation[5], to increase the appearance discrepancy between positive
pairs, but without changing image semantics. In this way, the trained model is
encouraged to understand the semantics in images, instead of some trivial
features. Therefore, plausible asymmetric designs are significantly important
for the performance of contrastive learning. However, due to image overlap
between positive pair, there is still a mass of appearance similarity in the
existing contrastive learning methods, which degrades the representations.

Different from contrastive learning, MIM task follows the idea of masked
language modeling task (MLM) [26, 27, 2, 11] in natural language processing
(NLP), where partially masked data are fed into the model to predict the
invisible part of data in an auto-encoding manner. Due to the heavy spatial
redundancy of image, the highly random masked images in MIM task can still
effectively retain the semantics of the original images [17], which achieves
very promising performance in self-supervised learning. However, with the
similar semantics, the raw pixels or their tokens have a large fluctuation on
appearance, leading to non-unique solutions to invisible patch reconstruction
from the random masked images, especially when the masked ratio is large. The
existing MIM methods attempt to map the highly masked image into a fixed
target, which inevitably introduces large fitting error, even if the
prediction is a plausible solution for the given input. We call it as _non-
unique target issue_ , which substantially limits the flexibility of the MIM
models.

Inspired by the above observations, we propose a novel asymmetric patch
sampling strategy, to introduce more asymmetry for contrastive learning and
alleviate the non-unique target issue suffered by the existing MIM methods at
the same time. Specifically, the improvement is two fold. First, to improve
the semantics of contrastive pretext task, the proposed sampling strategy
constructs positive pairs, where the two views are essentially semantic
consistent but with inappreciable similarity on appearance. Second, compared
to MIM methods, we replace the reconstruction objective with contrastive one,
which provides more flexible targets for training.

As shown in Figure 1, our proposed method respectively adopts two different
patch sampling strategies for two views of positive pair, named _asymmetric
patch sampling strategy_ (_APS_). For the first view, we conduct sparse patch
sampling [17] to obtain highly sparse patch sequences, which only contain
small portion of patches from the original image, e.g., 25%. This operation
aims to reduce spatial redundancy of image and encourage the Visual
Transformer (ViT) network to model long distance dependency among patches. For
the second view, we conduct a selective patch sampling, where the patches not
appearing in the first view are preferred to be sampled. In this way, the
appearance of the sampled views are essentially different from each other. To
minimizing the contrastive objective, the ViT model is required to understand
the semantics of images. Furthermore, we formally analyze the asymmetry of the
proposed method. To improve training stability of contrastive learning, we
further propose an adaptive gradient clip operation.

In summary, our main contribution is an asymmetric patch sampling strategy to
construct efficient positive pairs for contrastive learning, which allows two
views to represent the same object with essentially different appearances. The
trained model is hard to fit the contrastive objective by only simply
comparing low-level features. Thus, it is encouraged to extract semantic
representations from the asymmetric positive pairs. Experimental results
demonstrate that the proposed APS achieves excellent performance on
unsupervised representation learning. Specifically, for both ViT-Small/16 and
ViT-Base/16, our method significantly outperforms the previous best results on
ImageNet-1K. When pretrained on small datasets, CIFAR10 and CIFAR100, our
method respectively surpasses the previous state-of-the-art method by $1.2\%$
and $2.5\%$ using ViT-Tiny/2 backbone. Furthermore, our method also achieve
state-of-the-art performance on downstream tasks, object detection and
instance detection on COCO.

Figure 2: The overview of asymmetric patch sampling for contrastive learning.
Dual asymmetric patch sampling strategies, sparse sampling and selective
sampling, are conducted on the input image, to construct positive pairs with
large appearance discrepancy. The visual transformer encoder is trained to
minimize distance between positive pairs obtained by asymmetric sampling (hard
samples) and meanwhile maximize distance between negative pairs.

##  2 Related Work

###  2.1 Masked Image Modeling

Following MLM task [26, 27, 2, 11] in NLP, masked image modeling is proposed
as a novel pretext task for self-supervised learning in computer vision. In
this task, the masked raw pixels [17, 14, 36] or their tokens [1, 12, 34] are
used as the targets for model training. However, with the similar semantics,
the raw pixels or their tokens have a large fluctuation on appearance, which
may provide not so stable supervision signal. Moreover, the solutions of the
given masked image are generally not the unique one, especially when the
masked ratio is high, which may cause large prediction error on a plausible
prediction and effect the flexibility of model learning.

To this end, feature prediction based methods are proposed to alleviate the
above issue. iBOT [38] performs self-distillation to recover invisible patch
tokens from visible ones. CAE [7] adopts the representation alignment between
visible patches and invisible patches before decoder in MIM task. SIM [31]
introduces siamese network structure to align semantics between the tokens
from different augmented views by mask operation. In spite of the encouraging
results achieved, these methods still heavily rely on the unstable targets for
pixel/token reconstruction.

In contrast, the proposed method models masked images by learning instance
similarity between significantly different views sampled by asymmetric
sampling strategy, which provides more flexible and stable targets for self-
supervised learning.

###  2.2 Contrastive Learning

Contrastive learning conducts instance classification by maximizing
representation similarity under different distortions of a sample (positive
pair) and minimizing the one of different samples (negative pair), to learn
invariant representation of data under different distortions [35, 5, 18, 16,
9, 37, 33]. To avoid trivial solutions and learn valid representation,
asymmetric designs play a vital role in contrastive learning, which introduce
a series of variances on low-level features but without changing the semantics
of images [37, 33]. The most important asymmetric design is a series of data
augmentation techniques applied on positive pair, which is the most widely
adopted by popular contrastive learning methods [5, 8, 16, 3, 6]. For example,
color jitter, gray scale and solarize operation significantly change the color
of images in positive pairs, so the model in the contrastive setting is
required to capture the color-invariant representations. Then random crop
operation introduces variance on object parts and scale, which further removes
the dependency on object parts and scale for model. Therefore, the model is
trained to recognize objects using semantic features, instead of trivial ones.
The asymmetric designs are also introduced into network architectures, such as
prediction head [16, 9, 10] and momentum encoder [18, 8, 16, 10, 4], which
disturb the representation of positive pairs.

In this work, we introduce a novel effective asymmetric operation, asymmetric
patch sampling strategy. It constructs a series of significant different
positive pairs, each of which contains quite little appearance similarity with
their distorted version. To minimizing the contrastive objective, the model is
encouraged to learn more semantic and useful representation.

###  2.3 Hard Sample Mining

Hard sampling mining is widely adopted in object detection [30, 21], to
alleviate the extreme foreground-background class imbalance of dataset. OHEM
[30] selects the ROI with large loss to update the model during training.
Focal loss [21] dynamically scales cross entropy loss for training samples,
which encourages the model to focus on hard samples.

These methods tap hard samples according to the loss value during training for
object detection. However, they also boost the potential negative effect
caused by mislabeled samples. In contrast, our method directly constructs hard
positive pairs by sampling different patch combinations from the same objects,
which obtains informative samples with fewer potential mislabeled ones.

##  3 Method

###  3.1 Overview

In the setting of contrastive learning, it’s believed that positive pairs with
large appearance discrepancy can effectively regularize the representation
learning. To further improve appearance discrepancy, we adopt sparse patch
sampling strategy as masked image modeling methods, in which only small
portion of patches are sampled to construct positive pairs. In this way, two
views of positive pair have fewer overlapping patches and effectively improve
the asymmetry on low-level features.

As shown in Figure 2, the input image $x$ is randomly cropped into two sub-
images $x_{1}^{\prime}$ and $x_{2}^{\prime}$, and then processed by data
augmentation $\mathcal{T}_{1}(\cdot)$ and $\mathcal{T}_{2}(\cdot)$ to obtain
the positive pair for contrastive learning, which can be represented as

$\displaystyle[x_{1}^{\prime},B_{1}]=\mathcal{T}_{1}(x),$ (1)
$\displaystyle[x_{2}^{\prime},B_{2}]=\mathcal{T}_{2}(x),$

where $B_{i}(i=1,2)$ denotes the bounding box produced by random image crop.

To further reduce the appearance similarity on patches, we conduct asymmetric
sampling strategy $\mathcal{A}(\cdot)$, including sparse sampling and
selective sampling, to lower the overlapping probability between the two
target views as follows,

$\displaystyle[x_{1},x_{2}]=\mathcal{A}(x_{1}^{\prime},x_{2}^{\prime},B_{1},B_{2};s_{1},s_{2}),$
(2)

where $s_{i}(i=1,2)$ denotes the patch sampling ratio of $x_{i}^{\prime}$,
e.g., $s_{1}=s_{2}=0.25$.

Then, combined with projection head $\mathcal{G}(\cdot)$, the positive pair
$[x_{1},x_{2}]$ is respectively fed into visual transformer
$\mathcal{F}(\cdot)$ to obtain the representation
$z_{i}=\mathcal{G}(\mathcal{F}(x_{i}))$. Additionally, prediction head module
$\mathcal{H}(\cdot)$ is adopted, to increase the asymmetry between the
representations of positive pair. The output can be written as
$q_{i}=\mathcal{H}(z_{i})$.

Finally, we conduct contrastive learning between projection representation
$z_{i}$ and prediction representation $q_{i}$, to learn appearance-invariant
representations from positive pair with extremely asymmetric appearance.

Figure 3: The overlap ratio computation between two randomly cropped views.

###  3.2 Asymmetric Patch Sampling

As shown in Figure 3, the overlap ratio between patches from $x_{1}^{\prime}$
and $x_{2}^{\prime}$ can be computed as follows,

$\displaystyle r_{\rm overlap}=\frac{\mathcal{S}(P_{1}\cap
P_{2})}{\mathcal{S}(P_{2})},$ (3)

where $P_{i}(i=1,2)$ denotes the sampled patch in view $i$ and
$\mathcal{S}(\cdot)$ denotes the area of the given patch. To reduce the
sampling probability of overlapping patches, we propose a selective patch
sampling method, whose sampling probability density $p_{\rm sel}$ is computed
by

$\displaystyle p_{\rm sel}=(\gamma+1)\cdot s_{1}\cdot(1-r_{\rm
overlap})^{\gamma},$ (4)

where $\gamma$ is a hyper-parameter to tune the sensitivity of sampling. As
shown in Figure 4, the larger $\gamma$ is, the smaller possibility of the
sampled overlapping patches is. This selective sampling method and sparse
sampling form the asymmetric sampling strategy.

Since sparse sampling strategy uniformly samples patch from the first view
$x_{1}$, the probability of patches sampling in the overlapping area
$x_{1}\cap x_{2}$ is also $s_{1}$ as the one in $x_{1}$. Hence, the sampling
probability density is required to meet the following equation,

$\displaystyle\int_{0}^{1}p_{\rm sel}{\rm d}r_{\rm overlap}=s_{1},$ (5)

which guarantees that the total probability meets the ratio of our proposed
sampling strategy in $x_{1}\cap x_{2}$111More details can be found in the
supplementary material.. In other words, all areas under the curves in Figure
4 equal to $s_{1}$.

Figure 4: Asymmetric sampling curve. The higher the overlap ratio is, the
smaller the patch sampling probability is.

###  3.3 Asymmetry Analysis

In this section, we formally analyze the asymmetry of patch sampling strategy,
to quantify the appearance discrepancy between positive pairs. For
convenience, we focus on the spatial asymmetry, which is simply defined as the
non-overlap ratio between the images of positive pair. With consideration of
sampling randomness, we measure the spatial asymmetry by the expectation of
non-overlap ratio between positive pairs, where non-overlap ratio is defined
as $r_{\rm non}=1-r_{\rm overlap}$ and $r_{\rm overlap}$ denotes the overlap
between images from positive pair. The expectation of non-overlap ratio
$\mathbb{E}_{\rm non}$ can be obtained from the expectation of overlap ratio
$\mathbb{E}_{\rm overlap}$ by $\mathbb{E}_{\rm non}=1-\mathbb{E}_{\rm
overlap}$. For brevity, we focus on the analysis of $\mathbb{E}_{\rm overlap}$
as follows.

As shown in Figure 5 (a), a naive patch sampling strategy uniformly samples
patches from the same image crop to construct positive pairs for contrastive
learning. The sampling probability of each patch in crop $x_{1}^{\prime}$ and
$x_{2}^{\prime}$ can be regarded as their sampling ratios, $s_{1}$ and
$s_{2}$, respectively. This patch sampling strategy subjects to bernoulli
distribution, where each patch in the crop takes overlap probability of
$s_{1}\cdot s_{2}$ and non-overlap probability of $(1-s_{1}\cdot s_{2})$.
Therefore, the expectation of overlap ratio between positive pair by the naive
path sampling strategy can be computed as

$\displaystyle\mathbb{E}_{\rm overlap}^{\rm naive}=1\cdot s_{1}\cdot
s_{2}+0\cdot(1-s_{1}\cdot s_{2})=s_{1}\cdot s_{2}.$ (6)

This sparse patch sampling strategy can effectively reduce the overlap between
positive pair. For example, when $s_{1}=s_{2}=0.25$, the expectation of ratio
$\mathbb{E}_{\rm overlap}^{\rm naive}=0.0625$, which means only $6.25\%$
region is overlapped in the two views of positive pairs on average and
significantly improves asymmetry.

Figure 5: Asymmetry analysis. Two alternative patch sampling strategies. (a)
randomly sample patches from the same cropped image from origin image; (b)
combine random crop operation and random patch sampling one.

To further improve the asymmetry between positive pair, we combine the random
image crop operation and the asymmetric patch sampling strategy as Figure 5
(b). So the overlap ratio between the patches from $x_{1}$ and $x_{2}$ varies
from 0 to 1 ($r_{\rm overlap}\in[0,1]$), instead of discrete values, $0$ and
$1$, as naive patch sampling strategy, which allows introducing more
asymmetry. Specifically, we conduct asymmetric patch sampling using Eq. 11. As
naive patch sampling, the overlap probability density of asymmetric sampling
can be computed by $p_{\rm overlap}=p_{\rm sel}\cdot s_{2}$. So the
expectation of overlap ratio between positive pair can be obtained by

$\displaystyle\mathbb{E}_{\rm overlap}^{\rm sel}=\int_{0}^{1}{p_{\rm
overlap}\cdot r_{\rm overlap}{\rm d}r_{\rm overlap}}=\frac{s_{1}\cdot
s_{2}}{\gamma+2}.$ (7)

In other words, the overlap ratio of asymmetric patch sampling strategy is
$\frac{1}{\gamma+2}$ of the one by naive patch sampling strategy, which
effectively improves the asymmetry between positive pair. For example, when
$\gamma=3$, the expectation of overlap ratio is only $20\%$ of the naive one
according to Eq. 6 and Eq. 7.

###  3.4 Optimization

In this section, we use for reference loss function in [16, 9, 5, 10] to form
our contrastive objective as:

$\displaystyle\mathcal{L}_{\rm
contrast}=\tau\cdot\left[\mathcal{D}(q_{1},\operatorname{sg}(z_{2}))+\mathcal{D}(q_{2},\operatorname{sg}(z_{1}))\right],$
(8)
$\displaystyle\mathcal{D}(q,z)=-\sum_{i=1}^{N}\log\frac{\exp(\operatorname{sim}(q^{(i)},z^{(i)})/\tau)}{\sum_{j=1}^{N}\exp(\operatorname{sim}(q^{(i)},z^{(j)})/\tau)},$
$\displaystyle\operatorname{sim}(q,z)=\frac{q^{\rm T}\cdot
z}{\|q\|\cdot\|z\|},$

where $\tau$ and $N$ respectively denote temperature parameter and batch size,
$q^{(i)}$ and $z^{(i)}$ respectively denote the representation $q$ and $z$ of
$i$-th sample in mini-batch, $\operatorname{sg}(\cdot)$ denotes stop gradient
operation222Temperature $\tau$ in $\mathcal{L}_{\rm contrast}$ is used to
simplify the learning rate tuning under different temperature values. More
details can be found in the supplementary material. Compared to [16, 9, 5,
10], the main difference of our objective function is the combination of the
normalized temperature-scaled cross entropy loss and stop gradient operation.

As clarified in MoCo-v3 [10], contrastive learning methods for vision
transformer tend to suffer an unstable optimization issue, which significantly
hurts the performance. To stabilize the training, gradient clip strategy is a
popular technique, which scales $\ell_{2}$-norm of the excessive gradients to
the given maximal norm. However, this clip strategy fails to adapt to the
case, where the gradients progressively decay but sometimes abnormal
fluctuation during contrastive training. To solve this issue, we set an
adaptive threshold for step $t$ according to the exponential moving average of
gradient $\mathcal{G}_{t}$ as follows:

$\displaystyle\mathcal{G}_{t}=m\cdot\mathcal{G}_{t-1}+(1-m)\cdot g_{t},$ (9)

where $m\in[0,1)$ is a momentum coefficient and $g_{t}$ denotes the gradients
with respect to model parameters at step $t$. When
$\|g_{t}\|>\alpha\cdot\|\mathcal{G}_{t-1}\|$, the gradient $g_{t}$ is scaled
by the norm of threshold $\mathcal{G}_{t-1}$:

$\displaystyle\hat{g}_{t}=g_{t}\cdot\frac{\|\mathcal{G}_{t-1}\|}{\|g_{t}\|+\epsilon},$
(10)

where $\epsilon$ is set to $10^{-8}$ for better numerical stability, to adjust
the amplitude of gradients into a reasonable range and stabilize the training
of contrastive learning.

##  4 Experiments

In this section, we conduct experiments to evaluate the effectiveness of our
proposed method. First, we give the experiment settings, including model
backbone, optimization hyperparameters and other details. Then, we compare our
method with state-of-the-art methods on ImageNet-1K and CIFAR. Finally, we
further analyze the effectiveness of each component in our method by ablation
study.

Method | Backbone | Add. Module | #Views | Supervision | #Epochs | Finetune (%)
---|---|---|---|---|---|---
MoCo v3 [10] | ViT-S/16 | pred. head | 2 | contrastive rep. | 300 | 81.4
MAE [17] | ViT-S/16 | ViT decoder | 1 | invisible pixel | 1600 | 79.5
CAE [7] | ViT-S/16 |  DALL-E & ViT decoder | 2 |  invisible visual token | 300 | 81.8
BEiT [1] | ViT-S/16 |  DALL-E & ViT decoder | 2 |  invisible visual token | 300 | $81.7$
iBOT [38] | ViT-S/16 | proj. head | 2 | self distillation | 800 | 81.8
APS (ours) | ViT-S/16 | pred. head | 2 | contrastive rep. | 300 | 82.1
APS (ours) | ViT-S/16 | pred. head | 2 | contrastive rep. | 800 | 82.6
supervised [13] | ViT-B/16 | / | 1 | images from JFT | / | 79.9
MoCo v3 [10] | ViT-B/16 | pred. head | 2 | contrastive rep. | 300 | 83.0
DINO [4] | ViT-B/16 | pred. head | 12 | contrastive rep. | 400 | 82.8
MAE [17] | ViT-B/16 | ViT decoder | 1 | invisible pixel | 1600 | 83.6
CAE [7] | ViT-B/16 |  DALL-E & ViT decoder | 2 |  invisible visual token | 800 | 83.6
BEiT [1] | ViT-B/16 |  DALL-E & ViT decoder | 2 |  invisible visual token | 300 | 83.2
MixMIM [23] | ViT-B/16 | ViT decoder | 2 | invisible pixel | 300 | 83.2
SimMIM [36] | ViT-B/16 | pred. head | 1 | invisible pixel | 800 | 83.8
MaskFeat [34] | ViT-B/16 | pred. head | 1 | HOG feature | 1600 | 84.0
iBOT [38] | ViT-B/16 | proj. head | 14 | self distillation | 800 | 83.8
APS (ours) | ViT-B/16 | pred. head | 2 | contrastive rep. | 300 | 84.2
APS (ours) | ViT-B/16 | pred. head | 2 | contrastive rep. | 800 | 84.7
Table 1: Performance comparison on ImageNet-1K dataset. “pred. head” and
“proj. head” respectively denote prediction and projection head. “contrastive
rep.” denotes contrastive representation. “NA” denotes the result is not
available in the original paper.

|  |  |  | CIFAR10 | CIFAR100
---|---|---|---|---|---
Method | Backbone | Memory (G) | #FLOPs (G) | #Epochs | Accuracy (%) | #Epochs | Accuracy (%)
supervised | ViT-T/2 | / | / | 300 | 89.6 | 300 | 68.1
MoCo-v3 [10] | ViT-T/2 | 40.0 | 21.9 | 1600 | 96.3 | 1600 | 81.4
MAE [17] | ViT-T/2 | 13.3 | 4.8 | 1600 | 96.1 | 1600 | 76.3
SimMIM [36] | ViT-T/2 | 28.6 | 6.0 | 1600 | 95.3 | 1600 | 79.1
iBOT [38] | ViT-T/2 | 58.5 | 59.6 | 1600 | 96.3 | 1600 | 80.1
APS (ours) | ViT-T/2 | 9.0 | 4.2 | 1600 | 97.4 | 1600 | 83.9
APS (ours) | ViT-T/2 | 9.0 | 4.2 | 3200 | 97.4 | 3200 | 83.8
supervised | ViT-S/2 | / | / | 300 | 90.3 | 300 | 68.9
MoCo-v3 [10] | ViT-S/2 | 80.0 | 87.6 | 1600 | 97.3 | 1600 | 83.1
MAE [17] | ViT-S/2 | 25.5 | 19.2 | 1600 | 97.0 | 1600 | 80.0
SimMIM [36] | ViT-S/2 | 57.9 | 24.6 | 1600 | 97.1 | 1600 | 83.2
iBOT [38] | ViT-S/2 | 105.3 | 238.5 | 1600 | 97.0 | 1600 | 82.8
APS (ours) | ViT-S/2 | 16.2 | 16.2 | 1600 | 98.0 | 1600 | 84.8
APS (ours) | ViT-S/2 | 16.2 | 16.2 | 3200 | 98.1 | 3200 | 85.6
IN pretrained [32] | ViT-B/16 | / | / | 300 | 98.1 | 300 | 87.1
supervised | ViT-B/2 | / | / | 300 | 92.7 | 300 | 72.6
MoCo-v3 [10] | ViT-B/2 | 159.3 | 348.2 | 1600 | 97.2 | 1600 | 82.6
MAE [17] | ViT-B/2 | 50.8 | 76.2 | 1600 | 94.7 | 1600 | 72.3
SimMIM [36] | ViT-B/2 | 115.3 | 97.6 | 1600 | 97.1 | 1600 | 82.8
APS (ours) | ViT-B/2 | 33.1 | 66.2 | 1600 | 98.2 | 1600 | 85.1
APS (ours) | ViT-B/2 | 33.1 | 66.2 | 3200 | 98.2 | 3200 | 86.1

Table 2: Performance comparison on CIFAR dataset using finetune evaluation
protocol. “Memory” denotes GPU memory consumed during self-supervised
training, where batch size is set to 512 and automatic mixed precision is
applied. “#FLOPs” denotes the number of floating-point operations per
iteration during training, where batch size is set to 1. “IN pretrained”
denotes the model pretrained on ImageNet-1K using supervised cross entropy
loss and then finetuned on CIFAR with $224\times 224$ size.

###  4.1 Experimental Settings

####  4.1.1 Data Processing

For both ImageNet-1K [29] and CIFAR [20], we sample $25\%$ patches from the
given images, namely sampling ratio $s_{1}=s_{2}=0.25$. The sizes of input
image are $224\times 224$ and $32\times 32$ for ImageNet-1K and CIFAR,
respectively. Additionally, more data augmentations applied on ImageNet-1K and
CIFAR are clarified in the supplementary material.

####  4.1.2 Network Architecture

For ImageNet-1K, we adopt ViT-Small/16 and ViT-Base/16 [13] (respectively
denoted as _ViT-S/16_ and _ViT-B/16_) as the backbone, respectively. Following
MoCo-v3 [10], projection head and prediction head module are added and the
detail can be found in the supplementary material. During pretraining, we
adopt fixed sine-cosine position embedding and random initialized patch
embedding, which are combined with adaptive gradient operation to improve
training stability. The momentum encoder is also applied, whose ema
coefficient varies from $0.99$ to $1.0$ as cosine scheduler.

For CIFAR10 and CIFAR100, we conduct experiments on ViT-Tiny/2 and ViT-Small/2
(respectively denoted as _ViT-T/2_ and _ViT-S/2_), whose structures are
depicted in the supplementary material. Different from experiments on
ImageNet-1K, both position and patch embedding are learnable, without training
instability issue. Additionally, momentum encoder is not adopted on CIFAR.

####  4.1.3 Optimization

For loss function, the temperature $\tau$ is set to $0.1$ on both ImageNet-1K
and CIFAR. For ImageNet-1K, we use AdamW optimizer [24] with batch size
$4096$, learning rate $1.28\times 10^{-3}$, momentum $0.9$ and weight decay
$0.1$. During pretraining, we conduct learning rate warmup for $20$ epochs and
then follow a cosine learning rate decay schedule for the rest $780$ epochs.
To further stabilize the training, adaptive gradient clip operation is
implemented on each transformer block, where $m=0.4$ and $\alpha=1.05$. For
patch sampling, we set $\gamma=3$ to increase the appearance discrepancy
between positive pair.

For CIFAR10 and CIFAR100, we use AdamW optimizer with batch size $512$,
learning rate $1\times 10^{-3}$, momentum $0.9$ and weight decay $0.05$. The
model is trained by $1,600$ epochs, where the first $20$ epochs for warmup.
Additionally, gradient clip operation is not conducted on the model for CIFAR.

####  4.1.4 Evaluation

Linear probing has been a popular evaluation protocol for contrastive
learning. However, it fails to evaluate the quality of non-linear features [1,
17, 25]. For our proposed method, the input image fed into network is highly
sparse instead of complete one during training, which is significantly
different from the input samples during evaluation. Hence, we adopt finetune
protocol to evaluate the representation of our model, instead of linear
probing one. By default, all pretrained models are finetuned for 100 epochs.

###  4.2 Image Classification

####  4.2.1 Results on ImageNet-1K

As show in Table 1, our proposed method achieves $82.6\%$ finetune accuracy on
ViT-B/16, which significantly outperforms the previous SOTA method iBOT by
$0.8\%$ finetune accuracy. Moreover, our proposed method achieves $84.7\%$
finetune accuracy on ViT-B/16, which outperforms the previous SOTA method iBOT
by $0.9\%$ finetune accuracy. We believe that our method tends to extract more
semantic representation by modeling representation similarity between
asymmetric positive pairs. Moreover, our method does not require additional
tokenizer, such as DALL-E [28] to provide visual tokens as supervision signal,
which is more efficient and concise. Compared to previous state-of-the-arts,
MaskFeat and iBOT, our method requires shorter training schedule but achieves
better performance.

####  4.2.2 Results on CIFAR

As shown in Table 2, our method with ViT-T/2 achieves $97.4\%$ and $83.9\%$
finetune accuracy on CIFAR10 and CIFAR100, respectively, surpassing the
previous state-of-the-art method, iBOT by $1.1\%$ and $3.8\%$ on CIFAR10 and
CIFAR100, respectively. For ViT-S/2 backbone, our method consistently
outperforms the previous best finetune accuracy by $1.0\%$ on CIFAR10 and
$2.0\%$ on CIFAR100. For ViT-B/2 backbone, our proposed method achieves
$98.2\%$ and $86.1\%$ accuracy on CIFAR10 and CIFAR100, respectively, which
significantly reduces the performance gap with the model pretrained on large-
scale dataset: ImageNet-1K in a fully supervised manner. Especially on
CIFAR10, our method (input image size: $32\times 32$) evens outperform the
ImageNet-1K pretrained one using ViT-B/16 (input image size: $224\times 224$).
Overall, our method achieves the best performance on CIFAR dataset without
extra data, even outperforms the model pretrained by large-scale dataset:
ImageNet-1K.

Moreover, our method consumes the least memory during training, only about
$\frac{1}{6}$ memory of iBOT and $\frac{1}{5}$ memory of MoCo-v3 on both
ViT-T/2 and ViT-S/2. Therefore, our method is more friendly to hardware than
other self-supervised methods, which sheds some light on more training-
efficient self-supervised learning method. Meanwhile, our method is the most
computation-efficient self-supervised method, only about $\frac{1}{15}$
computation of iBOT and $\frac{1}{5}$ computation of MoCo-v3, thus requires
less training time.

###  4.3 Transfer Learning on Downstream Tasks

To further evaluate the transferability of our method, we conduct transfer
learning experiments on downstream tasks: object detection and instance
segmentation on COCO [22] by Mask RCNN [19] framework. As shown in Table 3,
our proposed APS achieves 51.8 box AP and 46.2 mask AP on objection detection
and instance segmentation, respectively. Compared to previous SOTA method
iBOT, our APS achieves 0.6 box AP and 2.0 mask AP improvement, which
illustrates the effectiveness of our method.

Method | #Pre-Epochs | APbox | APmask
---|---|---|---
MoCo v3 [10] | 300 | 47.9 | 42.7
MAE [17] | 1600 | 50.3 | 44.9
CAE [7] | 300 | 48 | 42.3
iBOT [38] | 1600 | 51.2 | 44.2
APS (ours) | 800 | 51.8 | 46.2
Table 3: Object detection and instance segmentation using pretrained ViT-B/16
model on COCO.

###  4.4 Ablation Study

In this section, we implement ablation study to validate the efficiency of
modules in our method. For convenience, the experiments are mainly conducted
on CIFAR dataset using ViT-T and partially on ImageNet dataset using ViT-S.

####  4.4.1 The Effect of Patch Sampling Ratio

To evaluate the effectiveness of patch sampling ratio, we compare the
performance using different ratios on CIAFAR10 and CIFAR100. As shown in Table
4, our method respectively reports $97.4\%$ and $83.9\%$ accuracy on CIFAR10
and CIFAR100, when sampling ratio $s=0.25$. This experimental result on
CIFAR100 outperforms the one with $s=1.0$ by $1.6\%$, which demonstrates that
patch sampling strategy can effectively improve the representation quality of
contrastive learning. In contrast, the experiment using extremely sparse
sampling ratio $s=0.15$, leads to a $1.9\%$ degradation on CIFAR100, which
significantly damages the performance. We believe that reasonable sparse
sampling strategy applied on images can effectively regularize the contrastive
representation learning.

sampling ratio ($s$) | CIFAR10 (%) | CIFAR100 (%)
---|---|---
1.0 | 96.4 | 82.3
0.75 | 96.9 | 81.9
0.5 | 97.0 | 83.2
0.25 | 97.4 | 83.9
0.15 | 96.7 | 82
Table 4: Performance comparison on different patch sampling ratios.

####  4.4.2 The Effect of Sampling Power

To analyze the effect of sapling power $\gamma$ in Eq. 11, we conduct
experiments on CIFAR10 and CIFAR100 using different values. As shown in Table
5, the model achieves the best performance on both CIFAR10 and CIFAR100 when
$\gamma=3.0$. Especially on CIFAR100, the model with $\gamma=3.0$ outperforms
the one of $\gamma=0.0$ by $2.6\%$. It demonstrates that the model can benefit
from a reasonable sampling power $\gamma$, thus is inclined to extract
semantic representations from asymmetric positive pairs. However, when
$\gamma=4.0$, it does not achieve better performance than the one of
$\gamma=1.0$ or $3.0$. The possible reason is that excessive punishment on
highly overlapping patches may reduce the chance of learning some important
representations from the neglected patches.

For CIFAR10, the performance fluctuation of our method is pretty small under
different $\gamma$ values. This can be in part explained by that, due to less
category diversity of CIFAR10, the trained model can not apparently benefit
from asymmetric patch sampling strategy.

sampling power ($\gamma$) | CIFAR10 (%) | CIFAR100 (%)
---|---|---
0.0 | 97.0 | 81.3
1.0 | 97.0 | 82.0
2.0 | 97.3 | 82.5
3.0 | 97.4 | 83.9
4.0 | 97.0 | 82.8

Table 5: Performance comparison on different asymmetric sampling power values.

####  4.4.3 The Effect of Sampling View Number

Since only small portion of patches are used during the training, we reuse the
rest patches of loaded data to improve the data throughout and training
efficiency. Furthermore, we compare the performance when different numbers of
sampling views are used to training. As shown in Table 6, the experimental
results demonstrate that using all sampling views can significantly improve
the performance of model. Especially on CIFAR100, 1.5% accuracy improvement is
achieved. Additionally, we find that more sampling views can efficiently
improve training stability during experiments.

#sampling | CIFAR10 (%) | CIFAR100 (%)
---|---|---
1 | 96.8 | 82.4
2 | 96.9 | 82.5
3 | 97.2 | 83.3
4 | 97.4 | 83.9
Table 6: Performance comparison on different number of sampling views. In this
case, the sampling ratio is $0.25$ and up to $4$ views can be used for
training.

####  4.4.4 The Effect of Adaptive Gradient Clip

To investigate the effectiveness of adaptive gradient clip operation, we
conduct experiments on small datasets, CIFAR using ViT-T/2 and large-scale
dataset, ImageNet-1K using ViT-S/16 for 300 epochs. The experimental results
are shown in Table 7, demonstrating that the adaptive gradient clip achieves
1.2% accuracy improvement on ImageNet-1K. On the contrary, applying adaptive
gradient clip operation on CIFAR degrades the performance. Due to larger
sample diversity of ImageNet-1K, the dramatic sample variation among
consecutive sample batches is easy to cause instability of model training. The
adaptive gradient clip can effectively improve the training stability of model
on large-scale dataset. Additionally, we find that clip momentum $m=0.4$ can
significantly improve the model performance on ImageNet-1K.

grad. clip ($m$) | CIFAR10 (%) | CIFAR100 (%) | ImageNet-1K (%)
---|---|---|---
0 | 97.4 | 83.9 | 80.9
0.2 | 97.0 | 82.8 | 81.6
0.4 | 97.0 | 82.1 | 82.1
0.8 | 96.8 | 81.9 | 81.8

Table 7: Performance comparison on different gradient clip momentum
coefficients.

##  5 Conclusion

In this paper, we propose a novel asymmetric patch sampling strategy for
contrastive learning. This strategy significantly improves the appearance
discrepancy between positive pairs and the hardness of pretext task for self-
supervised learning. Due to fewer clues to similarity on low-level feature
between positive pairs, the model is encouraged to learn semantic
representations. Then, we formally analyze the asymmetry metric of our method
and compare it with the baseline. Afterwards, we give the optimization
objective of our model. Finally, we propose a novel adaptive gradient clip
operation to stabilize the training of our model. Experimental results
demonstrate that the proposed method is superior to the previous state-of-the-
art on both ImageNet-1K, CIFAR and COCO, while consuming less memory and
computation. For future work, we plan to explore more effective and efficient
asymmetric designs to boost the performance of contrastive learning.

## References

  * [1] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  In International Conference on Learning Representations (ICLR), 2022\.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems (NeurIPS), 33:1877–1901, 2020.
  * [3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems (NeurIPS), 33:9912–9924, 2020.
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9650–9660, 2021.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International Conference on Machine Learning (ICML), pages 1597–1607. PMLR, 2020.
  * [6] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  Advances in Neural Information Processing Systems (NeurIPS), 33:22243–22255, 2020.
  * [7] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang.  Context autoencoder for self-supervised representation learning.  arXiv preprint arXiv:2202.03026, 2022.
  * [8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [9] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750–15758, 2021.
  * [10] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9640–9649, 2021.
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (ACL), pages 1877–1901, 2019.
  * [12] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu.  Peco: Perceptual codebook for bert pre-training of vision transformers.  arXiv preprint arXiv:2111.12710, 2021.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [14] Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei.  Corrupted image modeling for self-supervised visual pre-training.  arXiv preprint arXiv:2202.03382, 2022.
  * [15] Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao.  Convmae: Masked convolution meets masked autoencoders.  arXiv preprint arXiv:2205.03892, 2022.
  * [16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems (NeurIPS), 33:21271–21284, 2020.
  * [17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022.
  * [18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729–9738, 2020.
  * [19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2961–2969, 2017.
  * [20] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [21] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  pages 740–755. Springer, 2014.
  * [23] Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li.  Mixmim: Mixed and masked image modeling for efficient visual representation learning.  arXiv preprint arXiv:2205.13137, 2022.
  * [24] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations (ICLR), 2019\.
  * [25] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  pages 69–84. Springer, 2016.
  * [26] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  OpenAI blog, 2018.
  * [27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [28] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning (ICML), pages 8821–8831. PMLR, 2021.
  * [29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [30] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 761–769, 2016.
  * [31] Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai.  Siamese image modeling for self-supervised vision representation learning.  arXiv preprint arXiv:2206.01204, 2022.
  * [32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning (ICML), pages 10347–10357. PMLR, 2021.
  * [33] Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen.  On the importance of asymmetry for siamese representation learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16570–16579, 2022.
  * [34] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14668–14678, 2022.
  * [35] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3733–3742, 2018.
  * [36] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9653–9663, 2022.
  * [37] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning (ICML), pages 12310–12320. PMLR, 2021.
  * [38] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  In International Conference on Learning Representations (ICLR), 2022\.

##  Appendix A Network Structures

As shown in Table 8, we give the details of backbones used in our experiments.
For ImageNet-1K with input size $224\times 224$, we adopt standard vision
transformer architectures, ViT-Small and ViT-Base, where the patch size for
tokenization is $16\times 16$. For CIFAR10 and CIFAR100 with input size
$32\times 32$, we modify the patch size of standard vision transformer
architectures from $16\times 16$ to $2\times 2$, to adapt the small input
images. We also introduce a more lightweight vision transformer architecture,
ViT-Tiny, which only has half head number and half token dimension of ViT-
Small.

Dataset | Network | Patch Size | #Blocks | #Heads | Token Dim | #Params (M)
---|---|---|---|---|---|---
ImageNet-1K | ViT-Small | 16 | 12 | 6 | 384 | 21.6
| ViT-Base | 16 | 12 | 12 | 768 | 85.7
CIFAR | ViT-Tiny | 2 | 12 | 3 | 192 | 5.4
| ViT-Small | 2 | 12 | 6 | 384 | 21.3
| ViT-Base | 2 | 12 | 12 | 768 | 85.1
Table 8: The structure of visual transformer backbones. “#Blocks” denotes the number of standard transformer blocks in backbone. “Token Dim” denotes the dimension of visual token vector.  Dataset | Layer |  Projection Head |  Prediction Head
---|---|---|---
ImageNet-1K | 1 | Linear (4096) + BN + ReLU | Linear (4096) + BN + ReLU
| 2 | Linear (4096) + BN + ReLU | Linear (256) + BN*
| 3 | Linear (256) + BN* |
CIFAR | 1 | Linear (512) + BN + ReLU | Linear (512) + BN + ReLU
| 2 | Linear (512) + BN + ReLU | Linear (512) + BN + ReLU
| 3 | Linear (128) + BN* | Linear (128) + BN*
Table 9: The structure of projection and prediction heads. “Linear ($m$)”
denotes linear layer with output size $m$. “BN” and “ReLU” denote batch
normalization and rectify linear unit operation, respectively. “BN*” denotes
batch normalization without learnable parameters.

As shown in Table 9, we further present the structure of projection and
prediction head adopted during self-supervised pretraining. For ImageNet-1K,
there are 3 linear layers in projection head and 2 linear layers in prediction
heads. The first two linear layers are followed by batch normalization and
rectify linear unit in turn, and the output sizes of them are both 4096. The
last linear of both heads are followed by only batch normalization, and output
sizes of them are 256. For CIFAR10 and CIFAR100, the configurations of
projection and prediction head are similar to the ones of ImageNet-1K. The
main differences can be summarized as follows. First, the output sizes of
projection and prediction are both modified to 128. Second, the sizes of
hidden units are both modified to 512.

##  Appendix B Data Augmentations

As shown in Table 10, we describe the parameters of data augmentations used
during self-supervised pretraining. For ImageNet-1K, 6 data augmentation
techniques are applied to the input images, including random crop and resize,
horizontal flip, color jittering, gray scale, Gaussian blurring, as well as
solarization. There are two differences between the augmentations for the
construction of positive pair. First, the probability of Gaussian blurring is
1.0 in augmentation $\mathcal{T}_{1}(\cdot)$, but 0.1 in augmentation
$\mathcal{T}_{2}(\cdot)$. Second, solarization is only used in augmentation
$\mathcal{T}_{2}(\cdot)$. For CIFAR10 and CIFAR100, due to small image size,
Gaussian blurring is not used. Additionally, solarization is also not adopted.

|  | ImageNet-1K | CIFAR
---|---|---|---
Augmentation |  Parameter |  Aug. $\mathcal{T}_{1}(\cdot)$ |  Aug. $\mathcal{T}_{2}(\cdot)$ |  Aug. $\mathcal{T}_{1}(\cdot)$ |  Aug. $\mathcal{T}_{2}(\cdot)$
random crop and resize | area of the crop | [0.08, 1.0] | [0.08, 1.0] | [0.15, 1.0] | [0.15, 1.0]
| aspect ratio of the crop | [$\frac{3}{4}$, $\frac{4}{3}$] | [$\frac{3}{4}$, $\frac{4}{3}$] | [$\frac{3}{4}$, $\frac{4}{3}$] | [$\frac{3}{4}$, $\frac{4}{3}$]
random horizontal flip | horizontal flip probability | 0.5 | 0.5 | 0.5 | 0.5
random color jittering | color jittering probability | 0.8 | 0.8 | 0.8 | 0.8
| max brightness adjustment intensity | 0.4 | 0.4 | 0.4 | 0.4
| max contrast adjustment intensity | 0.4 | 0.4 | 0.4 | 0.4
| max saturation adjustment intensity | 0.2 | 0.2 | 0.4 | 0.4
| max hue adjustment intensity | 0.1 | 0.1 | 0.1 | 0.1
random gray scale | color dropping probability | 0.2 | 0.2 | 0.2 | 0.2
random Gaussian blurring | Gaussian blurring probability | 1.0 | 0.1 | / | /
| sigma of Gaussian blurring | [0.1, 2.0] | [0.1, 2.0] | / | /
random solarization | solarization probability | 0.0 | 0.2 | / | /

Table 10: The parameters of data augmentations applied during self-supervised
training. “[·, ·]” denotes the range for uniform sampling.

##  Appendix C Asymmetric Patch Sampling

To reduce the probability of sampling in the overlapping patch, we adopt an
asymmetric sampling strategy, where the sampling can be represented as

$\displaystyle p_{\rm sel}=\rho\cdot(1-r_{\rm overlap})^{\gamma},$ (11)

where $\rho$ is the coefficient required to determined. So, as shown in Figure
4 of the main paper, the larger $\gamma$ is, the smaller possibility of the
sampled overlapping patches is. Due to uniform sampling conducted in the first
view $x_{1}^{\prime}$, the probability of patch sampling in the union region
$x_{1}^{\prime}\cap x_{2}^{\prime}$ can also be regarded as $s_{1}$. In other
words, this condition can be presented as

$\displaystyle\int_{0}^{1}p_{\rm sel}{\rm d}r_{\rm overlap}=s_{1}.$ (12)

Combining with Eq. 11, we obtain

$\displaystyle\int_{0}^{1}p_{\rm sel}{\rm d}r_{\rm overlap}=s_{1}$
$\displaystyle\Rightarrow\int_{0}^{1}\rho\cdot(1-r_{\rm overlap})^{\gamma}{\rm
d}r_{\rm overlap}=s_{1}$ $\displaystyle\Rightarrow\frac{\rho}{\gamma+1}=s_{1}$
$\displaystyle\Rightarrow\rho=(\gamma+1)\cdot s_{1}$ (13)

Combining with Eq. 11, we can obtain $p_{\rm sel}=(\gamma+1)\cdot
s_{1}\cdot(1-r_{\rm overlap})^{\gamma}$, namely Eq. 11 of the main paper.

##  Appendix D Algorithm

To clearly clarify our proposed method, we give the overall algorithm as shown
in Algorithm 1.

Algorithm 1 Asymmetric Patch Sampling for Contrastive Learning

1:Unlabeled training data $\mathcal{X}=\\{x\\}$.

2:The parameters of ViT model $\mathcal{F}(\cdot)$.

3:Initialize the parameters of ViT model $\mathcal{F}(\cdot)$, projection head
$\mathcal{G}(\cdot)$ and prediction head $\mathcal{H}(\cdot)$;

4:for number of training iterations do

5: Obtain augmented views $[x_{1}^{\prime},x_{2}^{\prime}]$ and crop boxes
$[B_{1}^{\prime},B_{2}^{\prime}]$ by applying augmentation
$\mathcal{T}_{1}(\cdot)$ and $\mathcal{T}_{2}(\cdot)$, respectively;

6: Obtain view $x_{1}$ by uniformly sampling $s\times 100$ percent of the
patches from $x_{1}^{\prime}$;

7: Compute patch overlap ratio $r_{\rm overlap}$ between view $x_{1}$ and view
$x_{2}^{\prime}$;

8: Sample $s\times 100$ percent of the patches from $x_{2}^{\prime}$ to obtain
another view $x_{2}$ by Eq. 11;

9: Feed $x_{1}$ and $x_{2}$ into ViT model $\mathcal{F}(\cdot)$ and projection
head $\mathcal{G}(\cdot)$, to obtain $z_{1}=\mathcal{G}(\mathcal{F}(x_{1}))$
and $z_{2}=\mathcal{G}(\mathcal{F}(x_{2}))$;

10: Feed $z_{1}$ and $z_{2}$ into prediction head $\mathcal{H}(\cdot)$, to
obtain $q_{1}=\mathcal{H}(z_{1})$ and $q_{2}=\mathcal{H}(z_{2})$;

11: Compute contrastive loss $\mathcal{L}_{\rm contrast}$ cross $z_{j}$ and
$q_{k}$ by Eq. 8, where $j,k\in\\{1,2\\}$;

12: Update the parameters of $\mathcal{F}(\cdot)$, $\mathcal{G}(\cdot)$ and
$\mathcal{H}(\cdot)$.

13:end for

##  Appendix E The Effect of Training Epochs

To validate the effect of training epochs during self-supervised pretraining,
we conduct experiments on ViT-T/2 with different numbers of pretraining
epochs. As shown in Figure 6, we can find the proposed method achieves better
performance with longer training schedule. Meanwhile, the performance
improvement progressively reaches to saturation with the increment of
pretraining epochs. Hence, we adopt 1600 pretraining epochs in the paper,
where the performance is close to saturation.

(a) CIFAR10 (b) CIFAR100

Figure 6: Training with different numbers of epochs on CIFAR dataset.


§ ACRONYMS

The following acronyms are used in this manuscript:

CPU     Central Processing Unit

GPU     Graphic Processing Unit

TPU     Tensor Processing Unit

CNN     Convolutional Neural Network

ML     Machine Learning

MLP     Multi Layer Perceptron

AI     Artificial Intelligence

PDE     Partial Differential Equation

NN     Neural Network

GPGPU     Graphics Processing Unit

SIMD     Single Instruction Multiple Data

SIMT     Single Instruction Multiple Thread

FLOPS     Floating Point Operation per Seconds

XFS     Extents File System

RGB     Red Green Blue

EFF     Electronics Frontiers Foundation

FPGA     Field Programmable Gate Array

DPS     Digital Signal Processing

TDP     Thermal Design Power

PDF     Protein Data Bank

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

booktitleBroadband Coverage in Germany; 15th ITG-Symposium
titleImplications of the COVID-19 Pandemic on the Internet Traffic

titleScientific computing: an introduction with parallel computing

Springer Science & Business Media

titleEncyclopedia of parallel computing

booktitle2018 International Conference on Advanced Science and Engineering (ICOASE)
titleDistributed cloud computing and distributed parallel computing: A review

journaltitleThe Journal of Supercomputing
titleHierarchical parallel processing of large scale data clustering on a PC cluster with GPU co-processing

booktitle2009 IEEE International Conference on Cluster Computing and Workshops
titleGPU clusters for high-performance computing

Pouyan Press

journaltitleJournal of Soft Computing in Civil Engineering
titleEvolutionary algorithm performance evaluation in project time-cost optimization

titleOptimizing a hybrid wind-PV-battery system using GA-PSO and MOPSO for reducing cost and increasing reliability

journaltitleCluster Computing
titleHybrid genetic algorithm for engineering design problems

journaltitleJournal of cloud Computing
titleAn efficient parallel genetic algorithm solution for vehicle routing problem in cloud implementation of the intelligent transportation systems

booktitleProceedings of the 5th ACM International Conference on Nanoscale Computing and Communication
titleQuantum Internet: From communication to distributed computing!

journaltitlePhysical Review X
titleArchitectures for quantum simulation showing a quantum speedup

National Acad Sciences

journaltitleProceedings of the National Academy of Sciences
titleToward the first quantum simulation with quantum speedup

booktitleComputational Science and Its Applications - ICCSA 2018 - 18th International Conference, Melbourne, VIC, Australia, July 2-5, 2018, Proceedings, Part V
seriesLecture Notes in Computer Science
titleThe ECTN Virtual Education Community Prosumer Model for Promoting and Assessing Chemical Knowledge

booktitleComputational Science and Its Applications - ICCSA 2020 - 20th International Conference, Cagliari, Italy, July 1-4, 2020, Proceedings, Part VII
seriesLecture Notes in Computer Science
titleBinary Classification of Proteins by a Machine Learning Approach

booktitleComputational Science and Its Applications - ICCSA 2020 - 20th International Conference, Cagliari, Italy, July 1-4, 2020, Proceedings, Part I
seriesLecture Notes in Computer Science
titleSkin Cancer Classification Using Inception Network and Transfer Learning

booktitleComputational Science and Its Applications - ICCSA 2019 - 19th International Conference, Saint Petersburg, Russia, July 1-4, 2019, Proceedings, Part I
seriesLecture Notes in Computer Science
titleAn Approach for Improving Automatic Mouth Emotion Recognition

booktitleComputational Science and Its Applications - ICCSA 2019 - 19th International Conference, Saint Petersburg, Russia, July 1-4, 2019, Proceedings, Part II
seriesLecture Notes in Computer Science
titleSharing Linkable Learning Objects with the Use of Metadata and a Taxonomy Assistant for Categorization

journaltitleIEEE Transactions on Power Systems
titleShort term load forecasting using a multilayer neural network with an adaptive learning algorithm

booktitle2009 International Conference on Computational Intelligence and Natural Computing
titleThe improved training algorithm of back propagation neural network with self-adaptive learning rate

booktitleInternational Conference on Engineering Applications of Neural Networks
titleA novel adaptive learning rate algorithm for convolutional neural network training

booktitle2020 IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT)
titleSecure Smart Cities Framework Using IoT and AI

booktitleProceedings of the AAAI Conference on Artificial Intelligence
titleGenerating multiple diverse responses for short-text conversation

journaltitleSustainable Cities and Society
titleOn the social shaping dimensions of smart sustainable cities: A study in science, technology, and society

IGI Global

titleTechnology Integration in Higher Education: Social and Organizational Aspects: Social and Organizational Aspects

abstractA substantial amount of money is spent on technology by schools, families, and policymakers with the hope of improving educational outcomes. This chapter explores the theoretical and empirical literature on the impacts of technology on educational outcomes. The literature focuses on two primary contexts in which technology may be used for educational purposes: (i) classroom use in schools and (ii) home use by students. Theoretically, information and communications technology (ICT) investment and computer-aided instruction (CAI) use by schools and the use of computers at home have ambiguous implications for educational achievement: expenditures devoted to technology necessarily offset inputs that may be more or less efficient, and time allocated to using technology may displace traditional classroom instruction and educational activities at home. However, much of the evidence in the schooling literature is based on interventions that provide supplemental funding for technology or additional class time, and thus favor finding positive effects. Nonetheless, studies of ICT and CAI in schools produce mixed evidence with a pattern of null results. Notable exceptions to this pattern occur in studies of developing countries and CAI interventions that target math rather than language. In the context of home use, early studies based on multivariate and instrumental variables approaches tend to find large positive (and in a few cases negative) effects while recent studies based on randomized control experiments tend to find small or null effects. Early research focused on developed countries while more recently several experiments have been conducted in developing countries.
seriesHandbook of the Economics of Education
titleChapter 5 - Technology and Education: Computers, Software, and the Internet

journaltitleComputers & Education
titleA meta-analysis of the impact of technology on learning effectiveness of elementary students

booktitleComputational Science and Its Applications - ICCSA 2020 - 20th International Conference, Cagliari, Italy, July 1-4, 2020, Proceedings, Part VII
seriesLecture Notes in Computer Science
titleTeaching Math with the Help of Virtual Reality

abstractMillennials interact with technology like no other generation before them and this is affecting how they want to be taught in higher education and how they want to lead and expect to be led in organizations, after graduating. Though stating that they want to be enlightened in academia, some qualitative comments from millennials indicate the opposite, namely that they want to be prepared but also to be kept naïve as to what business really entails. This is expected to help keep motivation levels high, as motivation is seen to be the key element to success in life. Millennials expect also to be led authentically and to be treated as valued human beings. This is in contrast to the current autocratic leadership profile found predominantly in Portuguese organizations, at the time of writing. This study had a sample of one hundred and eleven millennial students who answered a survey on attitudes towards leadership and their desired approach to higher education. Three interviews with seasoned executives were also performed, to establish a contrast and see other perspectives. With this research, we conclude that we may be in the presence of a hard working millennial generation, contrary to previous research findings which has indicated that they are lazy. Finally, information technology (IT) is a precious partner in class, in particular Padlet.com, Moodle, and online News Forums, as well as the challenge to create original videos about course content. Future research should focus on how technology has made society more transparent with employees wanting more democratic leaders in times when hierarchies are seen to hinder rather than aid productivity levels.
journaltitleTelematics and Informatics
titleThe social impact of technology on millennials and consequences for higher education and leadership
Leadership,Higher education,Information technology,Romanticism,Enlightenment,Millennials,Success

booktitleSoftware Usability
titleLearning Mathematics in an Immersive Way

titleSmart working: Creating the next wave

Available at SSRN

titleSmart-working: work flexibility without constraints
volumeCESifo Working Paper N. 8165

booktitleComputational Science and Its Applications - ICCSA 2021 - 21th International Conference, Cagliari, Italy, September 13-16, 2021, Proceedings, Part I
seriesLecture Notes in Computer Science
titleImplementing a scalable and elastic computinge nvironment based on Cloud Containers

London, United Kingdom

Association for Computing Machinery

abstractLet's Encrypt is a free, open, and automated HTTPS certificate authority (CA) created to advance HTTPS adoption to the entire Web. Since its launch in late 2015, Let's Encrypt has grown to become the world's largest HTTPS CA, accounting for more currently valid certificates than all other browser-trusted CAs combined. By January 2019, it had issued over 538 million certificates for 223 million domain names. We describe how we built Let's Encrypt, including the architecture of the CA software system (Boulder) and the structure of the organization that operates it (ISRG), and we discuss lessons learned from the experience. We also describe the design of ACME, the IETF-standard protocol we created to automate CA–server interactions and certificate issuance, and survey the diverse ecosystem of ACME clients, including Certbot, a software agent we created to automate HTTPS deployment. Finally, we measure Let's Encrypt's impact on the Web and the CA ecosystem. We hope that the success of Let's Encrypt can provide a model for further enhancements to the Web PKI and for future Internet security infrastructure.
booktitleProceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security
seriesCCS '19
titleLet's Encrypt: An Automated Certificate Authority to Encrypt the Entire Web
automation,web PKI,certificate authority,usability

titleA Performance Comparison of CUDA and OpenCL

booktitle2011 International Conference on Parallel Processing
titleA Comprehensive Performance Comparison of CUDA and OpenCL

booktitle2021 8th International Conference on Electrical and Electronics Engineering (ICEEE)
titleImpact of CUDA and OpenCL on Parallel and Distributed Computing

booktitle2010 International Conference on Computational Science and Its Applications
titleThe AES Implantation Based on OpenCL for Multi/many Core Architecture

Springer International Publishing

abstractWe present a performance per watt analysis of CUDAlign 4.0, a parallel strategy to obtain the optimal alignment of huge DNA sequences in multi-GPU platforms using the exact Smith-Waterman method. Speed-up factors and energy consumption are monitored on different stages of the algorithm with the goal of identifying advantageous scenarios to maximize acceleration and minimize power consumption. Experimental results using CUDA on a set of GeForce GTX 980 GPUs illustrate their capabilities as high-performance and low-power devices, with a energy cost to be more attractive when increasing the number of GPUs. Overall, our results demonstrate a good correlation between the performance attained and the extra energy required, even in scenarios where multi-GPUs do not show great scalability.
booktitleBioinformatics and Biomedical Engineering
titleSmith-Waterman Acceleration in Multi-GPUs: A Performance per Watt Analysis

Boston, MA, USA

Association for Computing Machinery

abstractOver the past few years machine learning has seen a renewed explosion of interest, following a number of studies showing the effectiveness of neural networks in a range of tasks which had previously been considered incredibly hard. Neural networks' effectiveness in the fields of image recognition and natural language processing stems primarily from the vast amounts of data available to companies and researchers, coupled with the huge amounts of compute power available in modern accelerators such as GPUs, FPGAs and ASICs. There are a number of approaches available to developers for utilizing GPGPU technologies such as SYCL, OpenCL and CUDA, however many applications require the same low level mathematical routines. Libraries dedicated to accelerating these common routines allow developers to easily make full use of the available hardware without requiring low level knowledge of the hardware themselves, however such libraries are often provided by hardware manufacturers for specific hardware such as cuDNN [9] for Nvidia hardware or MIOpen [5] for AMD hardware.SYCL-DNN is a new open-source library dedicated to providing accelerated routines for neural network operations which are hardware and vendor agnostic. Built on top of the SYCL open standard and written entirely in standard C++, SYCL-DNN allows a user to easily accelerate neural network code for a wide range of hardware using a modern C++ interface. The library is tested on AMD's OpenCL for GPU, Intel's OpenCL for CPU and GPU, ARM's OpenCL for Mali GPUs as well as ComputeAorta's OpenCL for R-Car CV engine and host CPU. In this talk we will present performance figures for SYCL-DNN on this range of hardware, and discuss how high performance was achieved on such a varied set of accelerators with such different hardware features.
booktitleProceedings of the International Workshop on OpenCL
titleAccelerated Neural Networks on OpenCL Devices Using SYCL-DNN
SYCL,GPGPU,machine learning,OpenCL,neural networks

booktitleComputational Science and Its Applications - ICCSA 2020 - 20th International Conference, Cagliari, Italy, July 1-4, 2020, Proceedings, Part VII
seriesLecture Notes in Computer Science
titleAn Immersive Open Source Environment Using Godot

Springer International Publishing

abstractEmerging applications such as Deep Learning are often data-driven, thus traditional approaches based on auto-tuners are not performance effective across the wide range of inputs used in practice. In the present paper, we start an investigation of predictive models based on machine learning techniques in order to optimize Convolution Neural Networks (CNNs). As a use-case, we focus on the ARM Compute Library which provides three different implementations of the convolution operator at different numeric precision. Starting from a collation of benchmarks, we build and validate models learned by Decision Tree and naive Bayesian classifier. Preliminary experiments on Midgard-based ARM Mali GPU show that our predictive model outperforms all the convolution operators manually selected by the library.
booktitleComputational Science and Its Applications – ICCSA 2019
titleTowards a Learning-Based Performance Modeling for Accelerating Deep Neural Networks

CRC Press

booktitleFeynman and computation
titleSimulating physics with computers
family=Deutsch, D.,

Royal Society

journaltitleProc. R. Soc. Lond.
titleQuantum theory, the Church–Turing principle and the universal quantum computer

The Royal Society London

journaltitleProceedings of the Royal Society of London. A. Mathematical and Physical Sciences
titleQuantum theory, the Church–Turing principle and the universal quantum computer

journaltitlePhysical Review A
titleQuantum coding

The Royal Society London

journaltitleProceedings of the Royal Society of London. Series A: Mathematical and Physical Sciences
titleRapid solution of problems by quantum computation

journaltitlePhysical review letters
titleQuantum mechanics helps in searching for a needle in a haystack

booktitleThe development of the number field sieve
titleThe number field sieve

booktitleThe development of the number field sieve
titleA general number field sieve implementation

journaltitleSIAM review
titlePolynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer

journaltitleNature News
titleD-wave upgrade: How scientists are using the world’s most controversial quantum computer

Wiley Online Library

journaltitleFortschritte der Physik: Progress of Physics
titleThe physical implementation of quantum computation

journaltitleIEEE Access
titleChallenges for the Repeatability of Deep Learning Models

booktitleComputational Science and Its Applications - ICCSA 2021 - 21th International Conference, Cagliari, Italy, September 13-16, 2021, Proceedings, Part I
seriesLecture Notes in Computer Science
titleA new method for binary classification of proteins with Machine Learning

New York, NY, USA

Association for Computing Machinery

abstractThe General Data Protection Regulation (GDPR) is considered by some to be the most important change in data privacy regulation in 20 years. Effective May 2018, the European Union GDPR privacy law applies to any organization that collects and processes the personal information of EU citizens within or outside the EU. In this work, we seek to quantify the progress the GDPR has made in improving privacy policies around the globe. We leverage our data mining tool, PrivacyCheck, to automatically compare three corpora (totaling 550) of privacy policies, pre- and post-GDPR. In addition, to evaluate the current level of compliance with the GDPR around the globe, we manually studied the policies within two corpora (450 policies). We find that the GDPR has made progress in protecting user data, but more progress is necessary—particularly in the area of giving users the right to edit and delete their information—to entirely fulfill the GDPR’s promise. We also observe that the GDPR encourages sharing user data with law enforcement, and as a result, many policies have facilitated such sharing after the GDPR. Finally, we see that when there is non-compliance with the GDPR, it is often in the form of failing to explicitly indicate compliance, which in turn speaks to an organization’s lack of transparency and disclosure regarding their processing and protection of personal information. If Personally Identifiable Information (PII) is the “currency of the Internet,” these findings mark continued alarm regarding an individual’s agency to protect and secure their PII assets.
journaltitleACM Trans. Manage. Inf. Syst.
titleThe Effect of the GDPR on Privacy Policies: Recent Progress and Future Promise

abstractThe long-awaited General Data Protection Regulation (GDPR) of the EU was provisionally agreed in December 2015.1 The final details are still being ironed out, but publication of the final version of the regulation is expected around July 2016.2 There will then be a two-year waiting period until every organisation that does business in, or with, the EU must comply with the regulation. Since it is a regulation, not a directive, compliance is mandatory, without the need for each member state to ratify it into its own legislation. The GDPR expands the scope of data protection so that it applies to anyone or any organisation that collects and processes information related to EU citizens, no matter where they are based or where the data is stored. Colin Tankard of Digital Pathways examines what effect the new regulation is likely to have on organisations.
journaltitleNetwork Security
titleWhat the GDPR means for businesses

journaltitleInternational Journal of Market Research
titleThe EU General Data Protection Regulation (GDPR): European Regulation that has a Global Impact

booktitleComputational Science and Its Applications - ICCSA 2019 - 19th International Conference, Saint Petersburg, Russia, July 1-4, 2019, Proceedings, Part I
seriesLecture Notes in Computer Science
titleMobile Localization Techniques Oriented to Tangible Web