# What Do
Self-Supervised Vision Transformers Learn?

Namuk Park1 Wonjae Kim2 Byeongho Heo2 Taekyung Kim2 Sangdoo Yun2
1Prescient Design, Genentech 2NAVER AI Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Most of this work was done while the author was at NAVER AI Lab.

###### Abstract

We present a comparative study on how and why contrastive learning (CL) and
masked image modeling (MIM) differ in their representations and in their
performance of downstream tasks. In particular, we demonstrate that self-
supervised Vision Transformers (ViTs) have the following properties: (1) CL
trains self-attentions to capture longer-range global patterns than MIM, such
as the shape of an object, especially in the later layers of the ViT
architecture. This CL property helps ViTs linearly separate images in their
representation spaces. However, it also makes the self-attentions collapse
into homogeneity for all query tokens and heads. Such homogeneity of self-
attention reduces the diversity of representations, worsening scalability and
dense prediction performance. (2) CL utilizes the low-frequency signals of the
representations, but MIM utilizes high-frequencies. Since low- and high-
frequency information respectively represent shapes and textures, CL is more
shape-oriented and MIM more texture-oriented. (3) CL plays a crucial role in
the later layers, while MIM mainly focuses on the early layers. Upon these
analyses, we find that CL and MIM can complement each other and observe that
even the simplest harmonization can help leverage the advantages of both
methods.

##  1 Introduction

Contrastive Learning (CL) (He et al., 2020; Chen et al., 2020a; b; 2021) has
been the most popular self-supervised learning methods until recently. It aims
to learn the invariant semantics of two random views (Tian et al., 2020a; b)
by making global projections of representations similar for positive samples
and dissimilar for negative samples. Since CL exploits the globally projected
representations to contrast each other, it can be deemed as an _“image-level”_
self-supervised learning approach.

Deviating from CL, masked image modeling (MIM) (Bao et al., 2022; Xie et al.,
2022b; He et al., 2022) has risen as a strong competitor of CL in the era of
Vision Transformers (ViTs) (Dosovitskiy et al., 2021) with its impressive
performances of downstream tasks. MIM trains ViTs by reconstructing the
correct semantics of masked input patches. Unlike CL, it learns the semantics
of patch tokens and this can be deemed as a _“token-level”_ self-supervised
learning approach. Since MIM outperforms CL in fine-tuning accuracy, it may
appear prima facie as a more effective pre-training method than CL. However, a
different trend is observed for linear probing accuracy with CL outperforming
MIM (See LABEL:fig:behavior). For further exposition on CL and MIM, we refer
the reader to Appendix B.

Then, which method—CL or MIM—should we use for the self-supervised learning of
ViTs? Although both methods are widely used, little is known about what they
learn. This paper sheds light on their nature by showing that ViTs trained
through CL and MIM learn opposite knowledge. In particular, we raise questions
to better understand self-supervised learning, and then find the answers that
can potentially affect future improvements. The questions posed can be divided
into the following properties of Vision Transformers: the behavior of self-
attentions, the transformation of the representations, and the position of
lead role components. Our key questions and findings are elaborated below.

How do self-attentions behave? (Section 2) We find that CL primarily captures
global relationships, while MIM captures local relationships. This implies
that the representations of CL contain more global patterns, such as object
shapes, than those of MIM. On the one hand, this property helps CL recognize
objects and distinguish images. On the other hand, however, it also suggests
that CL struggles to preserve local information. In particular, we observe
that self-attentions of CL in the later layers for all query tokens and heads
collapse into homogeneous attention maps. In such cases, most self-attention
maps focus on object boundaries, meaning that they can capture object shapes
but may lose interaction diversity between tokens. Consequently, CL and MIM
each have advantages over different tasks: CL works well for linear probing
and classification tasks with smaller models, whereas MIM outperforms CL in
fine-tuning and dense prediction tasks with larger models.

How are representations transformed? (Section 3) CL transforms representations
mainly based on image-level information, and its self-attentions collect
information on object shape over entire tokens. This process makes tokens
similar rather than diversifying them. As a result, CL distinguishes images
well but has difficulty distinguishing tokens. On the contrary, MIM preserves
and amplifies token-level information. Thus, the self-attentions for each
token are substantially different and prohibit each token from including
redundant information. We observe the consistent property from our Fourier
analysis: CL primarily utilizes the low-frequency signals, but MIM utilizes
high-frequencies. This observation suggests that CL is shape-biased and MIM is
texture-biased. In sum, self-supervised models trained with CL and MIM learn
the representations in different levels of detail.

Which components play an important role? (Section 4) Analyses of the
importance of each CL and MIM layer demonstrate that the later layers in CL
and early layers in MIM play a key role. We interpret this as a consistent
observation since early layers are usually known to capture low-level
features—e.g., local patterns, high-frequency signals, and texture
information—and later layers capture global patterns, low-frequency signals,
and shape information (Dosovitskiy et al., 2021; Raghu et al., 2021; d’Ascoli
et al., 2021; Graham et al., 2021; Dai et al., 2021; Park & Kim, 2022b).

From the above analyses and insights, we find that CL and MIM can complement
each other and show in Section 5 that even the simplest implementation, such
as a linear combination of CL and MIM objectives, can take advantage of both
methods. Surprisingly, the hybrid models outperform those pre-trained with
either CL or MIM both in terms of fine-tuning and linear probing accuracy.

##  2 How Do Self-Attentions Behave?

We point out that CL and MIM may not be silver bullets for all tasks, as shown
in LABEL:fig:behavior. CL generally outperforms MIM in linear probing, while
MIM dominates CL in the fine-tuning scheme. However, when we dissect the size
of the model, CL outperforms MIM after fine-tuning for small models (cf. (Wang
et al., 2022)), while MIM performs better on large models. Also, MIM yields
effective representations for dense prediction tasks, such as object
detection, but CL falls short on those tasks. This section explains these
phenomena by investigating the behavior of self-attentions.

Our analyses mainly compare ViT-B/16 pre-trained on ImageNet-1K (Russakovsky
et al., 2015) with MoCo v3 (Chen et al., 2021) and SimMIM (Xie et al., 2022b).
We use the ImageNet validation images for our experiments. We observe that
other methods, e.g., DINO (Caron et al., 2021), BEiT (Bao et al., 2022), and
MAE (He et al., 2022), have consistent properties (See Figure C.1).

### CL mainly captures global relationships.

Figure 1:  Self-attentions of CL (MoCo) capture global information, but they
collapse into homogeneous attention maps for all query tokens and heads. Self-
attentions of MIM (SimMIM) mainly focus on local areas and similar tokens. We
visualize the attention maps for two different query tokens in the beginning
through the end layers. We omit the results for self-attention heads, which
show mostly consistent results. _Left_ : Self-attentions of CL capture global
patterns and the shape of an object. However, all attention maps capture the
same shape information regardless of the query tokens. _Right_ : Self-
attentions of MIM capture local patterns and are correlated with queries.

Figure 2:  Effective receptive fields of CL are global, but those of MIM are
local. This is particularly evident in the later layers.

Figure 3:  Self-attentions of CL have little to do with query tokens.
Normalized MI of CL is significantly lower than that of MIM in the later
layers.

We measure the ranges of self-attentions via attention distance (Dosovitskiy
et al., 2021). Attention distance is defined as the average distance between
the query tokens and key tokens considering their self-attention weights.
Therefore, it conceptually corresponds to the size of the receptive fields in
CNNs.

Figure 3 shows that the attention distance of CL (MoCo) is significantly
higher than that of MIM (SimMIM), especially in the later layers. As seen in
Figure 1, the qualitative visualization, this implies that the representations
of CL contain global patterns and shape information, so CL can help ViTs
distinguish between objects of images. Conversely, the self-attentions of MIM
mainly capture local relationships; i.e., MIM may have difficulty recognizing
whole objects and their shapes. Section 3 also discuss this claim from a
representational perspective.

### Self-attentions of CL collapse into homogeneity.

We observe an interesting behavior of CL in Figure 1, which shows the
attention maps for query tokens from two different spatial locations. The
self-attentions of CL surprisingly indicate _almost identical_ object shapes
for the two query tokens, compared to that of MIM. We describe this phenomenon
as _an attention collapse into homogeneity_. This collapsing trend in the
self-attentions of CL is observed across all the heads and query tokens. In
contrast, the self-attentions of MIM are more faithful to the two query
tokens, as expected.

We use normalized mutual information (NMI) (Strehl & Ghosh, 2002) to measure
the attention collapse. Let $p(q)$ be a distribution of query tokens, and
assume that these query tokens are uniformly distributed since a single query
token is given for each spatial coordinate, i.e., $p(q)=\nicefrac{{1}}{{N}}$
where $N$ is the number of the tokens. Then the joint distribution of query
and key tokens is $p(q,k)=\pi(k|q)p(q)$ where $\pi(k|q)$ is the softmax-
normalized self-attention matrix. Thus, the normalized mutual information is
$\frac{I(q,k)}{\sqrt{H(q)H(k)}}$ where $I(\cdot,\cdot)$ is the mutual
information and $H(\cdot)$ is the marginal entropy. Low mutual information
values show that attention maps are less dependent on the query tokens,
implying an attention collapse into homogeneity. Conversely, high mutual
information means that the attention maps strongly depend on the query tokens.

Figure 3 shows the degree of attention collapse in terms of the normalized
mutual information (NMI). Results show that the mutual information of CL is
significantly lower than that of MIM in the later layers, suggesting that the
self-attentions of CL tend to collapse into homogeneous distributions.

(a) Between heads

(b) Between depths

(c) Between tokens

Figure 4:  CL lacks representational diversity in the later layers. We measure
cosine similarities of representations in the self-attentions between the
heads (left), depths (middle), and spatial coordinates (right). All of the
results show that the representational similarity of later self-attentions of
CL is higher than that of MIM. Increasing heads or depths of CL is not
effective in improving the diversity. _Left_ : The similarity of
representations from two heads in self-attention. _Middle_ : The similarity
between representations before and after self-attentions transform them.
_Right_ : The similarities of representations at two spatial coordinates.
ViT-{S, L} is trained with 100 epochs.

### Attention collapse reduces representational diversity.

We conjecture that the self-attention collapse into homogeneity eventually
leads to homogeneous token representations. To support this argument, we
measure representational cosine similarities. In particular, we design three
similarities: between different self-attention heads (heads), between the
before and after self-attention layers (depths), and between different tokens
(tokens).

Figure 4 shows the results, reporting the representation similarities for
heads, depths, and tokens. As expected, the similarities of CL are notably
higher than those of MIM in the later layers, indicating that the
representations of CL have significant homogeneity. Even increasing the model
size does not solve the problem CL has and may rather worsen it. Increasing
the number of heads (ViT-S to ViT-B; Figure 4(a)) improves the
representational diversity of MIM, but hardly improves the diversity of CL.
Increasing the depth of CL (ViT-B to ViT-L; Figure 4(b)) only adds redundant
modules.

### Implications of the behaviors we observed.

In conclusion, the self-attention of CL captures global patterns and shapes of
objects. However, CL suffers from the problem of attention collapse into
homogeneity, which reduces the diversity of token representations. On the
other hand, MIM primarily captures local patterns and thus does not suffer
from the attention collapse problem.

The behaviors mentioned above can explain the phenomena we observed in
LABEL:fig:behavior:

  * •

CL outperforms MIM in linear probing tasks because it captures shapes, which
helps recognize objects and distinguish images. Although MIM preserves the
texture and diversity of representations, their correlation with objects or
content may not be as strong as shapes do.

  * •

The attention collapse prohibits CL from fully exploiting heads, depths, and
tokens of ViTs. Since homogeneous representations are not very helpful in
improving token representations, ViTs trained with CL waste a large part of
network capability. Therefore, the fine-tuning accuracy of MIM is
significantly higher than CL in large models.

  * •

CL is not suitable for dense prediction since the token features are
homogeneous with respect to their spatial coordinates.

We further investigate the self-attention’s behavior with restricted receptive
fields in Figure D.1. As shown in the experiment, locally restricted self-
attentions lead to lower linear probing but higher fine-tuning accuracy, which
is consistent with our observations.

##  3 How Are Representations Transformed?

(a) MoCo (one image)

(b) MoCo (two images)

(c) SimMIM (one image)

Figure 5:  Self-attention layers of CL and MIM transform representations
differently. We visualize 196 spatial representation tokens for an example
validation image in a representation space. The blue ($\bullet$) and red
($\bullet$) data points denote the tokens before and after the self-attention
transformation. _Left_ : The self-attentions of CL (e.g., MoCo) translate all
the tokens equally, so the distances between the tokens of an image do not
increase. _Middle_ : However, CL moves the “centers of representations
(represented by $\times$)” away from each other. Therefore, the images are
linearly separable. The circle ($\bullet$) and triangle ($\triangle$) data
represent tokens from different images. _Right_ : The self-attentions of MIM
(e.g., SimMIM) transform representations differently according to query
tokens, thus increasing the distances between tokens. See Figure 7 for
quantitative analyses.

In this section, we analyze the token representations of ViTs pre-trained with
CL and MIM to demonstrate how the properties of self-attentions we observed in
Section 2 affect the representations differently. We use the same pre-trained
ViT-B/16 models by default default just as we did in Section 2.

### CL transforms all tokens in unison, while MIM does so individually.

To show how CL and MIM transform token representations, we visualize them in
representation space. Figure 5 shows 196 (14$\times$14 patches) tokens before
and after self-attention modules from a single image sample of the ImageNet
validation set. We use the three large singular vectors obtained via singular
value decomposition (SVD) as the bases of the space. To better visualize this,
we display the representation of MoCo and SimMIM in their crucial layers—the
last layer and the first layer, respectively.

Figure 5(a) visualizes the changes that occur in the tokens of CL when
transformed by self-attention module; it indicates that the self-attentions of
CL translate all tokens in unison. This phenomenon occurs because the self-
attention maps of CL are homogeneous, i.e., self-attention is almost
independent of the spatial coordinates and query tokens. Therefore, the
modules add near-constant to all the token representations. As a result, the
inter-representation distance and the volume of representations do not
increase, which implies that CL cares less about individual tokens.

Nevertheless, self-attentions are essential for the discriminative power of
CL. As shown in Figure 5(b), they help distinguish images by moving “the
centers of the representation distribution” away from each other. In short,
this figure suggests that CL makes the image linearly separable even though it
loses the ability to distinguish tokens.

In contrast, MIM applies a different transformation to individual tokens, as
shown in Figure 5(c), because different self-attentions are assigned to the
individual spatial tokens. Thus, MIM alters the distance between tokens of a
single image as well as the volume of the representation distribution.

(a) Singluar value spectum of tokens from a single image (token-level)

(b) Singluar value spectum of images (image-level)

Figure 7:  CL barely changes or even decreases the distribution volume of
tokens from a single image, implying that it hardly distinguishes between
token. Instead, it significantly increases the distribution volume of images.
To demonstrate these properties, we visualize singular value spectra, the
singular values of the distribution of representations sorted by the
magnitude. The higher a singular value, the larger the volume of a
distribution. The right of this figure shows the $64^{\text{th}}$ and
$128^{\text{th}}$ highest singular value for depth. _Top_ : Singular value
spectra of tokens from a single image. CL decreases the singular values of the
tokens, but MIM increases. _Bottom_ : Singular value spectra of images. CL
significantly increases the volumes occupied by images, but MIM hardly does
so.

We find consistent results in quantitative analysis. Inspired by Jing et al.
(2022), Figure 7 visualizes singular value spectra for tokens and images. A
singular value spectrum provides singular values of a representation
distribution obtained by SVD, so we can use it to represent the effective
volume of distributions in a representation space. The higher the singular
value in a spectrum, the larger the volume of a representation distribution.
To calibrate the scale, we use the relative log singular value ($\Delta$ Log
singular value), the difference with the (second) largest singular value for a
depth.

Figure 7(a) shows singular value spectra of tokens from a single image. We
calculate them for each image in the ImageNet validation set and report
averaged singular values over the dataset. In this figure, the CL layers
hardly increase or even decrease the singular value; consistent with the
explanation above, this implies that CL hardly distinguishes tokens. In
contrast, MIM increases the singular value, meaning that it changes the volume
of tokens and can distinguish tokens. Another interesting observation is that
a few later layers of MIM decrease the volume, even though they capture local
patterns as shown in Figures 3 and 3. This is because they behave like
decoders. Section 4 discusses this in detail.

Figure 7(b) shows the singular value spectra of images. We average all tokens
in an image to build an image-level representation vector and conduct a
singular value spectrum over the collection of representations in the
validation set. As opposed to the previous case, the representational volume
of CL is larger than that of MIM, which implies that CL makes the image-level
representation separable.

### CL exploits low-frequencies, and MIM exploits high-frequencies.

We hypothesize that CL captures low-frequency and MIM captures high-frequency
information in spatial dimensions since CL provides image-level self-
supervision to capture global patterns, while MIM provides token-level self-
supervision to exploit local patterns. To support this argument from a
frequency perspective, we conduct a Fourier analysis of the representations as
following Park & Kim (2022b). In particular, we report the relative log
amplitude of Fourier-transformed representations by calculating the amplitude
difference between the highest and lowest frequencies of representations.

Figure 9 visualizes the relative amplitudes of CL and MIM. It shows that the
high-frequency amplitude of CL is significantly smaller than that of MIM,
suggesting that CL mainly utilizes low-frequency spatial information such as
global structures and shapes. On the contrary, MIM usually uses high-frequency
spatial information such as narrow structures and fine textures.

(a) Stylized ImageNet

(b) Robustness for noise frequency

Figure 8:  CL is biased toward shape, whereas MIM is biased toward texture.
We report the predictive results of models for linear probing tasks. However,
we observe consistent results in fine-tuned models (See Figure F.2). _Left:_
Result of classification on Stylized ImageNet. It shows that CL is more shape-
biased than MIM and even than the supervised pre-trained model. Vertical lines
represent averaged results for the shape categories. We also report the
results of supervised ViT with ImageNet-1K class labels for comparison.
_Right:_ Accuracy drops on images with frequency-based random noises. MIM
shows a more significant amount of accuracy drop than CL with high-frequency
noises, demonstrating MIM’s texture-biased property. The frequency window size
of the frequency-based noise is 0.1$\pi$.

Figure 9:  CL exploits low-frequency, but MIM exploits high-frequency.
Moreover, a few last layers of CL reduce high-frequency by capturing global
patterns. MIM also reduces it even though they capture local patterns, because
the later layers behave like decoders. See also Figure 11.

Another interesting finding is that the last few layers of MIM reduce the high
frequencies even though they only focus on local areas (See Figure 3). We
conjecture that MIM implicitly divides ViTs into the encoder-decoder structure
and allows intermediate layers to have linearly separable information. In
contrast, CL allows the last layer to have such information. This is further
elaborated in Figure 11.

### CL is shape-biased, but MIM is texture-biased.

Based on the results of the Fourier analysis, we assume that CL and MIM each
have a bias toward shapes and textures, respectively. To demonstrate this
claim, we use Stylized ImageNet (Geirhos et al., 2019), a texture-altered
dataset, by using AdaIN (Huang & Belongie, 2017). Figure 8(a) reports the
linear probing results on Stylized ImageNet to evaluate the shape and texture
biases of pre-trained models. Compared to the model pre-trained with
supervised learning, CL depends more on the shape and MIM depends on texture
of images to classify images. In other words, CL is robust to texture changes,
and MIM is vulnerable to them.

Figure 8(b) shows the consistent results. In this experiment, we follow Park &
Kim (2022a; b) and measure the decrease in accuracy on the ImageNet dataset
with frequency-based random noise. The results suggest that CL is robust to
high-frequency noises, but MIM is significantly more vulnerable to them. Since
high-frequency noises harm the fine details of images, we arrive at the same
conclusion that CL is more shape-biased and MIM is texture-biased. This can
explain the robustness of CL against adversarial perturbations (Bordes et al.,
2022).

##  4 Which Components Play an Important Role?

The previous sections consistently show through various perspectives that CL
exploits image-level global patterns while MIM captures token-level local
patterns. This section analyzes pre-trained ViTs from an architectural
perspective and shows that the key components in CL and MIM are different.

(a) Self-attention

(b) Fourier analysis

Figure 10:  The explicit decoder architecture of MAE helps ViTs effectively
leverage the advantages of MIM. We analyze the encoder and decoder of a pre-
trained model with a masking ratio of zero. The left side of each figure
represents the encoder and the right side the decoder. _Left:_ The mutual
information of MAE is lower than that of SimMIM in the encoder but higher in
the decoder. _Right:_ The decoder of MAE captures low-frequencies, while the
encoder captures high-frequencies. As a result, the decoder predicts blurred
images. Moreover, the later layers (excluding the last layer) of the encoder
do not reduce high-frequency information, while those of SimMIM do.

### Later layers of CL and early layers of MIM are important.

According to studies on ViT (Graham et al., 2021; Dai et al., 2021; Park &
Kim, 2022b), the later layers use high-level information, and the early layers
exploit low-level information. Since CL and MIM each exploit global and local
patterns, we expect that the later layers of CL and early layers of MIM play a
key role.

Figure 11:  Later layers of CL and early layers of MIM play a key role. We
report linear probing accuracies by using the representations of the
intermediate layers. CL outperforms MIM in later layers, and MIM outperforms
CL in early layers.

To evaluate the importance of each layer, we measure the linear probing
accuracy using intermediate representations with the configuration of Table
A.1. In Figure 11, we observe the following properties: First, the linear
probing accuracy of MIM is higher than that of CL at the beginning.
Conversely, CL outperforms MIM at the end of the model. Such result indicates
that the later layers of CL and early layers of MIM play an important role in
making linearly separable representations. Second, the accuracy of CL
increases with increasing depth as expected, but the accuracy of MIM
surprisingly decreases at the end of the model, i.e., the later layers of MIM
are not very helpful in separating representations. We explain this
observation as a phenomenon in which MIM methods with shallow prediction
heads, e.g., SimMIM, use later layers of the backbone as a decoder. Therefore,
MIM with a deep self-attention decoder, e.g., MAE (He et al., 2022), can be
useful for linear probing performance. Moreover, it also explains why SimMIM’s
high-frequency component and representational volumes drop in the later layers
as shown in Figures 9 and 7. Third, even the highest linear probing accuracy
of MIM is lower than that of CL.

### The explicit decoder helps ViTs further leverage the advantages of MIM.

Several previous observations find that the implicit decoder of MIM with a
shallow prediction head, such as SimMIM, can impair performance. MAE (He et
al., 2022) addresses this problem by introducing deep explicit ViT decoders
and reconstructing masked tokens only in the separate decoders.

In Figure 10, we analyze MAE to understand the properties of decoders more
deeply. Figure 10(a) shows the self-attention behaviors. The results indicate
that the mutual information of MAE is lower than that of SimMIM in the later
layers of the encoder but higher in the decoder, implying that the decoder
reconstructs masked tokens based on its neighborhood tokens.

Figure 10(b) shows the results of the Fourier analysis. As explained in Figure
9, the last four layers of SimMIM reduce the high-frequency components. In
contrast, the later layers (excluding the last layer) of MAE do not reduce
them. Instead, the decoder of MAE prioritizes low-frequency information
compared with the encoder, allowing the backbone to efficiently utilize high-
frequency information.

##  5 Are the Two Methods Complementary to Each Other?

We present comparative analyses on CL and MIM from three perspectives: self-
attentions, representation transforms, and the position of important layers.
All of our results indicate that CL and MIM train ViTs differently. These
differences naturally imply that combining CL and MIM to train a backbone may
help leverage the advantages of both methods.

(a) Performance

(b) Self-attention

(c) Fourier analysis

Figure 12:  The simple linear combination of CL (MoCo) and MIM (SimMIM)
objectives outperforms the vanilla CL and MIM. $\lambda$ is the importance
weight of CL, so $\lambda=0$ means SimMIM and $\lambda=1$ means MoCo. _Left:_
“CL + MIM” outperforms CL and MIM in both linear probing and fine-tuning
accuracy. _Middle:_ Mutual information of “CL + MIM” decreases at the end of
the model, suggesting that the self-attentions of later layers collapse into
homogeneity and capture the same object shape information. _Right:_ Fourier
analysis shows that “CL + MIM” amplifies high frequencies at the beginning and
reduces them at the end. It implies that “CL + MIM” exploits high-frequency
information at the beginning and low-frequency information at the end.

To show that CL and MIM are complementary, we introduce the simplest way to
harmonize CL and MIM by linearly combining two losses, i.e.,
$\mathcal{L}=(1-\lambda)\mathcal{L}_{\text{MIM}}+\lambda\mathcal{L}_{\text{CL}}$
where $\mathcal{L}_{\text{MIM}}$ and $\mathcal{L}_{\text{CL}}$ each indicate
the losses of MIM and CL, and $\lambda$ is the importance weight of CL. We
find that this simple hybrid model trained with combined losses efficiently
exploits the strengths of both methods. Figure 12(a) shows linear probing and
fine-tuning accuracy on ImageNet with varying $\lambda$. Surprisingly, the
hybrid models outperform MIM ($\lambda=0$) and CL ($\lambda=1$) in both
aspects. Figure 12(b) and Figure 12(c) can provide insights on how hybrid
models behave by analyzing the model with $\lambda=0.2$ in terms of self-
attentions in Section 2 and Fourier analysis in Section 3, respectively; both
results show that the hybrid model exploits MIM properties in the early layers
and CL properties in the later layers. In particular, Figure 12(b) indicates
that the self-attentions of the early layers are changed according to the
query token but those of the later layers are not. Likewise, Figure 12(c)
shows that the early layers exploit high-frequency, while the later layers try
to exploit low-frequency.

##  6 Conclusion

We conducted a comparative study highlighting various facets of two widely
used self-supervised learning methods for vision transformers: contrastive
learning (CL) and masked image modeling (MIM). The study demonstrated many
opposing properties of the two methods: image information (image-level _vs_.
token-level; as in Section 2), feature representations (low-frequency _vs_.
high-frequency; as in Section 3), and lead role components (later layers _vs_.
early layers; as in Section 4). Furthermore, we suggested a possible
application that exploits only the benefits from both methods and showed how a
combined model can outperform individual methods.

### Future directions.

Various future directions can be explored based on our study. We believe that
there are better ways than a simple linear combination of CL and MIM
objectives. For example, a novel self-supervised learning approach, in which
CL is applied in the later layers and MIM in the early layers, can be
considered. Moreover, we may extend our findings on self-supervision for
multi-stage ViTs, such as PiT (Heo et al., 2021) and Swin (Liu et al., 2021).
Another interesting direction is to enhance the individual properties of CL
and MIM. Techniques that help CL or MIM learn shapes or textures,
respectively, may also improve performance.

## References

  * Bao et al. (2022) Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  In _International Conference on Learning Representations_ , 2022.
  * Bordes et al. (2022) Florian Bordes, Randall Balestriero, and Pascal Vincent.  High fidelity visualization of what your self-supervised representation knows about.  _Transactions on Machine Learning Research_ , 2022.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _International Conference on Computer Vision_ , 2021.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning_ , 2020a.
  * Chen & He (2021) Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Chen et al. (2020b) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020b.
  * Chen et al. (2021) Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In _International Conference on Computer Vision_ , 2021.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Dai et al. (2021) Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  _Advances in Neural Information Processing Systems_ , 2021.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * d’Ascoli et al. (2021) Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  In _International Conference on Machine Learning_ , 2021.
  * Geirhos et al. (2019) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , 2019.
  * Graham et al. (2021) Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In _International Conference on Computer Vision_ , 2021.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 2020.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Heo et al. (2021) Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  In _International Conference on Computer Vision_ , 2021.
  * Huang et al. (2016) Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.  Deep networks with stochastic depth.  In _European Conference on Computer Vision_ , 2016.
  * Huang & Belongie (2017) Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * Jing et al. (2022) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _International Conference on Learning Representations_ , 2022.
  * Kim et al. (2018) Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al.  Nsml: Meet the mlaas platform with a real-world case study.  _arXiv preprint arXiv:1810.09957_ , 2018.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 2014.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _International Conference on Computer Vision_ , 2021.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Park & Kim (2022a) Namuk Park and Songkuk Kim.  Blurs behave like ensembles: Spatial smoothings to improve accuracy, uncertainty, and robustness.  In _International Conference on Machine Learning_ , 2022a.
  * Park & Kim (2022b) Namuk Park and Songkuk Kim.  How do vision transformers work?  In _International Conference on Learning Representations_ , 2022b.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Raghu et al. (2021) Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 2021.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 2015.
  * Strehl & Ghosh (2002) Alexander Strehl and Joydeep Ghosh.  Cluster ensembles—a knowledge reuse framework for combining multiple partitions.  _Journal of machine learning research_ , 2002.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * Tian et al. (2020a) Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In _European Conference on Computer Vision_ , 2020a.
  * Tian et al. (2020b) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  _Advances in Neural Information Processing Systems_ , 33:6827–6839, 2020b.
  * Tian et al. (2021) Yuandong Tian, Xinlei Chen, and Surya Ganguli.  Understanding self-supervised learning dynamics without contrastive pairs.  In _International Conference on Machine Learning_ , 2021.
  * Van der Maaten & Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 2008.
  * Van Horn et al. (2018) Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8769–8778, 2018.
  * Wang et al. (2022) Shaoru Wang, Jin Gao, Zeming Li, Jian Sun, and Weiming Hu.  A closer look at self-supervised lightweight vision transformers.  _arXiv preprint arXiv:2205.14443_ , 2022.
  * Wang et al. (2021) Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Wei et al. (2022a) Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022a.
  * Wei et al. (2022b) Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo.  Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.  _arXiv preprint arXiv:2205.14141_ , 2022b.
  * Xie et al. (2022a) Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao.  Revealing the dark secrets of masked image modeling.  _arXiv preprint arXiv:2205.13543_ , 2022a.
  * Xie et al. (2022b) Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022b.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _International Conference on Computer Vision_ , 2019.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.
  * Zhou et al. (2019) Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  _International Journal of Computer Vision_ , 127(3):302–321, 2019.
  * Zhou et al. (2022) Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  _International Conference on Learning Representations_ , 2022.

##  Appendix A Setup

We build the configurations based on Xie et al. (2022b) for fine-tuning and
Caron et al. (2021) for linear probing. Table A.1 summarizes the
configurations. Most analyzes of ViTs use the official ViT-B pre-trained
models, and some analyzes use ViT-{S, L} pre-trained with official
configurations but epochs of 100. Due to memory limitations, ViT-L is pre-
trained with a quarter batch size of the other models. The hybrid model
introduced in Section 5 uses the ViT backbone architecture of Xie et al.
(2022b) and employes a configureation based on their work for pre-training as
shown in Table A.1. For data augmentation and regularization, we adopt widely
used settings, e.g., Randaugment (Cubuk et al., 2020), label smoothing
(Szegedy et al., 2016), mixup (Zhang et al., 2018), cutmix (Yun et al., 2019),
stochastic depth (Huang et al., 2016). Layer decay (Bao et al., 2022) is also
used for fine-tuning. Neural network models are implemented in PyTorch (Paszke
et al., 2019). The code for analysis is available at https://github.com/naver-
ai/cl-vs-mim. All experiments use {1, 4, 8} NVIDIA A100 Tensor Core GPU. NSML
(Kim et al., 2018) has been used for experiments.

Table A.1: Training settings. Configuration | Linear Probing | Fine-tuning | Pre-training
---|---|---|---
optimizer | sgd | adamw | adamw
base learning rate | 1.0e-0 | 1.25e-3 | 1.0e-4
weight decay | 0.05 | 0.05 | 0.05
batch size | 1k | 2k | 1k
training epoch | 50 | 100 | 100
learning rate schedule | cosine | cosine | multistep
warmup epoch | 0 | 20 | 10
warmup schedule | $\cdot$ | linear | linear
randaugment | $\cdot$ | 9, 0.5 | 9, 0.5
label smoothing | $\cdot$ | 0.1 | 0.1
mixup | $\cdot$ | 0.8 | 0.8
cutmix | $\cdot$ | 1.0 | 1.0
stochastic depth | $\cdot$ | 0.1 | 0.1
layer decay | $\cdot$ | 0.65 | 1.0
gradient clip | $\cdot$ | 5.0 | 5.0

##  Appendix B Related Work

CL is a method based on comparing the global projection of two different
random views. However, this approach usually suffers from the collapsing
problem, where all representations collapse into constant solutions. To solve
this problem, various methods such as negative samples and InfoNCE (Oord et
al., 2018) have been proposed. Negative samples is an effective technique to
avoid the collapsing problems, but they cause dimensional collapse (Jing et
al., 2022) and require extra large batches (Chen et al., 2020a) or memory
queues (He et al., 2020; Chen et al., 2020b) to retrieve them. We mainly
analyze MoCo v3 (Chen et al., 2021), since the method includes these de facto
standard components—global projection, random views, and negative samples.

Some SSL methods, e.g. Grill et al. (2020); Caron et al. (2021), do not use
negative samples and use the projections of their Siamese representations as
the positives. Such self-distillation has been explored theoretically and
empirically (Chen & He, 2021; Tian et al., 2021) to prevent the collapsing
problem, but we do not discuss the distillation scheme in this paper. Wei et
al. (2022b) shows that feature distillation improves the fine-tuning
performance of CL by diversifying attention ranges; this observation is
consistent with our findings. While they focus on distillation to improve CL,
we reveal the fundamental nature of self-supervised learning by rigorously
comparing CL and MIM.

Compared with CL, MIM has been rarely explored in vision tasks. Various
methods, such as histograms of oriented gradients (Wei et al., 2022a) and
tokenization (Bao et al., 2022), have been proposed as part of porting masked
language models to the image domain with ViTs. Among them, SimMIM (Xie et al.,
2022b) and MAE (He et al., 2022) are simple yet effective methods to
reconstruct masked tokens without complicated pretext tasks. Because of its
simplicity and superior performance in downstream operations, MIM is
attracting attention as a promising technique in image processing.

Nevertheless, we find hints suggesting that CL and MIM utilize different
aspects of the data, making them complementary. For example, Zhou et al.
(2022); Wang et al. (2021); Yu et al. (2022) achieve high predictive
performance by harmonizing the image-level and the token-level self-supervised
learning. Xie et al. (2022a) also observe that, unlike supervised pre-trained
models or CL, self-attentions in SimMIM focus locally; this is a consistent
result with our findings.

##  Appendix C Our Insights Are Generalizable to Various Models

(a) Attention distance

(b) Normalized MI

Figure C.1:  MIM and CL methods each have consistent properties. To show this,
we visualize self-attention behaviors in terms of attention distance and
normalized mutual information (MI). SimCLR⋆, which was introduced in Chen et
al. (2021), stands for MoCo with a momentum coefficient of 0. _Left:_ The
attention distance of CL methods (namely MoCo, SimCLR⋆, and DINO) is higher
than that of MIM methods (namely SimMIM, BEiT, and MAE). This suggests that CL
methods consistently capture global patterns. _Right:_ The normalized mutual
information of MIM is higher than that of CL; i.e., the self-attentions of MIM
are more correlated with query tokens than CL.

(a) Attention distance

(b) Normalized MI

Figure C.2:  ViTs exhibit consistent self-attention patterns, regardless of
their size.  To better understand these patterns, we visualize the self-
attention behaviors of three ViTs—ViT-{Ti, S, B}—using two metrics: attention
distance and normalized mutual information (MI). _Left:_ All self-attentions
of MoCo capture global patterns in the later layers. In contrast, the self-
attentions of SimMIM capture local patterns. _Right:_ Likewise, all self-
attention maps of MoCo collapse into homogeneity in the later layers.

In the main text, we analyze ViT-B pre-trained using MoCo and SimMIM. We
observe consistent characteristics across various sizes of ViTs that have been
pre-trained using other self-supervised learning methods. To support this
claim, we delve into the properties of self-attentions.

Figure C.1 visualizes the self-attention behaviors of different self-
supervised learning methods in terms of attention distance and normalized
mutual information. As depicted in the figure, all CLs and MIMs exhibit
consistent properties. Similarly, Figure C.2 demonstrates that various sizes
of models also demonstrate consistent properties.

##  Appendix D Locality Inductive Bias Improves Fine-Tuning Accuracy of CL

Figure D.1:  Locality inductive bias harms linear probing but improves fine-
tuning.  We report the linear probing and fine-tuning accuracy of MoCo with
restricted self-attentions via attention masks.

In Section 2, we demonstrate that the homogeneity of self-attention map, i.e.,
attention collapse of CL, helps ViT distinguish images but harms fine-tuning
accuracy. As a result, we anticipate that incorporating a locality inductive
bias into CL will improve fine-tuning accuracy but degrade linear probing
accuracy. One simple method to inject locality into self-attentions is to
limit the receptive field of self-attention by using attention masks.

Figure D.1 shows the predictive performance of MoCo with restricted local
self-attentions. As expected, the results are similar to the performance of
MIM; As the kernel size decreases, the linear probing accuracy decreases but
the fine-tuning accuracy increases. These results are consistent with our
findings.

##  Appendix E A Closer Look at the Role of Self-supervised ViT Layers

The main text provides the key characteristics of CL and MIM. This section
delves deeper into the details not covered in the main text to provide a more
comprehensive understanding of the subjects.

(a) Standard deviations

(b) Distribution at 3${}^{\text{rd}}$ layer

Figure E.1:  The presence of an outlier head in MoCo raises the average of
normalized mutual information.  This observation explains how the normalized
mutual information in a couple of MoCo’s self-attention layers is similar to
or even surpasses that in SimMIM. _Left:_ We present the standard deviation of
the normalized mutual information. As depicted in this figure, the standard
deviation in SimMIM remains relatively consistent across different depths. In
contrast, the standard deviation in MoCo’s 3${}^{\text{rd}}$ or
4${}^{\text{th}}$ self-attention layer is notably higher than that in SimMIM.
_Right:_ Distribution of mutual information for the third self-attention layer
head. The visualization of this kernel density estimation shows that MoCo has
an outlier head with mutual information close to 1.0. The red rectangles
(${\color[rgb]{1,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0.5}\pgfsys@color@cmyk@stroke{0}{1}{0.50}{0}\pgfsys@color<EMAIL_ADDRESS>and blue triangles
(${\color[rgb]{0,0.44,0.57}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.44,0.57}\pgfsys@color@cmyk@stroke{0.98}{0.13}{0}{0.43}\pgfsys@color<EMAIL_ADDRESS>refer to the mutual information of heads in MoCo and SimMIM, respectively.

### The role of the early modules.

Figures 3 and 3 suggest that most layers of MoCo capture global patterns and
have only a weak correlation with query tokens. However, one or two of MoCo
layers exhibit unusual behavior. For example, the 3${}^{\text{rd}}$ layer of
MoCo focuses on local areas and its self-attention map is dependent on the
query. We explore this property in more detail.

Figure 1(a) provides the variance of normalized mutual information with
respect of heads. As the results show, the variance of SimMIM is consistent
across all depths whereas that of MoCo is not. In particular, the
3${}^{\text{rd}}$ layer of MoCo has high variance even though other layers do
not. This suggests that, while most of MoCo’s self-attention heads capture
global patterns and have weak correlation with query tokens, some heads
deviate from this behavior and exhibit a different pattern.

Figure 1(b) shows the distribution of normalized mutual information among
heads in the 3${}^{\text{rd}}$ layer to analyse this phenomenon. In this
figure, we use kernel density estimation with Gaussian kernel to visualize the
distribution. The results reveal several outlier heads in MoCo with mutual
information close to 1.0. As a result, these outliers significantly raises the
average value of normalized mutual information.

(a) MoCo

(b) SimMIM

Figure E.2:  The tokens of MoCo form a cluster for each image, while those of
SimMIM are intermingled.  This aligns with the finding that, compared to
SimMIM, MoCo is linearly separable. To demonstrate this property, we visualize
3,528 tokens (196 tokens$\times$18 images) from the representations of the
last layer via t-SNE, and find that a consistent pattern is observed even in
the representations of the intermediate layers. The colors represent three
different classes. See also Figures 5 and 7.

### A comprehensive view through visualization of tokens from multiple images.

Figure 5 visualizes how self-attention layers transform tokens from one or two
images in representation space. The figure demonstrates that MoCo transforms
all tokens in union while SimMIM transforms them individually. As a result,
MoCo separates the representations at the image-level and SimMIM separates
them at the token-level.

The t-SNE visualization (Van der Maaten & Hinton, 2008) in Figure E.2 provides
consistent results and offers even a more comprehensive perspective. In this
figure, we visualize the last representations of 3528 tokens from 18 images
that belong to three different classes. The visualization demonstrates that
MoCo separates the representations into distinct classes and even images,
while maintaining the tokens close together in compact image clusters. On the
other hand, SimMIM separates tokens from images, resulting in a wide
representation space for each image, but the images or even classes may be
challenging to linearly distinguish.

(a) Token visualization

(b) Token-level analysis

(c) Image-level analysis

Figure E.3:  The first layer of MoCo clumps tokens together.  We demonstrate
this property from two perspectives: qualitative visualization and singular
value of token distribution. _Left:_ Similar to Figure 5, we visualize tokens
of a sample image in a representation space. The blue and red data points
represent the tokens before and after the self-attention transformation. As
shown in this figure, the first self-attention layer clumps tokens into a
compact cluster. _Middle and Right:_ Similar to Figure 7, we visualize the
second largest log singular value (not $\Delta$ log singular value) for depth.
The singular value spectra demonstrate consistent results; the first layer of
MoCo (gray area) not only clumps tokens but also images into a compact
cluster.

### The first layer of MoCo aggregates tokens into compact clusters.

Figures 5 and 7 shows that all modules, except the first module, in MoCo
behave consistently. However, we observe that MoCo’s first module behaves
differently and unusually than the others. We elaborate the behaviour of the
first layer of module.

Figure 3(a) shows the qualitative visualization of tokens for a sample image,
similar to Figure 5. This visualization shows that the first MoCo layer
aggregates tokens into compact clusters. Although this figure only uses a
single image, the layer aggregates all images into a small representation
space as well. In terms of singular values, we observe consistent results.
Similar to Figure 7, Figures 3(c) and 3(b) report the second largest log
singular value, instead of the relative log singular value, to investigate the
absolute volume of the representations. As expected, most layers in both MoCo
and SimMIM increase the singular value, but surprisingly, the first layer of
MoCo reduces the singular value, meaning that the volumes of representations
are decreased at both the token-level and image-level. Based on these
observations, we conjecture that the first module of MoCo behaves like an
embedding component.

##  Appendix F Fine-Tuned Models Inherit the Properties of Pre-trained Models

The main text focuses on highlighting the key properties of pre-trained
models. This section demonstrates that these properties are also utilized by
fine-tuned models. As a result, we can safely apply the insights gained from
the main text to various situations.

(a) Normalized MI

(b) Fourier analysis

Figure F.1:  Self-attentions and representations in fine-tuned models exhibit
consistency with those of pre-trained models. Similar to Figures 3 and 9, we
present the normalized mutual information and the Fourier analysis results of
fine-tuned models. The abbreviation “ft” stands for “fine-tuned model.”
_Left:_ Similar to pre-trained models, the mutual information of MoCo’s self-
attention maps is generally lower compared to that of SimMIM. However, it is
noteworthy that the mutual information of the later self-attention maps in
SimMIM decreases significantly. This is because the later layers of a model
trained with supervision or fine-tuning tend to capture global information.
_Right:_ Similarly, SimMIM utilizes higher frequency information than MoCo.

(a) Stylized ImageNet

(b) Robustness for noise frequency

Figure F.2:  Fine-tuned ViTs inherit the robustness against frequency-based
noise. Similar to Figure 8(b), we measure the decrease in the accuracy of ViTs
fine-tuned with MoCo and SimMIM. _Left:_ Even with fine-tuned ViTs, MoCo is
relatively shape-biased and SimMIM relatively texture-biased. This bias is
just less apparent than in linear probing models. _Right:_ The robustness
against frequency-based random noise also suggests the same: MoCo is robust
against high-frequency noise, but SimMIM is not. In conclusion, fine-tuned
models inherit the properties of linear probing models.

### Consistent results in self-attention and Fourier analysis.

Figures 3 and 3 demonstrate that MoCo captures global areas and that its self-
attentions are less related to the query tokens, compared with SimMIM. In
addition, Figure 9 shows that MoCo captures low-frequency information as
opposite to SimMIM. These results are consistent in the fine-tuning scheme.

Figure 1(a) reveals the self-attention behaviours of fine-tuned MoCo and
SimMIM in terms of normalized mutual information. Similar to pre-trained
models, the fine-tuned self-attention maps of MoCo have generally lower mutual
information compared to those of SimMIM. The only significant difference is
that the mutual information of the later self-attention maps in fine-tuned
SimMIM decreases significantly, as later layers in models trained with
supervision or fine-tuning tend to capture more global information. As a
result, the gap between the two methods is reduced. This is also reflected in
the consistent results of Fourier analysis as shown in Figure 1(b). In this
analysis, SimMIM captures higher-frequency information compared to MoCo in
fine-tuning scheme as well. However, the later layers of SimMIM attempt to
capture low-frequency information. Therefore, the gap of fine-tuned models is
smaller than that of pre-trained models.

### CL is shape-biased and MIM is texture-biased in fine-tuning scheme.

In Figure 8, we demonstrate that linear probing model with CL (MoCo) is more
shape-biased and that with MIM (SimMIM) is texture-biased, compared with each
other. As in the experiment, we calculate the classification results of
ImageNet fine-tuned MoCo and SimMIM on Stylized-ImageNet, and measure the
decrease in accuracy against frequency-based random noise. As we would
expected, Figure F.2 shows that the property also extends to the fine-tuned
model. Even though we still observe the difference between MoCo and SimMIM,
the performance gap between MoCo and SimMIM is quite reduced compared to the
gap between the linear probing models.

(a) Early layer

(b) Later layer

Figure F.3:  The later layers of CL and early layers of MIM play a key role in
the fine-tuning scheme. To show this, we initialize a few blocks and measure
the decrease in the fine-tuning accuracy of pre-trained models.

### Later layers of CL and early layers of MIM are important in find-tuning
phases.

As shown in Figure 11, the later layers of the CL and the early layers of the
MIM are linearly separable. This finding suggests that these layers are
significant, however, it does not provide direct evidence that such properties
are preserved during fine-tuning phases. We demonstrate that these layers play
a crucial role in fine-tuning phases as well.

To support this claim, we conduct a study to measure the accuracy drop of
fine-tuned models using pre-trained models with a few blocks initialized. As
shown in Figure 3(a), the results indicate that the initializing a few early
blocks in the pre-training models of SimMIM significantly harms the fine-
tuning accuracy, compared to MoCo. These observations suggest that early
layers of SimMIM play an important role in fine-tuning. Conversely, Figure
3(b) shows that initializing later blocks in the pre-training models of SimMIM
does not significantly harms the fine-tuning accuracy, suggesting that they
are not important in fine-tuning compared with MoCo.

One limitation of this experiment is the evaluation of the accuracy drop in a
single run. Since the accuracy drop of MoCo is marginally higher than that of
SimMIM at the first initialization depth in Figure 3(b), additional
experiments may improve the results. In this experiment, we utilized the same
fine-tuning settings for both MoCo and SimMIM; but experiments with fine-
tuning settings tailored to each method may provide further insight.

##  Appendix G Hybrid Models Outperform CL and MIM in Downstream Tasks

Table G.1: Hybrid models of CL and MIM outperform both CL and MIM in various tasks. $\lambda$ (Importance of CL) | iNat-18 | ADE20k
---|---|---
$0.0$ (SimMIM) | 62.1 | 35.4
$0.2$ (SimMIM + MoCo) | 68.8 | 42.2
$1.0$ (MoCo) | 66.2 | 39.7

The claim that CL and MIM are complementary is demonstrated only on ImageNet
in Section 5. To validate this claim in tasks beyond ImageNet, we evaluated
the pre-trained models of the hybrid method introduced in Section 5 for
another classification task and a semantic segmentation task. In particluar,
we measured the accuracy on iNaturalist 2018 (Van Horn et al., 2018) and the
mIoU on ADE20K (Zhou et al., 2019). As shown in Table G.1, the hybrid model of
SimMIM and MoCo outperforms both SimMIM and MoCo in various downstream tasks.
Therefore, we conclude that the effectiveness of this claim extends beyond
ImageNet.


# Soft Labels for Rapid Satellite Object Detection

Matthew Ciolino PeopleTec, Inc, 4901 Corporate Dr NW, Huntsville, AL 35805,
USA Grant Rosario PeopleTec, Inc, 4901 Corporate Dr NW, Huntsville, AL 35805,
USA David Noever PeopleTec, Inc, 4901 Corporate Dr NW, Huntsville, AL 35805,
USA

###### Abstract

Soft labels in image classification are vector representations of an image’s
true classification. In this paper, we investigate soft labels in the context
of satellite object detection. We propose using detections as the basis for a
new dataset of soft labels. Much of the effort in creating a high-quality
model is gathering and annotating the training data. If we could use a model
to generate a dataset for us, we could not only rapidly create datasets, but
also supplement existing open-source datasets. Using a subset of the xView
dataset, we train a YOLOv5 model to detect cars, planes, and ships. We then
use that model to generate soft labels for the second training set which we
then train and compare to the original model. We show that soft labels can be
used to train a model that is almost as accurate as a model trained on the
original data.

###### Index Terms:

Soft Labels, Object Detection, Datasets

##  I Introduction

Learning representations is a powerful tool in artificial intelligence. Deep
learning has always been used to learn representations of images, text, and
audio but furthermore can be used for transfer learning [1] and pre-training
[2]. In this way one models strength can be used to improve performance on
another task. As is commonplace for many object detection backbones, a pre-
trained feature extraction network [3] is used to initialize the backbone of a
model. This stabilizes training, improves convergence speed, and improves
performance [4]. Soft labels attempt this by abstracting the transfer of
information to the dataset level.

Soft labels use a well-trained model to completely generate the training data
for another model. A clear use of soft labels is in model distillation [5]
where the final layer in a large neural network containing the class
probabilities is used as the ground truth to train a smaller network on. This
effectively teaches a smaller model what a large model learned in a teacher-
student [6] relationship. In this paper, we use soft labels to train an object
detection model on satellite imagery and then rapidly create a dataset of soft
labels.

###  I-A Background

Dataset creation can be a time-consuming and costly endeavor [7]. While a
handful of high-quality satellite object detection datasets exist [8], they
may not be sufficient for the task at hand. By automating the process of
annotations through soft labels, we can cut down on the time it takes to make
a dataset. Various problems and solutions arise with soft labels and here are
some examples:

((a)) Train Set 1

((b)) Train Set 2

Figure 1: Ground Truth Labels (green) vs Soft Labels (red)

####  I-A1 Missing Objects

One intuitive problem with soft labels for object detection is that low
confidence detections will be filtered not allowing a model to learn finer
grain details for objects. While this would leave the objects in question to
be designated as background objects, Wu et al. [9], show that with the right
modification to the bounding box loss function, a model can improve instead of
worsening. To prove this, they dropped 30% of the ground truth labels and
found a 5% drop in performance while when they weigh high-quality bounding
boxes higher an increase of 3% is found as compared to a baseline.

####  I-A2 Dataset Creation

One effective use of soft labels for object detection is subtyping. Subtyping
is the process of taking a class and breaking it down into smaller classes.
For example, a car can be broken down into a sedan, truck, and SUV. In our
past work, Rosario et al. [10] show that with a simple car detection model and
classifying cars by color, we can create a dataset of soft labels for car
colors.

####  I-A3 Overfitting

Overfitting is a common problem in deep learning. While there are many ways to
combat overfitting, Zhang et al. [11] proposed using an online label smoothing
to generate soft labels more reliably. In each epoch during training, they mix
hard labels and the previous epochs soft labels to iteratively improve the
soft labels of each object detected. That mixture is governed by the cross-
entropy classification loss with the original distribution of soft labels
being uniform. This method is shown to better define and separate classes on
image datasets bringing a 2.1% gain on performance for top-1 error on VOC [12]
and COCO.

####  I-A4 Student/Teacher Models

Using the soft labels from a teacher model, a student model can outperform
models trained on partially labeled COCO data. Xu et al. [13] in this semi-
supervised object detection (SSOD) paper, presented 2 techniques: soft
teacher, where the teacher model is actually updated by the student using an
exponential moving average, and a training strategy where the student/teacher
models are trained using images with different augmentations. This use of soft
labels improves state-of-the-art performance by 8.43% on 1% to 10% labeled
coco data.

###  I-B Contributions

The above methods are all examples of how soft labels can be used to improve a
model. In this paper, we approach a far simpler task of simply investigating
the performance drop-off using 100% soft labels as compared to the complete
dataset. Ground truth versus soft label dataset can be seen in [Figure 1]. In
this paper we attempt to answer 3 research questions that pertained to the
nuances of soft labels:

  * •

How does the soft labeling affect performance?

  * •

Can this be used to categorize datasets automatically?

  * •

What confidence value trains the best soft label model?

Figure 2: Expierment Flow TABLE I: Test Set Metrics Model | Conf | mAP50 | mAP95 | F1 Score | box_loss | obj_loss | class_loss
---|---|---|---|---|---|---|---
Train Set 1 | - | 0.76251 | 0.45069 | .71 @ .419c | 0.052841 | 0.03327 | 0.0058541
Train Set 1 Soft | 0.3 | 0.73275 | 0.42357 | .70 @ .482c | 0.042647 (-21.35%) | 0.029469 (-12.11%) | 0.0033020 (-55.74%)
Train Set 1 Soft | 0.5 | 0.70798 | 0.40687 | .69 @ .318c | 0.044096 (-18.04%) | 0.030910 (-7.35%) | 0.0041233 (-34.69%)
Train Set 2 | - | 0.77470 | 0.46090 | .72 @ .421c | 0.0415 | 0.027994 | 0.002451
Train Set 2 Soft | 0.3 | 0.72932 | 0.42006 | .70 @ .503c | 0.042767 (+3.01%) | 0.031548 (+11.93%) | 0.0033173 (+30.04%)
Train Set 2 Soft | 0.5 | 0.71681 | 0.41578 | .70 @ .308c | 0.044652 (+7.32%) | 0.030846 (+9.69%) | 0.0044479 (+57.89%)
TABLE II: Per Class Test Set Metrics Model | Class | P | R | mAP50 | mAP95
---|---|---|---|---|---
Train Set 1 | ship | 0.758 | 0.625 | 0.720 | 0.389
Train Set 1 | car | 0.689 | 0.693 | 0.735 | 0.371
Train Set 1 | plane | 0.876 | 0.674 | 0.823 | 0.585
Train Set 1 Soft 0.3 | ship | 0.677 | 0.645 | 0.694 | 0.355
Train Set 1 Soft 0.3 | car | 0.786 | 0.649 | 0.752 | 0.390
Train Set 1 Soft 0.3 | plane | 0.787 | 0.661 | 0.756 | 0.528
Train Set 1 Soft 0.5 | ship | 0.658 | 0.625 | 0.663 | 0.334
Train Set 1 Soft 0.5 | car | 0.742 | 0.656 | 0.743 | 0.389
Train Set 1 Soft 0.5 | plane | 0.836 | 0.647 | 0.717 | 0.498
Train Set 2 | ship | 0.732 | 0.641 | 0.728 | 0.393
Train Set 2 | car | 0.781 | 0.671 | 0.766 | 0.403
Train Set 2 | plane | 0.806 | 0.719 | 0.826 | 0.584
Train Set 2 Soft 0.3 | ship | 0.750 | 0.622 | 0.707 | 0.382
Train Set 2 Soft 0.3 | car | 0.694 | 0.700 | 0.716 | 0.364
Train Set 2 Soft 0.3 | plane | 0.828 | 0.665 | 0.765 | 0.514
Train Set 2 Soft 0.5 | ship | 0.755 | 0.588 | 0.681 | 0.362
Train Set 2 Soft 0.5 | car | 0.687 | 0.719 | 0.754 | 0.386
Train Set 2 Soft 0.5 | plane | 0.860 | 0.645 | 0.715 | 0.499

##  II Experiment

To answer these questions, we lay out the following experiments [Figure 2]
involving a subset of the xView dataset [14]. We sub-select xView for moving
objects (vehicles, planes, and ships) leaving 34,252 images collected from
parking lots, marine ports, and airports. Approximately one-third of the
pictures were background, meaning they contained no examples but contributed
to the training ”null” set. Our data contains 37,712 instances of ships,
174,779 instances of cars, and 18,052 instances of planes. These counts are
randomly split across 3 sets for train_1/train_2/valid sets containing
40/40/20 percent of the data respectively. More dataset information can be
found in the appendix.

YOLOv5s [15] containing 7,027,720 parameters was trained on the
train_1/train_2 set for 3 epochs with a batch size of 16 using the pretrained
coco [16] model as a checkpoint. Using those models, we soft-labeled the
opposite training set to generate soft-labeled train_2/train_1. All models
trained used the unseen valid set as the test set to compare metrics. We use
the mAP, F1, and losses to evaluate the models.

  * •

mAP is the mean of the average precision value for recall value over 0 to 1
for each class at a given IoU. [17]

  * •

The IoU threshold is the intersection over union of the predicted bounding box
and the ground truth bounding box. [18]

  * •

F1 is the harmonic mean of precision and recall at different confidence
thresholds. [19]

We also look at train/test loss. For loss, YOLOv5 uses a combination [20] of
bounding box loss, objectness loss, and classification loss to train the
model.

  * •

The bounding box loss is the mean squared error between the predicted bounding
box and the ground truth bounding box.

  * •

The objectness loss measures the probability that an object exists in a
proposed region of interest through binary cross entropy.

  * •

The classification loss is the cross-entropy loss between the predicted class
and the ground truth class.

A detailed summary of YOLOv5 can be found here [21]. The test set which the
results are all based on is broken down as follows: Across the 7,896 images
and 58,165 instances in the test set, 8,710 instances (15%) are ships, 45,025
instances (77%) are cars and 4,430 instances (8%) are planes.

In addition, during experimentation OWL-ViT [22] was released. This Open-
Vocabulary Object Detection model can be used to soft label. In the appendix
section, we compare YOLOv5 models with the COCO 2017 validation set with
ground truth, 0.1 confidence OWL-ViT inference, and 0.25 confidence OWL-ViT
inference.

##  III Results

While ground truth models performed the best [Table I], the soft label models
consistently came within a 6% difference for mAP and F1 score. Interestingly,
the F1 score remains very consistent (.68 to .70) with the optimal confidence
value acting inversely to the confidence threshold of the soft-labeled
training set. When looking at losses of the test set after the final epoch of
training, we see a more interesting story that might tell us why the metrics
are lower.

While the soft label trained model losses are both higher and lower than the
ground truth model, we can derive insight from the relative differences. Both
bounding box loss and objectness loss seem relatively consistent at an average
difference of 11.35%. This is in stark contrast to the classification loss
which averages a difference of 44.59%. Since classification loss has the
highest change (still the lowest component of the loss) it can be said that
the unlabeled objects in the soft label training sets account for the loss of
performance.

We can take a look at the per-class metrics [Table II and Figure 3] to get a
better look at what might be going on. As shown, the majority of the decrease
in performance comes from soft-labeled planes. For planes trained on 0.5
confidence soft labels, all metrics dropped on average 8.49%. Interestingly,
cars, which are the smallest but most abundant object in the dataset performed
only 0.65% worse across all metrics than the ground truth models. Overall,
per-class metrics dropped 4.44% across all metrics.

##  IV Discussion

While this performance drop is statistically significant in certain cases,
i.e., planes, the benefits of soft-labels outweigh this loss in performance by
providing additional data in a low-cost and efficient manner as well as
potentially increasing model knowledge in the cases of transfer learning.
Moreover, this loss in performance can usually be remedied by either balancing
the dataset or increasing data for lagging labels, as evidenced by the
statistically insignificant drop in car labeling performance.

##  V Conclusion

Models can be trained exclusively on soft labels with a less than 6% drop in
mAP as compared to ground truth labels on the same dataset. Regardless of the
confidence threshold used to create the soft labels, mAP/F1 scores remain
within 1% of each other. This suggests that the soft labels are not
overfitting to the training data. These results validate that rapid object
detection datasets can be created with soft labels and that soft labels can be
used to train models with a high degree of accuracy.

Figure 3: Per Class Metrics (Table II Plot)

## Future Work

In future work, we would like to use soft labels to improve the performance of
the model on the test set by supplementing existing data instead of training
solely on the soft labels. We would also like to use soft labels from one
generalized model that could categorize any satellite image you are looking at
into a dataset.

## Acknowledgment

The authors would like to thank the PeopleTec Technical Fellows program for
encouragement and project assistance.

## References

  * [1] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.
  * [2] H. H. Mao, “A survey on self-supervised pre-training for sequential transfer learning in neural networks,” _arXiv preprint arXiv:2007.00800_ , 2020.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learningfor image recognition,” _ComputerScience_ , 2015.
  * [4] Y. Chen, T. Yang, X. Zhang, G. Meng, X. Xiao, and J. Sun, “Detnas: Backbone search for object detection,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [5] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning efficient object detection models with knowledge distillation,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [6] C. H. Nguyen, T. C. Nguyen, T. N. Tang, and N. L. Phan, “Improving object detection by label assignment distillation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 1005–1014.
  * [7] R. Incze, “The cost of machine learning projects,” Sep 2019. [Online]. Available: https://medium.com/cognifeed/the-cost-of-machine-learning-projects-7ca3aea03a5c
  * [8] Chrieke, “Chrieke/awesome-satellite-imagery-datasets: list of satellite image training datasets with annotations for computer vision and deep learning.” [Online]. Available: https://github.com/chrieke/awesome-satellite-imagery-datasets
  * [9] Z. Wu, N. Bodla, B. Singh, M. Najibi, R. Chellappa, and L. S. Davis, “Soft sampling for robust object detection,” _arXiv preprint arXiv:1806.06986_ , 2018.
  * [10] G. Rosario, D. Noever, and M. Ciolino, “Soft-labeling strategies for rapid sub-typing,” _arXiv preprint arXiv:2209.12684_ , 2022.
  * [11] C.-B. Zhang, P.-T. Jiang, Q. Hou, Y. Wei, Q. Han, Z. Li, and M.-M. Cheng, “Delving deep into label smoothing,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 5984–5996, 2021.
  * [12] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge.” _Int. J. Comput. Vis._ , vol. 88, no. 2, pp. 303–338, 2010. [Online]. Available: http://dblp.uni-trier.de/db/journals/ijcv/ijcv88.html#EveringhamGWWZ10
  * [13] M. Xu, Z. Zhang, H. Hu, J. Wang, L. Wang, F. Wei, X. Bai, and Z. Liu, “End-to-end semi-supervised object detection with soft teacher,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3060–3069.
  * [14] D. Lam, R. Kuzma, K. McGee, S. Dooley, M. Laielli, M. Klaric, Y. Bulatov, and B. McCord, “xview: Objects in context in overhead imagery,” _arXiv preprint arXiv:1802.07856_ , 2018.
  * [15] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, Y. Kwon, K. Michael, and J. Fang, “ultralytics/yolov5: v6. 2-yolov5 classification models, apple m1, reproducibility, clearml and deci. ai integrations,” _Zenodo. org_ , 2022\.
  * [16] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [17] J. Hui, “Map (mean average precision) for object detection,” Apr 2019. [Online]. Available: https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173
  * [18] Baeldung, “Intersection over union for object detection,” Sep 2022. [Online]. Available: https://www.baeldung.com/cs/object-detection-intersection-vs-union
  * [19] “F-score,” May 2019. [Online]. Available: https://deepai.org/machine-learning-glossary-and-terms/f-score
  * [20] P. Lihi Gur Arie, “The practical guide for object detection with yolov5 algorithm,” Apr 2022. [Online]. Available: https://towardsdatascience.com/the-practical-guide-for-object-detection-with-yolov5-algorithm-74c04aac4843
  * [21] Ultralytics, “Yolov5 (6.0/6.1) brief summary · issue 6998 · ultralytics/yolov5.” [Online]. Available: https://github.com/ultralytics/yolov5/issues/6998
  * [22] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen _et al._ , “Simple open-vocabulary object detection with vision transformers,” _arXiv preprint arXiv:2205.06230_ , 2022.

## Appendix

Dataset information from the training sets. Instance counts bounding boxes,
bounding boxes, box center locations, and box width and height.

((a)) Train Set 1 Dataset Information

((b)) Train Set 1 Soft Labels

Figure 4: Dataset Information (a) and Example Predictions (b)

Figure 5: OWL-ViT YOLOv5 COCO-VAL GT, 0.1, 0.25 Training Metrics

As shown, mAP at 0.5 IoU for the ground truth is 0.2225, OWL-ViT at 0.25
confidence is .1844 (-17.12%), and OWL-ViT at 0.10 confidence is .1534
(-31.05%). Results for this experiment can be found at this public
tensorboard:
https://tensorboard.dev/experiment/J5prk5YeRKqZEuLPigU7jw/#scalars