# PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model
Pretraining

Yuting Gao1 Jinfeng Liu1,2,∗ Zihan Xu1,∗ Jun Zhang1 Ke Li1 Chunhua Shen3
1Tencent Youtu Lab 2Shanghai Jiaotong University 3Zhejiang University
<EMAIL_ADDRESS>The first three authors
contributed equally. This work was done when J. Liu was an intern at Tencent.

###### Abstract

Large-scale vision-language pre-training has achieved promising results on
downstream tasks. Existing methods highly rely on the assumption that the
image-text pairs crawled from the Internet are in perfect one-to-one
correspondence. However, in real scenarios, this assumption can be difficult
to hold: the text description, obtained by crawling the affiliated metadata of
the image, often suffer from semantic mismatch and mutual compatibility. To
address these issues, here we introduce PyramidCLIP, which constructs an input
pyramid with different semantic levels, and aligns visual elements and
linguistic elements in the form of hierarchy via intra-level semantics
alignment and cross-level relation alignment. Furthermore, we adjust the
objective function by softening the loss of negative samples (unpaired
samples) so as to weaken the strict constraint during the pre-training stage,
thus mitigating the risk of the model being over-confident. Experiments on
three downstream tasks, including zero-shot image classification, zero-shot
image-text retrieval and image object detection, verify the effectiveness of
the proposed PyramidCLIP. In particular, with the same amount of pre-training
data of 15 millions image-text pairs, PyramidCLIP exceeds CLIP by
19.2%$/$18.5%$/$19.6% respectively, with the image encoder being
ResNet-50$/$ViT-B32$/$ViT-B16 on ImageNet zero-shot classification top-1
accuracy. When scaling to larger datasets, the results of PyramidCLIP only
trained for 8 epochs using 128M image-text pairs are very close to that of
CLIP trained for 32 epochs using 400M training data.

##  1 Introduction

Recently, vision-language pre-training (VLP) has shown great success, which
aims to improve the accuracy of downstream vision-language tasks by pre-
training a model on large-scale image-text pairs harvested from the web
without any manual annotation. The mainstream VLP methods can be roughly
categorized into two paradigms, single-stream [1, 2, 3, 4, 5] and dual-stream
[6, 7, 8, 8, 9]. Compared to the single-stream counterpart, the dual-stream
paradigm decouples the image encoder and text encoder and extracts features
for image and text respectively, making the dual-stream paradigm more friendly
for downstream applications. Due to the advantages of performance and
efficiency, the dual-stream paradigm dominates. CLIP [6] preforms contrastive
image-language pre-training on 400M image-text pairs collected from the
Internet, which achieves astounding results. Later, methods such as DeCLIP
[10] and FILIP [8] improve CLIP by introducing self-supervision in the image
modal and text modal, and bringing in more fine-grained alignment on ViT [11]
patch tokens.

Although existing CLIP-alike methods have achieved very promising results on
downstream tasks, they strongly rely on the assumption that the image-text
pairs are of high quality: the image and text are of good matching. Ideally,
the pairs are in perfect one-to-one correspondence and have no correlation
with other unpaired samples. However, in practice this assumption is not easy
to satisfy as shown in Figure 1. Firstly, semantic mismatch between the visual
modality and linguistic modality often exists within image-text pairs, e.g.,
(a) Caption Redundancy: text describes too much redundant and fine-grained
details while image needs a more concise caption; (b) Image Redundancy: a
Region-of-Interest (ROI) corresponding to the text is only a sub-region of the
image; and (c) Cast Deficiency: text misses the descriptions of salient
objects in the image, while visual modeling needs to reason about the
relationship among instances. Besides, Mutual Compatibility (d) usually occurs
among pairs, i.e., pairs have more or less local similarities. For example,
the image$/$text of (d) can partially correspond to the text$/$image of (a).
However, existing approaches directly treat other pairs as negative samples
regardless the correlation, which can lead the model to overfit.

Figure 1:  Problems in the web-crawled image-text pairs. (a)(b) and (c) suffer
the semantic mismatch between the visual modality and the linguistic modality,
while the image-text of (d) has similarities with the unpaired text-image of
(a). Note that, (a) the red caption is redundant, (b) the image outside the
red bounding box is the redundant, (c) the descriptions for the casts in the
red boxes are missing and the red caption in (d) is compatible with (a).

In order to tackle the issues mentioned above, we propose PyramidCLIP in this
paper, which attempts to align image and text more precisely in the form of
hierarchy. PyramidCLIP constructs an input pyramid with different semantic
levels at both sides of the dual-stream network, i.e., the global image, local
image regions, and features of the salient instances in the image for visual
modeling; the original caption and text summarization for linguistic modeling.
Then we contrast visual elements and linguistic elements via intra-level
semantics alignment and cross-level relation alignment, tackling the issues of
(a)(b) and (c) respectively. Specifically, for intra-level semantics
alignment, since the global region of the image and text summarization both
contain global semantic information, while the local region and original
caption both contain more fine-grained semantic information, thus they are
treated as two pairs of positive samples. For cross-level relation alignment,
to avoid the visual encoder’s modelling of the object relationship being
overwhelmed by the scene semantics modelling, we explicitly align the instance
relation with linguistic elements. Moreover, for the issue of Mutual
Compatibility, we soften the loss term of the negative unpaired samples during
the contrast process to ease the strict constraint, alleviating the negative
effect of some local similarities.

Extensive experiments demonstrate the effectiveness of our proposed
PyramidCLIP. For fair comparison, when trained with the YFCC15M [12] dataset,
with ResNet-50 [13]$/$ViT-B32 [11]$/$ViT-B16 [11] as the image encoder and
Transformer as the text encoder, our model achieves the state-of- the-art
(SoTA) zero-shot classification on ImageNet [14], namely,
44.7%$/$41.4%$/$47.2% top-1 accuracy. In comparison, the CLIP baseline is
22.5%$/$22.9%$/$27.6%. Furthermore, when scaling to larger-scale dataset, the
results of PyramidCLIP only trained for 8 epochs on 128M image-text pairs are
very close to that of CLIP trained for 32 epochs on a 400M dataset,
significantly improving the training and data efficiency of CLIP.

Our main contributions are summarized as follows:

  * •

We propose PyramidCLIP for more accurate image-text alignment for vision-
language model pre-training, which effectively constructs an input pyramid at
both sides of the visual encoder and linguistic encoder, and then align the
visual elements and linguistic elements via intra-level semantic alignment and
cross-level relation alignment.

  * •

We soften the loss term of negative samples during the contrast process to
ease the strict constraint, so as to avoid the model being over-confident,
alleviating the negative effect caused by local similarities.

  * •

Extensive experiments demonstrate the effectiveness of PyramidCLIP. In the
case of the same amount of pre-training data, PyramidCLIP achieves SoTA
results, and significantly outperforms that of CLIP.

##  2 Related Work

Vision-Language Pre-training Vision-Language Pre-training (VLP) learns a
strong joint representation between two modalities by pre-training models on
large-scale image-text pairs, and it shows impressive transferability on a few
downstream vision and language tasks. In terms of the model architecture
design, the mainstream VLP models can be divided into two types: single-stream
(early fusion of the two input modalities) and dual-stream (late fusion). The
former one uses a single transformer to model both image and text
representations in a unified semantic space by concatenating image and text
input embeddings, including VisualBERT [2], UNITER [1], UNICODER [4], OSCAR
[3] and UNIMO [15]. While the latter one encodes images and texts separately
with a decoupled image encoder and a text encoder for high-level
representations, such as ViLBERT [16], LXMERT [9], ALIGN [7], CLIP [6], and
DeCLIP [10]. From a different perspective, the pre-training objective mainly
comprises two categories: image-text contrastive learning and masked token
tasks based on Language Modeling (LM). Among the methods mentioned above,
UNIMO, ALIGN, CLIP and DeCLIP adopt contrastive learning to pull closer
positive samples while pushing away negative samples. In contrast, VisualBERT,
UNITER, LXMERT, UNICODER use masked token tasks, including Masked
Language/Region Modeling and autoregressive LM. In this paper, we employ the
dual-stream architecture and contrastive learning for simplicity, flexibility,
and relatively cheaper computation.

Fine-grained Alignment Due to the semantic gap between image and text, there
may be some troubles in directly performing the alignment between these two
modalities. For example, the corresponding descriptions of the objects in an
image are not always available in the caption. On the other hand, some words
or phrases in the descriptions may be irrelevance to the images. Thus, finer-
grained alignment is indispensable, as it can reduce noises and provide more
accurate, richer supervision signals of multiple granularity, thus
significantly improving the training efficiency and model performance. FILIP
[8] improves the contrastive objective to achieve finer-level alignment, by
using a token-wise maximum similarity between visual and textual tokens. While
other methods [3, 17, 18, 19] construct multi-level semantic concepts for
finer-grained alignment. OSCAR [3] first introduces multi-level semantics,
captures object region features and the corresponding tags with a pre-trained
object detector, then concatenates text, object tags and region features to
learn the joint representations. VinVL [17] enhances the visual
representations of OSCAR by pre-trianing a more powerful object detector. Both
OSCAR and VinVL form the multi-level semantics only in the visual modality.
MVPTR [18] and X-VLM [19] obtain their multi-level semantics concepts in both
visual and linguistic modalities. MVPTR limits the interaction between object
tags and textual tokens, learns the object-tag alignment in an explicit
manner, and also models the nested property of language by learning phrase-
level semantics. X-VLM learns multi-level alignments by positioning vision
concepts using given texts, and makes alignments between these two parts.
However, in addition to the image encoder and the text encoder, the two
methods both have an additional cross-modal encoder, introducing computation
overhead.

In this paper, we follow the dual-stream design from CLIP and construct three
visual semantics levels and two linguistic semantics levels to form our
PyramidCLIP. Different from methods mentioned above, each level is input to
the corresponding encoder individually, without concatenating. The obtained
three visual representations and two linguistic representations are used to
compute four contrastive loss terms, which helps to achieve multi-level
alignments.

##  3 Method

In this section, we introduce the proposed PyramidCLIP for more accurate
alignment of image and text for vision-language model pre-training. We first
present the overall architecture in Section 3.1, and then introduce intra-
level semantics alignment and cross-level relation alignment in detail in
Section 3.2 and Section 3.3. Finally, the softened objective function is
described in Section 3.4.

Figure 2:  The overall architecture of the proposed PyramidCLIP which is a
dual-stream network. The input elements of visual modelling is of three-level
semantics, and the linguistic modelling has two-level semantics inputs. The
elements of the two modalities interact through intra-level semantics
alignment and cross-level relation alignment.

###  3.1 Overall Architecture

The entire framework of the proposed PyramidCLIP is presented in Figure 2.
PyramidCLIP is a dual-stream network, including a text encoder $h$ and an
image encoder $f=f_{2}\circ f_{1}$, where $f_{1}$ and $f_{2}$ denote the front
part and the rear part of the image encoder respectively. Each encoder
consists of a linear projector and a normalization operator in the end to
project the final class token into the unified dimension and then normalize
it, obtaining the corresponding visual or linguistic representation vector in
the same embedded space.

During the training, for each image-text pair $(I,T)$, the image $I$ is
transformed into two views, i.e., local view $L$ and global view $G$, through
random crop with different ratios, and text $T$ is input to a summary
extractor [20] to generate text summarization $T_{S}$ with higher level
semantics. The image global view $G$ and text summarization $T_{S}$ both
capture more global context information, while the image local view $L$ and
original text $T$ contain more detailed information. Therefore, $G$ and
$T_{S}$ are regarded as a pair of positive samples, while $L$ and $T$ are
regarded as another pair of positive samples, denoted as $(G,T_{S})$ and
$(L,T)$. Then these two pairs are input to the dual-stream encoder to extract
global and local representation pairs, $({\bm{v}^{g}},{\bm{l}^{s}})$ and
$({\bm{v}^{l}},{\bm{l}^{t}})$, where ${\bm{v}^{g}}=f(G)$,
${\bm{l}^{s}}=h(T_{S})$, ${\bm{v}^{l}}=f(L)$ and ${\bm{l}^{t}}=h(T)$. Finally,
$({\bm{v}^{g}}$, ${\bm{l}^{s}})$ and $({\bm{v}^{l}}$, ${\bm{l}^{t}})$ are
pulled together through contrastive learning loss  1 and  2 respectively
(refer to Figure 2), and other samples in the same batch are treated as
negative samples. We term this contrasting process as Intra-level Semantics
Alignment.

Furthermore, in order to explicitly model the relationship between salient
objects in the image, the ROI feature sequences
$\mathcal{R}=\\{{\bm{o}_{1}},{\bm{o}_{2}},...,{\bm{o}_{M}}\\}$ of $M$ detected
salient objects in the image $I$ are extracted through a pre-trained object
detector Faster R-CNN [21]. Then a linear embedding module $\varphi$ is used
to transform the ROI feature sequence $\mathcal{R}$ into the same dimension as
the output of the front part $f_{1}$ of the image encoder. The sequence is
successively fed into the rear part $f_{2}$, which contains one or more Multi-
head Self-attention (MHSA) layers [11], to adaptively capture the relation
between these salient instances, generating the final representation vector
${\bm{v}^{r}}$, i.e., ${\bm{v}^{r}}=f_{2}(\varphi(\mathcal{R}))$. To avoid the
visual model’s modelling of relation being overwhelmed by the context semantic
modelling and weakening the ability of reasoning, we have
$({\bm{v}^{r}},{\bm{l}^{s}})$ and $({\bm{v}^{r}},{\bm{l}^{t}})$ as another two
positive pairs, and the distance between ROI embedding and corresponding
language embedding are being narrowed through contrastive learning loss  3 and
4, termed as Cross-level Relation Alignment.

It is worth noting that at the inference stage, only the original image-text
pair $(I,T)$ is used, i.e., the visual representation ${\bm{v}^{i}}$ from $I$
and the linguistic representation ${\bm{l}^{t}}$ from $T$.

###  3.2 Intra-level Semantics Alignment

Now we present the details of the intra-level semantics alignment. As
mentioned above, the dual-stream vision-language contrastive learning methods
such as CLIP strongly rely on that the image-text pairs are of good quality of
one-to-one correspondence. However, semantic mismatch between images and text
captions often occurs in the automatically harvested data. Therefore, we
construct an input pyramid with multi-level semantics on both sides of the
dual-stream network, and then align image and text within the same semantics
level. Specifically, the image $I$ is transformed to the global view $G$ and
the local view $L$ through two random crops with different ratios. For the
text caption, besides the original caption $T$, text summarization $T_{S}$
with more compact semantics is extracted using a pre-trained text
summarization extractor.

Coarse-grained Global Contrast We set the random crop ratio for generating
global view $G$ to be $[0.9,1]$, which basically contains all the information
in the original image. Text summarization $T_{S}$ condenses the original
caption $T$, removing some redundant and overly detailed information in the
$T$. $G$ and $T_{S}$ both capture global information and can be used as pairs
of positive samples. The projected embedding ${\bm{v}^{g}}$ and ${\bm{l}^{s}}$
of $G$ and $T_{S}$ are pulled closer through contrastive learning.

Fine-grained Local Contrast Since the alignment of the global view $G$ with
the text summarization $T_{S}$ described above is relatively coarse, finer-
grained information is largely discarded. Intuitively, image sub-regions can
be aligned with some descriptions of the caption. To this end, we introduce
fine-grained local contrast. We set the random cropping ratio for generating
local view $L$ to be $[0.5,1]$, which focuses on the sub-region of the image
$I$. The original caption $T$ contains many detailed descriptions, so it is
more suitable to be regarded as a pair of positive samples with $L$. Then the
projected embeddings ${\bm{v}^{l}}$ and ${\bm{l}^{t}}$ of $L$ and $T$ are also
brought together through contrastive loss (refer to Section 3.4).

###  3.3 Cross-level Relation Alignment

To further improve the alignment precision, we introduce the ROI feature
sequence of salient objects in the image to provide more supervisions.
Specifically, given an image $I$ with $M$ salient objects, we use a pre-
trained object detector Faster R-CNN to extract the visual semantics of each
object region as $[{\bm{o_{m}}^{\prime}},{\bm{z}_{m}}]$, where $m$ denotes the
$m_{th}$ object, ${\bm{o_{m}}^{\prime}}$ is a 2048-dimensional feature vector
and ${\bm{z}_{m}}$ is a 4-dimensional normalized position vector indicating
the coordinates of top-left and bottom-right corners. By concatenating
${\bm{o_{m}}^{\prime}}$ and ${\bm{z}_{m}}$, we have the 2052-dimensional
position-sensitive ROI feature vector ${\bm{o}_{m}}$, forming the ROI feature
sequence $\mathcal{R}=\\{{\bm{o}_{1}},{\bm{o}_{2}},...,{\bm{o}_{M}}\\}$. Then
$\mathcal{R}$ $\in\mathbb{R}^{M\times 2052}$ is transformed into
$\mathbb{R}^{M\times d}$ using the projector in the embedding module
$\varphi$, where $d$ indicates the latent dimension of the MHSA layers in the
image encoder. A randomly initialized $d$-dimensional class token is
additionally appended at the front, resulting $\mathcal{F}$
$\in\mathbb{R}^{(1+M)\times d}$ which is further feed into the rear part
$f_{2}$ of the image encoder to compute the normalized ROI relation embedding
${\bm{v}^{r}}$, i.e., $\mathcal{F}=\varphi(\mathcal{R})$ and
${\bm{v}^{r}}=f_{2}(\mathcal{F})$. Note that feature sequence $\mathcal{F}$ is
position-sensitive following $\mathcal{R}$. Thus, positional embedding is not
applied before it enters into the MHSA layer. To enhance the text encoder’s
capability of modelling the relation of concepts, while avoiding weakening the
reasoning ability of the visual encoder, (${\bm{v}^{r}}$, ${\bm{l}^{s}}$) and
(${\bm{v}^{r}}$, ${\bm{l}^{t}}$) are used as another two positive pairs, and
the distance between ${\bm{v}^{r}}$ and ${\bm{l}^{s}}$ and the distance
between ${\bm{v}^{r}}$ and ${\bm{l}^{t}}$ are minimized simultaneously. Since
the instance-level input used by the visual modality is very fine-grained,
while the input employed by the linguistic modality is complete sentences
(both text summarization and the original caption), we term this training
process cross-level relation alignment.

Figure 3: (a) The schematic of CNN-based image encoder. (b) The schematic of
ViT-based image encoder. (c) The structure of LeFF module in ViT.

In the case that the visual model is a convolution neural network (CNN), the
traditional pooling layer is replaced by attention pooling, which actually is
a MHSA layer. So the embedded ROI feature sequence $\mathcal{F}$ is input to
the attention pooling layer, i.e., $f_{2}$, which indicates the final
attention pooling layer, as shown in Figure 3(a). For the transformer-based
visual model (ViT), the sequence $\mathcal{F}$ can be directly input to a
transformer layer. Considering that $\mathcal{F}$ already encodes high-level
visual semantics, we feed it into the rear part $f_{2}$ of the ViT encoder,
see Figure 3(b).

Besides, as pointed out in [22, 23], the standard ViT may not fully leverage
the local context information, which limits the visual representation capacity
of the ViT-based image encoder. Following [23], we incorporate a depth-wise
convolution into the Feed-Forward module of the ViT architecture, termed
Locally-enhanced Feed-Forward module (LeFF), improving the patch-level local
perception and interaction. The structure of LeFF is shown in Figure 3(c).
First, the patch tokens are projected into a higher dimension through a linear
projection layer and reshaped. Next, a $3\times 3$ depth-wise convolution is
utilized to capture the local information. Then the feature maps are flattened
to a token sequence and re-projected into the initial dimension. While the
class token is identical during the process and is concatenated with locally-
enhanced patch tokens, generating the final output. As depicted in Figure
3(b), LeFF is only applied in the front part $f_{1}$ of the ViT-based image
encoder, since it is clearly not suitable for the embedded ROI feature
sequence $\mathcal{F}$. In Figure 3(b), $L$ denotes the total number of
transformer layers in ViT and $L_{s}$ is the number of transformer layers with
LeFF in $f_{1}$. The influence of different settings of $L_{s}$ can be seen in
Section 4.5.

###  3.4 Softened Objective Function

For a batch of $N$ image-text pairs $\\{(I_{i},T_{i})\\}_{i=1}^{N}$, where $i$
indicates the $i_{th}$ pair, the normalized embedded vectors
$\\{{\bm{v}_{i}^{g}},{\bm{v}_{i}^{l}},{\bm{v}_{i}^{r}},{\bm{l}_{i}^{s}},{\bm{l}_{i}^{t}}\\}_{i=1}^{N}$
of the same dimension are obtained by the dual-stream encoders. In this
formulation, ${\bm{v}_{i}^{g}}$, ${\bm{v}_{i}^{l}}$, and ${\bm{v}_{i}^{r}}$
are generated by the image encoder from global-crop image $G$, local-crop
image $L$ and ROI feature sequence $\mathcal{R}$ respectively, while
${\bm{l}_{i}^{s}}$, ${\bm{l}_{i}^{t}}$ are generated by the text encoder from
the text summarization $T_{S}$ and the original text $T$ respectively. Then we
use this vector group to construct four supervision signals $\mathcal{L}_{{\rm
GS}}$, $\mathcal{L}_{{\rm LT}}$, $\mathcal{L}_{{\rm RS}}$ and
$\mathcal{L}_{{\rm RT}}$ for in-batch contrastive learning, which can be
calculated with
$\\{({\bm{v}_{i}^{g}},{\bm{l}_{i}^{s}})\\}_{i=1}^{N}$,$\\{({\bm{v}_{i}^{l}},{\bm{l}_{i}^{t}})\\}_{i=1}^{N}$,$\\{({\bm{v}_{i}^{r}},{\bm{l}_{i}^{s}})\\}_{i=1}^{N}$
and $\\{({\bm{v}_{i}^{r}},{\bm{l}_{i}^{t}})\\}_{i=1}^{N}$ respectively. Our
four contrastive losses, with the formulation of InfoNCE [24], are designed to
achieve the alignment between visual representation and linguistic
representation from disparate semantic levels.

Take the first loss term $\mathcal{L}_{{\rm GS}}$ from
$\\{({\bm{v}_{i}^{g}},{\bm{l}_{i}^{s}})\\}_{i=1}^{N}$ as an example. For the
$i_{th}$ pair, the normalized vision-to-language similarity
${\bm{p}}_{i}^{{A}}(G)=\\{p_{ij}^{{A}}(G)\\}_{j=1}^{N}$ and language-to-vision
similarity ${\bm{p}}_{i}^{{B}}(T_{S})=\\{p_{ij}^{{B}}(T_{S})\\}_{j=1}^{N}$ can
be calculated through:

$\displaystyle p_{ij}^{{A}}(G)$
$\displaystyle=\frac{\mathrm{exp}(\mathrm{sim}({\bm{v}_{i}^{g}},{\bm{l}_{j}^{s}})/\tau)}{\sum_{j=1}^{N}\mathrm{exp}(\mathrm{sim}({\bm{v}_{i}^{g}},{\bm{l}_{j}^{s}})/\tau)},$
(1) $\displaystyle p_{ij}^{{B}}(T_{S})$
$\displaystyle=\frac{\mathrm{exp}(\mathrm{sim}({\bm{l}_{i}^{s}},{\bm{v}_{j}^{g}})/\tau)}{\sum_{j=1}^{N}\mathrm{exp}(\mathrm{sim}({\bm{l}_{i}^{s}},{\bm{v}_{j}^{g}})/\tau)},$
(2)

where $\tau$ is a learnable temperature parameter initialized with $0.07$ and
the function $\mathrm{sim}(\cdot)$ conducts dot product to measure the
similarity scores.

In standard practice, the corresponding label vectors of the ground-truth
labels ${\bm{y}}_{i}^{{A}}(G)=\\{y_{ij}^{{A}}(G)\\}_{j=1}^{N}$ and
${\bm{y}}_{i}^{{B}}(T_{S})=\\{y_{ij}^{{B}}(T_{S})\\}_{j=1}^{N}$, with positive
pair denoted by $\mathrm{1}$ and negatives by $\mathrm{0}$, are used as the
targets to calculate cross-entropy. This kind of hard targets assumes there is
absolutely no similarity between unpaired image and text. However, within a
large-size batch, unpaired image and text may have more or less local
similarity, i.e., some local regions in the image may be matched with some
words or phases in other unpaired text. To address this problem for better
generalization, we use label smoothing to soften the hard targets. The
corresponding soften targets ${\widetilde{\bm{y}}}_{i}^{{A}}(G)$ and
${\widetilde{\bm{y}}}_{i}^{{B}}(T_{S})$ for the $i_{th}$ pair can be written
as:

$\displaystyle{\widetilde{\bm{y}}}_{i}^{{A}}(G)$
$\displaystyle=(1-\alpha){\bm{y}}_{i}^{{A}}(G)+\alpha/(N-1),$ (3)
$\displaystyle{\widetilde{\bm{y}}}_{i}^{{B}}(T_{S})$
$\displaystyle=(1-\alpha){\bm{y}}_{i}^{{B}}(T_{S})+\alpha/(N-1),$ (4)

where $\alpha$ is the smoothing hyper-parmeter set to $0.2$ in our
experiments. Then the loss term $\mathcal{L}_{{\rm GS}}$ can be formulated as:

$\displaystyle\mathcal{L}_{{\rm
GS}}=-\frac{1}{2N}\sum_{i=1}^{N}\sum_{j=1}^{N}(\widetilde{y}_{ij}^{{A}}(G)\cdot\mathrm{log}(p_{ij}^{{A}}(G))+\widetilde{y}_{ij}^{{B}}(T_{S})\cdot\mathrm{log}(p_{ij}^{{B}}(T_{S}))).$
(5)

The other three loss terms $\mathcal{L}_{{\rm LT}}$, $\mathcal{L}_{{\rm RS}}$
and $\mathcal{L}_{{\rm RT}}$ can be calculated similarly. Therefore, the
overall objective function of PyramidCLIP is

$\displaystyle\mathcal{L}=(1-\lambda-\mu-\gamma)\mathcal{L}_{{\rm
GS}}+\lambda\mathcal{L}_{{\rm LT}}+\mu\mathcal{L}_{{\rm
RS}}+\gamma\mathcal{L}_{{\rm RT}},$ (6)

where the loss weights $\lambda$, $\mu$ and $\gamma$ are all set to $0.25$ in
our experiments.

##  4 Experiments

###  4.1 Datasets

Pre-training Datasets All the pre-training datasets that we used and the
corresponding number of image-text pairs are listed in Table 1.

Table 1: Pre-training datasets. Datasets | SBU | CC3M | CC12M | YFCC15M-V1 | LAION99M | Total
---|---|---|---|---|---|---
#$\langle$Image, Text$\rangle$ Pair | 1M | 3M | 10M | 15M | 99M | 128M

SBU [25] is a relatively small image-text dataset, which contains 1 million
image-text pairs obtained from Flickr. YFCC15M [12] is a commonly used subset
of YFCC100M [12] and there are mainly two versions of YFCC15M, V1 and V2.
YFCC15M-V1 is obtained by applying the same filtering rule on YFCC100M as CLIP
[6], while YFCC15M-V2 is collected by DeCLIP [10] with a different filtering
strategy. In addition to a subset of YFCC, the V2 dataset also contains some
additional data crawled from the Internet and is of higher quality than V1.
Note that YFCC15M-V1 is used in our experiments. CC3M [26] and CC12M [27]
conduct the same image-text filter pipeline on Internet webpage sources, and
the difference is that the filtering method of the latter is more relaxed.
LAION400M [28] is currently one of the largest openly available image-text
datasets. We rank the image-text pairs by their similarity scores, which are
pre-computed by the producer using a pre-trained CLIP model, and pick up a 99M
subset with the highest similarity scores. Combining all the above datasets,
we finally have a dataset of 128M image-text pairs.

Extraction of Text Summarization We use the publicly released T5 model [20] to
extract text summarization for all texts.

Extraction of Object Region Features We utilize the object detector Faster
R-CNN pre-trained by VinVL [17] to extract salient object features in the
image. Specifically, we resize the image to the size of $640\times 640$, then
input it into the detector, and take 10 objects with the highest confidence
and extract the feature of each object from the ROI pooling layer of the
detector.

Downstream Datasets We validate the effectiveness of our proposed method on
three downstream tasks: the zero-shot image classification, zero-shot image-
text retrieval and image object detection. For zero-shot image classification,
experiments are carried out on ImageNet [14]. For the zero-shot image-text
retrieval task, experiments are conducted on MS-COCO [29]. For the object
detection task, the proposed method is verified on PASCAL VOC [30].

###  4.2 Implementation Details

Pre-training Settings We experiment on three different architectures,
PyramidCLIP-ResNet50, PyramidCLIP-ViT-B/32 and PyramidCLIP-ViT-B/16, according
to the choice of image encoder. Their detailed architecture designs follow
that of CLIP [6]. The input resolution of image encoder is 224$\times$224 and
the maximum context length of text encoder is 77. We train the three models
using an AdamW [31] optimizer and the cosine learning rate schedule with a
linear warmup. Specifically, the learning rate linearly increases from 0 to
$5\times 10^{-4}$ within $10\%$ of the total steps, and then decreases with a
cosine anneal strategy. The weight decay rate of AdamW is set to 0.2. To save
GPU memory, automatic mixed-precision [32] is used. We train PyramidCLIP from
scratch for either 8 or 32 epochs in our experiments. Besides, the training
batch size is set to 4096 on 15M dataset and 8192 on the large-scale dataset.

Downstream Zero-shot Image Classification Due to the fact that the labels of
common classification datasets are mostly nouns rather than natural language
descriptions, we follow the same prompt setting as used in CLIP, that is, for
every single class name, we generate 80 different textual descriptions with 80
prompt templates (such as “a photo of label”). The ensembles of these textual
representations are used in computing similarity between images and label
names.

Downstream Zero-shot Image-text Retrieval The image-text retrieval task can be
split into two sub-tasks, _i.e._ , image retrieval and text retrieval,
according to the target modality. We evaluate the zero-shot image-text
retrieval capabilities on the MS-COCO dataset, which is performed by ranking
image-text pairs according to their similarity scores.

Downstream Object Detection Following [33, 34], for the downstream object
detection task, we fine tune the whole model. The detector is trained for 24k
steps with a batch size of 40, and the initial learning rate is 0.02 with 100
warm-up iterations and decays by 10 at 18k, 22k steps. The scale of image is
randomly sampled from $[480,800]$ with interval 32 during the training and is
set to 800 for inference.

###  4.3 Fair Comparison against SoTA

We first validate our method on the ImageNet zero-shot classification task and
MS-COCO zero-shot image-text retrieval task using the same amount of pre-
training data.

####  4.3.1  Comparison on YFCC15M

Table 2: Zero-shot top1 accuracy on ImageNet. All the models are pre-trained
for 32 epochs.

Method | Image | ImageNet
---|---|---
Encoder | ZS Top1
Pre-trained on YFCC15M-V2
CLIP [6] | ResNet50 | 37.2†
SLIP [35] | ResNet50 | 28.5†
FILIP [8] | ResNet50 | 21.3†
DECLIP [10] | ResNet50 | 41.9
CLIP [6] | ViT-B/32 | 32.8†
SLIP[35] | ViT-B/32 | 34.3†
Pre-trained on YFCC15M-V1
CLIP⋄ | ResNet50 | 25.5
PyramidCLIP | ResNet50 | 44.7
CLIP⋄ | ViT-B/32 | 22.9
DeFILIP [36] | ViT-B/32 | 36.4†
PyramidCLIP | ViT-B/32 | 41.4
CLIP⋄ | ViT-B/16 | 27.6
PyramidCLIP | ViT-B/16 | 47.2
⋄ Our Implementation † Reported in [36]

For fair comparison, we conduct experiments on YFCC15M-V1 and the results are
shown in Table 4.3.1 and Table 3. It can be seen that our method significantly
exceeds the results of CLIP baseline when both are pre-trained on YFCC15M-V1.
In particular, as exhibited in Table 4.3.1, compared to the CLIP baseline,
PyramidCLIP improves the ImageNet zero-shot top-1 accuracy by
19.2%$/$18.5%$/$19.6% when the visual model is ResNet50$/$ViT-B/32$/$ViT-B/16.
Furthermore, PyramidCLIP exceeds DeFILIP by 5.0%, which is the combination of
DeCLIP and FILIP, when the visual model is ViT-B/32. Note that we also list
the results of other methods pre-trained on YFCC15M-V2 for reference. Since
the quality of V2 is higher than that of V1, the results on V2 are better than
those on V1, but our method still significantly exceeds them, showing the
superiority of PyramidCLIP. Furthermore, the results on the image-text
retrieval task in Table 3 also demonstrates the effectiveness of Pyramid.

Table 3: Zero-shot image-text and text-image retrieval results on MS COCO. All the models are pre-trained for 32 epochs, except SLIP for 100 epochs. Method |  Image Encoder | MS COCO
---|---|---
I2T R@1 | I2T R@5 | T2I R@1 | T2I R@5
Pre-trained on YFCC15M-V1 |  |  |
CLIP⋄ | ResNet50 | 21.0 | 45.2 | 13.4 | 31.2
PyramidCLIP | ResNet50 | 37.2 | 61.8 | 22.0 | 44.4
CLIP⋄ | ViT-B/32 | 17.7 | 40.4 | 10.7 | 27.1
PyramidCLIP | ViT-B/32 | 30.8 | 57.0 | 18.9 | 41.1
SLIP [35] | ViT-B/16 | 33.9 | 60.0 | 22.5 | 45.4
CLIP⋄ | ViT-B/16 | 22.1 | 45.8 | 13.0 | 30.9
PyramidCLIP | ViT-B/16 | 34.7 | 61.6 | 22.7 | 45.5
⋄ Our Implementation

####  4.3.2 Comparison on LAION15M

Since the distribution of different datasets can vary vastly, we not only
conduct experiments on the commonly used YFCC15M dataset, but also sample 15
millions image-text pairs from LAION400M for fair comparison. The results are
shown in Table 4, it can be seen that PyramidCLIP still outperforms the CLIP
baseline by a large margin, regardless of the visual encoder used.

Table 4: Zero-shot image-text and text-image retrieval results on MS COCO and zero-shot top1 accuracy on ImageNet. All the models are pre-trained on LAION15M for 32 epochs. Method |  Image Encoder | MS COCO | ImageNet
---|---|---|---
I2T R@1 | I2T R@5 | T2I R@1 | T2I R@5 | ZS Top1
CLIP⋄ | ResNet50 | 31.5 | 57.0 | 18.9 | 39.8 | 35.6
PyramidCLIP | ResNet50 | 31.2 | 57.7 | 21.4 | 44.5 | 41.1
CLIP⋄ | ViT-B/32 | 25.9 | 49.8 | 15.6 | 34.8 | 32.6
PyramidCLIP | ViT-B/32 | 25.3 | 50.8 | 18.8 | 39.9 | 39.2
CLIP⋄ | ViT-B/16 | 31.4 | 56.2 | 19.2 | 40.6 | 38.3
PyramidCLIP | ViT-B/16 | 30.6 | 57.3 | 22.7 | 45.8 | 45.0
⋄ Our Implementation

###  4.4 Comparison on Large-scale Datasets

In this section, we validate the effectiveness of our method on a larger
dataset, i.e. 128M image-text pairs, and the ImageNet zero-shot classification
accuracy results are shown in Table 5. Due to the time and computational
resource constraints, PyramidCLIP is only pre-trained for 8 epochs. With the
same amount of pre-training data and the number of training epochs,
PyramidCLIP significantly exceeds the results of the CLIP baseline by 4.7% and
6.8% using ResNet50 and ViT-B/32 as the image encoder, respectively.
Furthermore, when the vision model is ResNet50, the result of PyramidCLIP only
trained on 128M data for 8 epochs are already very close to that of CLIP
trained on 400M data for 32 epochs.

Table 5: Zero-shot top1 accuracy on ImageNet. All models are pre-trained on large-scale dataset. Method |  Image Encoder | Pre-training Setting | ImageNet
---|---|---|---
# $\langle$Image, Text$\rangle$ Pair | Epoch | ZS Top1
CLIP [6] | ResNet50 | 400M | 32 | 59.6
DECLIP [10] | ResNet50 | 88M | 32 | 62.5
CLIP⋄ | ResNet50 | 128M | 8 | 53.0
PyramidCLIP | ResNet50 | 128M | 8 | 57.7(+4.7)
CLIP [6] | ViT-B/32 | 400M | 32 | 63.2
DECLIP [10] | ViT-B/32 | 88M | 32 | 66.2
FILIP [8] | ViT-B/32 | 340M | 30 | 68.8
CLIP⋄ | ViT-B/32 | 128M | 32 | 55.5
CLIP⋄ | ViT-B/32 | 128M | 8 | 49.7
PyramidCLIP | ViT-B/32 | 128M | 8 | 56.5(+6.8)
CLIP [6] | ViT-B/16 | 400M | 32 | 68.6
CLIP⋄ | ViT-B/16 | 128M | 8 | 54.4
PyramidCLIP | ViT-B/16 | 128M | 8 | 62.4(+8.0)
⋄ Our Implementation

###  4.5 Ablation Study

Figure 4: Zero-shot performance with different settings of $L_{s}$.

In this section, we further explore the impact of each component in
PyramidCLIP. Due to the time and resource constraints, all the ablation
experiments are pre-trained for 8 epochs on YFCC15M-V1, and then transfer to
the downstream zero-shot ImageNet classification task.

##### The Influence of Different $L_{s}$ Settings

We first explore the influence of the transformer layers $L_{s}$ in the front
part of ViT-based image encoder. Note that the total number of transformer
layers $L$ in ViT is 12. The corresponding results with different $L_{s}$
values are shown in Figure 4. It can be found that setting $L_{s}$ to 9
achieves the best result, hence $L_{s}=9$ in our experiments. Besides,
$L_{s}=0$ represents that the feature sequence $\mathcal{F}$ is input to the
first transformer layer of ViT and all the layers are without using LeFF.
While $L_{s}=12$ indicates that all the 12 layers are with LeFF and the raw
sequence $\mathcal{F}$ is directly input to the final projector without being
processed by transformer, which is the reason why $L_{s}=12$ shows such poor
performance.

##### The Effectiveness of Each Component

We further verify the effectiveness of each component in PyramidCLIP and the
results are shown in Table 6. It is worth noting that $\mathcal{L}_{\rm CLIP}$
represents the original CLIP loss, which is actually the contrastive loss
between the image global view and the original text. Obviously
$\mathcal{L}_{{\rm LT}}$ performs better than $\mathcal{L}_{\rm CLIP}$, since
the local view statistically removes some unimportant information in the
image, i.e., some sub-regions not described in the text. This is why we use
the image local view and the original text to construct the local contrastive
loss. Besides, it can also be seen that other components, including
$\mathcal{L}_{{\rm GS}}$, $\mathcal{L}_{{\rm RT}}$, $\mathcal{L}_{{\rm RS}}$,
LeFF of ViT and soften objectives in PyramidCLIP, can all bring significant
gains individually.

Table 6: Ablation study of each component on ImageNet zero-shot classification task. $\mathcal{L}_{\rm CLIP}$ represents the original CLIP loss. “LeFF” represents the operation of incorporating depth-wise convolution into the FFN layer of ViT. “Soften” means that all the objectives are softened. Image Encoder | Components | ImageNet
---|---|---
$\mathcal{L}_{\rm CLIP}$ | $\mathcal{L}_{{\rm LT}}$ | $\mathcal{L}_{{\rm GS}}$ | LeFF | $\mathcal{L}_{{\rm RT}}$ | $\mathcal{L}_{{\rm RS}}$ | Soften | ZS Top1 | ZS Top5
ResNet50 | ✓ |  |  | - |  |  |  | 29.2 | 53.6
| ✓ |  | - |  |  |  | 30.3(+1.1) | 54.8(+1.2)
| ✓ | ✓ | - |  |  |  | 34.7(+5.5) | 60.6(+7.0)
| ✓ | ✓ | - | ✓ |  |  | 35.9(+6.7) | 63.1(+9.5)
| ✓ | ✓ | - | ✓ | ✓ |  | 37.2(+8.0) | 63.9(+10.3)
| ✓ | ✓ | - | ✓ | ✓ | ✓ | 38.5(+9.3) | 64.7(+11.1)
ViT-B/16 | ✓ |  |  |  |  |  |  | 25.6 | 49.0
| ✓ |  |  |  |  |  | 27.3(+1.7) | 50.7(+1.7)
| ✓ | ✓ |  |  |  |  | 33.4(+7.8) | 58.5(+9.5)
| ✓ | ✓ | ✓ |  |  |  | 36.0(+10.4) | 61.7(+12.7)
| ✓ | ✓ | ✓ | ✓ |  |  | 38.6(+13.0) | 65.2(+16.2)
| ✓ | ✓ | ✓ | ✓ | ✓ |  | 39.6(+14.0) | 66.1(+17.1)
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 41.3(+15.7) | 68.1(+19.1)

###  4.6 Downstream Task: Object Detection

In order to verify that our model can better exploit the relationship between
objects in the image, we validate our models on the object detection task.
Specifically, we take the visual model ResNet50 and then use it as the
backbone of C4 based Faster R-CNN. The results are shown in the Table 7. It
can be seen that compared against the CLIP baseline, our method performs
better both for the cases that it is pre-trained on YFCC15M and LAION15M, and
it is more significant on YFCC15M. Furthermore, the penultimate line denotes
that PyramidCLIP only adopts semantics contrastive loss without using the
relations, and it can be seen that cross-level relation contrast brings more
improvement in detection than intra-level semantics contrast, indicating the
relation contrast can indeed enhance the relation reasoning ability of visual
models, which is expected.

Table 7: Object detection results on the VOC07 test with ResNet-50 backboned C4 based Faster R-CNN. Method | Pre-training Dataset | PASCAL VOC
---|---|---
$AP^{bb}$ | $AP^{bb}_{50}$ | $AP^{bb}_{75}$
Comparison against CLIP, 32ep |  |  |
CLIP⋄ | YFCC15M | 45.0 | 72.7 | 46.6
PyramidCLIP | YFCC15M |  49.2(+4.2) |  77.1(+4.4) |  52.5(+5.9)
CLIP⋄ | LAION15M | 46.8 | 74.9 | 49.5
PyramidCLIP | LAION15M |  48.6(+1.8) |  76.4(+1.5) |  52.5(+3.0)
Ablation on YFCC15M, 8ep |  |  |
CLIP⋄ | YFCC15M | 43.5 | 72.4 | 45.3
PyramidCLIP (w/o Relation) | YFCC15M | 45.2(+1.7) | 74.1(+1.7) | 46.9(+1.6)
PyramidCLIP | YFCC15M |  46.8(+3.3) |  75.6(+3.2) |  49.8(+4.5)
⋄ Our Implementation

##  5 Conclusion

In this paper, we have proposed a hierarchical pre-training method, termed
PyramidCLIP, to achieve improved alignment between visual and linguistic
modalities. It resolves the issue that the webly-crawled data is not in
perfect one-to-one correspondence by explicitly constructing pyramidal
semantic inputs at the both sides of dual-stream network. We also show that
softened intra-level semantics alignment and cross-level relation alignment
can interact between two modalities and are beneficial. PyramidCLIP achieves
the state-of-the-art results on three downstream tasks, which demonstrates the
superiority.

## References

  * [1] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in _Proc. European Conf. Computer Vision_ , 2020, pp. 104–120.
  * [2] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” _arXiv preprint arXiv:1908.03557_ , 2019.
  * [3] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei _et al._ , “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in _Proc. European Conf. Computer Vision_ , 2020, pp. 121–137.
  * [4] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang, “Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 336–11 344.
  * [5] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, “Vl-bert: Pre-training of generic visual-linguistic representations,” _arXiv preprint arXiv:1908.08530_ , 2019.
  * [6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _Proc. International Conf. Machine Learning_ , 2021, pp. 8748–8763.
  * [7] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4904–4916.
  * [8] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, “Filip: Fine-grained interactive language-image pre-training,” _arXiv preprint arXiv:2111.07783_ , 2021.
  * [9] H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,” _arXiv preprint arXiv:1908.07490_ , 2019.
  * [10] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, “Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,” _arXiv preprint arXiv:2110.05208_ , 2021.
  * [11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [12] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “Yfcc100m: The new data in multimedia research,” _Communications of the ACM_ , vol. 59, no. 2, pp. 64–73, 2016.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [15] W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang, “Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning,” _arXiv preprint arXiv:2012.15409_ , 2020.
  * [16] J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” _Proc. Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [17] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” in _Proc. IEEE/CVF Conf. Computer Vision & Pattern Recognition_, 2021, pp. 5579–5588.
  * [18] Z. Li, Z. Fan, H. Tou, Z. Wei, and X. Huang, “Mvptr: Multi-stage vision-language pre-training via multi-level semantic alignment,” _arXiv preprint arXiv:2201.12596_ , 2022.
  * [19] Y. Zeng, X. Zhang, and H. Li, “Multi-grained vision language pre-training: Aligning texts with visual concepts,” _arXiv preprint arXiv:2111.08276_ , 2021.
  * [20] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _arXiv preprint arXiv:1910.10683_ , 2019\.
  * [21] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” _Proc. Advances in Neural Information Processing Systems_ , vol. 28, 2015.
  * [22] J. Guo, K. Han, H. Wu, C. Xu, Y. Tang, C. Xu, and Y. Wang, “Cmt: Convolutional neural networks meet vision transformers,” _arXiv preprint arXiv:2107.06263_ , 2021.
  * [23] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu, “Incorporating convolution designs into visual transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 579–588.
  * [24] A. Van den Oord, Y. Li, O. Vinyals _et al._ , “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , vol. 2, no. 3, p. 4, 2018.
  * [25] V. Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs,” _Proc. Advances in Neural Information Processing Systems_ , vol. 24, 2011.
  * [26] P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 2556–2565.
  * [27] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3558–3568.
  * [28] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” _arXiv preprint arXiv:2111.02114_ , 2021.
  * [29] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Proc. European Conf. Computer Vision_ , 2014, pp. 740–755.
  * [30] M. Everingham, S. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [31] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [32] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in _International Conference on Learning Representations_ , 2018.
  * [33] Z. Fang, J. Wang, L. Wang, L. Zhang, Y. Yang, and Z. Liu, “Seed: Self-supervised distillation for visual representation,” _arXiv preprint arXiv:2101.04731_ , 2021.
  * [34] Y. Gao, J.-X. Zhuang, K. Li, H. Cheng, X. Guo, F. Huang, R. Ji, and X. Sun, “Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning,” _arXiv preprint arXiv:2104.09124_ , 2021.
  * [35] N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” _arXiv preprint arXiv:2112.12750_ , 2021.
  * [36] Y. Cui, L. Zhao, F. Liang, Y. Li, and J. Shao, “Democratizing contrastive language-image pre-training: A clip benchmark of data, model, and supervision,” _arXiv preprint arXiv:2203.05796_ , 2022.