# ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens

$\text{Yangyang Guo}^{\natural}$, $\text{Haoyu Zhang}^{\natural}$,
$\text{Yongkang Wong}^{\natural}$, $\text{Liqiang Nie}^{\dagger}$,
$\text{Mohan Kankanhalli}^{\natural}$ ${}^{\natural}\text{National University
of Singapore}$, ${}^{\dagger}\text{Harbin Institute of Technology (Shenzhen)}$

###### Abstract

Learning a versatile language-image model is computationally prohibitive under
a limited computing budget. This paper delves into the _efficient language-
image pre-training_ , an area that has received relatively little attention
despite its importance in reducing computational cost and footprint. To that
end, we propose a vision token pruning and merging method ELIP, to remove less
influential tokens based on the supervision of language outputs. Our method is
designed with several strengths, such as being computation-efficient, memory-
efficient, and trainable-parameter-free, and is distinguished from previous
vision-only token pruning approaches by its alignment with task objectives. We
implement this method in a progressively pruning manner using several
sequential blocks. To evaluate its generalization performance, we apply ELIP
to three commonly used language-image pre-training models and utilize public
image-caption pairs with 4M images for pre-training. Our experiments
demonstrate that with the removal of 30$\%$ vision tokens across 12 ViT
layers, ELIP maintains significantly comparable performance with baselines
($\sim$0.32 accuracy drop on average) over various downstream tasks including
cross-modal retrieval, VQA, image captioning, _etc_. In addition, the spared
GPU resources by our ELIP allow us to scale up with larger batch sizes,
thereby accelerating model pre-training and even sometimes enhancing
downstream model performance. Our code will be released at link.

##  1 Introduction

Recent advancement in various benchmarks benefits primarily from large model
pre-training. These pre-trained models stand out for their versatility and
generalization ability, and are further encouraged by the scaling law [23,
18], which tells that expanding model size and training data leads to
increasingly better performance. Nevertheless, the use of pre-trained large
models often incurs a noticeable footprint and faces great challenges for
deployment in resource-constrained environments. As a result, many efforts
have been devoted to optimizing the efficiency-effectiveness trade-off of
large models [51, 21, 45].

Figure 1: Visualization of attention map discrepancy between ViT and BLIP
models and pipeline of our proposed method ELIP. (a) When presented with the
same image, ViT and BLIP often see different regions, resulting in a large KL
divergence of their attention maps. (b) ELIP achieves efficient language-image
pre-training by pruning less important vision tokens.

Conventional efficient learning approaches, _e.g_. knowledge distillation [50,
17], low-rank approximation [52] and quantization [9, 34], are commonly
employed to compress a cumbersome model into a lightweight one. By this means,
the computational overhead and memory cost are thereby reduced, despite the
complexity involved in developing these compression algorithms. Since the
emergence of Vision Transformers (ViTs) [11], recent research focus has been
tailored to a more explainable and effective approach, _i.e_. _vision token
pruning_. ViTs embed images using non-overlapped patches, which is distinct
from the traditional approach of CNNs that explicitly incorporates spatial
inductive bias [20]. This operation often leads to redundant vision tokens
that can be safely removed without significantly compromising models’ accuracy
[36, 43, 54]. However, existing pruning methods in the vision-only domain
universally rely on an objective-free approach, whereby the pruning mask is
learned from signals of current or preceding layers [36, 54]. This approach
may entail the risk of removing tokens that play a crucial role in achieving
the task objective, especially for vision-language models.

We notice that there is relatively little literature on efficient language-
image pre-training [35]. In general, the natural correspondence between
language and image mutually describes which token(s) is dispensable for
training a generalizable multi-modal model. Besides, recent methods often
employ separate pre-trained encoders for the two input modalities, wherein the
encoding operation is asynchronous111The complex parallel computing, though
feasible, usually prohibits researchers from encoding language and image
simultaneously.. This allows us to leverage the output from the text encoder
as supervision for removing vision tokens (refer to Fig. 1(b)), which differs
significantly from that in the vision-only domain (In addition, the vision-
only model and language-image model usually concentrate on different regions,
as shown in Fig. 1(a)). The language tokens, on the other hand, are less
redundant in their representation due to short context (_e.g_. 20 words per
sentence) and high information density [15]. We therefore only ablate language
token pruning for completeness [35].

Our method does not require _any incremental trainable parameters_ beyond
backbone language-image models. Building on the observation that the attention
map on vision tokens becomes increasingly concentrated with deeper layers (see
Fig. 2), we implement vision token pruning and merging in a progressive and
multi-stage way. We integrate our ELIP as a plug-and-play module into _three_
popular language-image pre-training models and pre-train them from scratch on
datasets with image-caption pairs using 4M images, where the datasets consist
of MSCOCO Caption [37], Visual Genome [26], SBU [42], and Conceptual Captions
[47]. Through our experimental results, we demonstrate that removing $\sim$30%
vision tokens can well maintain the model performance ($\sim$0.32 accuracy
drop on average) on downstream tasks including image-text retrieval, visual
question answering, visual entailment, image captioning, and natural language
for visual reasoning. In addition, the spared GPU memory by our method enables
model scaling up with larger batch sizes and even sometimes slightly boosts
downstream model fine-tuning. We also validate the effectiveness of combining
our pre-training method with several parameter-efficient transfer learning
approaches. _It is worth noting that we do not apply our proposed method to
CLIP[44] and its successors due to two reasons:_ I) the lack of accessibility
of pre-trained datasets and II) the inflexibility of adapting CLIP models to
non-matching language-image downstream tasks such as VQA. To summarize, our
ELIP represents an initial attempt to achieve efficient language-image pre-
training with fewer vision tokens. We believe that our approach provides
valuable insights for future language-image pre-training to develop more
advanced models whilst reducing computational cost and footprint.

##  2 Related Work

Figure 2: Token similarity and attention maps across different ViT layers of
BLIP [30], as well as the FLOPs proportion of different modules for three
typical language-image pre-trained models. (a) The attention distribution over
image tokens grows from uniform to concentrated with layers going deeper.
Besides, the token similarity initially decreases but then significantly
increases, indicating that more vision tokens become redundant. (b) Notably,
the vision encoder (VE) accounts for the majority of the computational cost of
language-image models (compared to the text encoder - TE and modal fusion -
MF).

###  2.1 Pruning in Neural Networks

Network pruning is leveraged to remove unnecessary or less important
components in models [56]. By removing some connections or parameters, the
original dense network reduces to a sparse one, in which the required capacity
for storage will dwindle as well as the volume of computations [25, 58]. Based
on the granularity of reduction, existing methods can be roughly grouped into
unstructured pruning and structured pruning. The former refers to pruning less
salient components, such as neurons or connections between layers [58, 38,
57]. In contrast, the latter aims to remove a large bundle of parameters [16,
25], on which we mainly discuss in this section. Previous structured pruning
methods mostly target removing less influential Transformer heads [7], layers
[59], and convolutional channels [16]. With the startling success of ViT [11],
increasing research has been devoted to pruning input tokens of each layer due
to the following two reasons. First, the input tokens from different layers
have different redundancies and only a small number of them contribute
significantly to the accuracy of models [25, 43, 5]. Second, pruning tokens
leads to more visual explainability as compared to other elements such as
heads or layers.

Perhaps the most relevant work to ours is TRIPS [22]. It is worth noting that
our method sets it apart from TRIPS by four merits: We propose to employ an
enhanced pruning approach by leveraging multi-modal information, whereas TRIPS
solely relies on text; Our method achieves improved efficiency; We conduct a
more comprehensive evaluation to validate the generalizability of the proposed
method (we consider three models while TRIPS only uses one); We further
validate the effectiveness of combining our method with other parameter-
efficient transfer learning techniques (refer to the supplementary material).

###  2.2 Vision-Language Transformers

The past few years have witnessed the popularity of Transformers in natural
language processing and computer vision [55, 10, 11]. Given its sweeping trend
and overwhelming performance in these related domains, researchers have
actively extended this technique to vision-language tasks. In detail, a _pre-
train then fine-tune_ paradigm is adopted by mainstream methods and the models
are often pre-trained on certain large-scale vision-language datasets [10].

Unlike previous single modality model pre-training, the vision-language domain
requires two heterogeneous inputs. The ubiquitous image-text pairs, _i.e_.
textual caption regarding an image, serve as the key data format for pre-
training due to their easy availability. Common datasets include Conceptual
Captions [47], Visual Genome [27], COCO Captions [37], and LAION-400M [46]. At
the bedrock of vision-language Transformers lies the embedding behavior of the
two modalities. Pertaining to the vision embedding, the feature extraction has
grown from grid [3], region features [2] of CNN models, to the recent patch
features of Transformers [28]. In contrast, the text tokenization promptly
changed from traditional Word2Vec to BERT-style pre-trained embeddings after
the prevalence of modern language modeling [55, 10]. On top of the embedding
process, there are generally two types of modal fusion approaches: dual-stream
and single-stream. The former adopts a late fusion strategy, where the vision
and text are separately encoded until a fusion operation to combine these two
[40, 1, 53, 13, 19, 30]. The single-stream fusion method presents to encode
the text and vision with a unified Transformer model, wherein the modal fusion
is performed beforehand [49, 48, 8, 33, 24]. To enable the training on these
large-scale captioning datasets, some pretext objectives are carefully
designed, such as masked language modeling [10, 40], masked vision modeling
[8, 53], and image-text matching [30, 40].

##  3 Method

###  3.1 Preliminary

Figure 3: Overview of our proposed ELIP method. ELIP is composed of four
sequential blocks and the corresponding numbers of layers are respectively 2,
2, 6, and 2. To reduce the computational overhead, we prune and merge less
influential vision tokens based on the features of the vision [CLS] and text
[CLS] tokens for the last three blocks.

####  3.1.1 Overview of language-image pre-trained models

Transformers have grown into a fundamental building block of many modern
language-image models [29, 24, 30, 12]. According to the common training
paradigm, existing models can be split into three modules: vision encoder,
text encoder, and multi-modal fusion. In this way, the language and vision
encoders can be aligned with separate pre-trained models to facilitate
knowledge transfer.

Vision Encoder. Recent language-image pre-training models often leverage the
ViT model [11] as the vision encoder. ViT first partitions an RGB image
$I\in\mathbb{R}^{3\times H\times W}$ into $M\times M$ non-overlapping patches.
Together with a class token [CLS], these image patches are thereafter fed into
$N$ layers with self-attention as the basic operation. To this end, a set of
query, key, and value matrices are transformed from the patch embedding to
token features $\mathbf{X}^{v}\in\mathbb{R}^{(1+M^{2})\times d}$, where $d$
denotes the embedding size, followed by several feedforward layers and
residual connections.

Text Encoder. After tokenizing the input sentence according to the BERT
approach [10], current methods often employ a special [CLS] token at the
beginning and a [SEP] token at the end of the sequence. These tokens serve to
delimit the sentence and enable the BERT encoder to extract the token
encoding, which is represented by $\mathbf{X}^{t}\in\mathbb{R}^{(2+T)\times
d}$, where $T$ denotes the sentence length.

Modal Fusion. The model fusion module leverages frameworks that are similar to
those utilized by the Transformer decoder [55]. In particular, the common
practice includes the cross-attention between the vision and text encoders
[29, 30], as well as merged attention techniques [12].

Pre-training Objectives. Pre-training of language-image models on large-scale
image-caption datasets [37, 47] is made possible by several pretext
objectives. One such instance is masked language modeling, which aims to
reconstruct masked text tokens given the remaining ones and the image feature.
Additionally, the image-text matching (ITM) objective is employed to classify
the correspondence between a given image and its accompanying text.
Nevertheless, masked image modeling objectives have been largely abandoned by
recent approaches due to their convergence difficulty and lack of usefulness
for downstream fine-tuning performance [30, 24, 12].

####  3.1.2 Research Motivation

While achieving state-of-the-art results on downstream tasks, language-image
pre-training models can suffer from computational inefficiency. To approach
this problem, we first leverage Fig. 2 to illustrate two critical observations
that motivate this work:

###### Remark 1

Fig. 2(b) indicates that the vision encoder usually accounts for the majority
of overhead in a language-image model, especially for ALBEF [29] and BLIP
[30]. Given this observation, reducing the computational cost of the vision
encoder will yield great improvement in model training efficiency.

###### Remark 2

The vision tokens from ViT are redundant in their representations [36, 4], as
is the case for these language-image models. Moreover, the attention
distribution becomes increasingly concentrated for deeper ViT layers (as seen
in Fig. 2(a)). One insight from this observation is that we can progressively
remove these tokens that are less useful for the image-text matching objective
to achieve computational efficiency.

###  3.2 Method Architecture

In light of the above two observations, this paper aims to study the
_efficient language-image pre-training_ by means of _vision token pruning and
merging_. We do not remove image patches in the input space [35] as we believe
some background patches still provide useful features for cross-modal semantic
understanding. Instead, we propose to prune the vision tokens that are less
influential for the matching goal of the given image-text pair.

As noted in Remark 2, the redundancy of vision tokens increases as the depth
of Transformer layers grows. In view of this, we design a progressive pruning
strategy with multiple-stage blocks that follow the hierarchical ViT
structures, such as Swin Transformer [39] and MetaFormer [60]. Specifically,
our approach involves dividing a standard ViT encoder into four distinct and
non-overlapping blocks, as illustrated in Fig. 3:

  * •

Block I remains unaltered for the first two ViT layers. Unlike the vision-only
domain, both causal and background features contribute a lot to the semantic
understanding in a language-image model. It is thus preferable to leave these
layers close to the input unchanged.

  * •

Block II consists of two layers and prunes a few of all the vision tokens
(_e.g_. 10%) that are redundant.

  * •

Block III removes much more tokens (such as 25%) preceding the next six
layers. Fig. 2 shows that the attention maps tend to exhibit increasingly
concentrated for deeper layers, indicating that the model focuses primarily on
a small number of representative image regions.

  * •

Block IV further performs token pruning and keeps $\alpha$ (_e.g_. 40$\%$) of
the entire vision tokens with the last two layers, which are crucial for the
subsequent multi-modal fusion. Inspired by MaskAE [15] approaches, we
demonstrate that we can maintain a comparable fine-tuning model performance by
retaining only a small group of vision tokens.

Table 1: Comparison of text retrieval (TR) and image Retrieval (IR)
performance on Flickr30K and MSCOCO datasets. The TFLOP calculations are based
on a batch size of 36, and the memory usage estimates are only applicable to
the tested backbones and our proposed methods. Latency: ms; Mem: GB.

Model | TFLOPs | Latency | Mem | Flickr30K | MSCOCO
---|---|---|---|---|---
TR | IR | TR | IR
R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
ViLT [24] | 9.74 | 573 | - | 83.5 | 96.7 | 98.6 | 64.4 | 88.7 | 93.8 | 61.5 | 86.3 | 92.7 | 42.7 | 72.9 | 83.1
UNITER [8] | 0.20 | 31 | - | 87.3 | 98.0 | 99.2 | 75.6 | 94.1 | 96.8 | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0
VILLA [13] |  $\sim$0.60 |  $\sim$93 | - | 87.9 | 97.5 | 98.8 | 76.3 | 94.2 | 96.8 | - | - | - | - | - | -
UNIMO [32] | - | - | - | 89.4 | 98.9 | 99.8 | 78.0 | 94.2 | 97.1 | - | - | - | - | - | -
METER [12] | 8.66 | 494 | 90.0 | 89.6 | 98.3 | 99.4 | 77.0 | 94.5 | 97.5 | - | - | - | - | - | -
\- EViT | 4.68 | 325 | 64.8 | 60.5 | 86.6 | 92.6 | 44.9 | 77.4 | 86.6 | - | - | - | - | - | -
\- ELIP | 6.43 | 420 | 70.4 | 89.3 | 98.8 | 99.6 | 76.0 | 94.7 | 97.4 | - | - | - | - | - | -
ALBEF [29] | 9.65 | 594 | 88.1 | 93.6 | 99.1 | 99.9 | 81.0 | 96.0 | 97.8 | 72.2 | 91.8 | 96.1 | 55.9 | 81.4 | 88.8
\- EViT | 3.21 | 262 | 50.8 | 87.7 | 97.8 | 98.6 | 75.4 | 93.1 | 96.7 | 65.7 | 88.4 | 94.0 | 49.7 | 77.1 | 85.8
\- ToMe | 6.66 | 450 | 69.6 | 92.1 | 98.7 | 99.6 | 78.1 | 94.6 | 97.6 | 68.8 | 90.1 | 94.9 | 51.9 | 79.1 | 87.1
\- ELIP | 8.50 | 518 | 69.6 | 93.4 | 99.3 | 99.8 | 80.6 | 95.4 | 97.7 | 71.8 | 91.6 | 95.7 | 55.0 | 80.8 | 88.4
BLIP [30] | 11.03 | 1,102 | 90.8 | 94.2 | 99.1 | 99.9 | 81.4 | 95.6 | 98.1 | 72.8 | 92.1 | 96.1 | 56.6 | 81.7 | 88.9
\- EViT | 4.80 | 536 | 60.8 | 87.3 | 98.5 | 99.4 | 75.1 | 93.5 | 96.4 | 66.8 | 88.9 | 93.9 | 50.8 | 77.9 | 86.3
\- ToMe | 6.98 | 740 | 72.0 | 91.5 | 98.8 | 99.4 | 80.5 | 95.6 | 97.9 | 71.5 | 91.6 | 95.9 | 55.3 | 81.2 | 88.7
\- ELIP | 9.34 | 960 | 74.7 | 92.2 | 99.1 | 99.7 | 80.3 | 96.0 | 98.0 | 72.0 | 91.9 | 95.9 | 56.3 | 81.2 | 88.7

###  3.3 Vision Token Pruning and Merging

Fig. 1 illustrates the encoding process typically used in language-image
models [29, 12], in which text and image inputs are processed separately. This
non-parallel operation allows us to leverage the output of the text encoder to
help remove irrelevant tokens in the vision encoder, which can provide
significant benefits over vision-only pruning models [4, 36]. Moreover, the
alignment between the image and text is determined by the features extracted
from the [CLS] token. As a result, we employ the fusion of these two sets of
features to jointly decide which tokens are influential for each given block.
We outline the process of the algorithm in the supp. Specifically, the number
of tokens reduces from 1 + $M_{i}$ to 2 + $\alpha M$ for each block with the
pruning and merging approach. Here, $M_{i}$ and $M$ represent the token
numbers of the current block and the input, respectively. The retaining ratio
$\alpha$ is defined in Sec. 3.2 and is always less than 1.0, _e.g_. 0.4 for
the last block.

To this end, we first replace the image [CLS] token features with the fusion
of itself and the text [CLS] features,

$\mathbf{X}^{v}_{[CLS]}=\lambda\mathbf{X}^{v}_{[CLS]}+(1-\lambda)\mathbf{X}^{t}_{[CLS]},$
(1)

where $\lambda$ is a coefficient hyper-parameter balancing the contribution of
vision and text tokens. In the next, we perform token pruning and merging
without considering gradients, _i.e_. in a _stop-gradient_ fashion. All the
vision token features $\mathbf{X}^{v}$ are thereafter fed to each layer of the
current block and only the attention values of [CLS] from the final layer,
namely $\mathbf{\xi}$ are preserved,

$\mathbf{X}^{v},\mathbf{\xi}=\text{BLOCK}_{i}(\mathbf{X}^{v};\mathbf{\Theta}),$
(2)

where $\Theta$ represents the involved parameters and with no gradient during
this computation. We then retain those token features based on a pre-defined
retaining ratio $\alpha$.

$\overline{\mathbf{X}}^{v}=concat(\mathbf{X}^{v}_{[CLS]};\\{\mathbf{X}^{v}_{j}\\}_{j\in\\{\text{top-}\alpha
M(\mathbf{\xi})\\}}),$ (3)

where $\text{top-}n()$ denotes the index set with the largest $n$ values. For
the remaining tokens, we merge them into a single token according to their
attention after re-normalization,

$\begin{cases}\hat{\mathcal{M}}_{i}=\mathcal{M}_{i}\setminus\\{\text{top-}\alpha
M(\mathbf{\xi})\\},\\\
\hat{\mathbf{\xi}}=norm(\\{\mathbf{\xi}_{j}\\}_{j\in\hat{\mathcal{M}}_{i}}),\\\
\mathbf{X}^{v}_{merge}=\sum_{k\in\hat{\mathcal{M}}_{i}}\mathbf{X}^{v}_{k}\hat{\mathbf{\xi}}_{k},\end{cases}$
(4)

where $\mathcal{M}_{i}$ represents the overall token index set from
$1\rightarrow M_{i}$ of the current block $i$. We finally concatenate it with
the remaining tokens after pruning. This approach ensures that the subsequent
ViT layers will consider a smaller number of tokens, leading to more efficient
processing.

###  3.4 Method Analysis

####  3.4.1 An In-depth Understanding

Our method leverages the multi-modal information (_i.e_. weighted sum) for
pruning, as demonstrated in Eqn. 1. We illustrate two extreme cases where
$\lambda$ takes the values of 0 or 1. On the one hand, when $\lambda=0$, our
method degrades to vision-only pruning, wherein there is no supervision from
the text. On the other hand, when $\lambda=1$, similar to TRIPS [22], the
pruning is solely determined by the text, resulting in a significant drop in
model performance. We relate this result to that of momentum update in MoCo
[14]. Specifically, replacing the vision [CLS] with the text [CLS] introduces
a substantial modality gap, which confuses the model’s learning process in
terms of which tokens it should focus on. In contrast, a slowly evolving
vision [CLS] ($0<\lambda<1$) serves as the core to harmoniously maintain
modality consistency.

Table 2: Performance comparison on VQA and $\text{NLVR}^{2}$ tasks.

Model | VQA | $\text{NLVR}^{2}$
---|---|---
test-dev | test-std | dev | test-P
VisualBERT [31] | 70.80 | 71.00 | 67.40 | 67.00
ViLT [24] | 71.26 | - | 75.24 | 76.21
LXMERT [53] | 72.42 | 72.54 | 74.90 | 74.50
UNITER [8] | 72.70 | 72.91 | 77.18 | 77.85
12-in-1 [41] | 73.15 | - | - | 78.87
ALBEF [29] | 74.57 | 74.79 | - | -
\- ELIP | 74.33 | 74.48 | - | -
METER [12] | 74.72 | 74.71 | 78.69 | 79.66
\- ELIP | 74.16 | 74.31 | 78.41 | 79.36

####  3.4.2 Complexity Analysis

To explain the efficiency of our method, let us consider the ViT-Base model,
which contains a total of $\pi$ vision tokens. Given that the ViT model
comprises twelve layers, the overall memory complexity can be roughly
estimated as $\mathcal{O}(f(12\times\pi))$, where $f$ denotes the token
processing in a single layer. Let us assume the retaining ratios are 90%, 65%,
and 40% for the last three blocks222The ratios are defined based on the
overall tokens., respectively. With our ELIP method, the resulting memory
complexity with respect to tokens is reduced to $\mathcal{O}(f((2+2\times
0.9+6\times 0.65+2\times 0.4)\times\pi))\approx\mathcal{O}(f(8.5\times\pi))$,
which corresponds to a reduction of approximately 30% in memory usage relative
to the original baseline model. This reduction enables the pre-training of
language-image models using larger batch sizes or deeper layers, while also
reducing the computational complexity as fewer tokens are taken in the self-
attention operation.

##  4 Experiments

###  4.1 Experimental Settings

####  4.1.1 Pre-training

We utilized four publicly available large-scale datasets for pre-training:
MSCOCO Caption [37], Visual Genome [26], SBU [42], and Conceptual Captions
[47], which together provide image-text pairs with $\sim$4M images. We
employed five downstream vision-language tasks in this work. To evaluate the
generalization performance, we applied our ELIP method to three recent popular
language-image pre-trained models, _i.e_. ALBEF [29], BLIP [30], and METER
[12]. For each individual model, we trained it from scratch using four NVIDIA
A5000 GPUs and kept most of the experimental settings untouched, except for
reducing the batch size due to resource constraints. We fixed the coefficient
parameter $\lambda$ to 0.8 in Eqn. 1 for all models. The detailed
implementation of each model can be found in the supplementary material.
Moreover, we adopted the PyTorch profiler API to quantify the FLOPs and
latency consumed by each model.

Table 3: Performance comparison on SNLI-VE (a) and Image Captioning (b) tasks.

(a) Results of both val and test sets on SNLI-VE.

Model | val | test
---|---|---
12-in-1 [41] | - | 76.95
UNITER [8] | 78.59 | 78.28
ALBEF [29] | 79.33 | 79.41
\- EViT | 78.54 | 78.75
\- ToMe | 78.69 | 78.76
\- ELIP | 79.24 | 79.38
METER [12] | 79.94 | 79.41
\- EViT | 74.65 | 73.89
\- ELIP | 79.59 | 79.10

(b) Following BLIP, we compare only with a single representative baseline that
has been pre-trained on large-scale datasets while excluding these
conventional comparatively weak approaches.

Model | CIDEr$\uparrow$
---|---
Enc-Dec [6] | 110.9
BLIP [30] | 121.7
\- EViT | 117.5
\- ToMe | 121.4
\- ELIP | 122.5

####  4.1.2 Compared Baselines

As the efficient language-image models are quite sparse in literature, we thus
adapted two SOTA vision token pruning baselines in the vision-only domains for
comparison – EViT [36] and ToMe [4]. Both methods prune the vision tokens for
each ViT layer in an unsupervised manner. Specifically, EViT leverages the
corresponding class token attention, while ToMe merges redundant tokens based
on a bipartite soft matching algorithm.

###  4.2 Overall Results

In Table 1, 2, and 3(b), we present the performance comparison of our approach
with other state-of-the-art methods on five downstream language-image tasks,
involving six datasets in total. The reported TFLOP values (both forward and
backward) are estimated with a batch size of 36, and the GPU memory usage is
calculated based on four A5000 GPUs (only backbone with and without ELIP
method). We excluded some experiments due to resource reasons, _e.g_. VQA on
the BLIP approach, or incompatibilities, such as METER and ToMe. The main
observations are as follows:

  * •

Previous strong language-image pre-training methods, such as UNITER [8] and
VILLA [13] often employ pre-extracted object features for vision encoder.
While these approaches can be less computationally expensive in terms of
TFLOPs, the retrieval results are often inferior to the recent models with ViT
encoders such as ALBEF [29] and BLIP [30].

  * •

The two baseline methods – EViT and ToMe, though reduce the model complexity
by a large margin, often trade drastic model performance over these downstream
tasks. For example, when applying EViT to the METER model, there is a
significant drop of 20 to 30 points in R@1 for both text and image retrieval.

  * •

Unlike the two baseline methods and other compared approaches, our ELIP model
achieves a superior efficiency-effectiveness trade-off. Specifically, across
all five downstream tasks, our model yields an average accuracy drop of less
than 0.33 for the three backbone models, evidently demonstrating its
effectiveness and generalization ability.

Table 4: The effect of three text pruning approaches on retrieval results of
the Flickr30K dataset.

Model | TR | IR
---|---|---
R@1 | R@5 | R@1 | R@5
METER | No Pruning | 84.2 | 97.7 | 69.5 | 92.3
Random | 73.8 | 93.6 | 57.9 | 88.1
Post | 76.1 | 94.6 | 61.6 | 89.0
Learned | 76.9 | 94.8 | 60.9 | 89.2
BLIP | No Pruning | 91.5 | 98.8 | 77.9 | 94.6
Random | 89.9 | 98.2 | 75.8 | 94.1
Post | 91.6 | 98.9 | 78.2 | 94.5
Learned | 91.1 | 98.8 | 77.7 | 94.5

###  4.3 Ablation Study

####  4.3.1 Text Token Removal

In typical language-image pre-training datasets, such as Conceptual Captions
[47], the text is often accompanied by a short context, consisting of
approximately 20 words per sentence. Moreover, previous studies have shown
that language tokens are typically less redundant and have a higher
information density in their representation [15, 35]. As a result, in our
method, we did not introduce text token pruning and only performed ablation
experiments to investigate its influence on model performance. Specifically,
we preserved the first half of all Transformer layers and pruned 40% of text
tokens in the second half to achieve a balance between efficiency and
effectiveness.

We designed three pruning strategies for this experiment: _Random_ – The
tokens are randomly pruned; _Post_ – We prioritized pruning the post tokens;
and _Learned_ – We used the [CLS] token features from the vision encoder to
guide the pruning of text tokens. We run these models for three pre-training
epochs and report the results in Table 4. One can observe that: I) Among the
three approaches, random pruning tends to perform unsatisfactorily due to the
unexpected removal of crucial text tokens, leading to inferior performance.
II) Pertaining to text token pruning, the BLIP model is less affected than the
METER model. For instance, with the post-pruning approach, the BLIP model’s
performance even slightly surpasses that of the non-pruned model.

Figure 4: Component effect on the text retrieval performance over the
Flickr30K dataset. Left: Performance comparison of pruning-only and pruning-
then-merging approaches. Right: Performance change with respect to the feature
combination coefficient parameter $\lambda$ in Eqn. 1.

####  4.3.2 Token Merging v.s. Pruning

One alternative way to deal with the inattentive tokens is to directly prune
them. Note that this pruning-only strategy leads to a slight efficiency
improvement compared to the merging one. To study its effectiveness in
downstream fine-tuning, we removed the merging operation in Eqn. 4 and
observed the performance change of this model. As shown in Fig. 4, we can see
that compared with the token merging, the pruning-only strategy usually
results in inferior downstream performance. This finding implies that these
less attentive tokens still contribute to the final model performance, and
removing them, especially from the input image space (as proposed in [35]) may
lead to sub-optimal downstream fine-tuning results.

####  4.3.3 Effect of Coefficient Parameter $\lambda$

We also experimented with different coefficient values in Eqn. 1. We conducted
this experiment with three pre-training epochs and show the downstream
performance change using different $\lambda$ in Fig. 4. The figure indicates
that using the vision or text [CLS] token only for the supervision of token
pruning leads to inferior outcomes. On the other hand, the combination of
these two, _i.e_. when $\lambda=0.6$ consistently outperforms the other values
tested. This result supports the validity of leveraging multi-modal feature
interaction for token pruning and merging in our method.

In addition, we also conducted experiments on the influence of downstream
pruning and pre-training epochs, and reported the results in the supplementary
material.

Table 5: Performance of scaling ELIP pre-training to larger batch sizes. We
also show the relative improvement of $\text{ELIP}_{{\color[rgb]{0,0,1}+}}$
over the original Base models on the retrieval results.

Model | batch size$\uparrow$ | latency$\downarrow$ | TR | IR
---|---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
METER | Base | 36$\times$4 | 595m | 89.6 | 98.3 | 99.4 | 77.0 | 94.5 | 97.5
ELIP | 36$\times$4 | 570m | 89.3 | 98.8 | 99.6 | 76.0 | 94.7 | 97.4
$\text{ELIP}_{{\color[rgb]{0,0,1}+}}$ | 54$\times$4 | 500m | 88.7 | 98.4+.1 | 99.4 | 75.8 | 94.2 | 97.2
ALBEF | Base | 40$\times$4 | 441m | 93.6 | 99.1 | 99.9 | 81.0 | 96.0 | 97.8
ELIP | 40$\times$4 | 406m | 93.4 | 99.3 | 99.8 | 80.6 | 95.4 | 97.7
$\text{ELIP}_{{\color[rgb]{0,0,1}+}}$ | 58$\times$4 | 369m | 93.7+.1 | 99.3+.2 | 100.0+.1 | 81.1+.1 | 95.6 | 98.0+.2
BLIP | Base | 42$\times$4 | 722m | 94.2 | 99.1 | 99.9 | 81.4 | 95.6 | 98.1
ELIP | 42$\times$4 | 628m | 92.2 | 99.1 | 99.7 | 80.3 | 96.0 | 98.0
$\text{ELIP}_{{\color[rgb]{0,0,1}+}}$ | 56$\times$4 | 587m | 92.7 | 99.2+.1 | 99.7 | 80.7 | 95.4 | 98.0

Figure 5: Visualization of pruning results with respect to two ViT depths: 2
and 10. Note that the effective vision tokens are gradually decreased by our
method. We omit the merged tokens and show only the attention maps of the
remaining ones for a clear illustration.

###  4.4 Pre-training Scaling

Our method uses fewer vision tokens during training compared to baseline
models, allowing us to spare GPU memory and scale the model to larger batch
sizes. To study this effect, we carefully increased the pre-training batch
size while ensuring that the required GPU memory remained less than the
original pre-training. Besides, we also estimated the latency of each pre-
training epoch. We performed this test on the Flickr30K dataset and
illustrated the results in Table 5. Our observations for this result are
three-fold:

  * •

Our ELIP method is able to maintain performance similar to the base model,
while also accelerating the pre-training process and reducing the required GPU
memory usage.

  * •

The spared GPU memory enables us to scale the pre-training with larger batch
sizes, _i.e_. $\text{ELIP}_{{\color[rgb]{0,0,1}+}}$ approach. For example,
with METER, we increased the batch size from 36$\times$4 to 54$\times$4,
resulting in a significant improvement in training efficiency, and a reduction
in BLIP pre-training time by approximately 15%.

  * •

In terms of fine-tuning, our $\text{ELIP}_{{\color[rgb]{0,0,1}+}}$ surpasses
the ELIP by a large margin, and even slightly outperforms the base model in
some cases. These results are rather promising as scaling model pre-training
brings significant improvement in both downstream performance and efficiency.

###  4.5 Visualization

As illustrated in Sec. 3.2, our method consists of four blocks, wherein we
perform pruning and merging in the last three blocks. To quantitatively
demonstrate the effectiveness of our pruning approach, we randomly selected
two cases and presented them in Fig.5. In particular, we mainly show the
pruned attention map for two ViT layers: 2 and 10, and the effective vision
tokens are gradually reduced with deeper ViT layers. From this figure, we can
observe that our method progressively removes less important vision tokens
with deeper ViT layers. For example, in the first case, the model gradually
filters out the background information and focuses more on the _sheep_. A
similar observation can also be found in the second case where the _kite_
gains more attention in the 10$th$ ViT layer. More visualizations can be found
in the supplementary material.

##  5 Conclusion and Future Work

In this paper, we propose a novel approach to achieve efficient language-image
pre-training without introducing any additional trainable parameters. We
progressively prune and merge less influential vision tokens based on the
language output using multiple stages. Despite its simplicity, we show that
our approach helps remove 30% vision tokens whilst maintaining comparable
performance with backbones over diverse downstream fine-tuning tasks. Our
method offers valuable insights for future research in language-image pre-
training under limited computing resources, and may potentially benefit other
multi-modal pre-training tasks such as video-language pre-training.

While our method demonstrates effectiveness in efficiency and scalability, one
limitation is the lack of flexibility in the pruning ratio definition.
Therefore, an adaptive approach may be more helpful as different images often
exhibit varying degrees of information sparsity. In addition, our method can
be seamlessly integrated with other efficient techniques, such as mixed-
precision computation and checkpointing, and thus build an even more efficient
and lightweight language-image pre-training model.

## References

  * Alberti et al. [2019] Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter.  Fusion of detected objects in text for visual question answering.  In _Empirical Methods in Natural Language Processing_ , pages 2131–2140. ACL, 2019.
  * Anderson et al. [2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Computer Vision and Pattern Recognition_ , pages 6077–6086. IEEE, 2018.
  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh.  VQA: visual question answering.  In _International Conference on Computer Vision_ , pages 2425–2433. IEEE, 2015.
  * Bolya et al. [2022] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  _CoRR_ , abs/2210.09461, 2022.
  * Chang et al. [2023] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou.  Making vision transformers efficient from A token sparsification view.  _CoRR_ , 2023.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Computer Vision and Pattern Recognition_ , pages 3558–3568. IEEE, 2021.
  * Chen et al. [2021] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang.  Chasing sparsity in vision transformers: An end-to-end exploration.  In _Advances in Neural Information Processing Systems_ , pages 19974–19988, 2021.
  * Chen et al. [2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: universal image-text representation learning.  In _European Conference on Computer Vision_ , pages 104–120. Springer, 2020.
  * Courbariaux et al. [2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.  Binaryconnect: Training deep neural networks with binary weights during propagations.  In _Advances in Neural Information Processing Systems_ , pages 3123–3131, 2015.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics_ , pages 4171–4186. ACL, 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_. OpenReview.net, 2021.
  * Dou et al. [2022] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng.  An empirical study of training end-to-end vision-and-language transformers.  In _Computer Vision and Pattern Recognition_ , pages 18145–18155. IEEE, 2022.
  * Gan et al. [2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  In _Advances in Neural Information Processing Systems_ , 2020.
  * He et al. [2019] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.  Momentum contrast for unsupervised visual representation learning.  _CoRR_ , abs/1911.05722, 2019.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick.  Masked autoencoders are scalable vision learners.  In _Computer Vision and Pattern Recognition_ , pages 15979–15988. IEEE, 2022.
  * He et al. [2017] Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In _International Conference on Computer Vision_ , pages 1398–1406. IEEE, 2017.
  * Hinton et al. [2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  _CoRR_ , abs/1503.02531, 2015.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  _CoRR_ , abs/2203.15556, 2022.
  * Huang et al. [2021] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In _Computer Vision and Pattern Recognition_ , pages 12976–12985. IEEE, 2021.
  * Jelassi et al. [2022] Samy Jelassi, Michael Sander, and Yuanzhi Li.  Vision transformers provably learn spatial structure.  In _Advances in Neural Information Processing Systems_ , pages 37822–37836, 2022.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pages 709–727. Springer, 2022.
  * Jiang et al. [2022] Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, and Songfang Huang.  TRIPS: efficient vision-and-language pre-training with text-relevant image patch selection.  In _Empirical Methods in Natural Language Processing_ , pages 4084–4096. ACL, 2022.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _CoRR_ , 2020.
  * Kim et al. [2021] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR, 2021.
  * Kong et al. [2022] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, Minghai Qin, and Yanzhi Wang.  Spvit: Enabling faster vision transformers via latency-aware soft token pruning.  In _European Conference on Computer Vision_ , pages 620–640. Springer, 2022.
  * Krishna et al. [2017a] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123(1):32–73, 2017a.
  * Krishna et al. [2017b] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123(1):32–73, 2017b.
  * Li et al. [2022a] Feng Li, Hao Zhang, Yi-Fan Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, PengChuan Zhang, and Lei Zhang.  Vision-language intelligence: Tasks, representation learning, and large models.  _CoRR_ , abs/2203.01922, 2022a.
  * Li et al. [2021a] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In _Advances in Neural Information Processing Systems_ , pages 9694–9705, 2021a.
  * Li et al. [2022b] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.  BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022b.
  * Li et al. [2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _CoRR_ , abs/1908.03557, 2019.
  * Li et al. [2021b] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 2592–2607. ACL, 2021b.
  * Li et al. [2020] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer, 2020.
  * Li et al. [2021c] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  BRECQ: pushing the limit of post-training quantization by block reconstruction.  In _International Conference on Learning Representations_. OpenReview.net, 2021c.
  * Li et al. [2022c] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  _CoRR_ , abs/2212.00794, 2022c.
  * Liang et al. [2021] Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie.  Evit: Expediting vision transformers via token reorganizations.  In _International Conference on Learning Representations_ , 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In _European Conference on Computer Vision_ , pages 740–755. Springer, 2014.
  * Liu et al. [2021a] Xinyu Liu, Baopu Li, Zhen Chen, and Yixuan Yuan.  Exploring gradient flow based saliency for DNN model compression.  In _International Conference on Multimedia_ , pages 3238–3246. ACM, 2021a.
  * Liu et al. [2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _International Conference on Computer Vision_ , pages 9992–10002. IEEE, 2021b.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , pages 13–23, 2019.
  * Lu et al. [2020] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In _Computer Vision and Pattern Recognition_ , pages 10434–10443. IEEE, 2020.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.  Im2text: Describing images using 1 million captioned photographs.  In _Advances in Neural Information Processing Systems_ , pages 1143–1151, 2011.
  * Pan et al. [2021] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai.  Scalable vision transformers with hierarchical pooling.  In _International Conference on Computer Vision_ , pages 367–376. IEEE, 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108, 2019.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: open dataset of clip-filtered 400 million image-text pairs.  _CoRR_ , abs/2111.02114, 2021.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 2556–2565. ACL, 2018.
  * Su et al. [2020] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations_. OpenReview.net, 2020.
  * Sun et al. [2019a] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _International Conference on Computer Vision_ , pages 7463–7472. IEEE, 2019a.
  * Sun et al. [2019b] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.  Patient knowledge distillation for BERT model compression.  In _Empirical Methods in Natural Language Processing_ , pages 4322–4331. ACL, 2019b.
  * Sung et al. [2022] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  VL-ADAPTER: parameter-efficient transfer learning for vision-and-language tasks.  In _Computer Vision and Pattern Recognition_ , pages 5217–5227. IEEE, 2022.
  * Tai et al. [2016] Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E.  Convolutional neural networks with low-rank regularization.  In _International Conference on Learning Representations_ , 2016.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  LXMERT: learning cross-modality encoder representations from transformers.  In _Empirical Methods in Natural Language Processing_ , pages 5099–5110. ACL, 2019.
  * Tang et al. [2022] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao.  Patch slimming for efficient vision transformers.  In _Computer Vision and Pattern Recognition_ , pages 12155–12164. IEEE, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008, 2017.
  * Wang et al. [2022] Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu.  Recent advances on neural network pruning at initialization.  In _Thirty-First International Joint Conference on Artificial Intelligence_ , pages 5638–5645. ijcai.org, 2022.
  * Weerakoon et al. [2022] Dulanga Weerakoon, Vigneshwaran Subbaraju, Tuan Tran, and Archan Misra.  Softskip: Empowering multi-modal dynamic pruning for single-stage referring comprehension.  In _International Conference on Multimedia_ , pages 3608–3616. ACM, 2022.
  * Xu et al. [2021] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  In _Empirical Methods in Natural Language Processing_ , pages 9514–9528. ACL, 2021.
  * Yu et al. [2022a] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui.  Width & depth pruning for vision transformers.  In _AAAI Conference on Artificial Intelligence_ , pages 3143–3151. AAAI Press, 2022a.
  * Yu et al. [2022b] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In _Computer Vision and Pattern Recognition_ , pages 10809–10819. IEEE, 2022b.