* Topping et al. (2022b) Topping M. W., Stark D. P., Endsley R., Plat A., Whitler L., Chen Z., Charlot S., 2022b, http://dx.doi.org/10.3847/1538-4357/aca522 , https://ui.adsabs.harvard.edu/abs/2022ApJ…941..153T 941, 153
  * Tornatore et al. (2007a) Tornatore L., Ferrara A., Schneider R., 2007a, http://dx.doi.org/10.1111/j.1365-2966.2007.12215.x , https://ui.adsabs.harvard.edu/abs/2007MNRAS.382..945T 382, 945
  * Tornatore et al. (2007b) Tornatore L., Borgani S., Dolag K., Matteucci F., 2007b, http://dx.doi.org/10.1111/j.1365-2966.2007.12070.x , https://ui.adsabs.harvard.edu/abs/2007MNRAS.382.1050T 382, 1050
  * Treu et al. (2017) Treu T. L., et al., 2017, Through the Looking GLASS: A JWST Exploration of Galaxy Formation and Evolution from Cosmic Dawn to Present Day, JWST Proposal ID 1324. Cycle 0 Early Release Science
  * Treu et al. (2022) Treu T., et al., 2022, http://dx.doi.org/10.3847/1538-4357/ac8158 , https://ui.adsabs.harvard.edu/abs/2022ApJ…935..110T 935, 110
  * Trinca et al. (2022) Trinca A., Schneider R., Valiante R., Graziani L., Zappacosta L., Shankar F., 2022, http://dx.doi.org/10.1093/mnras/stac062 , https://ui.adsabs.harvard.edu/abs/2022MNRAS.511..616T 511, 616
  * Trussler et al. (2022a) Trussler J. A. A., et al., 2022a, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2022arXiv220714265T p. arXiv:2207.14265
  * Trussler et al. (2022b) Trussler J. A. A., Conselice C. J., Adams N. J., Maiolino R., Nakajima K., Zackrisson E., Ferreira L., 2022b, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2022arXiv221102038T p. arXiv:2211.02038
  * Tsuribe & Omukai (2006) Tsuribe T., Omukai K., 2006, http://dx.doi.org/10.1086/504290 , https://ui.adsabs.harvard.edu/abs/2006ApJ…642L..61T 642, L61
  * Vanzella et al. (2020) Vanzella E., et al., 2020, http://dx.doi.org/10.1093/mnrasl/slaa041 , https://ui.adsabs.harvard.edu/abs/2020MNRAS.494L..81V 494, L81
  * Vanzella et al. (2023) Vanzella E., et al., 2023, http://dx.doi.org/10.3847/1538-4357/acb59a , https://ui.adsabs.harvard.edu/abs/2023ApJ…945…53V 945, 53
  * Ventura et al. (2023) Ventura E. M., Trinca A., Schneider R., Graziani L., Valiante R., Wyithe J. S. B., 2023, http://dx.doi.org/10.1093/mnras/stad237 , https://ui.adsabs.harvard.edu/abs/2023MNRAS.520.3609V 520, 3609
  * Visbal et al. (2012) Visbal E., Barkana R., Fialkov A., Tseliakhovich D., Hirata C. M., 2012, http://dx.doi.org/10.1038/nature11177 , https://ui.adsabs.harvard.edu/abs/2012Natur.487…70V 487, 70
  * Visbal et al. (2014) Visbal E., Haiman Z., Terrazas B., Bryan G. L., Barkana R., 2014, http://dx.doi.org/10.1093/mnras/stu1710 , https://ui.adsabs.harvard.edu/abs/2014MNRAS.445..107V 445, 107
  * Visbal et al. (2020) Visbal E., Bryan G. L., Haiman Z., 2020, http://dx.doi.org/10.3847/1538-4357/ab994e , https://ui.adsabs.harvard.edu/abs/2020ApJ…897…95V 897, 95
  * Wang et al. (2022) Wang X., et al., 2022, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2022arXiv221204476W p. arXiv:2212.04476
  * Weingartner & Draine (2001) Weingartner J. C., Draine B. T., 2001, http://dx.doi.org/10.1086/320852 , https://ui.adsabs.harvard.edu/abs/2001ApJS..134..263W 134, 263
  * Welch et al. (2022) Welch B., et al., 2022, http://dx.doi.org/10.1038/s41586-022-04449-y , https://ui.adsabs.harvard.edu/abs/2022Natur.603..815W 603, 815
  * Wise et al. (2012) Wise J. H., Abel T., Turk M. J., Norman M. L., Smith B. D., 2012, http://dx.doi.org/10.1111/j.1365-2966.2012.21809.x , https://ui.adsabs.harvard.edu/abs/2012MNRAS.427..311W 427, 311
  * Wise et al. (2014) Wise J. H., Demchenko V. G., Halicek M. T., Norman M. L., Turk M. J., Abel T., Smith B. D., 2014, http://dx.doi.org/10.1093/mnras/stu979 , https://ui.adsabs.harvard.edu/abs/2014MNRAS.442.2560W 442, 2560
  * Wolcott-Green et al. (2011) Wolcott-Green J., Haiman Z., Bryan G. L., 2011, http://dx.doi.org/10.1111/j.1365-2966.2011.19538.x , https://ui.adsabs.harvard.edu/abs/2011MNRAS.418..838W 418, 838
  * Woosley & Weaver (1995) Woosley S. E., Weaver T. A., 1995, http://dx.doi.org/10.1086/192237 , https://ui.adsabs.harvard.edu/abs/1995ApJS..101..181W 101, 181
  * Xu et al. (2013) Xu H., Wise J. H., Norman M. L., 2013, http://dx.doi.org/10.1088/0004-637X/773/2/83 , https://ui.adsabs.harvard.edu/abs/2013ApJ…773…83X 773, 83
  * Xu et al. (2016a) Xu H., Norman M. L., O’Shea B. W., Wise J. H., 2016a, http://dx.doi.org/10.3847/0004-637X/823/2/140 , https://ui.adsabs.harvard.edu/abs/2016ApJ…823..140X 823, 140
  * Xu et al. (2016b) Xu H., Ahn K., Norman M. L., Wise J. H., O’Shea B. W., 2016b, http://dx.doi.org/10.3847/2041-8205/832/1/L5 , https://ui.adsabs.harvard.edu/abs/2016ApJ…832L…5X 832, L5
  * Yoshida et al. (2003) Yoshida N., Abel T., Hernquist L., Sugiyama N., 2003, http://dx.doi.org/10.1086/375810 , https://ui.adsabs.harvard.edu/abs/2003ApJ…592..645Y 592, 645
  * Yoshida et al. (2008) Yoshida N., Omukai K., Hernquist L., 2008, http://dx.doi.org/10.1126/science.1160259 Science, https://ui.adsabs.harvard.edu/abs/2008Sci…321..669Y 321, 669
  * Yoshii et al. (2022) Yoshii Y., Sameshima H., Tsujimoto T., Shigeyama T., Beers T. C., Peterson B. A., 2022, http://dx.doi.org/10.3847/1538-4357/ac8163 , https://ui.adsabs.harvard.edu/abs/2022ApJ…937…61Y 937, 61
  * Zhukovska et al. (2008) Zhukovska S., Gail H. P., Trieloff M., 2008, http://dx.doi.org/10.1051/0004-6361:2007778910.48550/arXiv.0706.1155 , https://ui.adsabs.harvard.edu/abs/2008A&A…479..453Z 479, 453
  * Zubko et al. (2004) Zubko V., Dwek E., Arendt R. G., 2004, http://dx.doi.org/10.1086/382351 , https://ui.adsabs.harvard.edu/abs/2004ApJS..152..211Z 152, 211
  * de Bennassuti et al. (2014) de Bennassuti M., Schneider R., Valiante R., Salvadori S., 2014, http://dx.doi.org/10.1093/mnras/stu1962 , https://ui.adsabs.harvard.edu/abs/2014MNRAS.445.3039D 445, 3039
  * de Bennassuti et al. (2017) de Bennassuti M., Salvadori S., Schneider R., Valiante R., Omukai K., 2017, http://dx.doi.org/10.1093/mnras/stw2687 , https://ui.adsabs.harvard.edu/abs/2017MNRAS.465..926D 465, 926
  * van den Hoek & Groenewegen (1997) van den Hoek L. B., Groenewegen M. A. T., 1997, http://dx.doi.org/10.1051/aas:1997162 , https://ui.adsabs.harvard.edu/abs/1997A&AS..123..305V 123, 305

##  Appendix A SFRD comparison with small-scale simulations

Here we compare our predictions for the SFRD to a wider set of simulations
collected in Table 4. Data is sorted by adopted box side length in order to
separate simulations performed on more than ten cMpc scales from smaller
boxes. For each study, we provide in Table 4 the reference paper, code name,
adopted numerical scheme212121For hydrodynamical simulations, these are either
particle-based (SPH, e.g. Springel 2005; Springel et al. 2021), grid-based
Adaptive Mesh Refinement (AMR, e.g. Rosdahl et al. 2013) or Meshless Finite
Mass (MFM, e.g. Hopkins et al. 2018) schemes. Two Semi-Analytical Models (SAM,
i.e. Trinca et al. in prep., Visbal et al. 2020) are also included in the
list., box side length and mass of DM particles. Figure 13 compares the
predicted $\Psi_{\rm III}(z)$ resulting from small-scale/high-mass-resolution
simulations ($L\lesssim 4h^{-1}$ cMpc, see second group in Table 4); a
comparison with large-scale simulations ($L\geq 10h^{-1}$ cMpc, first group in
Table 4) has been already presented in Figure 2.

The summary in Table 4 shows that recent studies have adopted different
numerical strategies and physical assumptions, and explore Pop III formation
at different scales. Predictions of $\Psi_{\rm III}(z)$ at scales $L\geq
10h^{-1}$ cMpc (i.e. more similar to the one of dustyGadget) are shown in Fig.
2. These correspond to the results by Pallottini et al. (2014), Sarmento et
al. (2018) and by the semi-analytical model CAT (Trinca et al. in prep.); even
though the last model does not have a proper definition of simulated volume,
it spans a large number of Pop III star-forming environments, representative
of a large-scale cosmological volume. Also note that the quoted $\geq
10h^{-1}$ cMpc simulations differ by a factor $>100$ in the simulated volumes,
and by a remarkably different DM particle mass.

In terms of radiative feedback, all large-scale models adopt a global ionizing
UV background. The semi-analytic code CAT also accounts for feedback from a LW
background. Apart from studies based on GIZMO (Jaacks et al., 2019; Liu &
Bromm, 2020b) and the FiBY simulations (Johnson et al., 2013), which implement
a small-scale LW and UV background either with or without a semi-analytic
treatment of local sources and shielding, all the other small-scale
hydrodynamical simulations adopt a combination of a background with proper RT
from stellar populations, at least for photons in the LW/UV bands222222Maio et
al. (2016), for example, follows local radiation from 150 frequency bins for
photons originated by stars according to Pop III and Pop II Spectral Energy
Distributions (SED).. All hydrodynamical simulations but Pallottini et al.
(2014) implement primordial chemistry. With the exception of simulations based
on the ENZO code (explicitly including an evaluation of the H2 fraction in
their star formation recipe), all models adopt an H-based star formation
recipe.

Large differences are also found in the implementation of chemical feedback:
Johnson et al. (2013), Maio et al. (2016), dustyGadget simulations and CAT
semi-analytic model create metals from stars by adopting mass and metallicity-
dependent yields and follow single atomic species during metal pollution and
spreading; RAMSES, ENZO and GIZMO-based methods derive instead the metal mass
through an average yield. Moreover, apart from dustyGadget and CAT, no models
account for cosmic dust.

Figure 13: Average Pop III SFRD $\Psi_{\mathrm{III}}$ (computed as in the left panel of Figure 1) compared with the results of other small-scale (box sizes $\lesssim 4h^{-1}~{}$\mathrm{c}\mathrm{M}\mathrm{p}\mathrm{c}$$) models and simulations, i.e. Johnson et al. (2013) (dark-blue, dotted line), Maio et al. (2016) (light-blue, dashed-dotted line), Xu et al. (2016b) (gold lines \- the “Normal”, “Void” and “Rarepeak” simulations are shown respectively in solid, thin-dotted and thin-dashed linestyle), Jaacks et al. (2019) (purple, dashed line), Liu & Bromm (2020b) (hotpink, dotted line), Skinner & Wise (2020) (orange, dashed-dotted line), Visbal et al. 2020 (green, solid line) and Sarmento & Scannapieco (2022) (brown, dashed lines). See Table 4 for a compilation of the models used as a comparison here and in Figure 2, with their main features. Table 4: Independent theoretical models accounting for Pop III star formation. Table columns show: model reference, code name, type of adopted numerical scheme, box size $L$ $[$\mathrm{c}\mathrm{M}\mathrm{p}\mathrm{c}$/h]$, adopted DM particle mass $m_{\mathrm{DM}}$ $[10^{5}~{}$\mathrm{M}_{\odot}$/h]$. All the listed theoretical models are compared in Figures 2 and 13. See Figure 13 for references. | Code | Scheme | $L$ | $m_{\mathrm{DM}}$
---|---|---|---|---
This work | dustyGadget | SPH | 50 | 353.00
Sarmento+18 | RAMSES | AMR | 12 | 0.99
Pallottini+14 | RAMSES | AMR | 10 | 4.70
Trinca+23 | CAT | SAM | - | -
Xu+16 | ENZO | AMR | 4.3 | 0.21
Jaacks+19 | GIZMO | MFM | 4 | 0.29
Liu+20 | GIZMO | MFM | 4 | 0.36
Sarmento+22 | RAMSES-RT | AMR | 3 | 0.12
Johnson+13 | Gadget-2 | SPH | 2.84 | 0.04
Visbal+20 | - | SAM | 2.01 | 0.05
Skinner+20 | ENZO | AMR | 0.67 | 0.01
Maio+16 | Gadget-3 | SPH | 0.5 | 0.04

A comparison of our predictions with small-scale/high-mass-resolution
simulations ($L\lesssim 4$ cMpc, Figure 13) provides interesting clues on the
contribution of well resolved populations of star-forming mini-halos and
proper radiative feedback. Pop III star formation in small structures follows
a quasi-flat evolution in the redshift range $13\leq z\leq 19$, with values
differing by more than one order of magnitude across model predictions. At
$z\leq 13$, most of the models predict a flattening or a gentle decline of Pop
III star formation, and the different behaviour might be ascribed to
differences in the implemented hydrodynamical and/or RT scheme, as well as
metal mixing/spreading. Jaacks et al. (2019) and Sarmento & Scannapieco (2022)
are notable exceptions to these trends. Despite the relatively small simulated
volume, Jaacks et al. (2019) find a Pop III SFRD higher than our predictions
down to $z\sim 7$, probably because of the relatively inefficient
radiative/mechanical feedback model implemented in their simulation232323These
aspects were extensively commented in the original paper, and were addressed
in later simulations by Liu & Bromm (2020a, b), where they reduced the
efficiency of star formation, added SN-driven winds from Pop II, and enhanced
radiative feedback through a local LW contribution from stellar sources.
Indeed, $\Psi_{\rm III}(z)$ predicted by Liu & Bromm (2020b) appears
substantially reduced, and closer to other small-scale model predictions..
Conversely, Sarmento & Scannapieco (2022) find a trend similar to our own down
to $z\sim 11-12$, with a higher level of flattening at lower redshifts and a
gentle decline below $z\sim 8$, although they seem to be finding slightly
lower values of the Pop III SFRD than our model at all redshifts.

A good level of agreement, especially at $z\sim 10-14$, is found between our
simulations and e.g. the FiBY simulations (Johnson et al., 2013),
notwithstanding the absence of LW feedback in dustyGadget. This can be
understood by reminding that LW radiation has an effect on H2 star-forming
environments (i.e. mini-halos), unresolved by mass in our simulations. As a
result, dustyGadget finds that below $z\simeq 14-15$, our predictions of
$\Psi_{\rm III}$ are not dramatically different from those of models
accounting for LW-suppressed star formation in mini-halos. However, we
emphasize again that this also leads to an underestimation of Pop III stars at
very high redshifts ($z\gtrsim 14-15$), where the LW is not yet very
efficient, and mini-halos provide the dominant contribution to Pop III star
formation.

The study of Xu et al. (2016b) deserves a particular mention because it
provides an interesting comparison between different environments selected
from the Reinassance simulation. The authors adopt a zoom-in technique to re-
simulate small regions of interest ($\sim
4.3\;$\mathrm{c}\mathrm{M}\mathrm{p}\mathrm{c}$h^{-1}$) with increased
resolution from a bigger, low-resolution cube of $\sim
28\;$\mathrm{c}\mathrm{M}\mathrm{p}\mathrm{c}$h^{-1}$. The chosen sub-regions
exhibit different values of the density contrast, and are described as
“Normal” ($\langle\delta\rangle\simeq 0.09$), “Void”
($\langle\delta\rangle\simeq-0.26$) and “Rarepeak”
($\langle\delta\rangle\simeq 0.68$). Their $\Psi_{\rm III}(z)$ are shown in
the right panel of Figure 13 as gold lines, to visualize the expected spread
among different environments. At $11\lesssim z\lesssim 15$, dustyGadget
predictions are consistent with the results of the “Normal” region (gold,
solid line).

The large differences in both the amplitude and redshift evolution of
$\Psi_{\rm III}(z)$ found by small-scale simulations indicate that (i) the
estimates of $\Psi_{\rm III}(z)$ are heavily impacted by the bias introduced
by the specific scale, (ii) the capability to properly resolve star-forming
mini-halos is required to correctly estimate $\Psi_{\rm III}(z)$ at $z\geq
13$, and (iii) the implementation of detailed radiative/chemical feedback is
crucial to determine the late-time evolution of Pop III star-forming
environments and the Pop III/II transition. Models differing in both chemical
and RT scheme can even predict a sudden and fast suppression of Pop III star
formation at $z\sim 9$, with extremely low values ($\Psi_{\rm III}\leq
10^{-6.2}$ in $9<z<13$), despite their mass resolution (Maio et al., 2016).

Local radiative feedback, in particular, may have strong implications for gas
evolution in star-forming regions. The first three-dimensional radiative
simulations studying pristine-gas collapse and the role of UV radiation have
been presented in Machacek et al. (2001), Yoshida et al. (2003), O’Shea &
Norman (2004), O’Shea & Norman (2008), Wise et al. (2012), Wise et al. (2014)
and Regan et al. (2016); Regan et al. (2017). They show that, depending on the
radiative prescriptions adopted, different outcomes for gas collapse are
possible and feedback effects are dominant actors during structure formation.
Photo-ionization feedback is also discussed in Kannan et al. (2014), while LW
radiation, which is crucial for star formation quenching and black-hole birth,
is instead discussed by e.g. Habouzit et al. (2016), Maio et al. (2019) and
Latif et al. (2021). In general, their findings suggest that feedback
prescriptions are among the main causes of differences in the final results
and the formation of Pop III stars is tightly linked or even alternative to
the birth of massive black holes. In practice, strong radiative fields can
locally heat the medium, dissociate $\mathrm{H}_{2}$ and inhibit cooling and
star formation. This process, when happening in unpolluted gas, could lead to
the formation of a direct-collapse black hole instead of a Pop III star.

We note that the chosen Pop III IMF and critical metallicity
$Z_{\mathrm{crit}}$ can also play a role on the Pop III/II transition (see
e.g. the discussion in Maio et al. 2010). In the present work, we do not
explore the impact of a different Pop III IMFs and/or the critical metallicity
on our results, but note that different assumptions have been made by all the
models collected here. Pallottini et al. (2014) and Maio et al. (2016)
provided results using different IMFs: for Pallottini et al. (2014), we only
display their fiducial model, i.e. a Larson-Salpeter IMF (Larson, 1998) in the
range [0.1, 100] $\mathrm{M}_{\odot}$ (“SALP” case); for Maio et al. (2016),
we show the “TH-1e5K” case, i.e. a top-heavy IMF over [100, 500]
$\mathrm{M}_{\odot}$ with slope -2.35, same as ours. Note that while the
TH-1e5K case in Maio et al. (2016) was found to differ by up to 2 dex from the
SL-1e4K case (i.e. a Salpeter IMF over the range [0.1, 100]
$\mathrm{M}_{\odot}$)242424The different behaviour might be ascribed both to
different lifetimes and different SN/radiative feedback resulting from Pop III
stars with different masses. For example, a more powerful spectral energy
distribution (i.e. a black body with effective temperature
$T_{\mathrm{eff}}\sim 10^{5}$ K) is assumed in Maio et al. (2016) for Pop III
sources with a top-heavy IMF, while more typical $T_{\mathrm{eff}}\sim
10^{4}-4\times 10^{4}$ K sources were considered for the standard-Salpeter
case., rather surprisingly, Pallottini et al. (2014) found essentially no
difference between their fiducial model and the model assuming a top-heavy IMF
in the range [100, 500] $\mathrm{M}_{\odot}$ (“PISN” case). As for the
critical metallicity, most models have assumed
$Z_{\mathrm{crit}}=10^{-4}~{}$\mathrm{Z}_{\odot}$$ (similarly to us), although
a lower metallicity threshold is assumed e.g. in Sarmento et al. (2018) and
Sarmento & Scannapieco (2022)
($Z_{\mathrm{crit}}=10^{-5}~{}$\mathrm{Z}_{\odot}$$), and in Skinner & Wise
(2020) ($Z_{\mathrm{crit}}=5\times 10^{-6}~{}$\mathrm{Z}_{\odot}$$).


These authors contributed equally to this work.

These authors contributed equally to this work. Haiyang Xu${}^{\text{2}*}$
Shikun Zhang${}^{\text{1}*}$

[1]National Engineering Research Center for Software Engineering, Peking
University, Beijing, China

2]DAMO Academy, Alibaba Group, Beijing, China

# Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch
Selection

Wei Ye${}^{\text{1}}$<EMAIL_ADDRESS>Chaoya Jiang${}^{\text{1}}$
<EMAIL_ADDRESS><EMAIL_ADDRESS>Qinghao Ye${}^{\text{2}}$
<EMAIL_ADDRESS>Chenliang Li${}^{\text{2}}$ lcl193798@alibaba-
inc.com Ming Yan${}^{\text{2}}$<EMAIL_ADDRESS><EMAIL_ADDRESS>Songfang Huang${}^{\text{2}}$<EMAIL_ADDRESS>Fei
Huang${}^{\text{2}}$<EMAIL_ADDRESS>*  [

###### Abstract

Vision Transformers (ViTs) have become increasingly popular in large-scale
Vision and Language Pre-training (VLP) models. Although previous VLP research
has demonstrated the efficacy of ViTs, these efforts still struggle with
computational inefficiencies caused by lengthy visual sequences. To address
this challenge, we introduce an efficient VLP approach called TRIPS, which
stands for Text-Relevant Image Patch Selection. TRIPS progressively reduces
the visual sequence using a text-guided patch-selection layer in the visual
backbone, thereby accelerating both training and inference processes. This
patch-selection layer dynamically computes text-dependent visual attention,
enabling it to identify attentive image tokens with text guidance and fuse
inattentive ones in an end-to-end fashion. Importantly, TRIPS does not add any
extra parameters and generalizes to most ViT-based VLP models. We incorporate
TRIPS into three representative VLP models covering single-stream, dual-
stream, and generative paradigms, and conduct extensive experiments on five
widely-used multi-modal benchmark datasets. Our experimental results reveal
that TRIPS delivers a 40% speedup, while maintaining competitive or superior
performance on downstream tasks.

###### keywords:

Vision-Language pre-training, Patch Selection, Efficiency

##  1 Introduction

Figure 1:  Sub-figure (a) presents the VQA cases for ALBEF [24] fine-tuned on
the VQA task, where input image tokens are directly selected based on the
attention weight of the image [CLS] token in relation to other image tokens.
We visualize the attention distribution of the image [CLS] token, which, as
seen, naturally concentrates on objects within the images while disregarding
the backgrounds. If the question pertains to the objects in the images,
accurate predictions can be obtained. Sub-figure (b) compares VQA predictions
between ALBEF, which directly selects image tokens guided by the image [CLS]
token, and our model, TRIPS-ALBEF. As illustrated, when questions relate to
image backgrounds, the former produces incorrect answers, whereas the latter
provides the right responses by preserving text-relevant image tokens.

In recent years, Vision-Language Pre-training (VLP) [45, 5, 32, 17, 27, 6, 60,
24, 57] has experienced remarkable growth, emerging as a dominant paradigm for
addressing Vision-Language (VL) tasks. Conventional VLP models [45, 5, 32, 27]
employ pre-trained object detectors [38, 37, 15] to extract region-based image
features but suffer from extensive annotation and expensive computation of
object detector training. Inspired by the success of the Vision Transformer
(ViT) [10] and its variants [30, 53, 51] in the computer vision domain, more
recent VLP models [24, 35, 20, 50, 42] have adopted ViT as the visual encoder
or cross-modal fusion encoder without using region features from the pre-
trained object detector.

However, ViT-based VLP methods necessitate the modeling of lengthy visual
sequences, e.g., from high-resolution images, to achieve robust vision
understanding, which results in quadratic computational complexity
proportional to the visual sequence length. Furthermore, recent studies [10,
46] have started to investigate vision-language foundation models, focusing on
scaling up both model and data size. Consequently, there is an increasing need
to reduce the substantial computational costs associated with ViT-based VLP
models. As demonstrated in Figure 1 (a), we observe that eliminating the
inattentive patch tokens of the image [CLS] token generally does not impact
the Visual Question Answering (VQA) [1] predictions. Based on this
observation, we hypothesize that not all image tokens in the visual encoder
positively contribute to the final prediction results of VLP models, and a
considerable number of redundant image tokens exist.

Recent studies [36, 55, 61, 28] have explored ViT model acceleration by
eliminating unrelated image tokens. However, these methods are specifically
tailored for computer vision tasks (e.g., image recognition) and remove
redundant tokens based on visual semantics while disregarding the aligned
knowledge in the text modality. Consequently, they are not suitable for VL
tasks. As illustrated in Figure 1 (b), removing patch tokens based solely on
the image [CLS] token, without the guidance of text knowledge, leads to
incorrect answers. This observation inspires us to reduce the number of image
tokens by merging less informative patch tokens under the guidance of the
aligned text context.

In this work, we introduce an efficient VLP model featuring Text-Relevant
Image Patch Selection (TRIPS) to progressively reduce redundant image tokens
with the guidance of text. TRIPS selects text-consistent image tokens through
a text-aware patch-selection layer, thereby minimizing the computational cost
of visual encoding and cross-modal fusion. This patch-selection layer
preserves attentive image tokens with text guidance and fuses inattentive
tokens into a single one by dynamically computing text-dependent visual
attention in an end-to-end manner. Consequently, we gradually decrease the
number of image tokens as the visual backbone deepens, alleviating the
computational burden of the visual encoder and enhancing the efficiency of
cross-modal fusion due to the reduced visual sequences. Additionally, model
efficiency can be flexibly controlled by adjusting the keep rate of image
tokens in the patch-selection layer without introducing any additional
parameters.

We evaluate TRIPS on five representative VL tasks consisting of visual
question answering (VQA), natural language visual reasoning (NLVR), cross-
modal retrieval, image captioning, and visual grounding. By incorporating
TRIPS into existing representative VLP models, we achieve approximately 40%
efficiency gains while maintaining competitive or superior downstream task
performance. For example, equipped with TRIPS, ALBEF [24], a recent robust
two-stream VLP model, can accelerate by 40.98% and even improve by 0.1 on the
VQA test-dev and 0.2 on the NLVR Dev (see Table 3). Moreover, by increasing
the input image resolution while maintaining the same computational cost,
TRIPS can enhance performance by 0.4 on the VQA test-dev and 0.6 on the NLVR
Dev.

Our main contributions can be summarized as three-fold:

  * •

We introduce an efficient vision-and-language pre-training model featuring
Text-Relevant Image Patch Selection (TRIPS). To the best of our knowledge,
this is the first attempt to reduce the computational cost of VLP models by
diminishing image tokens with the assistance of linguistic context.

  * •

We propose a novel text-relevant patch-selection layer, which can dynamically
compute text-dependent visual attention to identify attentive (or critical)
image tokens and merge inattentive ones with text guidance in an end-to-end
manner.

  * •

Comprehensive experiments demonstrate that our TRIPS can enhance the training
and inference efficiency of VLP models while incurring lower computational
costs. Moreover, by increasing the input image resolution, TRIPS leverages
additional image tokens to achieve superior performance without increasing
computational expenses.

##  2 Related Work

###  2.1 Vision-Language Pre-training

Current approaches to VLP can be broadly divided into two categories in terms
of visual representation extraction. The first category is detector-based VLP
methods [32, 26, 45, 27, 6, 57]. These methods primarily adopt a two-step
training pipeline: they extract visual features using a pre-trained object
detector and then train the cross-modal pre-training model to align text and
visual features. Some region-based methods aim to reduce computational costs
with lightweight model architectures [48, 13] or knowledge distillation [12].
However, these methods still face several challenges, including expensive
computation and time consumption for object/region detection and error
propagation problems caused by the two-step pre-training strategy. The main
challenge for these methods is to balance effectiveness and efficiency. The
second category consists of more recent CNN-based [17, 54] or ViTs-based [24,
20, 35, 50] methods, especially patch-based ViT. These methods eliminate the
need for a complex object detector in feature extraction, enabling end-to-end
VL learning. However, few works have focused on reducing the high
computational cost of ViT-based VLP models.

In terms of fusion schemes for modeling cross-modal interaction, typical VLP
approaches [45, 24, 27, 52, 20, 25, 50] can also be categorized into dual
stream and single stream. Single-stream approaches [27, 20, 52, 50] assume
that the potential correlation and alignment between the two modalities are
relatively straightforward and can be learned by a single transformer encoder.
As a result, the architecture concatenates text embeddings and image features,
along with special embeddings that indicate their respective positions and
modalities. This concatenated feature set is then fed into a transformer-based
encoder for further processing. Dual-stream approaches [45, 23, 25, 11, 54]
assume that the intra-modal interaction and cross-modal interaction need to be
separated to obtain better multimodal representations. Thus, they employ two
single-modal encoders to separately encode images and text. Additionally,
previous VLP methods [26, 5, 43, 50] were typically only capable of performing
tasks related to vision-language understanding and reasoning (e.g., VQA and
NLVR). However, recent multimodal generation models [25, 23, 52] have started
to adopt encoder-decoder style generative approaches for accomplishing tasks
associated with multimodal text generation. Our method can easily generalize
to most ViT-based VLP models, including single-stream, dual-stream, and
generative approaches.

###  2.2 ViTs Acceleration

Numerous studies have focused on proposing more efficient attention mechanisms
[49, 21, 7] or compressing Transformer structures [30, 16, 51] to accelerate
the computation of transformer-based models [47]. Recently, some approaches
have aimed to accelerate ViTs by reducing the number of tokens involved in ViT
inference. For instance, [40] proposed TokenLearner to expedite ViTs, in which
a relatively small number of tokens are learned by aggregating the entire
feature map weighted by dynamic attention. [36] introduces a method to reduce
tokens for a fully trained ViT, where an extra learnable neural network is
added to ViT to select a subset of tokens. [28] proposes to reduce the
computational overhead of inference by introducing a token reorganization
method to progressively reduce and reorganize image tokens. However, these
methods are not suitable for VLP, as they reduce image tokens without
considering the text context. In contrast, our proposed approach takes into
account the textual context, making it more appropriate for VLP tasks.

Figure 2: Sub-figure (a) showcases the overall architecture of the VLP model
(TRIPS-ALBEF) presented in this paper. Sub-figure (b) provides a visual
overview of the ViT augmented with a Text-Relevant Image Patch Selection
module (ViT-TRIPS). We assume the ViT-TRIPS comprises 12 layers, and the 5th
and 10th layers serve as the patch selection layers. Sub-figure (c) depicts
the design of the Text-Relevant Image Patch Selection layer.

##  3 Method

Our method aims to generalize to most ViT-based VLP models covering single-
stream, dual-stream, and generative VLP paradigms. In this section, we mainly
use the dual-stream ALBEF [24] framework as a representative base model to
introduce the technical design of our TRIPS (named TRIPS-ALBEF). We will first
present the model architecture, and then introduce the ViT-based visual
backbone with the acceleration module of the text-relevant patch-selection
layer. We will also briefly discuss how we incorporate TRIPS into a recent
generative VLP model mPLUG [23] and single-stream model ViLT [20]. Finally, we
will introduce the pre-training objectives.

###  3.1 Model Architecture

As shown in Figure 2 (a), TRIPS-ALBEF contains a visual encoder with the text-
relevant patch-selection layer, a text encoder, and a multimodal fusion
encoder. The visual encoder takes a Vision Transformer (ViT), where text-
relevant patch-selection layers are used to progressively reduce and
reorganize image tokens, namely ViT-TRIPS. The text encoder adopts BERTbase
transformer [9]. Similar to [24], the multimodal fusion encoder is a
transformer encoder that performs the cross-modal interaction and fusion
through a cross-attention mechanism.

Formally, given an input image-text pair, we first feed the input text to the
text encoder and represent it as a sequence of embeddings
$T=\\{t_{cls},t_{1},t_{2},\cdots,t_{m}\\}$, where $t_{cls}$ is the embedding
of the text [CLS] token to summarize the input text. Then, we divide the input
image into patches $P=\\{p_{cls},p_{1},p_{2},\cdots,p_{u}\\}$, and encode them
with the image encoder ViT-TRIPS. It takes the text [CLS] embedding $t_{cls}$
and image patches $\\{p_{cls},p_{1},p_{2},\cdots,p_{u}\\}$ as input, and
outputs the image sequence $V=\\{v_{cls},v_{1},v_{2},\cdots,v_{e}\\}$. Note
that $e<u$, since we apply the text-relevant patch-selection layer to select
the text-aware image tokens and fuse the redundant tokens, allowing us to
reduce the total visual sequence length for efficiency. Finally, the text
features $\\{t_{cls},t_{1},t_{2},\cdots,t_{m}\\}$ and the image features
$\\{v_{cls},v_{1},v_{2},\cdots,v_{e}\\}$ encoded by the image encoder are
fused by cross attention at each layer of the multimodal encoder as in ALBEF
[24]. The output of the multimodal encoder is used to pre-train and finetune
downstream tasks.

###  3.2 Text-Relevant Image Patch Selection

Existing works in computer vision [36, 28] select patches by using only the
image [CLS] token from the ViT backbone. However, as shown in Figure 1 (b),
the selection of image tokens in cross-modal tasks is closely related to
textual context, and different texts for a single image may focus on different
parts of the visual content. Selecting the image tokens with the guidance of
aligned textual content can help the VLP model focus on the key parts of the
image for more effective and efficient cross-modal fusion. Here, we present a
text-relevant patch-selection layer that can dynamically select the image
patches with the guidance of textual input, yet with no additional parameters
introduced.

As shown in Figure 2 (b), for a ViT with $L$ standard Transformer layers and
$t$ patch-selection layers in total, the interval length is obtained as
$s=L/(t+1)$. Then, we choose the layer index $j=i*s+1$ as the $i_{th}$ patch-
selection layer, so that patch-selection layers are uniformly inserted into
the ViT-TRIPS backbone. In each patch-selection layer, as shown in Figure
2(c), we adopt standard self-attention (SA), Text-aware Dynamic Attention (TD-
ATT), and Inattentive Token Fusion (ITF) modules to progressively reduce image
tokens.

Specifically, for the $i_{th}$ patch-selection layer, image features $v^{j-1}$
= $\\{v^{j-1}_{cls},$ $v^{j-1}_{1},\cdots,v^{j-1}_{n}\\}$ are first fed to the
$j_{th}$ visual self-attention layer:

$v^{j}=LN(SA(v^{j-1})+v^{j-1})$ (1)

where LN is short for layer normalization, and $n$ is the number of patch
tokens in the $j-1$ visual transformer layer. Next, we will illustrate how to
use the Text-aware Dynamic Attention mechanism (TD-ATT) to select the text-
aware image patch tokens. The text [CLS] embedding $t_{cls}$ is linearly
projected to the query vector denoted as $q_{text}$ by the shared query linear
layer of the $j_{th}$ visual self-attention layer. We compute the text-to-
image attention feature map excluding the image [CLS] token as follows:

$a_{cls}=softmax(\frac{q_{text}\cdot v^{j}[1:]^{T}}{\sqrt{d}})$ (2)

We identify and preserve the attentive image tokens corresponding to the $k$
largest elements in the attention map $a_{cls}=\\{a_{1},..a_{n}\\}$, where
$k=n\times r$, and $r$ is the keep rate of this layer. The selected image
tokens are kept, and the un-selected image tokens are further fused by an
inattentive token fusion operation ITF.

The remaining inattentive patch tokens
$\\{v_{z_{1}},v_{z_{2}},\cdots,v_{z_{n-k}}\\}$ are treated as text-irrelevant
tokens. However, the fixed keep rate may remove some useful tokens, so we fuse
inattentive tokens to one token $v_{f}$ by a weighted sum operation to
supplement attentive ones as follow:

$v_{f}=\sum\limits^{n-k}\limits_{i=1}a_{cls,z_{i}}\cdot\hat{v}_{z_{i}}$ (3)

After fusing the inattentive patch tokens, we reconstruct the $j_{th}$ visual
sequence as
$v^{j}=\left[v^{j}_{cls},v^{j}_{1},\cdots,v^{j}_{k},{v}^{j}_{f}\right]$, which
consists of the image [CLS] token embedding, the selected text-aware image
patch embedding, and fused inattentive patch embedding. Then the new visual
sequence is fed to the feed-forward network (FFN).

Figure 3: Sub-figure (a) depicts the ViT-TRIPS backbone of the single-stream
VLP model. Sub-figure (b) is the illustration of the Text-Relevant Image
Patch-Selection layer.

###  3.3 Extension to Generative VLP Models

Our method can be easily applied to generative vision-language models. We
extend TRIPS to the latest state-of-the-art dual-stream generative model mPLUG
[23], referred to as TRIPS-mPLUG. Specifically, we replace the ViT-based
visual backbone in mPLUG with ViT-TRIPS while keeping other settings
unchanged.

###  3.4 Extension to Single-stream VLP Models

The proposed Text-Relevant Image Patch Selection layer can also be extended to
single-stream models, which employ the [CLS] token of the multimodal encoder
to preserve attentive image tokens and fuse inattentive tokens to accelerate
training and inference. In this paper, we implement this model, denoted as
TRIPS-ViLT, based on the ViLT [20] framework, a widely used single-stream
model. As shown in Figure 3, the single-stream model has a global [CLS] token
for aggregating visual and linguistic information, allowing us to directly
select text-aware visual tokens and fuse unrelated image patch tokens based on
the attention weights of the [CLS] tokens to image patch tokens.

###  3.5 Pre-training Objectives

Table 1: Pre-training tasks of different TRIPS variants. Model | ITC | ITM | MLM | PrefixLM
---|---|---|---|---
TRIPS-ViLT | ✗ | ✓ | ✓ | ✗
TRIPS-ALBEF | ✓ | ✓ | ✓ | ✗
TRIPS-mPLUG | ✓ | ✓ | ✓ | ✓

We pre-train our models with the following standard objectives: Image-Text
Contrastive learning (ITC), Image-Text Matching (ITM), and Masked Language
Modeling (MLM). These pre-training tasks are optimized jointly. As shown in
Table 1, the three TRIPS variants differ in the combination of pre-training
tasks.

Image-Text Contrastive (ITC) For TRIPS-ALBEF and TRIPS-mPLUG, we follow [24]
and apply ITC to align the image representation and text representation from
the unimodal encoders.

Image-Text Matching (ITM) The goal of image-text matching is to predict
whether the input image and text are matched, which is a shared task among all
model variants. We follow the design of [24] and select hard negative image-
text pairs based on the contrastive text-image similarity. We take the text
[CLS] embedding of the multimodal encoder’s output as the joint
representation, followed by a Multi-Layer Perceptron (MLP) layer for
prediction.

Masked Language Modeling (MLM) The task setup is essentially the same as in
BERT [9], where we randomly mask 15% of tokens in text and the model is asked
to predict these masked words with the cross-modal representations.

Prefix Language Modeling (PrefixLM). This task is specific to mPLUG [23],
which aims to generate a caption given an image and predict the text segment
subsequent to the cross-modal context as in [3]. It optimizes a cross-entropy
loss by maximizing the likelihood of text in an autoregressive manner.

##  4 Experiment Settings

###  4.1 Implementation Details

We pre-train TRIPS-ALBEF and TRIPS-mPLUG for 30 epochs, and TRIPS-ViLT for 20
epochs with a total batch size of 512 on 8 NVIDIA V100 GPUs. We initialize the
visual encoder using CLIP (ViT-B/16) [35] pre-trained on 400M noisy image-text
pairs and employ the AdamW [31] optimizer with a weight decay of 1e-2. The
learning rate is warmed up to 1e-5 (ViT-B/16) and 1e-4 (BERT${base}$) in the
first 1000 iterations. During pre-training, we use input images with a
resolution of $256\times 256$ and increase the image resolution during fine-
tuning. For TRIPS-ALBEF and TRIPS-mPLUG, we utilize a 6-layer Transformer for
both the text encoder and the cross-modal fusion network. Following [24], we
initialize the text encoder with the first 6 layers of BERT${base}$ [9] and
the cross-modal network with the last 6 layers of BERTbase. For image-text
contrastive learning, we set the queue size to 65,536 and the momentum
coefficient to 0.995.

In the proposed TRIPS-ViT backbone, we designate the 5th and 10th layers as
patch-selection layers and set the keep rate for each layer to 70%, achieving
a balance between downstream task performance and model inference speed.
Details of fine-tuning hyperparameters can be found in Appendix B.

We use the AdamW optimizer [31] with a weight decay of 0.02 to train our
models. For TRIPS-ALBEF and TRIPS-mPLUG, we warm up the learning rate to 1e-5
(ViT-B/16) and 1e-4 (BERTbase) in the first 1000 iterations, then decay it to
1e-6 according to a cosine schedule. Pre-training TRIPS-ALBEF takes
approximately 60 hours, TRIPS-mPLUG takes around 80 hours, and TRIPS-ViLT
requires about 40 hours; all performed on 8 V100-32G GPUs using a 4M pre-
training dataset for approximately 20 epochs.

To enhance the generalization of vision encoders during pre-training, we apply
RandAugment [8] to random image crops of size 256 $\times$ 256\. In the fine-
tuning stage for VQA, image captioning, and visual grounding tasks, we
increase the image resolution. For image-text contrastive learning, we set the
queue size to 65,536 and the momentum coefficient to 0.995.

###  4.2 pre-training data

Table 2: Statistics of the pre-training datasets. | COCO | VG | SBU | CC3M
---|---|---|---|---
image | 113K | 100K | 860K | 3M
text | 567K | 769K | 860K | 3M

We construct our pre-training data using two web datasets (Conceptual Captions
[41], SBU Captions [33]) and two in-domain datasets (MSCOCO [29] and Visual
Genome [22]). The total number of unique images is 4.0M, and the number of
image-text pairs is 5.1M. Table 2 shows the statistics of the 4M images with
texts used in the pre-training stage.

##  5 Experiment Results

###  5.1 Overall Performance

We evaluate TRIPS-ALBEF, TRIPS-mPLUG, and TRIPS-ViLT on five vision-language
downstream tasks: visual question answering (VQA), natural language for visual
reasoning (NLVR), image-text retrieval, image captioning, and visual
grounding. Our baselines cover 15 VLP models, which is detailed in Section A.
We will first analyze their overall performances on these tasks.

####  5.1.1 Visual Question Answering

The VQA task [1] requires the model to answer natural language questions given
an image. Following [24], we treat VQA as an answer generation problem. We
report test-dev and test-std scores by submitting our results to the
evaluation server111https://eval.ai/web/challenges/challenge-
page/830/overview, as shown in Table 3. Looking into Table 3, VLP models
equipped with ViT-TRIPS improve performance on the VQA task (e.g., ViLT [20]
achieves 71.26 on Test-dev, while TRIPS-ViLT scores 71.48; ALBEF [24] reaches
76.12, while TRIPS-ALBEF attains 76.23). These results are impressive,
considering they are achieved with significantly reduced training and
inference costs (see more details of efficiency in Section 5.2).

Table 3: Evaluation Results on VQA and NLVR2. More details about comparison models are in Appendix A.  model |  # Pre-train Data | VQA | NLVR2
---|---|---|---
Test-dev | Test-std | Dev | Test-P
E2E-VLP | 4M | 73.25 | 73.67 | 77.25 | 77.96
OSCARBase | 6.5M | 73.16 | 73.44 | 78.07 | 78.36
VinVLLarge | 5.65M | 76.52 | 76.60 | 82.67 | 83.98
ViLBERT | 3.3M | 70.63 | 70.92 | - | -
VisualBERT | 180K | 70.80 | 71.00 | 67.40 | 67.00
LXMERT | 180K | 72.42 | 72.54 | 74.90 | 74.50
UNITERLarge | 4M | 73.82 | 74.02 | 79.12 | 79.98
METERCLIP | 4M | 77.68 | 77.64 | 82.33 | 83.05
BLIPBase | 14M | 77.54 | 77.62 | 82.67 | 82.30
VLMoLarge | 4M | 76.64 | 76.89 | 82.77 | 83.34
SimVLMBase | 1.8B | 77.87 | 78.14 | 81.72 | 81.77
ViLTBase | 4M | 71.26 | 71.29 | 75.18 | 76.2
TRIPS-ViLT | 4M | 71.48 | 71.52 | 75.89 | 76.4
ALBEFBase | 4M | 76.12 | 76.32 | 82.21 | 83.1
TRIPS-ALBEF | 4M | 76.23 | 76.48 | 82.35 | 83.34
mPLUGGBase | 4M | 77.80 | 77.98 | 83.76 | 83.92
TRIPS-mPLUG | 4M | 77.89 | 78.23 | 83.87 | 84.04

####  5.1.2 Natural Language for Visual Reasoning

The NLVR2 [44] task requires the model to predict whether a sentence
accurately describes a pair of images, which is a binary classification task.
For TRIPS-ALBEF and TRIPS-mPLUG, we follow [24] and use two cross-attention
layers to process the two input images; their outputs are merged and fed into
a Feed Forward Network (FFN). An MLP classifier is then applied to the output
embedding of the text [CLS] token. For TRIPS-ViLT, we follow [20] and use the
pairing method. The input is reformulated into two pairs (question, image1)
and (question, image2), and each pair goes through TRIPS-ViLT. The
classification head takes the concatenation of two pooled representations as
input and outputs the binary prediction. As shown in Table 3, TRIPS
outperforms existing VLP methods.

Table 4: Evaluation results of image-text retrieval on Flickr30K [34] and COCO datasets [29]. Models | # Pre-train | MSCOCO (5K test set) | Flickr30K (1K test set)
---|---|---|---
data | TR | IR | TR | IR
|  | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
E2E-VLP | 4M | - | - | - | - | - | - | 86.2 | 97.5 | 98.92 | 73.6 | 92.4 | 96.0
OSCARBase | 4M | 70.0 | 91.1 | 95.5 | 54.0 | 80.8 | 88.5 | - | - | - | - | - | -
VinVLLarge | 5.65M | 75.4 | 92.9 | 96.2 | 58.8 | 83.5 | 90.3 | - | - | - | - | - | -
ViLBert | 3.3M | - | - | - | - | - | -s | - | - | - | 58.2 | 84.9 | 91.5
UNITERLarge | 4M | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0 | 87.3 | 98.0 | 99.2 | 75.6 | 94.1 | 96.8
METERCLIP | 4M | 76.2 | 93.2 | 96.8 | 57.1 | 82.7 | 90.1 | 94.3 | 99.6 | 99.9 | 82.2 | 96.3 | 98.4
VLMoLagre | 4M | 78.2 | 94.4 | 97.4 | 60.6 | 84.4 | 91.0 | 95.3 | 99.9 | 100.0 | 84.5 | 97.3 | 98.6
BLIPBase | 14M | 80.6 | 95.2 | 97.6 | 63.1 | 85.3 | 91.1 | 96.6 | 99.8 | 100.0 | 87.2 | 97.5 | 98.8
ViLT | 4M | 61.5 | 86.3 | 92.7 | 42.7 | 72.9 | 83.1 | 83.5 | 96.7 | 98.6 | 64.4 | 88.7 | 93.8
TRIPS-ViLT | 4M | 63.2 | 88.2 | 94.1 | 43.9 | 74.1 | 84.6 | 85.4 | 98.3 | 98.4 | 64.4 | 89.2 | 94.9
ALBEFBase | 4M | 77.8 | 94.3 | 97.4 | 60.3 | 84.7 | 91.0 | 95.9 | 99.8 | 100.0 | 85.6 | 97.5 | 98.9
TRIPS-ALBEF | 4M | 78.1 | 94.8 | 97.6 | 61.3 | 84.3 | 91.4 | 96.3 | 99.8 | 100.0 | 85.8 | 98.1 | 99.0
mPLUGBase | 4M | 80.5 | 95.4 | 97.9 | 63.3 | 85.3 | 91.2 | 96.7 | 99.8 | 100.0 | 86.5 | 97.5 | 98.9
TRIPS-mPLUG | 4M | 80.8 | 95.7 | 98.0 | 63.6 | 85.5 | 91.5 | 97.0 | 99.8 | 100.0 | 86.9 | 97.8 | 99.1
Table 5: Evaluation Results on image captioning on COCO Karpathy test split [19]. B@4: BLEU@4, M: METEOR, C: CIDEr, S: SPICE.  Models |  # Pre-train Data | COCO Caption
---|---|---
Cross-entropy Optimization | CIDEr Optimization
B@4 | M | C | S | B@4 | M | C | S
E2E-VLP | 4M | 36.2 | - | 117.3 | - | - | - | - | -
OSCARBase | 6.5M | 36.5 | 30.3 | 123.7 | 23.1 | 40.5 | 29.7 | 137.6 | 22.8
VinVLLarge | 5.65M | 38.5 | 30.4 | 130.8 | 23.4 | 41.0 | 31.1 | 140.9 | 25.2
BLIPBase | 14M | 38.6 | - | 129.7 | - | - | - | - | -
SimVLMBase | 1.8B | 39.0 | 32.9 | 134.8 | 24.0 | - | - | - | -
mPLUGGBase | 4M | 39.3 | 30.1 | 132.4 | 23.34 | 41.2 | 31.0 | 140.8 | 25.2
TRIPS-mPLUG | 4M | 39.4 | 30.5 | 132.8 | 23.82 | 41.5 | 30.9 | 141.2 | 25.4

####  5.1.3 Image-Text Retrieval

We conduct experiments for both image-to-text retrieval (TR) and text-to-image
retrieval (IR) on MSCOCO [29] and Flickr30K [34] datasets. We jointly optimize
the ITC loss and the ITM loss during fine-tuning. Results are reported in
Table 4. As illustrated in Table 4, our model achieves comparable performance
to other VLP baselines.

####  5.1.4 Image Captioning

To investigate the text generation capacity of TRIPS-mPLUG, we evaluate it on
the image captioning task. Since there is no textual input for this task, we
select visual semantic-aware patches based on the attention weight of the
image [CLS] token to other image tokens and fuse other image tokens based on
the attention weight. We perform this operation based on the observation that
the image [CLS] token in ViTs pays more attention (i.e., having a larger
attention value) to class-specific tokens than to tokens on non-object regions
[4]. Following [27], we fine-tune TRIPS-mPLUG with cross-entropy loss and then
with CIDEr optimization for an additional 5 epochs. Our experiments, as shown
in Table 5, demonstrate that TRIPS-mPLUG achieves comparable results with
state-of-the-art models.

####  5.1.5 Visual Grounding

Table 6: Evaluation results of visual grounding on ReferCOCO+. We use the accuracy of IOU 0.5 on visual grounding (a prediction is right if the IoU between the grounding-truth box and the predicted bounding box is larger than 0.5) Model | RefCOCO+
---|---
testA | testB | val
UNITERLarge | 75.90 | 81.45 | 66.70
ViLBERT | 72.34 | 78.52 | 62.61
VILLA | 76.17 | 81.54 | 66.84
MDETR | 79.52 | 84.09 | 70.62
UNICORN | 80.30 | 85.05 | 71.88
mPLUGBase | 80.07 | 85.21 | 71.03
TRIPS-mPLUG | 80.11 | 86.03 | 71.21

Following the setting of mPLUG [23], we also evaluate TRIPS-mPLUG on the
visual grounding task, which requires models to localize the referred object
in the image based on a given text query. Instead of directly regressing the
bounding boxes, we concatenate visual features and attended textual features
and feed them into the decoder to predict the coordinates. Table 6
demonstrates the performance of TRIPS-mPLUG in the visual grounding task.
TRIPS-mPLUG achieves comparable results with competitive baseline methods.

Figure 4: The figure below visualizes the performance and inference speed distribution of different VLP models. The y-axis represents the accuracy of the VLP model on the test-dev dataset of VQA, while the x-axis represents the throughput of processing image-text pairs on a single 32G V100 GPU. Table 7: The comparison of the efficiency of different models. FLOPs, throughput, and latency are reported here. Since FLOPs are proportional to input size, for a fair comparison, we use same the input size with [20], which is 197 for image patches length and 40 for text tokens length. We also keep the same setting when calculating throughput and latency. Models | Latency | FLOPs | Throughput
---|---|---|---
ViLBERT | 880ms | 958.10 | 6.40
VisualBERT | 910ms | 425.20 | 6.10
LXMERT | 980ms | 952.20 | 6.38
UNITERBase | 870ms | 949.90 | 6.42
OSCARBase | 860ms | 956.40 | 6.35
VinVLBase | 640ms | 1023.30 | 7.32
E2E-VLP | 70ms | 144.30 | 80.23
ViLTBase | 15ms | 55.40 | 247.53
TRIPS-ViLT |  8ms | 32.38 | 394.32
ALBEFBase | 22ms | 33.42 | 197.52
TRIPS-ALBEF | 11ms | 20.89 | 343.05
mPLUGBase | 24ms | 36.63 | 186.42
TRIPS-mPLUG | 13ms | 25.04 | 313.71

###  5.2 Efficiency of TRIPS

To investigate the efficiency of Text-Relevant Image Patch Selection, we first
compare the computational complexity of various models. We report the Floating
Point Operations Per Second (FLOPs), a widely used evaluation metric for model
computational complexity. In addition, we evaluate the computational speed of
our model by comparing the throughput and latency of different models. We use
a Xeon Platinum 8163 CPU and an NVIDIA V100 GPU to calculate the latency and
throughput. As shown in Table 7, TRIPS not only exhibits the lowest
computational complexity but also achieves the fastest computational speed
(e.g., 343.05 throughput and 11ms latency for TRIPS-ALBEF vs. 197.52
throughput and 22ms latency for ALBEF).

To provide a more intuitive illustration of how TRIPS balances model
efficiency and effectiveness, we visualize the performance and inference speed
distribution of different VLP models on the VQA task in Figure 4. The figure
clearly demonstrates that incorporating TRIPS consistently results in a
speedup while maintaining competitive or superior performance across various
VLP base models.

Similar to this visualization, the following further analyses will mainly be
based on the VQA task to make the tables and figures more reader-friendly.
Note that most conclusions apply to other tasks.

Table 8: Results of pre-training and finetuning TRIPS-ALBEF with different locations and keep rates. we report the text-dev score results of VQA, FLOPs and Throughput. In this table, we set the input image size to 384 $\times$ 384 and the length of input text is 40. The throughput (image-text/s) is measured on an NVIDIA V100 GPU using the largest possible batch size for our model.  Locations | Keep rates | Overall Keep rate | VQA test-dev | FLOPs (G) | Throughput
---|---|---|---|---|---
- | - | 100% | 76.12 | 76.03 | 79.32
[2] | 50% | 50% | 75.60 | 42.17 | 161.26
[10] | 50% | 50% | 76.19 | 63.84 | 96.66
[2,4] | 50% | %25 | 74.21 | 28.00 | 238.41
[4,8] | 50% | %25 | 74.93 | 38.82 | 165.30
[5,10] | 50% | %25 | 75.29 | 44.22 | 143.37
[6,12] | 50% | %25 | 75.48 | 49.63 | 126.46
[2,4] | 70% | %49 | 74.87 | 43.96 | 153.92
[4,8] | 70% | %49 | 75.94 | 51.72 | 125.55
[5,10] | 70% | %49 | 76.23 | 55.60 | 115.01
[6,12] | 70% | %49 | 76.24 | 59.48 | 106.07
[2,6,10] | 70% | %34 | 74.92 | 42.66 | 156.13
[3,6,9] | 70% | %34 | 75.09 | 43.49 | 151.40
[4,8,12] | 70% | %34 | 75.23 | 49.74 | 129.81
Table 9: Results of TRIPS-ALBEF finetuning on VQA and NLVR task with different resolution images. When calculating FLOPs, the input length of the text is kept at 40 and the settings for calculating throughput are the same as Table8. Selection Layer | Keep rate | image size | VQA test-dev | FLOPs(G) | Troughout
---|---|---|---|---|---
- | - | $384\times 384$ | 76.12 | 76.03 | 79.32
[5,10 ] | 70% | $224\times 224$ | 75.23 | 20.89 | 343.05
[5,10 ] | 70% | $256\times 256$ | 75.84 | 26.61 | 258.03
[5,10 ] | 70% | $304\times 304$ | 76.13 | 36.62 | 189.07
[5,10 ] | 70% | $384\times 384$ | 76.23 | 55.60 | 115.01
[5,10 ] | 70% | $464\times 464$ | 76.54 | 74.83 | 81.0
[5,10 ] | 70% | $512\times 512$ | 76.83 | 84.13 | 72.10

###  5.3  The Impact of Patch-section Location and Keep Rate

To assess the impact of patch-selection layer location and keep rate on the
model’s efficiency and effectiveness, we train TRIPS-ALBEF with varying patch-
selection locations and numbers of selected tokens, and test the yielding
models on the VQA task. The results presented in Table 8 reveal two key
observations.

First, positioning the patch-selection layer in shallower layers reduces
computational complexity but negatively affects accuracy. For instance, when
the patch-selection layer is placed before the third layer (i.e., at the
second layer), accuracy significantly drops despite a notable increase in
throughput. A possible explanation for this is that the attention maps between
the text [CLS] embedding and the input tokens may be unreliable during the
early processing of input tokens in shallow layers.

Second, fusing too many image tokens in the patch-selection layer can lead to
a considerable decline in downstream task performance. For example, if we
designate the 2nd and 4th layers in ViT as patch-selection layers and set the
keep rate to 50%, performance on the VQA task decreases to 74.21, compared to
76.12 for the model without a patch-selection layer.

(a) VQA results of Different Resolutions (b) Throughput of Different
Resolutions (c) FLOPs of Different Resolutions

Figure 5: Sub-figure (a) visualizes the VQA results of three models (TRIPS-
ViLT, TRIPS-ALBEF, TRIPS-mPLUG) at different image resolutions, sub-figure (b)
visualizes the throughput of three different models at different image
resolutions, and sub-figure (c) visualizes the FLOPs of the three models at
different image resolutions.

###  5.4 Fine-tuning on Higher Resolution Images

We can manipulate the computational cost by fusing varying numbers of
inattentive tokens. To demonstrate this, we fine-tune TRIPS-ALBEF, TRIPS-ViLT,
and TRIPS-mPLUG on the VQA task, which uses images with different resolutions
as input. The results are presented in Table 9 and Figure 5. These
experimental findings indicate that by increasing the input image resolution,
we can enhance the model’s performance by incorporating more image tokens. For
instance, by fine-tuning TRIPS-ALBEF with images of size 464x464, we can
achieve a score of 76.54 on VQA, surpassing the baseline fine-tuned with
images of 384x384 while maintaining a similar computational complexity.

Table 10: The result of ablations. We finetune TRIPS-ViLT, TRIPS-ABLEF, and TRIPS-mPLUG on VQA and report test-dev results. The setting for calculating FLOPs and throughput is the same as Table 8. The same with the settings of TRIPS we present in the main results, we select the 5th and 10th as the patch selection layer, and each layer will keep 70% image tokens.  model | VQA | FLOPs(G) | Throughput
---|---|---|---
ALBEF-TRIPS | 76.23 | 55.60 | 115.01
-$w/o$ ITF | 75.92 | 55.14 | 118.04
-$w/o$ TD-ATT | 75.23 | 53.44 | 120.23
ALBEF-mPLUG | 77.89 | 67.02 | 98.38
-$w/o$ ITF | 77.29 | 65.23 | 103.38
-$w/o$ TD-ATT | 76.89 | 63.34 | 109.38
ALBEF-ViLT | 71.48 | 59.42 | 166.74
-$w/o$ ITF | 70.63 | 57.19 | 172.01
Figure 6: The visualization of the selected text-aware image patches in
different selection layers of TRIPS-mPLUG. We set the 5th and 10th layers in
the vision backbone as the patch-selection layer, and we keep 70% image
patches in each layer.  Figure 7: The visualization of the selected text-aware
image patches in different selection layers of TRIPS-mPLUG, conditioned on
different text queries in the VQA scenario. We set the 5th and 10th layers in
the vision backbone as the patch-selection layer, and we keep 70% image
patches in each layer.

###  5.5 Ablation Study

We also conduct ablation studies to investigate the effects of inattentive
image token fusion and Text-aware Dynamic Attention (TD-ATT). In Table 10,
$w/o$ ITF denotes that the inattentive tokens are directly discarded without
fusion. As demonstrated in Table 10, fusing inattentive tokens leads to better
performance compared to the model without inattentive tokens. Although the
improvement is modest, no additional computational overhead is introduced. We
further examine the impact of Text-aware Dynamic Attention. Specifically,
$w/o$ TD-ATT indicates that we remove the TD-ATT from the patch-selection
layer and select the image tokens based on the image [CLS] token. As TRIPS-
ViLT is a single-stream model and the [CLS] token aggregates both image and
text information, we cannot perform this ablation for it. As demonstrated in
Table 10, selecting image patch tokens with the image [CLS] token without
considering the linguistic context leads to a decline in the model’s
performance. This result supports our initial motivation that directly
removing patch tokens based on the image [CLS] without incorporating text
knowledge is unsuitable for VLP models.

###  5.6 Visualization

The proposed TRIPS accelerates VLP through a novel patch selection module that
identifies text-consistent image tokens in the vision backbone and preserves
the attentive image tokens. To further investigate our model’s
interpretability, we visualize the process of text-relevant image patch
selection during both the pre-training and inference stages.

Figure 6 displays the selected text-relevant image patches during the pre-
training stage. It is evident that as the network goes deeper, the inattentive
tokens are gradually removed or fused, while the text-relevant tokens are
selected and preserved.

Figure 7 visualizes the selected text-aware image patches in the VQA scenario
to demonstrate the effectiveness of the text-relevant image patch selection
module during the downstream task inference stage. We input various queries
and visualize the text-aware image patches chosen by the text-relevant image
patch selection module. As depicted in Figure 7, the selected image patches
are highly relevant to the query texts, enabling our model to make accurate
predictions.

##  6 Conclusion

We have presented TRIPS, an efficient VLP model with Text-Relevant Image Patch
Selection, which progressively eliminates redundant image tokens with the
guidance of text. TRIPS incorporates a novel patch selection module within the
vision backbone to select text-consistent image tokens. This module preserves
attentive image tokens with text guidance and combines inattentive tokens into
a single token by dynamically computing text-dependent visual attention in an
end-to-end manner. Experiments demonstrate that our method not only reduces
the computational cost of VLP but also enhances the efficiency of cross-modal
fusion due to the decreased visual sequences, while maintaining or even
improving the performance of downstream image-text tasks.

## References

  *   * Agrawal et al [2015] Agrawal A, Lu J, Antol S, et al (2015) Vqa: Visual question answering. International Journal of Computer Vision 123:4–31
  * Agrawal et al [2018] Agrawal H, Desai K, Wang Y, et al (2018) nocaps: novel object captioning at scale. CoRR abs/1812.08658. URL http://arxiv.org/abs/1812.08658, 1812.08658
  * Bi et al [2020] Bi B, Li C, Wu C, et al (2020) Palm: Pre-training an autoencoding&autoregressive language model for context-conditioned generation. arXiv preprint arXiv:200407159
  * Caron et al [2021] Caron M, Touvron H, Misra I, et al (2021) Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp 9630–9640
  * Chen et al [2019] Chen YC, Li L, Yu L, et al (2019) Uniter: Learning universal image-text representations. ArXiv abs/1909.11740
  * Chen et al [2020] Chen YC, Li L, Yu L, et al (2020) Uniter: Universal image-text representation learning. In: ECCV
  * Choromanski et al [2021] Choromanski K, Likhosherstov V, Dohan D, et al (2021) Rethinking attention with performers. ArXiv abs/2009.14794
  * Cubuk et al [2020] Cubuk ED, Zoph B, Shlens J, et al (2020) Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 702–703
  * Devlin et al [2019] Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv abs/1810.04805
  * Dosovitskiy et al [2021] Dosovitskiy A, Beyer L, Kolesnikov A, et al (2021) An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv abs/2010.11929
  * Dou et al [2021] Dou ZY, Xu Y, Gan Z, et al (2021) An empirical study of training end-to-end vision-and-language transformers. arXiv preprint arXiv:211102387
  * Fang et al [2021] Fang Z, Wang J, Hu X, et al (2021) Compressing visual-linguistic model via knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1428–1438
  * Gan et al [2021] Gan Z, Chen YC, Li L, et al (2021) Playing lottery tickets with vision and language. In: AAAI Conference on Artificial Intelligence
  * Goyal et al [2017] Goyal Y, Khot T, Summers-Stay D, et al (2017) Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6904–6913
  * He et al [2017] He K, Gkioxari G, Dollár P, et al (2017) Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV) pp 2980–2988
  * Heo et al [2021] Heo B, Yun S, Han D, et al (2021) Rethinking spatial dimensions of vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp 11916–11925
  * Huang et al [2020] Huang Z, Zeng Z, Liu B, et al (2020) Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. ArXiv abs/2004.00849
  * Jia et al [2021] Jia C, Yang Y, Xia Y, et al (2021) Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:210205918
  * Karpathy and Fei-Fei [2015] Karpathy A, Fei-Fei L (2015) Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3128–3137
  * Kim et al [2021] Kim W, Son B, Kim I (2021) Vilt: Vision-and-language transformer without convolution or region supervision. In: ICML
  * Kitaev et al [2020] Kitaev N, Kaiser L, Levskaya A (2020) Reformer: The efficient transformer. ArXiv abs/2001.04451
  * Krishna et al [2016] Krishna R, Zhu Y, Groth O, et al (2016) Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123:32–73
  * Li et al [2022a] Li C, Xu H, Tian J, et al (2022a) mplug: Effective and efficient vision-language learning by cross-modal skip-connections
  * Li et al [2021] Li J, Selvaraju RR, Gotmare AD, et al (2021) Align before fuse: Vision and language representation learning with momentum distillation. In: NeurIPS
  * Li et al [2022b] Li J, Li D, Xiong C, et al (2022b) Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:220112086
  * Li et al [2019] Li LH, Yatskar M, Yin D, et al (2019) Visualbert: A simple and performant baseline for vision and language. ArXiv abs/1908.03557
  * Li et al [2020] Li X, Yin X, Li C, et al (2020) Oscar: Object-semantics aligned pre-training for vision-language tasks. In: ECCV
  * Liang et al [2022] Liang Y, Ge C, Tong Z, et al (2022) Not all patches are what you need: Expediting vision transformers via token reorganizations. ArXiv abs/2202.07800
  * Lin et al [2014] Lin TY, Maire M, Belongie SJ, et al (2014) Microsoft coco: Common objects in context. In: ECCV
  * Liu et al [2021] Liu Z, Lin Y, Cao Y, et al (2021) Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp 9992–10002
  * Loshchilov and Hutter [2019] Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: ICLR
  * Lu et al [2019] Lu J, Batra D, Parikh D, et al (2019) Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS
  * Ordonez et al [2011] Ordonez V, Kulkarni G, Berg TL (2011) Im2text: Describing images using 1 million captioned photographs. In: NIPS
  * Plummer et al [2015] Plummer BA, Wang L, Cervantes CM, et al (2015) Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision 123:74–93
  * Radford et al [2021] Radford A, Kim JW, Hallacy C, et al (2021) Learning transferable visual models from natural language supervision. In: ICML
  * Rao et al [2021] Rao Y, Zhao W, Liu B, et al (2021) Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: Neural Information Processing Systems
  * Redmon et al [2016] Redmon J, Divvala SK, Girshick RB, et al (2016) You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 779–788
  * Ren et al [2015] Ren S, He K, Girshick RB, et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39:1137–1149
  * Rennie et al [2017] Rennie SJ, Marcheret E, Mroueh Y, et al (2017) Self-critical sequence training for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1179–1195, 10.1109/CVPR.2017.131
  * Ryoo et al [2021] Ryoo MS, Piergiovanni AJ, Arnab A, et al (2021) Tokenlearner: Adaptive space-time tokenization for videos. In: NeurIPS
  * Sharma et al [2018] Sharma P, Ding N, Goodman S, et al (2018) Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL
  * Singh et al [2021] Singh A, Hu R, Goswami V, et al (2021) Flava: A foundational language and vision alignment model. ArXiv abs/2112.04482
  * Su et al [2020] Su W, Zhu X, Cao Y, et al (2020) Vl-bert: Pre-training of generic visual-linguistic representations. ArXiv abs/1908.08530
  * Suhr et al [2019] Suhr A, Zhou S, Zhang I, et al (2019) A corpus for reasoning about natural language grounded in photographs. ArXiv abs/1811.00491
  * Tan and Bansal [2019] Tan HH, Bansal M (2019) Lxmert: Learning cross-modality encoder representations from transformers. ArXiv abs/1908.07490
  * Touvron et al [2021] Touvron H, Cord M, Douze M, et al (2021) Training data-efficient image transformers & distillation through attention. In: ICML
  * Vaswani et al [2017] Vaswani A, Shazeer NM, Parmar N, et al (2017) Attention is all you need. ArXiv abs/1706.03762
  * Wang et al [2020a] Wang J, Hu X, Zhang P, et al (2020a) Minivlm: A smaller and faster vision-language model. arXiv preprint arXiv:201206946
  * Wang et al [2020b] Wang S, Li BZ, Khabsa M, et al (2020b) Linformer: Self-attention with linear complexity. ArXiv abs/2006.04768
  * Wang et al [2021a] Wang W, Bao H, Dong L, et al (2021a) Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. ArXiv abs/2111.02358
  * Wang et al [2021b] Wang W, Xie E, Li X, et al (2021b) Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp 548–558
  * Wang et al [2021c] Wang Z, Yu J, Yu AW, et al (2021c) Simvlm: Simple visual language model pretraining with weak supervision. ArXiv abs/2108.10904
  * Wu et al [2021] Wu H, Xiao B, Codella NCF, et al (2021) Cvt: Introducing convolutions to vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp 22–31
  * Xu et al [2021a] Xu H, Yan M, Li C, et al (2021a) E2e-vlp: End-to-end vision-language pre-training enhanced by visual learning. ArXiv abs/2106.01804
  * Xu et al [2021b] Xu Y, Zhang Z, Zhang M, et al (2021b) Evo-vit: Slow-fast token evolution for dynamic vision transformer. ArXiv abs/2108.01390
  * Yang et al [2021] Yang Z, Gan Z, Wang J, et al (2021) Crossing the format boundary of text and boxes: Towards unified vision-language modeling. CoRR abs/2111.12085. URL https://arxiv.org/abs/2111.12085, 2111.12085
  * Yu et al [2021] Yu F, Tang J, Yin W, et al (2021) Ernie-vil: Knowledge enhanced vision-language representations through scene graph. In: AAAI
  * Yu et al [2016] Yu L, Poirson P, Yang S, et al (2016) Modeling context in referring expressions. In: European Conference on Computer Vision, Springer, pp 69–85
  * Zhang et al [2021] Zhang P, Li X, Hu X, et al (2021) Vinvl: Revisiting visual representations in vision-language models. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp 5575–5584
  * Zhou et al [2020] Zhou L, Palangi H, Zhang L, et al (2020) Unified vision-language pre-training for image captioning and vqa. ArXiv abs/1909.11059
  * Zong et al [2021] Zong Z, Li K, Song G, et al (2021) Self-slimmed vision transformer. ArXiv abs/2111.12624

##  Appendix A Comparison Models

  * •

E2E-VLP [54]: proposes the first end-to-end VLP method for both V+L
understanding and generation, with a unified Transformer encoder-decoder
architecture.

  * •

VinVL [59]: pre-trains a large-scale object-attribute detection model with
much larger amounts of supervised data on four public object detection
datasets for extracting better region-based visual features.

  * •

OSCAR [27]: proposes to use object tags detected in images as anchor points to
ease the learning of cross-modal alignments, where the input to the
Transformer is a combination of image, text, and object tags.

  * •

METER [11]: systematically investigates how to design and pre-train a fully
transformer-based VL model in an end-to-end manner.

  * •

VLMo [50]: presents a unified vision-language pre-trained model that jointly
learns a dual encoder and a fusion encoder with a modular Transformer network.

  * •

SimVLM [52]: different from previous VLP methods that only use limited
(4M-10M) image-text pairs for pre-training, it proposes a simple VLP model
with a single prefix language modeling objective, which pre-trains on an
extremely large aligned cross-modal data of about 1.8B noisy image-text pairs.
This is also the latest state-of-the-art method of image captioning.

  * •

ALBEF [24]: introduces a contrastive loss to align the image and text
representations before fusing them through cross-modal attention, which
enables more grounded vision and language representation learning.

  * •

UNITER [6]: proposes an improved single-stream VLP method, by designing two
new pre-training strategies: 1) it uses conditional masking on pre-training
tasks instead of random masking strategy, 2) it designs a new word-region
alignment pre-training task via the use of optimal transport to explicitly
encourage fine-grained alignment between words and image regions.

  * •

ALIGN [18]: leverages a noisy dataset of over one billion image alt-text
pairs, obtained without expensive filtering or post-processing steps in the
Conceptual Captions dataset.

  * •

ViLT [20]: adopts linear projection and word embedding as the visual and
textual encoders, and uses the visual transformer as the cross-modal encoder
to align and fuse the features of both modalities in an end-to-end manner.

  * •

BLIP [25]: proposes a new VLP framework that transfers flexibly to both
vision-language understanding and generation tasks. It effectively utilizes
noisy web data by bootstrapping the captions.

  * •

UNICORN [56]: proposes a vision-language (VL) model that unifies text
generation and bounding box prediction into a single architecture.

  * •

LXMERT [45]: is the pioneering work to pre-train a dual-stream multi-modal
Transformer, which consists of an object relationship encoder, a language
encoder, and a cross-modality encoder. It is widely used as a baseline method
for VLP models.

  * •

ViLBERT [32]: proposes the first work that extends the BERT architecture to a
multi-modal dual-stream VLP model, which processes both visual and textual
inputs in separate streams that interact through co-attentional transformer
layers.

  * •

mPLUG [23]: is a vision-language foundation model for both cross-modal
understanding and generation and introduces an effective and efficient vision-
language architecture with novel cross-modal skip-connections.

##  Appendix B Downstream Task Details

We evaluate TRIPS on the four downstream vision-language tasks. The
hyperparameters that we use for finetuning on the downstream tasks are listed
in Table 11. Following [24], all tasks adopt RandAugment, AdamW optimizer with
a weight decay of 0.05 and a cosine learning rate schedule. Next, we introduce
the dataset settings in detail.

##### VQA.

The VQA task [1] requires the model to answer natural language questions given
an image. Most methods [45, 50, 27, 52] deal with visual question answering
tasks as multi-label classification on pre-defined answer sets. This strategy
achieves strong performance, but it is not suitable for real-world open
scenarios. We conduct an experiment on the VQA2.0 dataset [14], which contains
83k/41k/81k images for training/validation/test. Following [24], we use both
training and validation splits for training, and incorporate additional
training data from Visual Genome [22]. Following [27], we concatenate the
question with the object labels and OCR tokens extracted from the image.

Table 11: Finetuning hyperparameters for downstream tasks. $\dagger$ denotes two stages of fine-tuning. Task |  LR (ViT-L/BERTbase) | batch size | epochs
---|---|---|---
VQA | 2e-5/5e-6 | 1024 | 8
Captioning$\dagger$ | 1e-5&8e-7 | 256 | 5
Retrieval | 1e-5/2e-6 | 256 | 5
NLVR2 | 5e-5/5e-6 | 256 | 15
Visual Grounding | 2e-5/2e-6 | 512 | 120

##### Image Captioning.

Image captioning requires generating a descriptive and fluent caption for a
given image. We evaluate the performance of TRIPS on two popular datasets:
COCO Caption [29] and NoCaps [2]. We fine-tune TRIPS-mPLUG on the training set
of COCO Caption and test it on the same Karpathy split [27, 52] as well as the
NoCaps validation set. To fine-tune TRIPS-mPLUG on COCO Caption, we follow the
approach in [27] and first train the model with the cross-entropy loss for 5
epochs with a learning rate of 1e-5 and a batch size of 256. We then further
fine-tune the model with CIDEr optimization [39] for an additional 5 epochs
with a smaller learning rate of 8e-7. We use the best checkpoint on COCO
Caption to predict on the NoCaps validation set. During inference, we use beam
search with a beam size of 10 and set the maximum generation length to 20.

##### Image-Text Retrieval.

We conducted experiments on both image-to-text retrieval (TR) and text-to-
image retrieval (IR) using the COCO [29] and Flickr30K [34] datasets and used
the widely-used Karpathy split [19] for both. COCO contains 113k/5k/5k images
for train/validation/test, while Flickr30K contains 29k/1k/1k images for
train/validation/test. During fine-tuning, for TRIPS-ALBEF and TRIPS-mPLUG we
jointly optimized the ITC loss and the ITM loss following the approach in [24,
25]. During inference, we first selected the top-k candidates by computing the
dot-product similarity between the image and text encoder features (We set
$k=256$ for COCO and $k=128$ for Flickr30K). For the efficiency of coarse-
grained ranking, we directly selected the patch based on the attention weights
of the image [CLS] token to other patch tokens.

##### Visual Grounding.

The task of visual grounding involves localizing the referred object in an
image given a plain text query. Instead of directly regressing bounding boxes,
our approach concatenates visual features with textual features, which are
then fed into the multi-modal decoder to predict the object’s coordinates. We
evaluate our method on the referring expression grounding dataset:
RefCOCO+[58]. The RefCOCO+ dataset contains 19K images and 141K queries.

##### NLVR2.

The NLVR2 [44] task requires the model to predict whether a sentence. We
conduct experiments following the original train/val/test split in [44].
Following [25], we use two cross-attention layers to process the two input
images, and their outputs are merged and fed to the FFN. An MLP classifier is
then applied to the output embedding of the language [CLS] token.