# Learning Embeddings for Image Clustering:
An Empirical Study of Triplet Loss Approaches

Kalun Ho  Competence Center High Performance Computing, Fraunhofer ITWM,
Kaiserslautern, Germany  Data and Web Science Group, University of Mannheim,
Germany  Janis Keuper  Institute for Machine Learning and Analytics (IMLA),
Offenburg University, Germany  Franz-Josef Pfreundt  Competence Center High
Performance Computing, Fraunhofer ITWM, Kaiserslautern, Germany  Margret
Keuper  Data and Web Science Group, University of Mannheim, Germany

###### Abstract

In this work, we evaluate two different image clustering objectives, k-means
clustering and correlation clustering, in the context of Triplet Loss induced
feature space embeddings. Specifically, we train a convolutional neural
network to learn discriminative features by optimizing two popular versions of
the Triplet Loss in order to study their clustering properties under the
assumption of noisy labels.
Additionally, we propose a new, simple Triplet Loss formulation, which shows
desirable properties with respect to formal clustering objectives and
outperforms the existing methods. We evaluate all three Triplet loss
formulations for K-means and correlation clustering on the CIFAR-10 image
classification dataset.

##  I Introduction

When grouping data with missing or noisy labels, unsupervised approaches such
as clustering are crucial. While fully supervised classification methods might
result in a higher level of accuracy, they require a large amount of annotated
data. In contrast, clustering approaches leverage the intrinsic data
properties such as data density distributions or pairwise distances instead of
annotations in order to group. However, finding a suitable distance metric is
essential. One such a metric could be the Euclidean distance of two data
points based on some features. For instance k-means clustering is based on the
Euclidean distance to some centroid. Thus, the properties of the features
space play a crucial role. In computer vision, the simplest choice would be to
use raw image pixel. However, this is often neither effective nor feasible for
images with higher resolutions, and it shows poor generalization. An effective
way is to first reduce the dimension of the image (e.g. raw image pixels) into
a lower dimensional space (embedding) with a non-linear mapping function. This
can be achieved using a convolutional neural network (CNN). A popular loss
function for learning such an embedding is the Triplet Loss [1], where three
images are given such that the CNN learns to organize images of the same class
closer to one another in the embedding space than images of different classes.
Using the resulting embedding as features, one can run traditional clustering
methods such as k-means clustering as for example done in [2]. k-means
clustering assumes that the data points are evenly distributed around the
cluster centroids. Furthermore, it is required that the number of clusters has
to be specified beforehand. In many practical tasks, this specific scenario
might be unrealistic because, for example, the number of objects to be grouped
is simply unknown or because data points from the same class lie on a more
complex manifold. This motivates us to additionally consider a graph-based
approach, where no data specific knowledge is required, i.e. the _minimum cost
multicut_ problem, also known as _correlation clustering_ [3, 4].

Figure 1: Summary of the experiment setup in three steps: 1) dataset is
trained using the Triplet Loss. Here, we add random noises (in red) by
selecting wrong samples in the training data. 2) We cluster data using graph-
based approach based on the learned embedding features (in blue). 3)
Evaluation of clustering accuracy.

In the context of these two common clustering techniques, we want to study the
properties of embeddings resulting from two common variants of the Triplet
Loss [1, 5] and investigate their susceptibility to label noise in the
training data. Additionally, we propose and study a third variant of the
Triplet Loss, which shows promise in the context of both minimum cost
multicuts as well as k-means clustering and can be understood as a
simplification of the loss proposed in [5]. Both share the desired property to
directly allow the extraction of pairwise cut probabilities between data
points from the embedding space without an intermediate learning step.
Specifically, we train a CNN on the CIFAR-10 image classification dataset [6]
to learn discriminative features using the three variants of the Triplet Loss,
where we apply noise to the training labels for positive and negative samples.
We evaluate the resulting embeddings by comparing the resulting clustering
performance using minimum cost multicuts and k-means clustering. Figure 1
illustrates our experimental setup.

In summary, our contributions are:

  * •

We conduct a thorough study of the clustering behavior of two popular
clustering approaches, k-means and minimum cost multicuts, applied to learnt
embedding spaces from three Triplet Loss formulations on the CIFAR-10 [6]
dataset under a varying amount of label noise.

  * •

Our study reveals that, while the traditional Triplet Loss [1] is well suited
for k-means clustering, its performance drops under the looser assumptions
made by minimum cost multicuts.

  * •

We propose a simplification of the Triplet Loss from [5] (3), which allows to
directly compute the probability of two data points for belonging to disjoint
components and is robust against noise in both clustering scenarios.

  * •

Our proposed Triplet Loss variant outperforms both previous versions in terms
of clustering performance and stability under label noise on the CIFAR-10
dataset.

##  II Related Work

###  II-A Clustering

Many clustering approaches on computer vision problems are based on
dimensionality reduction, where a non-linear mapping function is applied. One
popular way is to use an autoencoder, where an input image is encoded into a
embedding of lower dimension and then the decoder attempts to reconstruct its
original. The embedding is then used as feature space for the clustering
methods: For instance, Xie et al. [2] first train an autoencoder and then use
the same dataset and fine-tune it by training it again using a KL-divergence
loss. Another approach based on autoencoder is [7], which uses the
reconstruction loss along with relative entropy to jointly train the network.
Similar approaches can be found in [8, 9, 10]. Recently, generative models
have been proposed for clustering tasks [11, 12]. A large scale study on
clustering is proposed by Caron et al. [13], which iteratively groups the
features with a k-means during the optimization.

###  II-B Correlation Clustering

Correlation Clustering, also referred to as the _minimum cost multicut
problem_ [3, 4] is a popular choice when the number of clusters are unknown.
One such practical scenario is multiple object tracking, where pedestrians are
tracked by just providing their detections [14, 15, 16, 17, 18]. Correlation
clustering allows to group the data points based on pairwise cut probabilities
without any cluster size bias and optimizes the number of clusters along with
the data association. The crucial part there is to define or learn cut
probabilities based on features. For instance [19] uses DeepMatching [20] as a
similarity measure, followed by a logistic regression. An alternative is to
use features from embeddings learnt through a Siamese network [21]. The
training is based on pairs of images where the network outputs a binary
decision.This is closely related to our work, since we also want to obtain
discriminative features by training a CNN with the Triplet Loss. However,
instead of a binary output as done in [21], we want our network to learn an
embedding of a fixed vector length, e.g. 32 dimensions.

###  II-C Deep Embedding Learning

The main idea of learning embeddings is to attract similar data points to one
another in a lower dimensional space while pushing dissimilar samples away
from one another. While the Contrastive Loss [22] fixes the positive and
negative pairs by a fixed distance, it can be restrictive to variations in the
embedding space [23]. In contrast, the Triplet Loss [1] captures the relative
similarity of pairs of data points instead of absolute similarities. It has
been widely used for embedding learning [24, 25, 26, 27]. However, Zhang et
al. [5] highlighted three major issues and thus proposed an _Improved Triplet
Loss_ by enforcing intra- and intercluster constraints. We compare these two
Triplet Loss formulations and propose a simplification of the formulation from
Zhang et al., which outperforms both other formulations in practice.

##  III Study Setup

In this section, we describe the setup of our study. First, we introduce the
architecture and the different variations of the Triplet Loss in Section III-A
and III-B. Then, in section III-C, we explain the correlation clustering
method, also called the minimum cost multicut problem. Section III-D describes
the used dataset as well as the evaluation metric.

###  III-A CNN-Architecture

We use AlexNet [28] as a CNN-backbone as done in [13]. Furthermore, we also
replaced the local response layers with batch normalization as done in [13].
However, in order to reduce the feature dimensionality, we changed the size of
the last two fully-connected layers from 4096 to 64 and 32, respectively.

###  III-B Loss Function

Given any model architecture with trainable parameters $\theta$, we want to
map an input image $x_{i}$ with a non-linear function $f_{\theta}:X\rightarrow
Z$, with $f(x_{i})\in\mathbb{R}^{d}$. In our case, $x_{i}$ is the image of the
$i$-th sample from the dataset with a total number of $n$ samples, ${x_{i}\in
X}_{i=1}^{n}$ and $d$ is the dimension of the embedding space. $d$ is much
smaller than the dimension of the input image. Given a set of three images,
$x_{i}^{a}$, $x_{j}^{n}$ and $x_{k}^{p}$, the embedding features are learned
by simply minimizing the Triplet Loss [1] over the parameters $\theta$ of our
deep neural network. Here, the parameter $\alpha$ sets the margin of the
similarity difference between the positive sample $x_{k}^{p}$ and negative
sample $x_{j}^{n}$ and the anchor image:

$L_{triplet}=\sum_{i=1}^{n}[\|f(x_{i}^{a})-f(x_{i}^{p})\|^{2}-\|f(x_{i}^{a})-f(x_{i}^{n}\|^{2})+\alpha]_{+}$
(1)

Figure 2: Visualization of a possible data distribution trained with Triplet
Loss [1]: different intra-cluster distances over different classes make it
impossible to learn one distance threshold at which data points should belong
to different components. A logistic regression model could thus not learn
cluster boundaries for graph partition. The correct decision boundaries are
marked by green lines while the red line shows a wrong separation of data. Our
aim is to propose new Triplet Loss to stabilize such distances.

Our approach is based on the assumption that embedding features, learned from
the regular Triplet Loss (1) can produce high variances in inter- and intra-
cluster distances, because it only considers relative differences between the
distances of positive and negative pairs. This objective is suitable for
k-means clustering. Yet, the attempt to learn whether two data points should
belong to the same or to a different class from their pairwise distances (for
example through a logistic regression model as in Section III-D) might fail,
when the intra- and inter cluster samples are equally far away. This is shown
in Figure 2 where the correct decision boundaries are marked by green lines.
In contrast, the red line, at the same Euclidean distance as the green lines,
indicates a false separation of data. This motivates us to consider losses
that preserve the distance equally between the positive pairs during the
optimization. Similar to [5], we therefore add an additional term to equation
(1), which we denote as Triplet Loss_2 [5]:

$L_{triplet\\_2}=L_{triplet}+[\|f(x_{i}^{a})-f(x_{i}^{p})\|^{2}-\beta]_{+}$
(2)

The additive term in equation (2) introduces an additional parameter $\beta$,
which sets the maximum distance between the positive pairs, e.g. the intra-
cluster distance. As the regular Triplet Loss only considers the distance
difference between the positive and negative pairs, set by the parameter
$\alpha$, we propose a third loss function, which considers the absolute
distance for the positive and negative pairs (instead of distance difference):

$L_{triplet\\_3}=[\alpha-\|f(x_{i}^{a})-f(x_{j}^{n})\|^{2}]_{+}+[\|f(x_{i}^{a})-f(x_{k}^{p})\|^{2}-\beta]_{+}$
(3)

We argue that this variant of the Triplet loss is more intuitive than Triplet
Loss_2 because it directly pushes positive pairs within a certain margin
$\beta$ while driving negative pairs apart with a minimum distance $\alpha$.
Within these margins, it still allows for varying distances, which is in
contrast to for example Siamese approaches or the contrastive loss [22].

###  III-C Minimum Cost Multicuts

We assume, we are given an undirected graph $G=(V,E)$, where nodes $v\in V$
represent images and edges $e\in E$ encode their respective connectivity.
Additionally, we are given real valued costs $c:E\rightarrow\mathbb{R}$
defined on all edges which represent the node affinities. The goal is to
determine _edge_ labels $y:E\rightarrow\\{0,1\\}$ defining a graph
decomposition such that every partition of the graph corresponds to exactly
one class. To infer such an edge labeling, we can solve instances of the
minimum cost multicut problem with respect to the graph G and costs c, defined
as follows [3, 4]:

$\displaystyle\min\limits_{y\in\\{0,1\\}^{E}}\sum\limits_{e\in E}c_{e}y_{e}$
(4)

$\displaystyle s.t.\quad\forall C\in cycles(G)\quad\forall e\in
C:y_{e}\leq\sum\limits_{e^{\prime}\in C\backslash\\{e\\}}y_{e^{\prime}}$ (5)

The objective is to minimize (4) with respect to the assigned real valued
costs of the edges and the corresponding cycle inequality constraint in Eq.
(5). The cycle inequality constraint ensures that the edge labeling $y$
induces a decomposition of $G$. In [3], it was shown to be sufficient to
enforce Eq. (5) on all _chordless_ cycles, i.e. all cycles. Typically, if cut
probabilities between pairs of nodes are available, the costs are computed
using the _logit_ function $\text{logit}(p)=\log\frac{p}{1-p}$ to generate the
real valued edge costs. With these costs set appropriately, the optimal
solution of minimum cost multicut problems not only yields an optimal cluster
assignment but also estimates the number of clusters automatically.
Furthermore, this problem is able to generate small clusters and does not
necessarily provide balanced sized clusters.

#### Optimization

The minimum cost multicut problem (4) is in the class of NP-hard [29] and even
APX-hard [4, 30] problems. Nonetheless, instances have been solved within
tight bounds, for example in [31] using a branch-and-cut approach. While this
can be reasonably fast for some easier problem instances, it can take
arbitrarily long for others. Thus, primal heuristics such as the one proposed
in [32, 33, 34] or [35] are often employed in practice and show convincing
results in various scenarios [32, 21, 36].

###  III-D Pairwise Cut Probabilities for Multicuts

We formulate the following task as a minimum cost multicut problem: each node
in the graph $G$ represents an image $x_{i}$ that we want to assign to a class
label. The weight of the edges between two nodes represents the similarity
between the two images. The similarity is computed based on the Euclidean
distance of in embedding feature using our CNN-model:

$d_{i,j}=\|f(x_{i})-f(x_{j})\|$ (6)

Once the similarity of two images is obtained, we seek to estimate their
probability to belong to distinct classes. A decision function can be learned
through a logistic regression using the label information, e.g. positive pairs
are mapped to the value 0 while negative pairs are mapped to 1.

However, with the Triplet Loss_2 (2) and the proposed Triplet Loss_3 in (3),
the distance threshold $\tau$ that decides whether two points should belong to
different components (compare the distances in Figure 2) can be automatically
derived from the parameters $\alpha$ and $\beta$. Since $\beta$ restricts the
maximum distance of positive pairs to exactly $\beta=\alpha/2$, the distance
threshold $\tau$ of the logistic function is computed as:

$\tau=\sqrt{(\alpha+\beta)/2}$ (7)

Note that this is not possible for the Triplet Loss from Eq. (1) because it
only considers relative distances.

A complete graph is built and the clusters are obtained by optimizing equation
(4) using [32]. We use clustering accuracy (ACC) as evaluation metric, where
the best map between predicted clusters and true label is found.

##  IV Experiments and Results on CIFAR-10

In this Section, we present the experiments and the results of our study.
CIFAR-10 [6] is a popular dataset for image classification tasks. It contains
50.000 train and 10.000 test data samples of tiny images (32px x 32px). Each
sample is assigned a label that belongs to one of the ten classes: airplane,
automobile, bird, cat, deer, dog, frog, horse, ship, and truck. All models are
trained for 100 epochs with a batch size of 100 and a learning rate of 0.001
using AdamOptimizer [37]. First, we compare the performance of the minimum
cost multicuts and k-means clustering using different Triplet Losses in
Section IV-A. Then, we present some insights related to inter- and intra-
cluster distances in Section IV-B. In Section IV-C, we present our study on
the feature learning under label noise. The results are shown in Table I. In
Section IV-D, we present some qualitative results.

Figure 3: Comparison of three different losses on clustering performance. The
numbers on the x-axis represent the triplet loss type while the y-axis shows
the average cluster accuracy on five runs. The black line indicates the
standard deviation. Triplet Loss_2 (2) and our Triplet Loss_3 (3) perform
better than the regular Triplet Loss for minimum cost multicuts. However,
k-means consistently achieves better results given the fact that the parameter
$k$ is set correctly (shown in right). For k-means, the Triplet Loss_2 shows
worse performance than the regular one [1], while the proposed, simpler
version, Triplet Loss_3, performs best in both scenarios.

a) Triplet Loss_1 (1) b) Triplet Loss_2 (2) c) Triplet Loss_3 (3)

Figure 4: Euclidean inter and intra cluster distance trained with all three
Triplet Loss variants (equation (1), (2) and (3)): the average distance shows
significant variance when trained with a) Triplet Loss_1 (1), especially for
the class _cat_ and _dog_. The overlap of the distances prevents the logistic
regression model to learn the cluster boundaries. Both other variants, b) and
c), produce more consistent and stable results, thus higher clustering
accuracy.

###  IV-A Evaluation of Cluster Accuracy

We compare the CNN models that are optimized with the three different losses.
Furthermore, we also run k-means clustering on the same embedding features
with $k$ correctly set to $k=10$ (thus introducing external knowledge to the
clustering process). For Tripet Loss (1), an additional regression model is
trained using the label information in order to estimate the threshold, while
for (2) and (3), the threshold is computed directly from the optimization
parameters using equation (7). All experiments are executed five times with
different random seeds and we report the average number over the clustering
accuracy.

#### Results

Figure 3 shows the clustering accuracy for different variants of the Triplet
Loss. First, we observe that Triplet Loss_2 (2) and Triplet Loss_3 (3)
outperform (1) on multicut clustering (left). However, K-Means (right)
performs better on all our experiments given the fact that $k=10$ is known.
The highest performance is achieved when we train the CNN-model with the
Triplet Loss_3 (3), where the average accuracy is 80.5% (red). For k-means,
the Triplet Loss_2 shows worse performance than the regular one [1], while the
proposed, simpler version, Triplet Loss_3, performs best in both scenarios.

###  IV-B Inter- and Intra-cluster distances

Figure 4 compares the cluster distances of the samples of all three Triplet
Losses. The red curve represents the average pairwise distances of the samples
within a cluster while the blue curve shows the average distances of one
cluster to its nearest cluster. Furthermore, the color range represents the
standard deviation $\sigma$ and $2\sigma$ of the cluster distances. As
illustrated in Figure 4 a), both distances show significant variances when
trained with the regular Triplet Loss (1). Furthermore, class cat and dog show
a significant overlap in their inter- and intra cluster distances, which
prevents the logistic regression from setting the right decision boundary as
explained in Figure 2. In contrast, Triplet Loss_2 (2) and Triplet Loss_3 (3)
produce more consistent and stable results, and produce higher clustering
accuracy (see Figure 3).

| Minimum Cost Multicuts
---|---
| Triplet Loss (1) | Triplet Loss (2) | Triplet Loss (3)
Noise % | random | 7% | 5% | 2% | 0% | random | 7% | 5% | 2% | 0% | random | 7% | 5% | 2% | 0%
20% | 73.85 | 73.64 | 74.43 | 76.28 | 75.18 | 60.13 | 65.61 | 68.05 | 67.26 | 64.39 | 75.23 | 74.39 | 75.00 | 72.80 | 75.27
10% | 77.90 | 77.02 | 76.93 | 78.04 | 77.87 | 74.23 | 75.64 | 75.23 | 74.75 | 75.48 | 77.10 | 76.53 | 76.47 | 76.06 | 76.29
5% | 78.65 | 76.73 | 78.30 | 77.73 | 78.87 | 75.62 | 76.27 | 76.06 | 75.80 | 76.63 | 77.07 | 78.94 | 80.16 | 77.44 | 77.75
0% | 78.14 | 77.76 | 78.31 | 78.56 | 78.44 | 77.07 | 77.46 | 77.47 | 80.41 | 79.96 | 80.25 | 80.38 | 80.65 | 80.42 | 80.51
| k-means Clustering
| Triplet Loss (1) | Triplet Loss (2) | Triplet Loss (3)
Noise % | random | 7% | 5% | 2% | 0% | random | 7% | 5% | 2% | 0% | random | 7% | 5% | 2% | 0%
20% | 79.83 | 80.26 | 80.29 | 80.60 | 80.55 | 71.71 | 74.70 | 74.86 | 74.62 | 74.61 | 79.66 | 80.54 | 77.83 | 78.20 | 80.53
10% | 81.55 | 80.95 | 81.10 | 81.53 | 81.21 | 75.33 | 75.80 | 75.89 | 75.54 | 76.48 | 81.93 | 81.78 | 80.59 | 80.45 | 82.06
5% | 81.62 | 80.82 | 81.27 | 81.19 | 81.68 | 76.79 | 77.01 | 76.14 | 76.97 | 76.93 | 82.07 | 82.11 | 81.98 | 81.90 | 82.08
0% | 81.59 | 81.33 | 81.53 | 81.51 | 81.82 | 80.25 | 80.80 | 81.11 | 81.39 | 80.73 | 82.15 | 82.14 | 81.93 | 81.64 | 81.72
TABLE I: Evaluation of clustering accuracy using minimum cost multicuts (top) and k-means (bottom) based on the embedding features of the CNN-model trained on three different Triplet Loss variants. The average accuracies of five runs are reported in %. Furthermore, the numbers in bold are the reported numbers from Figure 3. The x-axis shows the amount of label noise on the negative samples up to 7%, while y-axis shows the amount of label noise on the positive samples up to 20%, respectively. The column random selects the negative sample without employing the labels. On the CIFAR-10 dataset, the chances are 90.0% to retrieve correct negative samples.  | Minimum Cost Multicuts |  | K-means
---|---|---|---
noisy positive labels |  |  |
noisy negative labels |  |  |
noise of pos. and neg. labels |  |  |
Figure 5: Average cluster accuracy against the percentage of noise, that is
applied to the sampling. Noises on positive and negative pairs are applied on
first and second row, respectively while the last row is evaluated on an equal
amount of label noise of both, positive and negative pairs. The first column
shows the results of the minimum cost multicuts while the second column
employs k-means. The amount of noise (i.e. wrong pairs) is indicated in
x-axis.

###  IV-C Triplet Loss with Label Noise

Figure 6: TSNE-Visualization of the clusters on CIFAR-10 using a multicut
approach. In this example, the cluster accuracy is 80.27% trained on a CNN-
model with the Triplet Loss (3) and the total number of clusters is 44. The
color represents the 10 largest clusters found. There are 33 small clusters
($<10$ items), for instance bottom left and right are two clusters shown
containing 7 images each. False positives within the clusters are marked as
red. The cluster on the upper left corner contains images from bird, cat,
automobile (x3), planes and trucks.

In this experiment, we investigate the sensitivity of the different Triplet
Losses towards label noise. Specifically, we randomly select wrong triplets
during the training process of our CNN-model and evaluate the clustering
performance based on the embedding features, trained on all three loss
variants from Subsection III-B. The experiments were conducted repeatedly five
times with different seeds and we report the mean cluster accuracies in %. The
parameters $\alpha$ and $\beta$ are fixed to $0.8$ and $0.4$, respectively.

#### Results

Table I shows our complete evaluation with various setups: we applied
different amounts of label noise on triplets for positive and negative pairs.
The x-axis represents the noise for the negatives while the y-axis indicates
noise on positive pairs within the triplet. These wrong pairs are retrieved
randomly. All results are reported as average clustering accuracy over five
runs of training. In Table I top, we present the performance using minimum
cost multicuts while the bottom rows show the results of k-means clustering
Note that k-means requires to specify the number of clusters $k$, beforehand
(on CIFAR-10, we know $k=10$), while minimum cost multicuts do not require
this dataset specific knowledge. Without noise added, considering the CNN-
model trained with the same loss function, k-means seems more stable against
noise and outperforms the correlational clustering on average by 1-2% and the
highest clustering accuracy when no noises are added. In Figure 5, we give a
more detailed analysis of these results.

Remark: When sampling the triplets randomly on balanced dataset with
$k$-clusters, the chance to get a true positive and true negative pair is
$\frac{1}{k}$ and $\frac{k-1}{k}$ respectively.

Figure 5 shows the clustering accuracy against the percentage of noise that is
applied to the sampling. In Figure 5, top left, we only add noise to the
positive pairs while selecting correct negative samples and evaluate using
minimum cost multicuts, i.e. without introducing knowledge on the number of
classes. Our first observation is that Triplet Loss_2 (2) is the most
sensitive to noise among all three loss variants. This is shown in Figure 5 in
blue. The regular Triplet Loss (1) and Triplet Loss_3 (3) still achieve and
average clustering accuracy of 75.0%, respectively, even though 20% of wrong
samples are used for training. A similar behavior can be observed in Figure 5,
top right, where we evaluate the same embeddings using k-means clustering. The
observations are slightly different when adding label noise to the negative
pairs, Figure 5, second row. Even when introducing 10% noise, which
corresponds to drawing negative samples completely at random in a balanced 10
class classification problem, all loss variants are relatively robust,
especially for k-means clustering (Figure 5, second row, right). Yet, the
proposed Triplet Loss_3 again performs best. In the bottom row of Figure 5, we
consider an equal amount of noise on both positive and negative samples
(corresponding to the diagonal in Table I). In this setting, the proposed
Triplet Loss_3 again shows higher stability than the two previous variants
when clustering using minimum cost multicuts Figure 5(bottom left). For
k-means clustering, the improvement over the Triplet Loss [1] is marginal.
This result is actually expected: The traditional Triplet Loss [1] creates an
embedding such that, for every data point, data points from the same class are
closer than points from any other class. This fits well with the k-means
clustering objective, assigning every points to the nearest cluster center,
regardless their absolute distance. Yet, it can be problematic in the context
of correlation clustering, where the minimum absolute distance between two
clusters matters (compare again Figure 2).

###  IV-D Qualitative Results

Figure 6 shows a TSNE-visualization [38] of the embedding features learned
from the CNN-model using the Triplet Loss_3 (3) variant. We use the minimum
cost multicut approach to cluster CIFAR-10 test dataset. In the particular
experiment example, the total number of clusters are 44 with a cluster
accuracy of 80.27%. The different colors represent the found class labels
while in the ground truth, there are only 10 classes on the CIFAR10 dataset
(which is shown in the legend). Any other found clusters are considered as
false positives and thus lower the cluster accuracy. However, there are in
fact 34 small clusters that contain less than 10 images. Three examples of
such mini clusters are shown in bottom left and right as well as on the top
left corner. Even though there are false positives shown in the examples of
the smaller clusters, the multicut approach explores meaningful sub-clusters
within a class label, which may be desirable on real-world scenarios. For
instance, instead of finding the class horse (in cyan), a subclass white-
horses is also found.

##  V Conclusion

In this work, we presented an extensive study on three different variations of
the Triplet Loss. Specifically, we have studied the clustering behavior of
k-means and minimum cost multicut clustering, applied to learnt embedding
spaces from three Triplet Loss formulations on the CIFAR10 [6] dataset under a
varying amount of label noise. We find that, while the traditional Triplet
Loss [1] is well suited for k-means clustering, its performance drops under
the looser assumptions made by minimum cost multicuts. We proposed a
simplification of the Triplet Loss from [5], which allows to directly compute
the probability of two data points for belonging to disjoint components. In a
line of experiments on the CIFAR-10 dataset, we show that this proposed loss
is robust against label noise in both clustering scenarios and outperformes
both previous Triplet Loss versions in terms of clustering performance and
stability.

## References

  * [1] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 815–823.
  * [2] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised deep embedding for clustering analysis,” in _International conference on machine learning_ , 2016, pp. 478–487.
  * [3] S. Chopra and M. Rao, “The partition problem,” _Mathematical Programming_ , vol. 59, no. 1–3, pp. 87–115, 1993.
  * [4] E. D. Demaine, D. Emanuel, A. Fiat, and N. Immorlica, “Correlation clustering in general weighted graphs,” _Theoretical Computer Science_ , vol. 361, no. 2–3, pp. 172–187, 2006.
  * [5] S. Zhang, Y. Gong, and J. Wang, “Deep metric learning with improved triplet loss for face clustering in videos,” in _Pacific Rim Conference on Multimedia_. Springer, 2016, pp. 497–508.
  * [6] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009.
  * [7] K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang, “Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5736–5745.
  * [8] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards k-means-friendly spaces: Simultaneous deep learning and clustering,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 3861–3870.
  * [9] F. Tian, B. Gao, Q. Cui, E. Chen, and T.-Y. Liu, “Learning deep representations for graph clustering,” in _Twenty-Eighth AAAI Conference on Artificial Intelligence_ , 2014.
  * [10] P. Ji, T. Zhang, H. Li, M. Salzmann, and I. Reid, “Deep subspace clustering networks,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 24–33.
  * [11] S. Mukherjee, H. Asnani, E. Lin, and S. Kannan, “Clustergan: Latent space clustering in generative adversarial networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 4610–4617.
  * [12] K. Ghasedi, X. Wang, C. Deng, and H. Huang, “Balanced self-paced learning for generative adversarial clustering network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4391–4400.
  * [13] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [14] M. Keuper, S. Tang, B. Andres, T. Brox, and B. Schiele, “Motion segmentation & multiple object tracking by correlation co-clustering,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 1, pp. 140–153, 2018.
  * [15] H. Sheng, Y. Zhang, J. Chen, Z. Xiong, and J. Zhang, “Heterogeneous association graph fusion for target association in multiple object tracking,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 11, pp. 3269–3280, 2018.
  * [16] P. Chu and H. Ling, “Famnet: Joint learning of feature, affinity and multi-dimensional assignment for online multiple object tracking,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6172–6181.
  * [17] R. Henschel, L. Leal-Taixé, D. Cremers, and B. Rosenhahn, “Fusion of head and full-body detectors for multi-object tracking,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 1428–1437.
  * [18] R. Henschel, Y. Zou, and B. Rosenhahn, “Multiple people tracking using body and joint detections,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [19] S. Tang, B. Andres, M. Andriluka, and B. Schiele, “Multi-person tracking by multicut and deep matching,” in _European Conference on Computer Vision_. Springer, 2016, pp. 100–111.
  * [20] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “Deepflow: Large displacement optical flow with deep matching,” in _Proceedings of the IEEE international conference on computer vision_ , 2013, pp. 1385–1392.
  * [21] S. Tang, M. Andriluka, B. Andres, and B. Schiele, “Multiple people tracking by lifted multicut and person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3539–3548.
  * [22] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [23] C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl, “Sampling matters in deep embedding learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2840–2848.
  * [24] H. Oh Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep metric learning via lifted structured feature embedding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4004–4012.
  * [25] B. Zhuang, G. Lin, C. Shen, and I. Reid, “Fast training of triplet-based deep binary embedding networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5955–5964.
  * [26] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _arXiv preprint arXiv:1703.07737_ , 2017.
  * [27] X. Dong and J. Shen, “Triplet loss in siamese network for object tracking,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 459–474.
  * [28] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [29] N. Bansal, A. Blum, and S. Chawla, “Correlation clustering,” _Machine Learning_ , vol. 56, no. 1–3, pp. 89–113, 2004.
  * [30] A. Horňáková, J.-H. Lange, and B. Andres, “Analysis and optimization of graph decompositions by lifted multicuts,” in _ICML_ , 2017.
  * [31] B. Andres, T. Kröger, K. L. Briggman, W. Denk, N. Korogod, G. Knott, U. Köthe, and F. A. Hamprecht, “Globally optimal closed-surface segmentation for connectomics,” in _ECCV_ , 2012.
  * [32] M. Keuper, E. Levinkov, N. Bonneel, G. Lavoue, T. Brox, and B. Andres, “Efficient decomposition of image and mesh graphs by lifted multicuts,” in _ICCV_ , 2015.
  * [33] A. Kardoost and M. Keuper, “Solving minimum cost lifted multicut problems by node agglomeration,” in _ACCV 2018, 14th Asian Conference on Computer Vision_ , Perth, Australia, 2018.
  * [34] T. Beier, B. Andres, U. Köthe, and F. A. Hamprecht, “An efficient fusion move algorithm for the minimum cost lifted multicut problem,” in _ECCV_ , 2016.
  * [35] T. Beier, T. Kroeger, J. Kappes, U. Kothe, and F. Hamprecht, “Cut, glue, & cut: A fast, approximate solver for multicut partitioning,” in _CVPR_ , 2014\.
  * [36] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schieke, “Deepercut: A deeper, stronger, and faster multi-person pose estimation model,” in _European Conference on Computer Vision (ECCV)_ , 2016. [Online]. Available: http://arxiv.org/abs/1605.03170
  * [37] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [38] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of machine learning research_ , vol. 9, no. Nov, pp. 2579–2605, 2008.


# BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile Screenshot
Captioning

Ching-Yu Chiang, I-Hua Chang, Shih-Wei Liao

###### Abstract

This study aims to explore efficient tuning methods for the screenshot
captioning task. Recently, image captioning has seen significant advancements,
but research in captioning tasks for mobile screens remains relatively scarce.
Current datasets and use cases describing user behaviors within product
screenshots are notably limited. Consequently, we sought to fine-tune pre-
existing models for the screenshot captioning task. However, fine-tuning large
pre-trained models can be resource-intensive, requiring considerable time,
computational power, and storage due to the vast number of parameters in image
captioning models. To tackle this challenge, this study proposes a combination
of adapter methods, which necessitates tuning only the additional modules on
the model. These methods are originally designed for vision or language tasks,
and our intention is to apply them to address similar challenges in screenshot
captioning. By freezing the parameters of the image caption models and
training only the weights associated with the methods, performance comparable
to fine-tuning the entire model can be achieved, while significantly reducing
the number of parameters. This study represents the first comprehensive
investigation into the effectiveness of combining adapters within the context
of the screenshot captioning task. Through our experiments and analyses, this
study aims to provide valuable insights into the application of adapters in
vision-language models and contribute to the development of efficient tuning
techniques for the screenshot captioning task. Our study is available at
https://github.com/RainYuGG/BLIP-Adapter

## Introduction

Recently, in this era where everyone owns a smartphone, screenshot captioning
has attracted increasing attention. This task is aimed at producing natural
language descriptions of user behaviors captured within mobile screenshots.
Without these screenshot captioning systems, users are burdened with the task
of manually describing the UI of mobile applications whenever they need to
report issues to developers, or when creating application tutorials, etc. This
process can be both time-consuming and labor-intensive. Our objective is to
investigate efficient tuning strategies tailored for the screenshot captioning
task.

Machine learning has achieved significant success in both vision and language
tasks (Dosovitskiy et al. 2021; Devlin et al. 2019; Raffel et al. 2020; Liu et
al. 2021; Yang et al. 2022; Vaswani et al. 2017). Moreover, there have been
notable advancements in vision-language tasks (Radford et al. 2021; Li et al.
2022a, 2021, b, 2023; Zhou et al. 2019, 2022), such as image-text matching,
visual question answering, and image captioning. In these frameworks, Vision-
language models, which typically utilize both a visual model and a language
model, have greatly improved due to enhanced architecture designs and the
availability of large-scale high-quality datasets (Lin et al. 2014; Deng et
al. 2009; Krishna et al. 2017; Xu et al. 2016; Chen and Dolan 2011; Agrawal et
al. 2019). Despite advancements in architectural designs, the size of modern
vision-language models is rapidly increasing, leading to substantial memory
and storage requirements. Moreover, these models typically comprise a vast
number of parameters, which poses a significant challenge for training from
scratch and consumes considerable time and computational resources. Large-
scale high-quality datasets in image captioning tasks primarily consist of
general real-world scenes, with a lack of labeled datasets specifically
tailored to special domains, such as medical images (Subramanian et al. 2020),
earth observation images (Zia, Mohsin Riaz, and Ghafoor 2022), and mobile
screenshots (Wang et al. 2021). However, collecting large-scale datasets for
every visual task is labor-intensive and prohibitively expensive to scale.
Moreover, ensuring high-quality datasets also requires human resources for
data labeling. To overcome these problems, fine-tuning the pre-trained models
has been widely adopted modernly. The models pre-trained on large-scale
datasets, such as ImageNet (Deng et al. 2009), COCO (Lin et al. 2014), and
Visual Genome (Krishna et al. 2017), are used. These models are then
subsequently fine-tuned on smaller-scale specific downstream datasets. It
serves as a solution to leverage the pre-training knowledge from large-scale
datasets and adapt it to specific tasks at hand.

However, fine-tuning the entire model can still be resource-intensive,
consuming considerable computation power, storage, and time, especially for
large vision-language models. To address this, adapter-based fine-tuning
(Houlsby et al. 2019; Guo, Rush, and Kim 2020; Chen et al. 2023a; Sung, Cho,
and Bansal 2022) has emerged as a more parameter-efficient alternative.
Similarly, the model pre-trained on large-scale datasets are used as the
backbones and the adapters with light learnable weight will be inserted into
the model. As shown in Figure 1, by using adapters, the weights in the pre-
trained models are frozen, subsequently, fine-tuning the adapters on smaller-
scale downstream datasets. we can only conduct training on additional adapters
to get the fine-tuning effect. This approach enables the model to be adapted
to specific tasks while preserving the knowledge acquired during pre-training.
Additionally, it helps mitigate the limitations related to dataset
availability, computational resources, and storage requirements that are often
encountered when training models from scratch.

Figure 1:  Illustration of the adapter-based fine-tuning approach. Through the
insertion of adapters, we can selectively fine-tune the lightweight
components, which, in our case, comprise approximately 1.47% of the model’s
total parameters.

To address the screenshot captioning task, this study explores various methods
and techniques for implementing parameter-efficient tuning and evaluates their
effectiveness within this specific context. Modifications of the state-of-the-
art vision-language model, BLIP, are explored by employing parameter-efficient
tuning methods for task-specific fine-tuning in the mobile user interface (UI)
screenshot captioning domain. The impacts of various methods are assessed,
providing a comprehensive evaluation of their respective effects.
Additionally, a modification is incorporated into the BLIP model, similar to
the implementation in previous work, by inserting an intermediate layer
between the vision and language models. The effect of this alteration, coupled
with the use of parameter-efficient methods on the language model, is
analyzed. Furthermore, this study experiments with different combinations of
parameter-efficient tuning strategies on BLIP, evaluating their efficacy for
screenshot captioning tasks. Our contributions could be summarized as:

  * •

The evaluation of various parameter-efficient tuning strategies is conducted,
applied separately to vision and language tasks, on the state-of-the-art
captioning model, BLIP.

  * •

This study presents comprehensive transfer learning research applied to
Screen2Words, a dataset specifically tailored for image captioning tasks
within the mobile UI domain.

  * •

The demonstration that applying a combination of different parameter-efficient
tuning methods can achieve performance comparable to full fine-tuning, but
only requires updating 0.08% to 1.47% of the parameters.

## Related Work

### Vision-Language Models

Vision-language models are a category of models that blend vision and language
components to tackle tasks that cross both domains. These models’
architectures may vary depending on the specific task. For instance, in image-
text matching tasks, a Siamese network architecture, featuring both a visual
encoder and a text encoder, is often preferred. Conversely, image captioning
tasks typically employ an encoder-decoder architecture, which comprises a
visual encoder and a text decoder.

Over the years, a variety of vision-language models have emerged, reflecting
advancements in architectural designs and pre-training strategies, especially
in image captioning tasks. In 2015, Vinyals et al. (Vinyals et al. 2015)
introduced an image caption generator that synergizes Convolutional Neural
Networks (Krizhevsky, Sutskever, and Hinton 2012) as a visual encoder with
Recurrent Neural Networks as a text decoder. In 2018, Anderson et al.
introduced the Bottom-Up and Top-Down Attention model (Anderson et al. 2018).
Its primary innovation lies in the utilization of Faster R-CNN (Ren et al.
2015) for object detection, enabling the achievement of bottom-up attention by
obtaining the corresponding detected targets and labels. Further, Long-Short
Term Memory (LSTM) networks (Hochreiter and Schmidhuber 1997) were leveraged
in the decoder, dynamically adjusting the focus on the input image features
according to the output language. This attention mechanism allows the model to
pay greater attention to the more salient and important objects in the image,
thus creating a better description. However, after transformer-architecture-
based models (Vaswani et al. 2017; Devlin et al. 2019) demonstrated state-of-
the-art performance in natural language processing tasks, their impact
extended to the field of vision-language models. The introduction of the
Vision Transformer by Dosovitskiy et al. in 2020 (Dosovitskiy et al. 2021)
marked a significant milestone. The Vision Transformer applies the transformer
architecture to visual data by treating images as sequences of patches. This
approach achieved competitive results in image classification tasks and
influenced the development of vision-language models. Building on these
innovations, the BLIP model, introduced by Li et al. in 2022 (Li et al.
2022b), has achieved state-of-the-art performance across multiple vision-
language tasks. It achieves this through pre-training multimodal components
such as visual and text encoders, as well as a text decoder, which are
applicable to various vision-language tasks, including image-text matching,
visual question answering, and image captioning. During pre-training, BLIP
utilizes large-scale datasets, including COCO (Lin et al. 2014), Visual Genome
(Krishna et al. 2017), Conceptual Captions (Sharma et al. 2018), Conceptual
12M (Changpinyo et al. 2021), SBU captions (Ordonez, Kulkarni, and Berg 2011),
and LAION (Schuhmann et al. 2021) for comprehensive pre-training. In image
captioning tasks, the visual encoder and text decoder components are used,
which are ViT (Dosovitskiy et al. 2021) and BERT (Devlin et al. 2019)
respectively, followed by fine-tuning on the COCO Caption dataset (Lin et al.
2014). This convergence of ViT and BERT underlines the transformative role of
the transformer architecture in propelling advancements in vision-language
models. Such progress highlights the importance of architectural innovation
and refined pre-training strategies in further advancing the field of vision-
language understanding and generation.

### Adapter Approaches

Adapters, introduced by Houlsby et al. in 2019 (Houlsby et al. 2019), are a
parameter-efficient tuning technique employed in transfer learning. This
approach involves adding lightweight, task-specific layers or modules to a
pre-trained model without altering the original parameters. Instead of fine-
tuning the entire model, it’s feasible to tune only the small-scale adapters
to achieve the fine-tuning effect. By selectively updating these adapters
while freezing the remaining parameters, parameter efficiency is achieved
without significantly compromising the model’s performance. Moreover, by only
saving the additional weights of the adapters during training, greater storage
efficiency is achieved compared to saving the weights of the entire model.
These adapters learn task-specific information while keeping the pre-trained
model’s parameters intact, thus reducing computational and storage costs and
allowing for better generalization to different tasks.

Adapters have shown significant performance not only in natural language
processing tasks (Houlsby et al. 2019; He et al. 2021; Bapna, Arivazhagan, and
Firat 2019; Pfeiffer et al. 2020; Chen et al. 2023b) but also in various
vision tasks. (Chen et al. 2023a; Pan et al. 2022; Ermis et al. 2022; Yi-Lin
Sung 2022). The bottleneck adapters, as proposed by Houlsby et al., are
inserted into the transformer architecture, specifically after the feed-
forward layers. These adapters comprise a down projection, followed by a GELU
activation function, and then an up projection. Bapna et al. (Bapna,
Arivazhagan, and Firat 2019) modify the adapter architecture for translation
tasks by incorporating an additional layer normalization and replacing the
GELU activation function with a ReLU activation function. The Compacter
architecture, proposed by Mahabadi et al. in 2021 (Karimi Mahabadi, Henderson,
and Ruder 2021), modifies the adapter structure by replacing the linear down-
and up-projections with a parameterized hypercomplex multiplication layer.
Distinct from the linear layer, this hypercomplex multiplication layer
generates its weight matrix from two smaller matrices, consequently decreasing
the total number of parameters. Additionally, these matrices can be factorized
and shared across all adapter layers. Prefix Tuning, introduced by Li et al.
in 2021 (Li and Liang 2021), innovates by incorporating new parameters within
the multi-head attention blocks in each transformer layer. Specifically, it
enhances the model by prepending trainable prefix vectors to the keys and
values of the attention head input, which adds flexibility and adaptability to
the model’s attention mechanism. Low-Rank Adaptation (LoRA), introduced by Hu
et al. in 2021 (Hu et al. 2021), incorporates trainable low-rank decomposition
matrices into the layers of a pre-trained model. Specifically, LoRA targets
the attention weights within the transformer’s self-attention sub-layers.
Instead of introducing additional parameters, BitFit (Zaken, Ravfogel, and
Goldberg 2021) simply fine-tunes the bias terms within each module, allowing
for task-specific adaptation with minimal changes to the pre-trained model. In
object detection tasks, Liu et al. introduced Explicit Visual Prompting (EVP)
in 2023 (Liu et al. 2023), which employs an architecture similar to Houlsby’s
adapters. EVP distinguishes itself by incorporating handcrafted features as
input to the adapters and utilizing shared up-projection layers within the
adapter structure. In 2022, Sung et al. introduced VL-Adapter (Yi-Lin Sung
2022), wherein they experimented with integrating various adapters into
vision-language models and evaluated their performance. They primarily applied
these adapters to video question answering tasks. In contrast to our approach,
their model features both a visual encoder and a text encoder, as well as a
text decoder, while ours does not include a text encoder. Additionally, their
models are pre-trained on general image datasets and NLP tasks, whereas our
model is specifically pre-trained for the image captioning task and we further
fine-tuned for a specialized screenshot captioning dataset.

### Mobile Screenshot Captioning

Mobile screenshot captioning is a specialized subset of image captioning that
concentrates on generating textual descriptions for mobile screenshots. This
task is notably challenging due to the distinctive traits of mobile
screenshots, including the presence of various UI elements, the absence of
fixed layouts, and a wide range of UI elements and styles. Contrary to general
image captioning, which emphasizes describing the overall scene and objects,
mobile screenshot captioning focuses on articulating the functionality and
content of the UI elements. Additionally, in mobile screenshot captioning, the
layout of UI elements is more closely linked to captioning performance.

Screen2Words, introduced by Wang et al. in 2021 (Wang et al. 2021), is the
first open-source dataset for mobile screenshot captioning. This dataset is
built upon the Rico dataset (Deka et al. 2017), which is a large-scale dataset
containing mobile app interface images, and UI layouts. Screen2Words enhances
the mobile screenshots in the Rico dataset (Deka et al. 2017) by including
human-labeled textual descriptions that correspond to the screenshots. The
dataset consists of 22,417 unique Android screenshots, each accompanied by
five concise language descriptions that convey important content and
functionality of the mobile screens. These descriptions are valuable for
various language-based application scenarios. Additionally, they offer a model
that includes a ResNet encoder and Transformer decoder, which serves to
evaluate the performance of the dataset.

## Methodology

We explored the effectiveness of various parameter-efficient tuning strategies
as applied to screenshot captioning tasks. Our goal is to illustrate how
employing a combination of parameter-efficient tuning methods can contribute
to achieving parameter-efficient tuning for our caption models while
maximizing the performance of screenshot captioning systems. The BLIP Caption
model (Li et al. 2022b) is employed as our image caption model and is fine-
tuned by the Screen2Words dataset (Wang et al. 2021) as our baseline.
Subsequently, the following parameter-efficient tuning strategies were
evaluated separately on the visual encoder and text decoder of the BLIP
Caption model. Following this, different combinations of these strategies were
employed to ascertain the most effective approach for optimizing the model
architecture and fine-tuning the components of the model to achieve desired
outcomes. The specifics of these parameter-efficient tuning approaches and the
model architecture are detailed in the following sections.

### Parameter-Efficient Tuning approaches

As our model is a vision-language model, we had explored the usage of adapters
specifically designed for the vision and language components separately. These
adapters, including the Houlsby adapter (Houlsby et al. 2019), BitFit (Zaken,
Ravfogel, and Goldberg 2021), LoRA (Hu et al. 2021), and Explicit Visual
Prompting (Liu et al. 2023), were examined both individually and in various
combinations in the experiments.

  * •

Houlsby adapter is additional bottleneck module that consist of a down-
projection, GELU activation, an up-projection, and a skip connection. In our
implementation, Houlsby adapters were integrated into the three feed-forward
layers of each transformer block in the text decoder as shown in Figure 2 (a).

  * •

BitFit stands out by not necessitating the introduction of additional modules
to the model. Instead, it focuses on fine-tuning the biases of the existing
modules within the model. In these experiments, the effectiveness of applying
BitFit to either the visual encoder or the text decoder is examined.

  * •

LoRA consists of a down-projection and an up-projection that run in parallel
with the existing linear-projection layers. In our experiments, LoRA was
integrated into the attention modules of the transformer blocks in text
decoder as shown in Figure 2 (b). The attention module typically consists of
queries, keys, and values, with LoRA being employed in the queries and values.

  * •

Explicit Visual Prompting (EVP) adapters utilize a similar architecture to
Houlsby adapter, comprising a down-projection, GELU activation, and an up-
projection. However, there are notable differences in terms of input, weight
sharing, and network connection. In the case of EVP adapters, the input
includes both the projection of the input image and task-specific information.
Furthermore, the weight of the up-projection is shared among all EVP adapters,
allowing for efficient parameter sharing. These adapters are inserted in front
of each transformer block within the visual encoder.

Figure 2:  Illustration of the adapter insertion points in the BLIP Caption
model. (a) The modifications to the BLIP Caption model architecture. (b) The
architecture of parameter-efficient method modules.

### Model Architecture Modification

As we need to insert additional modules into the model and freeze the original
weights except for those we want to fine-tune, there are several ways to
modify the model architecture.

First, experiments were conducted using these parameter-efficient tuning
methods individually, either on the visual encoder or the text decoder, to
observe the impact of each method on the vision-language model. Since EVP
adapter is specifically tailored for object detection tasks and aligns well
with the Vision Transformer architecture, it was deployed on the visual
encoder. To this end, three different handcrafted features were experimented
with as inputs to EVP adapter: the Fast Fourier Transform of the original
image, as outlined in EVP paper, the original image itself, and its grayscale
version. The inclusion of the grayscale version is particularly beneficial for
screenshots, as the semantics of most screenshot elements are color-irrelated,
which is described in the Screen2Words paper. On the other hand, given that
the other methods are primarily engineered for NLP tasks, it was considered
fitting to apply them to the text decoder. This was done to determine whether
the method modified on the visual encoder or the text decoder is more
effective for the vision-language model when choosing only one of them. The
positions of insertions or modifications for each method within the model are
depicted in Figure 2 (a).

Second, a modification similar to the model implementation of the VL-adapter
(Sung, Cho, and Bansal 2022) was implemented, which involves inserting a
visual projection between the visual encoder and the language model as shown
in Figure 2 (a). A key difference between our model and that of the VL-adapter
is the absence of a text encoder in the former. Therefore, the linear
projection layer was inserted between our visual encoder and text decoder. In
this strategy, only the visual projection and associated modules of the
methods on the text decoder were fine-tuned. Given the freeze applied to the
entire visual encoder, only the methods on the text decoder were employed.
Moreover, an experiment was conducted using the Vision Transformer block as
the visual projection instead of the linear projection layer, inspired by the
implementation of the VL-adapter. This was done to determine whether employing
a visual projection and solely fine-tuning the text decoder could outperform
fine-tuning the entire model in this particular scenario.

Finally, an attempt was made to combine the methods on both the visual encoder
and text decoder to see if the combination could lead to enhanced results. EVP
adapters were integrated into the visual encoder using various handcrafted
features, and combined with Houlsby adapter and LoRA, which were implemented
on the text decoder. As the underlying principle of BitFit is not to introduce
additional modules but rather to fine-tune the biases, it was additionally
tested across the entire model.

Through these experiments, insights were gained into how different adapters
and methods affect the vision-language model’s performance. Our findings
provide guidance for selecting the optimal approach in fine-tuning and
architecting the model to achieve the desired results in screenshot captioning
systems.

## Evaluation

We begin by introducing the experimental settings, including dataset
selection, evaluation metrics, and training procedures. Subsequent utilization
of parameter-efficient tuning methods and evaluation of various model
structure modifications associated with these methods was carried out, as
detailed in the Methodology section. Initially, the performance of the methods
when applied to the visual and language components was evaluated individually.
Thereafter, the visual encoder was frozen and only the text decoder and the
inclusion of a visual projection were trained. Finally, the integration of the
methods into both the visual and language components of the BLIP caption model
was explored. In this three attempts, the entire model with approximately 223
million parameters, which underwent complete fine-tuning, served as a baseline
for performance comparisons. This enables the assessment of how closely each
modification and the methods can approach the baseline performance. Through
these evaluations, the aim was to determine the impact of each modification
and the methods on the performance and parameter efficiency of the model. This
comprehensive analysis enabled the identification of the most effective
strategies for enhancing the screenshot captioning system.

### Experiment Settings

In our experiments, the Screen2Words dataset (Wang et al. 2021) is utilized
for training, validation, and evaluation. The dataset was split according to
the guidelines provided in its release. Each screenshot in the dataset is
accompanied by five captions. To match the number of captions and ensure
sufficient training data, the screenshots were duplicated during the training
phase, resulting in five training instances for each. During the evaluation
phase, the five captions associated with each screenshot were used as
references. To gauge the performance of the models, the BLEU-4 and CIDEr
metrics (Papineni et al. 2002; Vedantam, Lawrence Zitnick, and Parikh 2015)
were employed, both being widely used benchmarks in the field of image
captioning. All experiments were conducted using the PyTorch deep learning
framework (Paszke et al. 2019) and were performed on a single Nvidia RTX A6000
GPU. During the training process, the AdamW optimizer with a weight decay of
0.1 was utilized, and the learning rate was set to 5e-5. The batch size for
training is 32, and the models were trained for a total of 30 epochs.

### Individual Tuning of Visual and Language Components

The parameter-efficient tuning approaches were employed for the visual encoder
and text decoder separately to observe the individual effects of each approach
on our captioning task. For the text decoder, methods such as BitFit, Houlsby
adapter, and LoRA were utilized. In this scenario, the entire visual encoder
was frozen and the training was focused on the components of the method
integrated into the text decoder. Conversely, for the visual encoder, methods
such as EVP, BitFit were employed, and the entire text decoder was frozen
while training exclusively focused on the components associated with the
method on the visual encoder. Additionally, due to the use of handcrafted
feature extraction in EVP, three different handcrafted features were employed:
the Fast Fourier Transform of the original image, along with the original
image and its grayscale version. In our notation, “FT” stands for fine-tuning,
“EVP” represents Explicit Visual Prompting, “EVP-gs” indicates Explicit Visual
Prompting with grayscale extraction, and “EVP-fft” signifies Explicit Visual
Prompting with Fast Fourier Transform extraction. Additionally, “(A)” denotes
the entire model, “(V)” corresponds to the visual encoder, and “(T)” signifies
the text decoder. The “Parameters(%)” column in the table denotes the
percentage of trainable parameters in the model. The results are shown in
Table 1.

We can observe that by fine-tuning the entirety of a model component if it has
a sufficient number of parameters, the performance can either approach the
baseline or even show potential for enhancement. Both fine-tuning the entire
visual encoder and the entire text decoder could achieve more than 90% BLEU-4
score of the baseline. Notably, when solely fine-tuning the entire visual
encoder, the performance exceeds expectations, with the CIDEr score surpassing
that achieved by fine-tuning the whole model. We guess that this can be
attributed to the text decoder, which remains frozen, already having been pre-
trained for the captioning task, thus proving sufficient in generating quality
captions. Additionally, the sufficient parameters of the visual encoder are
conducive to the learning of task-specific information. When employing
parameter-efficient tuning methods individually, Houlsby adapter stands out by
achieving the highest performance, reaching approximately 91.4% of the CIDEr
score attained by fine-tuning the entire model, thereby surpassing all other
methods. Additionally, a marked decline in performance was observed when these
methods were utilized in isolation, especially if they were not applied to the
text decoder. This can be attributed to the fewer parameters being adjusted,
which suggests that methods applied solely to the visual encoder may not
possess sufficient parameters to learn task-specific information, whereas
methods applied exclusively to the text decoder are more adept at learning how
to generate captions. Therefore, it is more effective to use the methods on
the text decoder and fine-tune it to achieve satisfactory results. This
enables the model to adjust and enhance its language generation capabilities
for the specific task, leading to getting close to the performance achieved by
fine-tuning the entire model.

| BLEU-4 | CIDEr | Parameters(%)
---|---|---|---
FT (A) | 20.6 | 88.3 | 100.0
FT (V) | 18.6 | 89.6 | 38.44
FT (T) | 19.0 | 82.3 | 61.56
EVP (V) | 11.6 | 65.1 | 0.29
EVP-gs (V) | 11.7 | 64.9 | 0.29
EVP-fft (V) | 12.1 | 66.1 | 0.29
BitFit (V) | 13.4 | 67.9 | 0.05
BitFit (T) | 14.8 | 70.4 | 0.08
Houlsby (T) | 18.0 | 80.7 | 1.18
LoRA (T) | 14.5 | 74.2 | 0.26

Table 1: Performance of individual component tuning using parameter-efficient
methods.

### Text Decoder and Visual Projection Tuning

The visual encoder was frozen and focus was exclusively put on training the
text decoder and the visual projection layer as illustrated in Figure 2 (a).
BitFit, Houlsby adapter, and LoRA are utilized with a visual projection layer
to investigate the potential performance enhancements this combination might
yield. The visual projection layer employed is a linear projection layer that
maps the visual features to the same dimensionality as the original features.
Moreover, an experiment was conducted using a Vision Transformer block (ViT
block) as an alternative to the linear projection layer. The results are shown
in Table 2.

We observe that when using a visual projection layer and training both the
projection and the entire text decoder, the performance significantly
declines, with the CIDEr score dropping to 67.3, which is approximately 76.2%
of the baseline. This decline can be attributed to the visual projection layer
being initialized with zeros, which destabilizes the pre-trained weights on
the text decoder during training. In contrast, due to the frozen pre-trained
weights on the text decoder, LoRA and Houlsby adapter only fine-tune the
additional parameters, making them more stable. In this case, both LoRA and
Houlsby adapter can achieve more than 88% CIDEr score of the baseline.
Moreover, utilizing ViT blocks leads to superior performance in comparison to
using a linear projection layer. This improvement can be attributed to the ViT
block’s enhanced capability in capturing the specific information of the
intermediate visual embeddings relative to the linear projection layer.
Houlsby adapter, in particular, attains a CIDEr score of 86.0, which
represents approximately 97.4% of the baseline. Furthermore, LoRA exhibits the
highest performance, achieving a CIDEr score of 88.1, equivalent to roughly
99.8% of the baseline. In this experiment, it’s evident that the use of visual
projection can effectively transfer task-specific features to a certain
extent.

| BLEU-4 | CIDEr | Parameters(%)
---|---|---|---
FT (A) | 20.6 | 88.3 | 100.0
linear & text decoder | 15.7 | 67.3 | 61.66
linear & Houlsby | 17.9 | 81.3 | 1.44
linaer & LoRA | 16.1 | 80.5 | 0.52
ViT block & text decoder | 16.6 | 77.9 | 62.74
ViT block & Houlsby | 18.2 | 86.0 | 4.18
ViT block & LoRA | 17.9 | 88.1 | 3.31

Table 2: Performance of text decoder tuning with visual projection layer using
parameter-efficient methods.

### Entire Model Tuning

Both types of parameter-efficient tuning methods were implemented
simultaneously on the visual encoder and text decoder. For the text decoder,
Houlsby adapter and LoRA were utilized, while EVP was employed for the visual
encoder. Additionally, BitFit is exclusively tested on the entire model. Given
that BitFit’s fundamental concept does not involve the introduction of
additional modules, but instead focuses on fine-tuning biases, it was deemed
pertinent to evaluate its impact across the whole model. The results are shown
in Table 3.

By utilizing the parameter-efficient tuning methods on the whole model, where
the trainable parameters are distributed sparsely throughout the model, we
observe that the results are reasonably robust. The lowest performance
achieved in this configuration is a CIDEr score of approximately 79.2, which
is about 89.7% of the baseline. But the lowest BLEU score is 15.4, which is
only 74.8% of the baseline. This discrepancy can be attributed to the BLEU
metric focusing solely on the n-gram overlap between the generated captions
and the reference captions, whereas the CIDEr metric takes into account not
only n-grams but also the Term Frequency Inverse Document Frequency (TF-IDF)
(Robertson 2004), which measures the similarity between the generated captions
and the reference captions. The combinations of EVP with LoRA may capture some
implicit information, but may not perform as well in representing explicit
details, leading to lower BLEU scores. However, when using the combination of
EVP with Houlsby adapter, the BLEU score could acheieve 18.2, which is 88.3%
of the baseline ans the CIDEr score could reach 85.2, which is 96.5% of the
baseline.

| BLEU-4 | CIDEr | Parameters(%)
---|---|---|---
FT (A) | 20.6 | 88.3 | 100.0
BitFit (A) | 17.6 | 80.4 | 0.13
EVP & Houlsby | 18.2 | 85.2 | 1.47
EVP-gs & Houlsby | 18.2 | 84.7 | 1.47
EVP-fft & Houlsby | 18.1 | 84.0 | 1.47
EVP & LoRA | 15.4 | 79.2 | 0.55
EVP-gs & LoRA | 15.4 | 79.2 | 0.55
EVP-fft & LoRA | 15.8 | 80.7 | 0.55

Table 3: Overall performance of parameter-efficient tuning methods on entire
model.

### Discussion

From our experimental results, we can observe that the handcrafted features
used in EVP do not contribute to the enhancement of the model’s performance in
our case. The performance of EVP in this scenario is inferior to the other
methods when applied individually to the visual encoder. However, when EVP is
combined with the Houlsby adapter or LoRA, effectively dispersing updatable
parameters across the entire model, we observe a substantial improvement in
performance.

BitFit, when applied across the entirety of the model, can yield decent
results without necessitating the insertion of additional modules. However,
when employed exclusively on a partial component of the model, its
effectiveness diminishes.

LoRA, in most cases, can achieve satisfactory results, yet its performance is
not on par with that of the Houlsby adapter. These can be attributed to the
fact that LoRA is only applied to the attention module within the Transformer
block, whereas the Houlsby adapter is implemented throughout the entirety of
the Transformer block. But when LoRA is applied with a ViT block, it can
achieve the highest performance, very close to the baseline. This may because
LoRA in the cross-attention modules directly captures the intermediate visual
features during tuning, which are more task-specific, and the ViT block can
capture more information than the linear projection layer.

Houlsby adapter, whether used individually on the text decoder, applied with a
visual projection layer, or combined with EVP, almost outperforms all other
methods. Houlsby adapter, even when deployed independently, comes close to
matching the performance achieved by fine-tuning the entire text decoder.
Moreover, when it is applied with EVP, its performance approximates the
effectiveness of fine-tuning the entire model, all while only necessitating
updates to a mere 1.47% of the model’s parameters. This demonstrates its
stability and flexibility across various scenarios.

As demonstrated in Figure 4, we examine the CIDEr performance of our
experimental approaches. Individually tuning the visual encoder and text
decoder using parameter-efficient methods doesn’t yield sufficiently
satisfactory results. This could be due to an inadequate number of parameters
available to learn task-specific information, or the inability to modify the
other component of the model, thereby resulting in model instability. As
depicted in Figure 3, the BLEU scores of our combined approaches or those
using visual projection are all approximately 18, reflecting closely
comparable performances. However, implementing the VL-adapter’s model
modification with a linear visual projection layer does have some impact, but
our application of the ViT block coupled with LoRA—with only 3.31% of the
entire model’s parameters—is more effective in capturing intermediate visual
features, thus achieving a higher score. The ViT block, having a larger
parameter count than the linear projection layer, could account for its
superior performance. Notably, the ViT block’s parameters constitute
approximately 3% of the entire model’s parameters, a quantity nearly tenfold
that of the EVP’s parameters. The use of EVP with Houlsby adapter, requiring
only 1.47% of the model’s parameters, yields a substantial score, reaching
96.5% of the baseline. We further delve into comparing the generated captions
from these two top-performing methods, as outlined in the Generated Captions
section.

Figure 3:  Comparison of the overall BLEU score of parameter-efficient tuning
methods.

Figure 4:  Comparison of the overall CIDEr score of parameter-efficient tuning
methods.

## Generated Captions

* In this section, we present a collection of example captions generated by our model, compared with human labels and the outputs from the Screen2Words paper’s model in the context of screenshot captioning tasks. These examples further elucidate our model’s capabilities in generating captions, extending beyond the quantitative language scores covered in the main sections of the paper. These selections have been curated to showcase the diversity and quality of captions generated across varied scenarios and conditions. We also include a few examples where our model fails to generate captions that are as accurate as the human labels, and discuss the possible reasons for these failures. The examples provided here encompass captions generated by our implementation of EVP with the Houlsby adapter, LoRA paired with a ViT block visual projection, and the fine-tuning of the entire model. For a comparative analysis, we also include the generated captions from the model implemented in the Screen2Words paper. It’s worth noting that the model from the Screen2Words paper is a multi-modal model that uses not only images but also the text and layout information present in the screenshots as inputs.

As depicted in Figure 5, our models are able to capture the global information
but struggle to retrieve the detailed information, as evidenced in the two top
examples. as evidenced in the two top examples. The failures observed could be
attributed to the abundance of text on the screenshot and the fact that we
solely use the image as input. In the middle-left example, our two adapter-
based models exhibit a comprehensive understanding of the context as fine-
tuning the. In the middle-right example, the caption generated by LoRA with
the ViT block visual projection aligns more closely with the human label than
the captions produced by either our implementation of EVP with the Houlsby
adapter or by fine-tuning the entire model. Interestingly, the latter two
models mistakenly interpret the album as a music player. This disfference
could potentially be attributed to the LoRA’s insertion into the attention
module, enabling the model to focus more on semantic features. In the bottom-
left example, our models grasp half of the context—they can identify it as a
login page but fail to recognize which app it is. This may be attributed to
the simplicity of the login page, which consists of only a few elements in the
page. Additionally, the models may struggle to comprehend the text within the
page. In the bottom-right example, our models manage to discern that the app
is a baby care app, but do not pick up on the text information displayed on
the screen. Notably, the caption describes a baby in the app, which may be due
to the pre-training of the model on diverse datasets, including classification
datasets. This pre-training may predispose the model to concentrate on objects
present in the image.

Figure 5: Example of the generated captions

## Conclusion

This study explores the effectiveness of various parameter-efficient tuning
strategies, originally designed for both visual and language tasks, within the
context of screenshot captioning and conduct various combinations of the
methods to identify the most effective combination for the BLIP Caption model.
Through our experiments, we find that using LoRA with a ViT block as a visual
projection achieved the highest CIDEr performance, reaching about 99.8% of the
performance achieved by fully fine-tuning the model, with only 3.31% of the
model’s parameters needing tuning. Similarly, the combination of EVP on the
visual encoder and the Houlsby adapter on the text decoder reached about 96.5%
of the performance of full fine-tuning, using only 1.47% of the model’s
parameters. This underscores that both EVP with Houlsby adapter and LoRA with
ViT block are viable choices, depending on the parameter constraints. It also
demonstrates the efficacy of parameter-efficient tuning strategies in
enhancing screenshot captioning performance. These findings serve as an
important benchmark for future research in this field. Moreover, the generated
captions by these models for the test set could be a valuable resource for
further studies related to caption generation.

## Acknowledgments

This project is funded by Google’s grant to National Taiwan University
numbered ”Google-NTU-112-6-00049”.

## References

  * Agrawal et al. (2019) Agrawal, H.; Desai, K.; Wang, Y.; Chen, X.; Jain, R.; Johnson, M.; Batra, D.; Parikh, D.; Lee, S.; and Anderson, P. 2019.  Nocaps: Novel object captioning at scale.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 8948–8957.
  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6077–6086.
  * Bapna, Arivazhagan, and Firat (2019) Bapna, A.; Arivazhagan, N.; and Firat, O. 2019.  Simple, scalable adaptation for neural machine translation.  _arXiv preprint arXiv:1909.08478_.
  * Changpinyo et al. (2021) Changpinyo, S.; Sharma, P.; Ding, N.; and Soricut, R. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3558–3568.
  * Chen and Dolan (2011) Chen, D.; and Dolan, W. 2011.  Collecting Highly Parallel Data for Paraphrase Evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 190–200. Portland, Oregon, USA: Association for Computational Linguistics.
  * Chen et al. (2023a) Chen, T.; Zhu, L.; Ding, C.; Cao, R.; Wang, Y.; Li, Z.; Sun, L.; Mao, P.; and Zang, Y. 2023a.  SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More.  arXiv:2304.09148.
  * Chen et al. (2023b) Chen, Z.-C.; Fu, C.-L.; Liu, C.-Y.; Li, S.-W. D.; and Lee, H.-y. 2023b.  Exploring efficient-tuning methods in self-supervised speech models.  In _2022 IEEE Spoken Language Technology Workshop (SLT)_ , 1120–1127. IEEE.
  * Deka et al. (2017) Deka, B.; Huang, Z.; Franzen, C.; Hibschman, J.; Afergan, D.; Li, Y.; Nichols, J.; and Kumar, R. 2017.  Rico: A mobile app dataset for building data-driven design applications.  In _Proceedings of the 30th annual ACM symposium on user interface software and technology_ , 845–854.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 248–255.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Ermis et al. (2022) Ermis, B.; Zappella, G.; Wistuba, M.; Rawal, A.; and Archambeau, C. 2022.  Continual Learning With Transformers for Image Classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 3774–3781.
  * Guo, Rush, and Kim (2020) Guo, D.; Rush, A. M.; and Kim, Y. 2020.  Parameter-efficient transfer learning with diff pruning.  _arXiv preprint arXiv:2012.07463_.
  * He et al. (2021) He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; and Neubig, G. 2021.  Towards a Unified View of Parameter-Efficient Transfer Learning.  _CoRR_ , abs/2110.04366.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural computation_ , 9(8): 1735–1780.
  * Houlsby et al. (2019) Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , 2790–2799. PMLR.
  * Hu et al. (2021) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Karimi Mahabadi, Henderson, and Ruder (2021) Karimi Mahabadi, R.; Henderson, J.; and Ruder, S. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34: 1022–1035.
  * Krishna et al. (2017) Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123: 32–73.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25.
  * Li et al. (2022a) Li, D.; Li, J.; Le, H.; Wang, G.; Savarese, S.; and Hoi, S. C. H. 2022a.  LAVIS: A Library for Language-Vision Intelligence.  arXiv:2209.09019.
  * Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li et al. (2022b) Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022b.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , 12888–12900. PMLR.
  * Li et al. (2021) Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; and Hoi, S. C. H. 2021\.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34: 9694–9705.
  * Li and Liang (2021) Li, X. L.; and Liang, P. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft COCO: Common Objects in Context.  In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., _Computer Vision – ECCV 2014_ , 740–755. Cham: Springer International Publishing.  ISBN 978-3-319-10602-1.
  * Liu et al. (2023) Liu, W.; Shen, X.; Pun, C.-M.; and Cun, X. 2023.  Explicit visual prompting for low-level structure segmentations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 19434–19445.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021\.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 10012–10022.
  * Ordonez, Kulkarni, and Berg (2011) Ordonez, V.; Kulkarni, G.; and Berg, T. 2011.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ , 24.
  * Pan et al. (2022) Pan, J.; Lin, Z.; Zhu, X.; Shao, J.; and Li, H. 2022.  ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition.  _arXiv preprint arXiv:2206.13559_.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 311–318.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Pfeiffer et al. (2020) Pfeiffer, J.; Rücklé, A.; Poth, C.; Kamath, A.; Vulić, I.; Ruder, S.; Cho, K.; and Gurevych, I. 2020.  AdapterHub: A Framework for Adapting Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 46–54.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1): 5485–5551.
  * Ren et al. (2015) Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Robertson (2004) Robertson, S. 2004.  Understanding inverse document frequency: on theoretical arguments for IDF.  _Journal of documentation_ , 60(5): 503–520.
  * Schuhmann et al. (2021) Schuhmann, C.; Vencu, R.; Beaumont, R.; Kaczmarczyk, R.; Mullis, C.; Katta, A.; Coombes, T.; Jitsev, J.; and Komatsuzaki, A. 2021.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2556–2565.
  * Subramanian et al. (2020) Subramanian, S.; Wang, L. L.; Mehta, S.; Bogin, B.; van Zuylen, M.; Parasa, S.; Singh, S.; Gardner, M.; and Hajishirzi, H. 2020.  Medicat: A dataset of medical images, captions, and textual references.  _arXiv preprint arXiv:2010.06000_.
  * Sung, Cho, and Bansal (2022) Sung, Y.-L.; Cho, J.; and Bansal, M. 2022.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5227–5237.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vedantam, Lawrence Zitnick, and Parikh (2015) Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4566–4575.
  * Vinyals et al. (2015) Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3156–3164.
  * Wang et al. (2021) Wang, B.; Li, G.; Zhou, X.; Chen, Z.; Grossman, T.; and Li, Y. 2021.  Screen2words: Automatic mobile UI summarization with multimodal learning.  In _The 34th Annual ACM Symposium on User Interface Software and Technology_ , 498–510.
  * Xu et al. (2016) Xu, J.; Mei, T.; Yao, T.; and Rui, Y. 2016.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5288–5296.
  * Yang et al. (2022) Yang, C.; Qiao, S.; Yu, Q.; Yuan, X.; Zhu, Y.; Yuille, A.; Adam, H.; and Chen, L.-C. 2022.  Moat: Alternating mobile convolution and attention brings strong vision models.  _arXiv preprint arXiv:2210.01820_.
  * Yi-Lin Sung (2022) Yi-Lin Sung, M. B., Jaemin Cho. 2022.  VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks.  In _CVPR_.
  * Zaken, Ravfogel, and Goldberg (2021) Zaken, E. B.; Ravfogel, S.; and Goldberg, Y. 2021.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  _arXiv preprint arXiv:2106.10199_.
  * Zhou et al. (2022) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9): 2337–2348.
  * Zhou et al. (2019) Zhou, L.; Palangi, H.; Zhang, L.; Hu, H.; Corso, J. J.; and Gao, J. 2019.  Unified Vision-Language Pre-Training for Image Captioning and VQA.  _CoRR_ , abs/1909.11059.
  * Zia, Mohsin Riaz, and Ghafoor (2022) Zia, U.; Mohsin Riaz, M.; and Ghafoor, A. 2022.  Transforming remote sensing images to textual descriptions.  _International Journal of Applied Earth Observation and Geoinformation_ , 108: 102741.