# Hack The Box: Fooling Deep Learning Abstraction-Based Monitors

Sara Hajj Ibrahim1 Mohamed Nassar1,2
1 American University of Beirut (AUB)
2 University of New Haven
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep learning is a type of machine learning that adapts a deep hierarchy of
concepts. Deep learning classifiers link the most basic version of concepts at
the input layer to the most abstract version of concepts at the output layer,
also known as a class or label. However, once trained over a finite set of
classes, some deep learning models do not have the power to say that a given
input does not belong to any of the classes and simply cannot be linked.
Correctly invalidating the prediction of unrelated classes is a challenging
problem that has been tackled in many ways in the literature. Novelty
detection gives deep learning the ability to output ”do not know” for
novel/unseen classes. Still, no attention has been given to the security
aspects of novelty detection. In this paper, we consider the case study of
abstraction-based novelty detection and show that it is not robust against
adversarial samples. Moreover, we show the feasibility of crafting adversarial
samples that fool the deep learning classifier and bypass the novelty
detection monitoring at the same time. In other words, these monitoring boxes
are hackable. We demonstrate that novelty detection itself ends up as an
attack surface.

##  1 Introduction

Machine learning algorithms are excellent at analyzing data and finding
interesting patterns. However, they give up to the so-called dimensionality
curse. It was shown that deep learning bypasses the traditional machine
learning algorithms in most learning tasks in the literature Goodfellow et al.
(2016); Nassar (2020). While deep learning yields remarkable results in the
field of raw data representation and classification, it suddenly becomes sub-
optimal when explaining decisions or recognizing a novel class of input. In
their default settings, supervised deep neural networks never say ”I don’t
know”, they can be just less or more confident about an outcome or decision.
The necessity of monitoring deep learning for novelty and anomaly detection is
directly visible.

Novelty detection can play a significant role and be leveraged for monitoring
and discovering new classes that were unseen during training time. However,
most work on novelty detection give no attention to its security aspects. In
this paper, we show that from a security perspective, novelty detection can be
easily attacked and may augment the attack surface of deep learning based
systems.

We distinguish between three interrelated and very close concepts, namely
anomaly detection, outlier detection and novelty detection. These terms are
sometimes interchangeably used in literature, but we suggest that they mean
different things and it is time to give each an appropriate definition. In our
terminology, anomaly detection stems from unsupervised one-class modeling or
supervised binary classification into normal and abnormal. Outlier detection
stems from unsupervised learning and consists on finding points that likely do
not belong to any clusters found in unlabeled data.

Novelty detection is the process of distinguishing between data inputs
belonging to one of the classes encountered during the training time and data
inputs belonging to classes that are previously unseen. It is different than
outlier detection since data have labels. It is also different from anomaly
detection since it is parameterized by both the data and the classifier
whereas anomaly detection is usually solely parameterized by the data. Novelty
detection in deep learning is a new and active research area. Abstraction-
based novelty detection is one of the main proposed approaches. This approach
summarizes training input and intermediate data representations into
statistical constructs that makes it easy to detect novelty at the testing
stage.

A white-box abstraction-based novelty detection method is proposed in
Henzinger et al. (2020). A monitor takes a one-by-one decision on each testing
sample and identifies it as either valid (i.e. the classifier prediction is
correct on assigning the sample to one of the training classes), or invalid
(i.e. the classifier prediction is rejected). The monitor decision is based on
verifying whether a special representation of the sample falls within one of
the boxes constructed during training or outside these boxes. This special
representation is based on values taken from internal neural nodes at hidden
layers. Each class has its own box or set of boxes. We take these monitors as
a case-study in this paper and show that: (1) these monitors are not efficient
when adversarial testing samples are presented, and (2) these monitors can
themselves be attacked by appending the adversarial generation process with
new constraints. In other words, these boxes are hackable.

The remaining of this paper is organized as follows: Section 2 summarizes our
terminology and literature review. Our attack methodology is presented in
section 3. Section 4 evaluates our experiments and findings. Finally Section 5
concludes the paper and sketches future work.

##  2 Background & Literature Review

First let’s be more precise about what is novelty detection and how it differs
from outlier and anomaly detection. An outlier is an ”Observation which
deviates so much from other observations as to arouse suspicion it was
generated by a different mechanism” Hawkins et al. (1980). Outlier detection
is the process of coining observations that significantly deviate from the
majority of data. Unsupervised algorithms extract statistical information
indicating how unlikely a certain observation is to occur, for example,
finding a point deviating far from the statistical means of other points as
illustrated in Figure 1.

Figure 1: An example of an outlier detection system.

An anomaly is a special case of outliers which is usually tied to special
information or reasons Aggarwal (2015). Anomalies indicate significant and
rare events that may prompt critical actions in a wide range of application
domains Ahmed et al. (2016). Anomaly detection may require labeled data and
employ supervised algorithms as illustrated in Figure 2. For example, we
consider the problem of malware/benign classification as a form of anomaly
detection.

Figure 2: An example of an anomaly detection system.

Novelty detection is the process of identifying inputs that belong to unknown
classes that were not provided during training time. Consider a supervised
learner having $c$ classes at training time but $c+u$ classes appear at
testing time. The goal of novelty detection is to invalidate the output of the
classification when samples from the $u$ classes are presented. Novelty
detection is different than the previously described anomaly and outlier
detection for two main reasons: (1) training data have labels, and (2) the
learner itself is an input to the detector algorithm.

Novelty detection can be achieved in white-box mode by taking the model
obtained after training and building a monitor on top of it. The monitor
fingerprints the behaviour of the model when training data are presented. For
the special case of deep learning, such a monitor can register the values of
hidden nodes given by forward-propagating the training samples. The monitored
values are abstracted into statistical constructs. Later, outlier detection
flag inputs having fingerprints that largely deviate from these constructs. In
other words, this approach transforms the novelty detection problem into an
outlier detection problem by projecting the data into a new hyper-dimensional
space. This projection is parameterized by the neural network model itself.
Different types of abstractions were proposed in Henzinger et al. (2020)
before concentrating on the evaluation of box abstractions in particular.
Figure 3 shows an example of a box abstraction-based novelty detection system.

(a)

(b)

Figure 3: An example of a novelty detection system based on box abstractions.

Next, we summarize the main approaches for addressing novelty detection in
deep learning from the literature.

Distance based methods

These methods compute novelty values or confidence scores based on distance
metric functions. In Mandelbaum and Weinshall (2017) the data are first
embedded as derived from the penultimate layer of the neural network. The
confidence score is based on the estimation of local density. Local density at
a point is estimated based on the Euclidean distance in the embedded space
between the point and its $k$ nearest neighbors in the training set. A similar
approach based on learning a local model around a test sample is proposed on
Bodesheim et al. (2015) for Multi-class novelty detection tasks in image
recognition problems.

Statistical Based Methods

Novelties are caused by differences in data distributions at training and
prediction time. Some of these methods require sampling the distribution at
run-time or an online adaptation of classifiers. In Pidhorskyi et al. (2018)
the underlying structure of the inlier distribution of the training data is
captured. The novelty is detected by the means of a hypothesis test or by
computing a novelty probability value.

Auto-Encoding and Reconstruction Based Methods

One way to proceed is to train a deep encoder-decoder network that outputs a
reconstruction error for each sample. The error is used to either compute a
novelty score or to train a one-class classifier. Pidhorskyi et al. (2018)
also uses an auto-encoder network but to derive a linearized manifold
representation of the training data. The manifold representation helps compute
a novelty probability that represents how likely it is that a sample was
generated by the inlier distribution. This is why we consider it as a
statistical based method in the same time.

Domingues et al. (2018) introduces an unsupervised model for novelty detection
based on Deep Gaussian Process Auto-Encoders (DGP-AE). The proposed auto-
encoder is trained by approximating the DGP layers using random feature
expansions, and by performing stochastic variational inference on the
resulting approximate model. Their work can be categorized under anomaly
detection in our terminology.

Bayesian methods

These methods use Bayesian formalism to detect anomalies and new classes in
addition to classification Roberts et al. (2019). The basic idea is to add a
”dummy” class at the root node. The class is considered under-represented in
the training set. The classifier gives a strong a posterior of being ”dummy”
for unseen instances.

Abstraction based methods

These methods consider a finite set of vectors $X$, and construct a set $Y$
that generalizes $X$ to infinitely many elements and has a simple
representation that is easy to manipulate and answer queries for. Examples of
these methods are ball-abstraction such as one-class support vector machines,
one-class neural networks Chalapathy et al. (2018) and box-abstraction
Henzinger et al. (2020).

However, little or none work has been conducted on the security of the
aforementioned approaches either in the presence of adversarial samples for
fooling the classifier or against especially crafted samples for fooling the
novelty detector and the classifier at the same time. For our study, we
consider the use-case of ”outside the box” Henzinger et al. (2020) and analyse
its security in several ways.

##  3 Methodology

In Henzinger et al. (2020), constructing an abstraction at layer $l$ of the
monitored network for class $y$ works as follows:

  1. 1.

Collect outputs at layer $l$ for inputs of class $y$.

  2. 2.

Divide collected vectors into clusters.

  3. 3.

Construct an abstraction for vectors in each cluster, e.g. an enclosing box.

Monitoring at layer $l$ works as follows:

  1. 1.

Predict class of input $\mathbf{x}$.

  2. 2.

Collect output at layer $l$ into a vector $\mathbf{v}$.

  3. 3.

Check if any of the abstraction of the predicted class contains $\mathbf{v}$.

  4. 4.

The prediction is rejected if the check returns empty.

We distinguish two types of attacks against this schema:

Attack 1 - from valid to invalid

Consider an input $\mathbf{x}$ which would normally be identified as valid by
the monitor and belongs to one of the training classes. This attack modifies
$\mathbf{x}$ in a slight and unperceivable way to make it get rejected by the
monitor. As an example of application of this attack, we would imagine a
denial of service for a legitimate user in a face recognition system.

Attack 2 - from invalid to valid

Consider an input $\mathbf{x}$ which would normally be rejected by the monitor
as not belonging to any of the classes seen during training. This attack
modifies $\mathbf{x}$ in a slight and unperceivable way to make it get
accepted by the monitor. The attack can be targeting a preset prediction, or
just going with any prediction output by the neural network. As an example of
application, we would imagine letting go an intruder in a face recognition
system. The intruder is identified as any of the legitimate white-list users.

In terms of implementation, we propose to formulate each attack as an
optimisation problem that can be solved by an iterative optimisation
algorithm. We adapt an idea that has been widely studied in the literature
(e.g. Zügner et al. (2018)): introducing a perturbation within a small
epsilon-ball or budget to maximize the confusion of the target function. We
also experiment with off-the-shelf adversarial attacks against neural networks
and assess their efficiency, as well as augmenting them by an optimisation
component to target a specific attack and make them more efficient. We detail
these approaches next.

###  3.1 Optimisation based attacks

#### Attack 1: from valid to invalid

Consider a neural network that is trained over only two classes, where each
class is represented by the monitor under one box. As shown in Figure 4($a$),
we push the images of valid points, as represented by the monitor, from both
classes  and  to fall outside their boxes and therefore be marked as novel
exactly as for the $\boldsymbol{\star}$ points.

(a)

(b)

Figure 4: Examples of attack types. Projections using two arbitrary dimensions
are shown.

More generally, consider a point $\mathbf{x^{0}}$ such as
$\text{monitor}(\mathbf{x^{0}},c)=1$ (accept as class $c$), our goal is to
find a point $\mathbf{x}$ as close as possible to $\mathbf{x^{0}}$ such that
$\text{monitor}(\mathbf{x},c)=0$ (reject). For measuring the distance between
the two points we either use the $L_{1}$ norm or the $L_{2}$ norm. In
addition, we require that $\mathbf{x}$ preserves the same prediction as of
$\mathbf{x^{0}}$ via the monitored network:
$\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})=c$.

In case of the $L_{1}$ norm, we replace the non-differentiable objective
function by a differentiable one through introducing a vector $\mathbf{z}$
having the same dimension as $\mathbf{x}$. We get a linear programming problem
as follows:

Minimize $\displaystyle\sum_{i=1}^{n}z_{i}$ (1) Subject to $\displaystyle
z_{i}+(x_{i}-x_{i}^{0})\geq 0$ $\displaystyle z_{i}-(x_{i}-x_{i}^{0})\geq 0$
$\displaystyle\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=0$
$\displaystyle\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})$

We next use the $L_{2}$ norm
(${\left\|\mathbf{x}-\mathbf{x^{0}}\right\|}_{2}$) to formulate a second
optimisation problem. The constraints for $L_{2}$ are same as for $L_{1}$
except that $L_{2}$ norm is already a differential objective function and can
be directly tuned by a linear solver. The $L_{2}$ norm attack focuses on
minimizing the square root of the sum of squared differences of
($\mathbf{x^{0}}$) and ($\mathbf{x}$) elements. We then get a linear
programming problem as follows:

Minimize $\displaystyle\sqrt{\sum_{i=1}^{n}{|x_{i}-x_{i}^{0}|}^{2}}$ (2)
Subject to
$\displaystyle\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=0$
$\displaystyle\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})$

#### Attack 2: from invalid to valid

In this attack, we push the images of invalid points $\boldsymbol{\star}$, as
represented by the monitor, towards the boxes representing legitimate classes.
The points will be marked by the monitor as valid if, at the same time, the
neural network prediction matches the box owner, either as  or . The 2d
projection of the monitor representation for a binary classifier is shown in
Figure 4($b$).

Given a point $\mathbf{x^{0}}$ where $\text{monitor}(\mathbf{x^{0}},c)=0$
(reject), our goal is to find a point $\mathbf{x}$ as close as possible to
$\mathbf{x^{0}}$ such that
$\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=1$ (accept as same
class of $\mathbf{x^{0}}$). For measuring the distance between the two points
we choose the $L_{1}$ norm or the $L_{2}$ norm.

For the $L_{1}$ norm, we formulate the following problem:

Minimize $\displaystyle\sum_{i=1}^{n}z_{i}$ (3) Subject to $\displaystyle
z_{i}+(x_{i}-x_{i}^{0})\geq 0$ $\displaystyle z_{i}-(x_{i}-x_{i}^{0})\geq 0$
$\displaystyle\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=1$
$\displaystyle\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})$

For the $L_{2}$ norm, we formulate the following problem:

Minimize $\displaystyle\sqrt{\sum_{i=1}^{n}{|x_{i}-x_{i}^{0}|}^{2}}$ (4)
Subject to
$\displaystyle\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=1$
$\displaystyle\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})$

The four formulated problems can be efficiently solved by constrained
optimisation numerical methods that are either local such as SLSQP Kraft and
others (1988) and COBYLA Powell (1994) or global such as Differential
Evolution (DE) Price (2013) and SHGO Endres et al. (2018). We used
implementations from the SciPy library Virtanen et al. (2020) to show case
these attacks as will be detailed later in section 4.

###  3.2 Adversarial attacks against neural networks

Another idea is to use known adversarial attacks against neural networks as a
starting point for attacking the monitor. Adversarial neural network attacks
aim at changing the prediction of an input $\mathbf{x^{0}}$ when replaced by a
close point $\mathbf{x}$ as of
$\text{predict}(\mathbf{x})\neq\text{predict}(\mathbf{x^{0}})$. This can be
done in white-box mode by taking a step in the opposite direction of the
gradient of the objective function at the point $\mathbf{x^{0}}$. It is
interesting to check whether adversarial samples would be detected as novel by
the novelty monitor. Whether should or must they be detected as novel is even
a more puzzling question. In case these samples are rejected because of the
novelty detector, hence making the attack fail, we may consolidate them by
optimisation based methods to make them pass as valid points. In other words,
we can take an adversarial sample as a starting point for our search for an
attack against both the neural network and the monitor. Our goal here is to
find a point $\mathbf{x}$ very close to $\mathbf{x^{0}}$ such that
$\text{predict}(\mathbf{x})\neq\text{predict}(\mathbf{x^{0}})$ and
$\text{monitor}(\mathbf{x},\text{predict}(\mathbf{x}))=1$ (accept).

We use the implementation of adversarial attacks from the Foolbox library
Rauber et al. (2020). We assess the performance of an abstraction based
monitor to flag adversarial samples as novel. We study the effect of tuning
”outside the box” parameters to enhance the robustness of the monitor. In a
second time, we use optimisation techniques to force adversarial samples to
bypass the monitor.

Figure 5: Success rate of different optimisation based methods for the four
proposed attacks with different network architectures.

(a) Original

(b) Novel

Figure 6: Example of an unsuccessful attack / noisy sample obtained by the DE
method.

##  4 Experiments

###  4.1 Optimisation solvers experiment

In this experiment, we evaluate the performance of different optimisation
solvers in solving the four proposed optimisation problems and successfully
generating adversarial samples. We evaluate four solvers from SciPy: COBYLA,
SLSQP, SHGO and DE. We experiment with several network architectures, the
following in particular:

  * •

Two very simple neural networks: a XOR-like circuit and another inspired from
https://playground.tensorflow.org/.

  * •

Two MNIST classifiers trained over a subset of the ten available classes. The
first one is trained over two classes, and the eight others are considered
novel. The second one is trained over three classes, and the seven others are
considered novel.

The novelty monitor follows ”outside the box” paradigm. For each network, we
tested over 20 random $\mathbf{x^{0}}$ samples and counted the times where the
optimiser found a solution, i.e. generated an effective attack point
$\mathbf{x}$. Results are shown in Figure 5.

(a) Before SHGO Optimisation Attack. Samples are decided as not novel.

(b) After SHGO Optimisation Attack. Samples are decided as novel.

Figure 7: Adversarial image examples obtained by SHGO method.

(a)

(b)

Figure 8: $L_{1}$-norm optimisation attacks using SHGO method.

Results show that local optimisation methods such as SLSQP and COBYLA were
unsuccessful when applied to the MNIST classifiers. We attribute this to the
$28*28$ dimensionality of the images. Changing many pixels result in a
relatively large distance from the original point, which made the search very
narrow around the given starting point. Global optimisation methods (SHGO, DE)
were much more successful. DE has more successful attempts than SHGO in some
cases, but the generated image samples appeared very noisy making the attack
easily perceived by a human observer (for an example, see Figure 6). SHGO is
the best method in terms of success rate and preserving the original digit
shape.

Figure 7 illustrates examples of MNIST images before and after ”Attack 1” type
(from valid to invalid). The classifier and monitor were trained over the ten
classes in this experiment.

Note that even that our objective function is linear, our constraints such as
$\text{monitor}(\mathbf{x})=0/1$ and
$\text{predict}(\mathbf{x})=\text{predict}(\mathbf{x^{0}})$ are strongly non-
linear, which hinders the task of the used linear solvers. SHGO shined since
it is a derivative-free optimiser that is most appropriate for black box
functions and leverages input/output pairs. Another comparison factor is the
optimiser runtime. We recorded large run-times ($10-15$ hours) for most
optimisers during the experiment using a 16GB RAM computer and a processor of
2.60 GHz frequency. However, SHGO converged in matter of few minutes. Figure 8
shows a 2d projection of successful generated examples using SHGO and the
$L_{1}$-norm formulation for both types of attacks.

###  4.2 Adversarial attack experiment

Adversarial attacks on deployments of neural networks are regularly published
Ibitoye et al. (2019); Aiken and Scott-Hayward (2019); Papadopoulos et al.
(2021). Let us consider the case of adversarial samples that are supposed to
fool the classifier decision. An important question is whether these samples
should be flagged as novelty or not. In principle, the goal of novelty
detection is not to detect and eradicate adversarial samples. It might be
reasonable to have the monitor accept the adversarial samples since they are
very close to their original counterparts. However, if an adversarial sample
is detected as novel, the classifier output will be rejected and the attack
will be considered a failure. Therefore, the goal of the attacker becomes to
bypass the novelty detection and fool the classifier in the same time.

In this experiment, we consider a neural network classifier trained over all
the classes of the MNIST dataset. ”Outside the box” novelty detection was not
designed with the goal of detecting adversarial samples. However, we check
whether such samples would be detected as novel or not. Over 100 MNIST
samples, Figure 9 shows the number of successful attacks (i.e., the prediction
was successfully flipped) and the number of successful attacks considered as
valid (i.e. the monitor validates the prediction considering the sample as not
novel). Results show that most of these samples succeeded into fooling the
monitor in addition to fooling the classifier. The initial environment
properties of the detector were used, hence a single cluster/box for each
class and $0$ tolerance. Increasing the number of clusters per class from only
one to 2 or 3 clusters per class has a positive impact on invalidating
adversarial samples. Conversely, increasing the tolerance factor from zero to
$0.1$ or $0.25$ resulted in accepting more adversarial samples.

Figure 9: Experiment measuring the number of successful and valid attack
attempts using neural network attacks.

(a)

(b)

Figure 10: Fooling the classifier and the monitor together.

Furthermore, the adversarial samples that were successfully detected as
invalid can undergo one of our optimisation attacks to pass the monitoring
test. For instance, we ran the $L_{1}$ norm ”Attack 2” using SHGO on top of
the Fast Gradient Sign Method (FGSM) attack Goodfellow et al. (2014) to
achieve $100\%$ attack success over 300 MNIST samples. Of course, we had to
suppress the $\text{predict}(\mathbf{x})\neq\text{predict}(\mathbf{x^{0}})$
constraint.

Figure 10 shows a 2d projection of the adversarial samples as per the monitor
definition. Before the optimisation attack, there are still adversarial
samples marked as $\boldsymbol{\star}$ points. After the attack, all
$\boldsymbol{\star}$ points disappear.

##  5 Conclusion and Future Work

In this paper, we demonstrated that novelty detection monitors are vulnerable
to fooling attacks. We were successfully able to mislead the monitor using
multiple methods. We formulated optimisation problems that can be solved
efficiently to find attack vectors. We also leveraged adversarial neural
networks attacks from the literature to fool the classifier and the monitor at
the same time. Adversarial neural network attacks combined with optimisation
techniques are shown to be a deadly combo.

The message of the paper is that security by design should be a requirement
for new novelty detection systems in deep learning, especially in critical
systems. We envision exploring ways to defend novelty detection against
adversarial attacks. In future work, we aim at proposing efficient defense
mechanisms for novelty detection monitors against both monitor fooling and
classifier-monitor fooling attacks.

## Acknowledgments

The authors would like to thank Mohamed Jaber and Yliès Falcone for the
introduction and the early discussion of the topic. The authors thank also the
anonymous reviewers for their valuable feedback.

## References

  * Aggarwal (2015) Charu C. Aggarwal.  Outlier Analysis, pages 237–263.  Springer International Publishing, Cham, 2015.
  * Ahmed et al. (2016) Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu.  A survey of network anomaly detection techniques.  Journal of Network and Computer Applications, 60:19–31, 2016.
  * Aiken and Scott-Hayward (2019) James Aiken and Sandra Scott-Hayward.  Investigating adversarial attacks against network intrusion detection systems in sdns.  In 2019 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), pages 1–7. IEEE, 2019.
  * Bodesheim et al. (2015) Paul Bodesheim, Alexander Freytag, Erik Rodner, and Joachim Denzler.  Local novelty detection in multi-class recognition problems.  In 2015 IEEE Winter Conference on Applications of Computer Vision, pages 813–820. IEEE, 2015.
  * Chalapathy et al. (2018) Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla.  Anomaly detection using one-class neural networks.  arXiv preprint arXiv:1802.06360, 2018.
  * Domingues et al. (2018) Rémi Domingues, Pietro Michiardi, Jihane Zouaoui, and Maurizio Filippone.  Deep gaussian process autoencoders for novelty detection.  Machine Learning, 107(8-10):1363–1383, 2018.
  * Endres et al. (2018) Stefan C Endres, Carl Sandrock, and Walter W Focke.  A simplicial homology algorithm for lipschitz optimisation.  Journal of Global Optimization, 72(2):181–217, 2018.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * Hawkins et al. (1980) D Hawkins, A Jain, R Dubes, et al.  Identification of outliers chapman and hall.  London:[Google Scholar], 1980.
  * Henzinger et al. (2020) Thomas A. Henzinger, Anna Lukina, and Christian Schilling.  Outside the Box: Abstraction-Based Monitoring of Neural Networks, volume 325 of Frontiers in Artificial Intelligence and Applications.  IOS Press, 2020.
  * Ibitoye et al. (2019) Olakunle Ibitoye, Omair Shafiq, and Ashraf Matrawy.  Analyzing adversarial attacks against deep learning for intrusion detection in iot networks.  In 2019 IEEE Global Communications Conference (GLOBECOM), pages 1–6. IEEE, 2019.
  * Kraft and others (1988) Dieter Kraft et al.  A software package for sequential quadratic programming.  DFVLR Obersfaffeuhofen, Germany, 1988.
  * Mandelbaum and Weinshall (2017) Amit Mandelbaum and Daphna Weinshall.  Distance-based confidence score for neural network classifiers.  arXiv preprint arXiv:1709.09844, 2017.
  * Nassar (2020) Mohamed Nassar.  Deep Learning Handbook.  Zenodo, 2020.  http://mnassar.github.io/deeplearninghandbook.
  * Papadopoulos et al. (2021) Pavlos Papadopoulos, Oliver Thornewill von Essen, Nikolaos Pitropakis, Christos Chrysoulas, Alexios Mylonas, and William J Buchanan.  Launching adversarial attacks against network intrusion detection systems for iot.  Journal of Cybersecurity and Privacy, 1(2):252–273, 2021.
  * Pidhorskyi et al. (2018) Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto.  Generative probabilistic novelty detection with adversarial autoencoders.  In Advances in neural information processing systems, pages 6822–6833, 2018.
  * Powell (1994) Michael JD Powell.  A direct search optimization method that models the objective and constraint functions by linear interpolation.  In Advances in optimization and numerical analysis, pages 51–67. Springer, 1994.
  * Price (2013) Kenneth V Price.  Differential evolution.  In Handbook of optimization, pages 187–214. Springer, 2013.
  * Rauber et al. (2020) Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel.  Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax.  Journal of Open Source Software, 5(53):2607, 2020.
  * Roberts et al. (2019) Ethan Roberts, Bruce A Bassett, and Michelle Lochner.  Bayesian anomaly detection and classification.  arXiv preprint arXiv:1902.08627, 2019.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, et al.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17:261–272, 2020.
  * Zügner et al. (2018) Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann.  Adversarial attacks on neural networks for graph data.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2847–2856, 2018.


# Cross-Domain Image Captioning with Discriminative Finetuning

Roberto Dessì
Meta AI / UPF
Michele Bevilacqua
Samaya AI
Eleonora Gualdoni
UPF
Nathanaël Carraz Rakotonirina
UPF
Francesca Franzon
UPF
Marco Baroni
UPF / ICREA

###### Abstract

Neural captioners are typically trained to mimic human-generated references
without optimizing for any specific communication goal, leading to problems
such as the generation of vague captions. In this paper, we show that fine-
tuning an out-of-the-box neural captioner with a self-supervised
discriminative communication objective helps to recover a plain, visually
descriptive language that is more informative about image contents. Given a
target image, the system must learn to produce a description that enables an
out-of-the-box text-conditioned image retriever to identify such image among a
set of candidates. We experiment with the popular ClipCap captioner, also
replicating the main results with BLIP. In terms of similarity to ground-truth
human descriptions, the captions emerging from discriminative finetuning lag
slightly behind those generated by the non-finetuned model, when the latter is
trained and tested on the same caption dataset. However, when the model is
used without further tuning to generate captions for out-of-domain datasets,
our discriminatively-finetuned captioner generates descriptions that resemble
human references more than those produced by the same captioner without
finetuning. We further show that, on the Conceptual Captions dataset,
discriminatively finetuned captions are more helpful than either vanilla
ClipCap captions or ground-truth captions for human annotators tasked with an
image discrimination task.111Our code is available at
https://github.com/facebookresearch/EGG/tree/main/egg/zoo/discriminative_captioner.

Figure 1: Setup of our discriminative finetuning method when applied to the
ClipCap captioner [28]. All CLIP encoders are frozen, while the language
generation modules (mapper and GPT-2) are updated based on reward values.

##  1 Introduction

The last decade has seen impressive progress on the task of automatically
generating image descriptions with deep neural networks [46, 48, 5, 41]. Most
of the proposed methods try to optimize the similarity of system-produced
captions with ground-truth human references, either through a standard cross-
entropy cost function [3], or by maximizing natural-language-generation (NLG)
metrics such as CIDEr [38, 33] through a reward-based objective. While
imitating human captions is a reasonable goal, it does not take into account
that, in concrete applications, an image description is produced for a purpose
[15, 21].

There are a multitude of context-dependent purposes a description might be
produced for, but a fundamental one is to correctly characterize an object so
that a hearer could discriminate it from other contextual elements [18]. This
ability to discriminate between referents is a core purpose of communication,
playing a fundamental role in its evolution and acquisition (e.g., [40, 6]).
We study here what happens when we take an out-of-the-box image captioner that
was trained to imitate human captions, and finetune its language components
with a discriminative objective using reinforcement learning. In particular,
we let the captioner play a discrimination game with an out-of-the-box
caption-based image retriever. The captioner generates a caption given a
target image, and the retriever (whose weights are not updated) uses the
caption to select the target among a set of candidates, as shown in Fig. 1.
This finetuning technique does not require annotated data (only a set of
images), and it’s agnostic to the underlying captioner and retriever
components.

We report two strong and novel results. First, we show that captions finetuned
in this way lead to better 0-shot cross-domain caption generation.222We note
that in the earlier literature, cross-domain captioning can rely on sets of
unpaired images and captions from the target domain (e.g., [7, 49, 52]). We
consider here a more challenging and realistic 0-shot transfer task in which
no captions from the target domain are available. Second, not only are the
finetuned captions good for neural text-based image retrieval (both in- and
across-domain), but they can also be more useful to human annotators, helping
discriminate the target from distractors more than human-generated ground-
truth captions do. We conclude the paper with an analysis of the finetuned
captions, comparing them to human-generated and non-finetuned ones from the
Conceptual Captions dataset. We find that discriminative finetuning undoes the
more abstract language that the underlying system had learned from the ground-
truth captions, leading to a more plainly descriptive style that we expect to
be more useful in practical applications.

##  2 Related Work

In recent years, deep learning has led most progress in image captioning [3,
12]. While early approaches relied on supervised learning to maximize the
likelihood of the model against human references[3, 46], other methods have
tried to maximize a reward based on standard language generation metrics [37,
36]. Our work belongs to a third tradition, exploring the idea of caption
learning or finetuning with a reward-based objective that is not (only) based
on a comparison with reference captions. Among the approaches more closely
related to ours, the system of Yu et al. [51] uses a ClipCap-like system for
caption generation, and CLIP to measure image-caption similarity, focusing on
generating captions in multiple styles. Cho et al. [8] use CLIP to finetune a
pre-trained captioner. Like [51], they use the CLIPScore image-caption
similarity measure [16] as a reward signal, rather than a discriminative
objective like the one we adopt. We show in Appendix B.2 that the
discriminative objective outperforms an image-caption similarity objective
similar to CLIPScore (cosine similarity in Table 9). Luo et al. [27] apply
discriminative finetuning to basic MLE-trained models. Their method requires
the use of ground-truth captions for concurrent CIDEr-based optimization. A
similar approach was presented by Liu et al. [26]. They also finetune a MLE-
pre-trained captioner with a mixture of discriminative and CIDEr-based rewards
(requiring ground-truth captions for the latter). Intriguingly, their method
produces a higher portion of novel and unique captions compared to a system
trained only to imitate human captions, pointing to the potential for
generalization that we confirm here. Finally, in another early paper using
discrimination-based caption learning, Dai and Lin [9] propose a loss that
explicitly pushes a model to make captions more discriminative with respect to
those of a reference model.

Our study builds on this earlier work. Our main novelties lie in the
development of a simple system to perform discriminative finetuning that only
requires unannotated images, out-of-the-box generation/retrieval components
and applying the vanilla REINFORCE algorithm, and in our out-of-domain and
human-based evaluations.

##  3 Discriminative Self-Supervised Training

We finetune with reinforcement learning a pre-trained image captioner to
generate a caption that is subsequently fed as input to a frozen
discriminator. The discriminator is then tasked to retrieve the original image
among a set of distractors. We refer to our tuning method as DiscriTune.

##### Captioner

We employ a multimodal language model as neural captioner. We experiment with
two different captioners, namely the ClipCap architecture from Mokady _et al_.
[28], and the captioner network of the BLIP model [24].333Additional results
with the recently introduced CaMEL [4] captioner are provided in Appendix C.

ClipCap is an image-conditioned publicly available GPT-2 model [35].
Importantly, Mokady _et al_. [28] showed that it performs comparably to other
state-of-the-art captioning systems while having less parameters and being
more efficient to train. Given an input image, ClipCap uses a frozen CLIP [34]
visual encoder to extract visual features. These features are then projected
through a trainable mapper network onto the GPT-2 embedding space, where they
are used as a prefix conditioning the generation of an image description. In
practice, they act as soft image prompts that are used to kickstart the
caption generation process.

We use the publicly available ClipCap checkpoints where the language model
weights were also updated during training. Such checkpoints were trained on
Conceptual Captions [39] (ClipCap-ConCap henceforth), and MS COCO [25]
(ClipCap-COCO henceforth), respectively. Given that these models were trained
with a frozen ViT-B/32-based CLIP as visual feature extractor, we use the same
visual encoder in our experiments.

To test whether our method generalizes to other models, we experiment with the
BLIP system [24], a large captioner pre-trained with web-mined data which
performed on par or better than other state-of-the-art systems on several
benchmarks and has shown strong retrieval and captioning performance [24].
BLIP is made of a text Transformer [44] and a vision Transformer [13]. The
text Transformer is trained to maximize the likelihood of reproducing a
ground-truth reference by autoregressively generating a caption. Visual
information is injected by cross-attending over the output of the vision
Transformer. The model is trained with multi-task learning with two other
multimodal alignment losses. We refer the reader to [24] for additional
details on BLIP. We use the BLIP-base version,444The model is publicly
available through the LAVIS library [23] pre-trained on a large dataset
including both COCO and Conceptual Captions.

##### Retriever

We used the standard CLIP model from Radford _et al_. [34] as our neural
retriever. CLIP is a multimodal dual encoder model that embeds text and images
and learns to maximize their similarity through a contrastive loss. It was
trained on a dataset of 400M human-curated image-text pairs. We refer the
reader to [34] for additional details about CLIP pre-training. We use the
publicly available original implementation of CLIP with a ViT-B/32 backbone
[13]. 555https://github.com/openai/CLIP

We perform retrieval by computing a matching score $\mathrm{match}(c,i)$ for
an image $i$ and a caption $c$ as the dot product between the embedded
representation of $c$ (computed by the frozen CLIP textual encoder) and the
embedded representation of $i$ (computed by the frozen CLIP visual encoder).
We consider an image $i$ correctly retrieved against a set of distractors
$D\subseteq I$ (taken from a larger image collection $I$) iff $i^{\prime}\in
D$, and $i\neq i^{\prime}$,
$\mathrm{match}(c,i)>\mathrm{match}(c,i^{\prime})$.

##### Optimization

Since the decoding process creates a discrete bottleneck, we cannot use a loss
function to backpropagate end-to-end from the retriever output. Thus, we use
REINFORCE [47] to optimize the captioner (ClipCap or BLIP) by using a reward
that is based on the matching scores between the caption $c$ and the set of
candidates $D\cup\\{i\\}$. In this process, the CLIP retriever is kept frozen
and, given the non-differentiable text generation step, there is no gradient
flowing from the retriever to the captioner.

To compute a reward, we normalize the retriever scores to a probability
distribution, and then use as reward the so-defined log probability of the
original image $i$: $R(c,i,D)=\mathrm{log}\
\frac{e^{\mathrm{match}(c,i)}}{\sum_{i^{\prime}\in
D\cup\\{i\\}}e^{\mathrm{match}(c,i^{\prime})}}$. We then compute the reward as
the cross-entropy loss between such distribution and the position of the
target image in the list of candidates. Our captioner, which, in essence,
models the probability $P_{\theta}(c|i)$ of a caption $c$ given an image $i$,
is, in reinforcement learning terms, the action policy, and the actions taken
are just the selected tokens. The policy is trained to minimize the negative
expected reward, i.e., $\mathbb{E}_{c\sim P_{\theta}(\cdot|i)}[-R(c,i,D)]$ and
we compute the gradient of this expectation as the expectation of the
gradient. To reduce variance of the gradient estimator we use as a baseline a
running mean of past rewards [43] (see Appendix B for more details).

In our experiments we let both our captioners freely generate for a maximum of
40 tokens, or until a full stop (or EOS token for BLIP) is produced. We set
such maximum length after observing that most captions were much shorter and
taking into account the fact that CLIP, our downstream retriever, does not
process contexts larger than 75 tokens.

##  4 Experiments

###  4.1 Setup

In this section, we report experiments with our DiscriTune method where we
finetune the pre-trained captioner with a self-supervised discriminative
reward provided by a frozen downstream CLIP receiver. We apply the method to
the ClipCap checkpoints introduced above, namely ClipCap-COCO and ClipCap-
ConCap. We finetune them with CLIP-provided discriminative reward using COCO
and Conceptual Captions data, respectively. We call the resulting finetuned
models DiscriTune-COCO and DiscriTune-ConCap. We also repeat the main
experiments using the BLIP checkpoint trained on COCO, which we further
finetune on COCO with our DiscriTune method.666We use the EGG library [19] to
perform all our experiments.

We finetune both our captioners with the discriminative reward using the Adam
[20] optimizer with a learning rate of $10^{-7}$ and a constant schedule. We
use a batch size of 100, with in-batch distractors, so that each target image
is mixed with 99 randomly sampled distractors for discriminative finetuning.
All our experiments were performed on a single Tesla V100 GPU. For ClipCap on
COCO, we finetune for $20$ epochs, with around 1.2K batches per epochs and
roughly 24K updates, whereas on Conceptual Captions we finetune for $2$ epochs
leading to around 28K batches per epoch and 56K updates. For BLIP, we finetune
on COCO for a single epoch corresponding to 1.2K updates, as preliminary
experiments showed that to be sufficient for convergence. We use greedy
decoding at training time and beam search at test time.

##### Hyperparameter search

We used the Flickr validation set [50], not used elsewhere in the paper, to
tune learning rate, reward function (discriminative- vs accuracy-based), and
REINFORCE baseline (see Appendix B).

##### Data

We used the standard MS COCO [25] and Conceptual Captions [39] datasets to
perform discriminative finetuning. MS COCO is on of the most commonly used
captioning and text-conditioned retrieval datasets, containing around 120K
images, each provided with 5 human-generated captions. We use the Karpathy
train and test set split [17]. Conceptual Captions is a collection of images
mined from the web and aligned with their alt-text descriptions. It contains
around 3M samples for training and around 16K images for validation, that we
used as our test set. After filtering out corrupted images due to outdated
download links, we are left with 2.8M samples in the train set and exactly 13K
images for testing.

To test 0-shot cross-domain generalization, we use the Flickr [50], nocaps [1]
and Concadia [21] datasets. We divided Flickr data according to the Karpathy
split and evaluate on the test set (1K samples), where each image is aligned
with 5 human captions. The nocaps dataset was introduced to test caption
generalization performance of models trained on COCO. It is divided into three
splits. We do not use the control in-domain split, since it contains the same
object classes as COCO. The near-domain split has images from COCO categories
as well as images from new categories. The out-domain split only contains
pictures of objects that are not present in COCO. Following prior work [28],
we use the validation set of nocaps for testing purposes. Concadia is a
dataset recently introduced to test difference in text generation performance
when producing captions compared to descriptions. The former are meant to
accompany an image in order to provide additional context, as in books or
newspapers, while the latter should be able to replace the image, an example
being descriptions for visually impaired people.777Note that in this paper we
follow instead the standard practice of using “caption” to refer to both
captions and descriptions in the sense of Kreiss _et al_. [21]. We use the
Concadia test split, which contains 9.6K images, each annotated with a caption
and a description.

We want to emphasize again that, when finetuning our models, we did not use
any human reference. Ground-truth captions were only used to compute NLG
metrics.

Model | COCO | ConCap | Flickr | nocaps near | nocaps out | Concadia
---|---|---|---|---|---|---
ClipCap-COCO | 74.2 | 73.0 | 65.9 | 77.3 | 73.9 | 53.74
DiscriTune-COCO | 84.8 | 83.6 | 79.4 | 86.0 | 82.5 | 64.79
ClipCap-ConCap | 73.4 | 82.5 | 76.7 | 78.1 | 73.6 | 59.17
DiscriTune-ConCap | 81.6 | 94.4 | 87.8 | 89.1 | 88.7 | 80.49
Human captions | 76.3 | 81.6 | 88.7 | 85.5 | 87.7 | 73.96
Table 1: ClipCap and DiscriTune percentage accuracy (P@1) when retrieving a
target image from a set of 100 candidates taken from the COCO, Concpetual
Captions and Concadida test set, and nocaps validation sets.

###  4.2 Results

####  4.2.1 Text-Conditioned Image Retrieval

Table 1 shows that, as expected, ClipCap trained with DiscriTune greatly
improves over vanilla ClipCap on the image retrieval tasks it was finetuned
on. The result however also extends to cross-domain retrieval. DiscriTune-COCO
(slightly) outperforms ClipCap-ConCap on Conceptual Captions, and DiscriTune-
ConCap greatly outperforms ClipCap-COCO on COCO. Moreover, both versions of
DiscriTune greatly outperform ClipCap on all other datasets. Interestingly,
DiscriTune almost always outperforms human-generated captions, confirming a
recent result by Dessì _et al_. [11] on how neural retrievers show better
performance with neural captions. We performed additional experiments on a
more challenging setup where hard distractors are either automatically mined
or selected from adjacent frames in videos and confirm the superior
performance of our DiscriTune method (Appendix D).

COCO
---
Model | B@4 | M | C | S
ClipCap-COCO | 32.60 | 27.50 | 108.55 | 20.33
DiscriTune-COCO | 32.31 | 26.05 | 105.40 | 20.03
Conceptual Captions
Model | B@4 | M | C | S
ClipCap-ConCap | 7.32 | 10.81 | 87.22 | 18.07
DiscriTune-ConCap | 3.92 | 8.79 | 55.26 | 15.40
Table 2: NLG metrics (BLEU@4[31], METEOR[10], CIDEr[45] and SPICE[2]) for
ClipCap and DiscriTune captions tested in-domain on COCO and Conceptual
Captions. COCO
---
Model | B@4 | M | C | S
ClipCap-ConCap | 8.50 | 13.29 | 37.03 | 9.77
DiscriTune-ConCap | 13.99 | 16.97 | 53.20 | 12.07
Conceptual Captions
ClipCap-COCO | 1.47 | 6.43 | 23.74 | 7.98
DiscriTune-COCO | 1.71 | 6.58 | 28.01 | 9.00
Flickr
ClipCap-COCO | 17.21 | 18.43 | 41.65 | 12.04
DiscriTune-COCO | 18.48 | 18.61 | 44.78 | 12.68
ClipCap-ConCap | 8.28 | 12.24 | 27.57 | 7.81
DiscriTune-ConCap | 13.01 | 15.16 | 36.35 | 9.44
nocaps-near
ClipCap-COCO | 30.47 | 24.36 | 69.66 | 10.89
DiscriTune-COCO | 32.87 | 24.11 | 70.63 | 10.98
ClipCap-ConCap | 10.41 | 13.25 | 30.47 | 5.72
DiscriTune-ConCap | 18.93 | 17.66 | 46.45 | 7.53
nocaps-out
ClipCap-COCO | 20.32 | 20.22 | 51.74 | 8.55
DiscriTune-COCO | 24.10 | 20.49 | 57.06 | 8.83
ClipCap-ConCap | 10.69 | 13.15 | 36.57 | 5.71
DiscriTune-ConCap | 16.76 | 17.03 | 54.03 | 7.56
Concadia-Descriptions
ClipCap-COCO | 1.94 | 5.57 | 14.99 | 6.44
DiscriTune-COCO | 2.03 | 5.65 | 16.70 | 7.15
ClipCap-ConCap | 0.62 | 3.69 | 12.77 | 5.82
DiscriTune-ConCap | 1.12 | 4.81 | 17.20 | 7.13
Concadia-Captions
ClipCap-COCO | 0.24 | 2.45 | 4.35 | 2.11
DiscriTune-COCO | 0.30 | 2.49 | 5.57 | 2.59
ClipCap-ConCap | 0.50 | 2.74 | 9.22 | 3.65
DiscriTune-ConCap | 0.30 | 2.79 | 9.37 | 4.20
Table 3: NLG metrics (BLEU@4, METEOR, CIDEr and SPICE) for ClipCap and
DiscriTune captions tested across domain on COCO (Conceptual Captions-based
models only), ConceptualCaptions (COCO-based models only), Flickr, nocaps,
Concadia.

####  4.2.2 Ground-Truth-Based Caption Quality Evaluation

The text-based caption retrieval results show that our approach is remarkably
good at the task it was finetuned on, also in the 0-shot cross-domain setup.
However, it is less clear that retrieval-based finetuning should improve the
captions’ faithfulness to human ground-truth image descriptions. Table 2 shows
that, indeed, discriminative finetuning leads to some decrease in caption
faithfulness, when testing on the dataset used for supervised pre-training.
However, this performance drop (which, in the case of COCO is very small) is
balanced by greater generalization performance, as shown in Table 3.
DiscriTune-COCO is the best domain transfer model across-the-board. In
addition, DiscriTune-ConCap is also consistently outperforming its vanilla
counterpart, ClipCap-ConCap, in this 0-shot cross-domain generalization setup.
Interestingly, for the Concadia dataset we have a (slightly) higher gain over
plain ClipCap with the descriptions split rather than with the captions one,
especially for DiscriTune-ConCap. Human references in the descriptions split
were generated with the goal of replacing an image by describing its contents.
This confirms that DiscriTune contributes to captions that are more suited for
the “communicative” purpose of characterizing the crucial aspects of an image
contents. Overall, the results suggest that, on the one hand, discriminative
finetuning leads to captions that drift apart somewhat from the human
descriptions that the model learnt to mimic. At the same time, though, this
might allow the model to get away from the idiosyncrasies of a specific
captioning style, leading to captions that better generalize to a wider set of
unseen images and domains. As the experiment in Section 4.2.4 below shows,
sometimes this drifting might be beneficial even when captioning images in the
same domain.

####  4.2.3 Applying Discriminative Finetuning to BLIP

To verify the robustness of DiscriTune finetuning, we next apply it to BLIP.
We finetune BLIP on COCO captions, one of the datasets it was pre-trained on.
Tables 4 (retrieval) and 5 (captioning) show that Discrirune-BLIP outperforms
its non-finetuned counterpart for both retrieval performance and caption
quality. The results confirm that our captioner-agnostic discriminative
finetuning is helpful even when applied to this latest-generation general-
purpose vision-and-language model. DiscriTune-BLIP outperforms its vanilla
counterpart on all out-of-domain datasets tested. Again, we note the greater
performance boost on the Concadia Descriptions split compared to the Captions
one, confirming the results obtained with ClipCap and the benefits of our
finetuning method to generate captions that are closer to human references
produced with the communicative intent of replacing an image. Given that BLIP
is not based on a CLIP encoder, this experiment also refutes the hypothesis
that DiscriTune performance gains are simply due to using CLIP both as visual
encoder for the captioner and as retriever, confirming the wider applicability
of our method.

Model | Flickr | nocaps | nocaps | Concadia
---|---|---|---|---
|  | near | out |
BLIP | 75.1 | 87.0 | 90.6 | 71.0
DiscriTune-BLIP | 80.8 | 90.2 | 92.6 | 74.9
Table 4: BLIP and DiscriTune-BLIP percentage accuracy on the cross-domain
retrieval task described in Section 4.2.1. Flickr
---
Model | B@4 | M | C | S
BLIP | 27.18 | 22.74 | 70.63 | 16.03
DiscriTune-BLIP | 28.19 | 23.61 | 74.77 | 16.9
nocaps-near
BLIP | 44.91 | 29.52 | 107.13 | 14.52
DiscriTune-BLIP | 45.46 | 29.81 | 108.70 | 14.97
nocaps-out
BLIP | 38.42 | 26.95 | 105.63 | 13.77
DiscriTune-BLIP | 37.34 | 27.13 | 106.23 | 14.14
Concadia-Descriptions
BLIP | 2.71 | 6.90 | 26.02 | 9.75
DiscriTune-BLIP | 2.90 | 7.18 | 27.20 | 9.98
Concadia-Captions
BLIP | 0.66 | 3.14 | 9.97 | 3.84
DiscriTune-BLIP | 0.74 | 3.35 | 10.84 | 3.96
Table 5: NLG metrics (BLEU@4, METEOR, CIDEr and SPICE) for BLIP and
DiscriTune-BLIP captions tested across domains.

####  4.2.4 Human Text-Based Image Retrieval

The results in tables 1 and 2 show that, when testing on Conceptual Captions,
DiscriTune-ConCap produces outputs that are less similar to human captions
than those of ClipCap-ConCap, but it greatly outperforms the latter in text-
based image retrieval accuracy. Recall that the ground-truth captions in
Conceptual Captions come from the alt-texts associated to images harvested
from the Web [39]. As discussed in more detail in the qualitative analysis
below, when taken out of context, such captions are often non-informative, and
thus it’s not clear that learning to reproduce their style as closely as
possible, like vanilla ClipCap-ConCap does, is a good idea. Consider for
example Fig. 2(c) below. ClipCap-ConCap is perfectly reproducing the ground-
truth description (“digital art selected for the #”), and yet this is not as
informative as the caption produced by DiscriTune-ConCap (“a boy in a pond
with a lot of stars”). We thus conjecture that, despite their lower
faithfulness to the human ground truth, DiscriTune-ConCap captions are more
informative than ClipCap captions, and possibly even more informative than the
original human captions in this dataset.

To verify this hypothesis, we designed an experiment in which human annotators
had to select a target image from a set of 10 candidates, based either on a
ground-truth human description, or a ClipCap-ConCap or DiscriTune-ConCap-
generated one. We sampled the data for this experiment from our Conceptual
Captions test set. To make sure that the captions needed to be genuinely
informative in order to allow successful target retrieval, we selected hard
distractors among the nearest visual neighbours of the target. The latter were
found by passing images through the CLIP ViT-B/32 visual encoder and computing
the cosine of the resulting representations.888We found that very high-
similarity neighbours are often near duplicates of the target image and we
excluded those with similarity above $0.8$ We collected human data for 500
target+distractor sets, for each of the three caption types. We recruited
Amazon Mechanical Turk participants who saw 100 sets
each.999https://www.mturk.com/ Experimental details (including ethical
approval) are in Appendix A.

Captions | accuracy
---|---
Human | 42.8
ClipCap-ConCap | 36.2
DiscriTune-ConCap | 47.6
Table 6: Percentage accuracy of human annotators when selecting a target image
among 9 distractors based on the target human-, ClipCap- or DiscriTune-
generated captions.

Results are reported in Table 6. They confirm that the task, due to the hard
distractors, is quite challenging, with the annotators failing to reach 50%
accuracy independently of caption type. However, even with the least
informative ClipCap-ConCap captions, humans are well above the 10% chance
level. Importantly, the DiscriTune-ConCap captions greatly outperform not only
their ClipCap counterparts, but also the human ones, with a solid 5% accuracy
boost. We thus confirm our conjecture that, given a relatively noisy dataset
such as Conceptual Captions, our finetuning procedure can produce captions
that are more informative than the original human-produced descriptions. In
the next section, we explore the differences between human and DiscriTune-
generated captions.

##  5 Caption Analysis

Figure 2: Example Conceptual Captions images with the corresponding captions
produced by humans (H), ClipCap (C) and DiscriTune (D), respectively. In all
cases, annotators were able to identify the target image (framed in green)
only when receiving the DiscriTune caption as input. The image on the right of
each pair is the distractor that was chosen by annotators when receiving the
human caption as input.

Fig. 2 shows 6 Conceptual Captions examples from the experiment with humans in
which human annotators were only able to guess the target given the
DiscriTune-generated captions (correct target image is on the left; image
picked by subjects in response to the human-generated caption is shown on the
right). These cases, selected among those where no human face is recognizable,
were hand-picked to illustrate various phenomena. They are however
representative of the data, which we manually inspected as a whole.

Starting with a comparison of DiscriTune and ClipCap captions,101010As the
focus of this analysis in on the Conceptual Captions-trained/finetuned models,
we will drop the -ConCap suffix. we observe that the latter tend to be more
vague (cf. examples (b), (c), (d) and (f)), where sometimes these vague
captions are actually matching the human-generated ones very closely (as in
(c)). The other issue with ClipCap captions is that they are often inaccurate,
as in example (a), where people are playing in the water, not on the beach;
and it’s hard to believe the desolated landscape in (b) is a tourist
attraction. Note that the use of a generic term such as tourist attraction in
this last example is a peculiarity that ClipCap inherited from the Conceptual
Captions style (the dataset was constructed by replacing place and people
names with generic hypernyms such as this one). Clearly, learning to reproduce
such Conceptual Captions-specific idiosyncrasies penalizes ClipCap when it’s
evaluated on other datasets. On the other hand, it’s remarkable that
discriminative finetuning, built on top of this very ClipCap system, was able
to steer the captions back towards a more descriptive and precise language.

Comparing DiscriTune with the human captions, we see that the former tend to
be more plainly descriptive and precise than the latter. Often, human captions
in the Conceptual Captions dataset contain non-discriminative “meta”
information about an image that is not useful to identify it, or might have
made sense in the context of the original web page, but becomes opaque once
the image and its alt-text are extrapolated. In example (a), the human caption
is just stating that we are on a beach with shallow water, so that the human
discriminator picked another shallow-beach picture. The DiscriTune caption
precisely reports that there are a woman and children in the water. The human
caption of example (b) might be accurate, but without more context it’s
difficult to recognize the white plateau as a frozen lake. Consequently, the
human discriminator wrongly chose a more stereotypical picture of a lake.
Concerning examples (c) and (f), the human captions report that they are
“digital art” and an ”avatar”, respectively, leading the annotators to pick
other potential exemplars of digital art and avatars from the distractors. The
DiscriTune caption for (f) is actually inaccurate, as the avatar is not a
bird, but the mention of a pink nose nevertheless helped the subject
identifying the right image. The human caption of example (d) refers to an
outdoor amphitheater that is not visible in the picture. Again, DiscriTune was
more helpful to the human discriminators by providing a plain description of
the picture contents. Finally, example (e) is interesting because the human
caption highlights the somewhat atypical belfry in the picture (and so the
human discriminator picked a photo with a more prominent belfry), whereas the
DiscriTune caption provides a more discriminative cue by mentioning the
colonial style of the landscape.

human | ClipCap | DiscriTune
---|---|---
new | young | red
other | old | green
musical | biological | blue
outdoor | close | black
small | white | pink
long | trendy | colorful
big | digital | yellow
low | funny | denim
large | aerial | purple
beautiful | general | high
Table 7: Top 10 adjective lemmas most associated to a caption type (human,
ClipCap or DiscriTune) according to the local Mutual Information association
statistics computed on all captions generated for our full Conceptual Captions
test set.

To get a more general sense of the language of the different caption types, we
lemmatized and part-of-speech tagged the full Conceptual Captions test set
caption corpora with Spacy.111111https://spacy.io/ We then computed the local
Mutual Information statistics [14] across all possible lemma/caption-type
pairs. We restricted the analysis to adjectives and nouns, as we found these
two parts of speech to include the most visually descriptive terms. The
adjective results are presented in Table 7 (noun analysis is in Appendix E).
The difference between the adjectives in human and DiscriTune captions is
striking: the latter are nearly all highly visual descriptive terms (in
particular, colours), whereas the former contain several terms that are hardly
providing any visual information (new, musical, other). Even the most concrete
human-caption adjectives are not as specific as those strongly associated with
DiscriTune (compare small, big, large, beautiful to red, blue, purple, denim).
The ClipCap list also contains few adjectives that might be genuinely useful
to discriminate a specific image (white, aerial), with most being either
abstract or very generic (biological, trendy, funny, general). It is
remarkable that self-supervised discriminative finetuning, probably by
exploiting the multimodal knowledge encoded in the pre-trained ClipCap
components, is able to recover a highly visually descriptive language, despite
the fact that it operates on a system that has been trained on human captions
that, as we have just seen, are not as plainly descriptive, and that the
system is not exposed to any new language during finetuning.

##  6 Conclusion

We presented a simple finetuning method to make model-generated captions more
discriminative. Given a pre-trained captioner, its text generation component
is finetuned on the task of helping a black-box text-based image retriever
picking a target image among distractors. The task only requires unannotated
images, and we were able to make the system work with the basic REINFORCE
algorithm. We leave the exploration of more sophisticated reinforcement
learning techniques as an obvious direction for future work. Our results are
reported using two captioners, ClipCap, a decoder-only model, and BLIP, an
encoder-decoder trained with multitask learning on web-scale data.

We found that the discriminatively finetuned captions do not improve over the
original ones in terms of similarity to human ground truth, when tested on the
same dataset the captioner was trained on. However, for both models and on a
variety of out-of-domain datasets, they consistently outperform those of the
original captioner. Discriminative pressure might be a strong enough signal to
“unlearn” some of the overfit on the caption style of the pre-training
dataset, and instead better capture the semantic content of the image. What’s
more, for the noisily annotated Conceptual Captions data-set (where we
observed the largest performance drop in terms of mimicking ground-truth
descriptions when finetuning ClipCap), discriminatively finetuned captions are
more helpful than ground-truth captions, not only to a neural retriever, but
also for humans tasked with a challenging image identification task. This
suggests that our system could be used as-is to generate Web image captions
that are on average more informative to users than alt-text descriptions
(which are the source of Conceptual Captions annotations).

Qualitatively, we find that, even when finetuning the Conceptual Captions-
trained captioner (that has learned to reproduce the somewhat abstract style
of alt-text descriptions), our discriminative finetuning procedure recovers a
more precise and plainly descriptive language. Compared to those of the
original captioner, it is also clear why these more descriptive captions, that
shed the idiosyncrasies of alt-text, will generalize better to other datasets.
We focused our analysis on ClipCap and Conceptual Captions, since this is the
setup where we observed the largest discrepancy between human and
discriminatively-generated captions, and a marked asymmetry in retrieval vs.
generation performance. We leave a thorough investigation of how the nature of
the captions used to train the backbone captioner affects our method to future
work. Given that several pre-trained text-based image retrievers are publicly
available, another interesting direction would be to alternate different
retrievers during finetuning. This might help the model further generalize, as
it would be less likely to overfit the quirks of one specific retriever.

Finally, there has been recent progress in training models to learn from human
feedback through reinforcement learning [30, 42, 29]. Given the costly human
annotation process required by this approach, our method could be seen as a
cheaper alternative, exploiting “neural” feedback to guide the finetuning of
an existing model. Future research directions should study the interplay
between human and neural feedback to improve the capabilities of current
systems.

## Acknowledgements

We thank the area chairs and reviewers for feedback. We thank Corentin
Kervadec for feedback and help in creating Figure 1, and Lorenzo Baraldi and
Manuele Barraco for replying to a Sunday night email during the rebuttal. EG,
NCR, FF and MB received funding from the European Research Council (ERC) under
the European Union’s Horizon 2020 research and innovation programme (grant
agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref.
PID2020-112602GB-I00/MICIN/AEI/10.13039/501100011033). This paper reflects the
authors’ view only, and the funding agencies are not responsible for any use
that may be made of the information it contains.

## References

  * [1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  Nocaps: Novel object captioning at scale.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8948–8957, 2019.
  * [2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 382–398, Cham, 2016. Springer International Publishing.
  * [3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of CVPR, pages 6077–6086, Salt Lake City, UT, 2018\.
  * [4] Manuele Barraco, Matteo Stefanini, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara.  CaMEL: Mean Teacher Learning for Image Captioning.  In International Conference on Pattern Recognition, 2022.
  * [5] Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank.  Automatic description generation from images: A survey of models, datasets, and evaluation measures.  Journal of Artificial Intelligence Research, 55:409–442, 2016.
  * [6] Paul Bloom.  How Children Learn the Meanings of Words.  MIT Press, Cambridge, MA, 2000.
  * [7] Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun.  Show, adapt and tell: Adversarial training of cross-domain image captioner.  In Proceedings of ICCV, pages 521–530, Venice, Italy, 2017.
  * [8] Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal.  Fine-grained image captioning with CLIP reward.  In Findings of NAACL, pages 517–527, Seattle, WA, 2022.
  * [9] Bo Dai and Dahua Lin.  Contrastive learning for image captioning.  In Proceedings of NIPS, pages 898–907, Long Beach, CA, 2017.
  * [10] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
  * [11] Roberto Dessì, Eleonora Gualdoni, Francesca Franzon, Gemma Boleda, and Marco Baroni.  Communication breakdown: On the low mutual intelligibility between human and neural captioning.  In Proceedings of EMNLP, Abu Dhabi, United Arab Emirates, 2022.  In press.
  * [12] Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):677–691, 2017.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [14] Stephanie Evert.  The Statistics of Word Cooccurrences.  Ph.D dissertation, Stuttgart University, 2005.
  * [15] Adam Fisch, Kenton Lee, Ming-Wei Chang, Jonathan Clark, and Regina Barzilay.  CapWAP: Image captioning with a purpose.  In Proceedings of EMNLP, pages 8755–8768, Online, 2020.
  * [16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: A reference-free evaluation metric for image captioning.  In Proceedings of EMNLP, pages 7514–7528, Punta Cana, Dominican Republic, 2021.
  * [17] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of CVPR, pages 3128–3137, Boston, MA, 2015.
  * [18] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  ReferItGame: Referring to objects in photographs of natural scenes.  In Proceedings of EMNLP, pages 787–798, Doha, Qatar, 2014.
  * [19] Eugene Kharitonov, Roberto Dessì, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni.  EGG: a toolkit for research on Emergence of lanGuage in Games.  https://github.com/facebookresearch/EGG, 2021.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts.  Concadia: Towards image-based text generation with a purpose.  In Proceedings of EMNLP, Abu Dhabi, United Arab Emirates, 2022.  In press.
  * [22] Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy.  Image retrieval from contextual descriptions.  In Proceedings of ACL, pages 3426–3440, Dublin, Ireland, 2022.
  * [23] Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi.  Lavis: A library for language-vision intelligence, 2022.
  * [24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  https://arxiv.org/abs/2201.12086, 2022.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In Proceedings of ECCV, pages 740–755, Zurich, Switzerland, 2014\.
  * [26] Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang.  Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data.  In Proceedings of ECCV, 2018.  Published online: https://openaccess.thecvf.com/ECCV2018.
  * [27] Ruotian Luo, Gregory Shakhnarovich, Scott Cohen, and Brian Price.  Discriminability objective for training descriptive captions.  In Proceedings of CVPR, pages 6964–6974, Salt Lake City, UT, 2018\.
  * [28] Ron Mokady, Amir Hertz, and Amit Bermano.  ClipCap: CLIP prefix for image captioning.  https://arxiv.org/abs/2111.09734, 2021.
  * [29] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332, 2021.
  * [30] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [31] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
  * [32] Jonathan Peirce, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv.  PsychoPy2: Experiments in behavior made easy.  Behavior Research Methods, 51(1):195–203, Feb. 2019.
  * [33] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai.  Tuning computer vision models with task rewards.  arXiv preprint arXiv:2302.08242, 2023.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Proceedings of ICML, pages 8748–8763, virtual conference, 2021\.
  * [35] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf, 2019\.
  * [36] Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba.  Sequence level training with recurrent neural networks.  CoRR, abs/1511.06732, 2016.
  * [37] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024, 2017.
  * [38] Ananya Sai, Akash Kumar Mohankumar, and Mitesh Khapra.  A survey of evaluation metrics used for NLG systems.  ACM Computing Surveys, 2022.  In press.
  * [39] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of ACL, pages 2556–2565, Melbourne, Australia, 2018\.
  * [40] Brian Skyrms.  Signals: Evolution, learning, and information.  Oxford University Press, Oxford, UK, 2010.
  * [41] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.  From show to tell: A survey on deep learning-based image captioning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.  In press.
  * [42] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  * [43] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of NIPS, pages 5998–6008, Long Beach, CA, 2017.
  * [45] R. Vedantam, C. Zitnick, and D. Parikh.  Cider: Consensus-based image description evaluation.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, Los Alamitos, CA, USA, jun 2015. IEEE Computer Society.
  * [46] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Computer Vision and Pattern Recognition, 2015.
  * [47] Ronald Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine learning, 8(3-4):229–256, 1992.
  * [48] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In Proceedings of ICML, pages 2048–2057, Lille, France, 2015.
  * [49] Min Yang, Wei Zhao, Wei Xu, Yabing Feng, Zhou Zhao, Xiaojun Chen, and Kai Lei.  Multitask learning for cross-domain image captioning.  IEEE Transactions on Multimedia, 21(4):1047–1061, 2019.
  * [50] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  * [51] Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, and Yejin Choi.  Multimodal knowledge alignment with reinforcement learning.  https://arxiv.org/abs/2205.12630, 2022.
  * [52] Wentian Zhao, Xinxiao Wu, and Jiebo Luo.  Cross-domain image captioning via cross-modal retrieval and model adaptation.  IEEE Transactions on Image Processing, 30:1180–1192, 2021.

##  Appendix A Crowdsourcing Experiment Details

We chose the stimuli for the human annotation experiment as follows. We
iterated over the images in our Conceptual Captions test set, and sampled, for
each image, the 9 closest neighbours, thus creating sets composed of 10
images: 1 target and 9 distractors. We set a threshold of maximum cosine
similarity to the nearest neighbour of $0.8$. We decided on this threshold
after visual inspection of the generated sets: we aimed at having sets
challenging for the annotators, yet not impossible to solve, and higher
thresholds could lead to sets containing almost identical images, such as
subsequent frames extracted from the same video or different croppings of the
same picture. Neither targets nor distractors were repeated in the sets and we
manually excluded disturbing images.

In the human retrieval experiment, we annotated each set with 3 types of
captions: human captions, or captions generated by DiscriTune(-ConCap) or
ClipCap(-ConCap), respectively. We randomly divided the entire set into blocks
of 100 questions containing mixed caption types. On each screen, the 10 images
from a set were presented at the center, arranged in two arrays of 5 images,
with the caption written above–see Figure 3. Participants were asked to click
on the image that matched the caption best. They were shown one example before
starting the task, and were also warned that some cases could contain
automatically-generated captions: we asked them to always reply with the
answer they found most plausible. Finally, they were warned that the
experiment contained some control items, used to ensure annotation quality.

Figure 3: Example of a screen shown to the participants, with a human caption.

Each subject was presented with one block of questions, plus 5 randomly placed
controls, designed to ensure that annotators were paying attention to the
task. These cases were made intentionally very simple: targets were surrounded
by random distractors not appearing in the other sets, and the associated
caption was a human generated one. We made sure internally that these sets
could be easily processed with 100% retrieval accuracy.

The data collection routine was written in Psychopy [32] and launched through
Pavlovia.121212https://pavlovia.org/ There was no time limit for completing
the study. We recruited participants via Amazon Mechanical
Turk.131313https://www.mturk.com We only accepted annotators from the US, with
HIT approval rate higher than 97%, and number of approved HITs higher than
1000. We informed them that we would not collect any personal data (except for
their workerID, necessary for their payment, that we would not make public),
and that the goal of the experiment was to study how well people identify
images based on descriptions. Before being able to access the link of the
experiment, participants had to complete an informed consent form, warning
them that our experiment would show images and descriptions sampled from the
web, and that could therefore contain upsetting content (although, as said
above, we manually ensured that images we personally found disturbing would be
excluded). They were able to quit the experiment at any time. We paid them
$13.5$$ for completing the task. The experiment was approved by the ethical
board of Universitat Pompeu Fabra in the context of the AMORE project (grant
agreement No. 715154). We excluded the data of participants that made more
than one mistake when scoring the controls, suggesting that they were not
paying enough attention to the task.

##  Appendix B Hyperparameter Exploration

To generate text, we use greedy decoding at train time and beam size with 5
beams at test time, without tuning these values. hyperparameter searches, we
use retrieval score as our selection metric, since its consistent with our
training objective and does not require annotated data. We perform our
hyperparameter searches on the Flickr validation set. Even though finetuning
on Flickr might be seen as favouring out-of-domain performance (the models
were pre-trained on COCO or Conceptual Captions), we we are confident it is
not a major factor: the following sections show that the type of REINFORCE
baseline and reward function do not have a big impact on performance, and for
learning rate we informally found that, as long as large values are avoided,
it does not greatly affect results, only convergence speed. Finally,
optimizing text-based models with reinforcement learning, especially when done
form scratch, can be a challenging problem due to sparse rewards and the vast
action space of selecting a token from a vocabulary. This can lead to
repetitions or other unnatural word sequences. In practice, we did not notice
issues such as repetitions or ungrammatical text. Indeed, we observed quite
the opposite, with DiscriTune consistently producing natural text, as
confirmed by the NLG metrics improvements. We believe this is due to starting
from pre-trained models that are producing fluent language. Evidently,
discriminative REINFORCE tuning does not degrade fluency.

###  B.1 Reinforce Baseline

Using REINFORCE [47], we can rewrite the gradient of the expected reward as
the expectation of the gradient, approximated by a single sample caption
$\hat{c}$:

$\begin{split}\nabla_{\theta}L(i,D,\theta)&=\nabla_{\theta}\mathbb{E}_{c\sim
P_{\theta}(\cdot|i)}[-R(c,i,D)]\\\ &=\mathbb{E}_{c\sim
P_{\theta}(\cdot|i)}[-R(c,i,D)\nabla_{\theta}\log P_{\theta}(c|i)]\\\
&\approx-R(\hat{c},i,D)\nabla_{\theta}\log P_{\theta}(\hat{c}|i)\end{split}$
(1)

where $i$ is the target image, $\hat{c}$ is the generated caption, $D$ is the
set candidates fed to the retriever and $R$ is the reward function. The
parameters $\theta$ can be optimized with regular (mini-batch) gradient
descent. To reduce variance of the gradient estimator when using REINFORCE, we
subtract a baseline term. We compared two different baselines. The first is a
running mean of past rewards values using greedy decoding. The second uses
beam with the baseline computed using the reward value given by CLIP when fed
captions generated with greedy decoding. We trained a ClipCap model on Flickr
for 10 epochs using the setup described in Section 4, and then evaluated its
performance on the validation set. Results are presented in Table 8. We found
that, without subtracting a baseline, the model performed poorly, achieving an
accuracy of 34.3%. Running mean and greedy decoding yield similar performance,
with running mean showing slightly higher accuracy (97.8% vs 97.4%). We thus
employed a running mean baseline with greedy decoding in all the main
experiments.

baseline type | P@1
---|---
no baseline | 34.3
greedy decoding (w/ beam search) | 97.4
running mean | 97.8
Table 8: ClipCap retrieval accuracy with 100 candidates on the Flickr
validation set using different REINFORCE baselines. The no baseline and
running mean methods were used employing greedy decoding to generate captions,
whereas when greedy decoding was the baseline, we let ClipCap produce captions
with beam search using 5 beams.

###  B.2 Reward Function

To find the best reward to train our captioner, we trained a ClipCap model on
Flickr for 10 epochs using the setup described in Section 4, and then
evaluated its performance on the validation set. We explored three different
reward functions. The cosine similarity reward computes the normalized dot
product between the target image embedding and the model-generated caption
representation. This is equivalent to the CLIPScore [16] and it is not
discriminative since it does not compare the target image with any distractor.
The accuracy-based reward computes a binary score which is 1 if CLIP assigned
the highest dot-product-based alignment score to the target image when fed a
caption, and 0 otherwise. The third reward type is the negative softmax-
normalized log probability of the match between a caption and each image in
the candidate list, as described in Section 3. As reported in Table 9, the log
probability reward performed best, although not by a large margin. Thus, we
run all the experiments presented in this work optimizing the captioner using
such reward.

reward function | P@1
---|---
cosine similarity | 85.2
accuracy | 85.3
log probability | 86.2
Table 9: ClipCap retrieval accuracy with 100 candidates on the Flickr
validation set with different reward functions.

##  Appendix C Finetuning CaMEL

We apply our DiscriTune method to the recently introduced CaMEL [4] captioner
model. This model is trained on COCO using a distillation loss based on a
model tracking the running mean of an online network, and concurrently
optimized with a reward-based objective after a first phase of supervised
learning against human references. The reward is computed using CIDEr [45]
(please see [4] for additional details on the model and its training setup).
NLG results in Table 10 confirm that, at the price of a small drop in in-
domain performance, DiscriTune is able to improve (by a small margin) when
tested on the Flickr out-of-domain dataset, confirming the benefits of
discriminatively finetuning a pre-trained captioner, even when the procedure
is applied to a “bleeding-edge” model of this sort.

COCO
---
Model | B@4 | M | C | S
CaMEL | 38.11 | 29.03 | 128.62 | 23.35
DiscriTune-CaMEL | 33.45 | 27.63 | 117.71 | 22.03
Flickr
CaMEL | 22.93 | 20.93 | 58.38 | 14.62
DiscriTune-CaMEL | 22.60 | 20.99 | 59.12 | 14.94
Table 10: NLG metrics (BLEU@4[31], METEOR[10], CIDEr[45] and SPICE[2]) for
CaMEL and DiscriTune-CaMEL captions on the COCO test split (in-domain, our
results when using CaMEL) and Flickr test split (out-of-domain).

##  Appendix D Image Retrieval with Hard Distractors

###  D.1 ImageCoDe

In order to test retrieval performance in a challenging setup, we use the
ImageCoDe dataset [22]. ImageCoDe was recently introduced as a testbed for
text-based image retrieval. It is formed by $10$-elements sets of target
images collected from consecutive video frames or by mining similar images to
a given target frame. For a fair comparison with prior work, we use the
validation images as test data.

In Table 11, we report ImageCoDe results with all our model-generated captions
as well as human-generated ones, when a CLIP model with ViT-B-32 was used as
the text-to-image retriever. The models are the ones trained with the setup
described in Section 4. The results are remarkable, reaching a new state of
the art (for either human or model-generated captions) on this dataset (best
previous result, obtained with human captions: 29.9% [22]). This shows that
our tuning method is beneficial to produce discriminative captions even in
contexts in which distinctions need to be very subtle. This suggests that our
method could be profitably applied to scenarios where such granular
discrimination is called for, such as in video understanding tasks.

Captions | ImageCoDe
---|---
ClipCap-COCO | 28.7
DiscriTune-COCO | 34.0
ClipCap-ConCap | 26.8
DiscriTune-ConCap | 36.2
Blip-COCO | 24.0
DiscriTune-COCO | 24.7
Human | 22.3
Table 11: Percentage accuracy (P@1) when retrieving a target image from the
validation image sets of ImageCoDe. Random chance is at 10%.

###  D.2 Hard Negative Mining

We perform an additional experiment where at training time the retrieval task
is performed using automatically-mined hard distractors. When testing, we
still randomly select all non-target candidates. We pick the k most similar
distractors based on the cosine similarity with a target using the CLIP visual
encoder (the remaining $99-k$ distractors are picked randomly, as usual). This
experiment is aimed at studying the impact of the distractors in
discriminative finetuning, with the idea that making the task harder should
lead to more discriminative captions. In Table 12 and Table 13 we report NLG
metrics and retrieval accuracy, respectively. Overall, we see a mixed picture.
With respect to the NLG metrics, hard distractors are helpful only for one of
the two OOD datasets (Flickr), but at the price of a larger performance drop
in-domain (COCO). Concerning retrieval accuracy, hard distractors give a
slight improvement over random ones for in-domain data (COCO) and only on
Flickr but not on Conceptual Captions for out-of-domain data.

We conjecture that the harder setup can lead to overfitting the quirks of the
frozen retriever, in some cases leading to (slightly) poorer generalization.
Studying the impact of the retrieval task with respect to number and type of
distractors is an interesting direction for future work.

COCO
---
Model | B@4 | M | C | S
ClipCap-COCO | 32.60 | 27.50 | 108.55 | 20.33
DiscriTune-COCO | 32.31 | 26.05 | 105.40 | 20.03
w/ 5 hard distractors | 29.85 | 25.53 | 100.25 | 19.50
w/ 10 hard distractors | 29.20 | 25.25 | 98.69 | 19.30
Conceptual Captions
ClipCap-COCO | 1.47 | 6.43 | 23.74 | 7.98
DiscriTune-COCO | 1.71 | 6.58 | 28.01 | 9.00
w/ 5 hard distractors | 1.47 | 6.22 | 25.11 | 8.50
w/ 10 hard distractors | 1.39 | 6.15 | 24.69 | 8.40
Flickr
ClipCap-COCO | 17.21 | 18.43 | 41.65 | 12.04
DiscriTune-COCO | 18.48 | 18.61 | 44.78 | 12.68
w/ 5 hard distractors | 18.75 | 18.95 | 45.15 | 13.00
w/ 10 hard distractors | 18.23 | 18.68 | 44.28 | 12.85
Table 12: NLG metrics (BLEU@4, METEOR, CIDEr and SPICE) for ClipCap and ClipCap-based DiscriTune captions on COCO, ConceptualCaptions and Flickr, after training with 5 or 10 automatically mined hard distractors and testing with randomly selected ones. COCO |
---|---
Model | P@1
ClipCap-COCO | 74.2
DiscriTune-COCO | 84.8
w/ 5 hard distractors | 84.9
w/ 10 hard distractors | 85.2
Conceptual Captions |
ClipCap-COCO | 73.0
DiscriTune-COCO | 83.6
w/ 5 hard distractors | 82.3
w/ 10 hard distractors | 82.0
Flickr |
ClipCap-COCO | 65.9
DiscriTune-COCO | 79.4
w/ 5 hard distractors | 79.8
w/ 10 hard distractors | 79.1
Table 13: P@1 retrieval accuracy for ClipCap and ClipCap-based DiscriTune
captions on COCO, Conceptual Captions and Flickr, after training with 5 or 10
automatically mined hard distractors and testing with randomly selected ones.

##  Appendix E Caption Analysis: Nouns

The patterns we encountered in the adjective analysis presented in section 5
are confirmed by the noun lemma analysis. In Table 14, we report the top 10
noun lemmas most strongly associated with the human, Clipcap and DiscriTune
captions in the Conceptual Captions dataset. DiscriTune favours words with
strong and precise visual content, such as dress, woman, pair, garden, field
and lake. Human captions tend to include more generic terms such as person and
background, as well as several nouns that might be describing images at a more
abstract level, that would probably not favour their precise identification
(image, time, summer, style, part). The preference for these more abstract
terms is even more pronounced in ClipCap (view, actor, portrait, illustration,
premiere, artist, vector, property).

human | ClipCap | DiscriTune
---|---|---
nouns
person | view | forest
time | illustration | night
day | actor | dress
water | premiere | garden
image | person | celebrity
summer | artist | field
instrument | portrait | woman
style | vector | lake
background | background | pair
part | property | group
Table 14: Top 10 noun lemmas most associated to a caption type (human, ClipCap
or DiscriTune) according to the local Mutual Information association
statistics computed on all captions generated for our full Conceptual Captions
test set.