#  Fold analysis of crumpled sheet using micro computed tomography

Yumino Hayase  Department of Physics, Kyushu University, Fukuoka 819-0395,
Japan  Hitoshi Aonuma  Research Institute for Electronic Science, Hokkaido
University, Sapporo 060-0812, Japan  Satoshi Takahara  Graduate School of
Information Science and Technology, Hokkaido University, Sapporo 060-0812,
Japan  Takahiro Sakaue  Department of Physical Sciences, Aoyama Gakuin
University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara, Kanagawa 252-5258, JAPAN
Shun’ichi Kaneko  Kazusa DNA Research Institute, 2-6-7 Kazusa-kamatari,
Kisarazu, Chiba 292-0818, Japan  Hiizu Nakanishi  Department of Physics,
Kyushu University, Fukuoka 819-0395, Japan

###### Abstract

Hand crumpled paper balls involve intricate structure with a network of
creases and vertices, yet show simple scaling properties, which suggests self-
similarity of the structure. We investigate the internal structure of crumpled
papers by the micro computed tomography (micro-CT) without destroying or
unfolding them. From the reconstructed three dimensional data, we examine
several power laws for the crumpled square sheets of paper of the sizes
$L=50\sim 300$ mm, and obtain the mass fractal dimension $D_{M}=2.7\pm 0.1$ by
the relation between the mass and the radius of gyration of the balls, and the
fractal dimension $2.5\lesssim d_{f}\lesssim 2.8$ for the internal structure
of each crumpled paper ball by the box counting method in the real space and
the structure factors in the Fourier space; The data for the paper sheets are
consistent with $D_{M}=d_{f}$, suggesting that the self-similarity in the
structure of each crumpled ball gives rise to the similarity among the balls
with different sizes. We also examine the cellophane sheets and the aluminium
foils of the size $L=200$ mm and obtain $2.6\lesssim d_{f}\lesssim 2.8$ for
both of them. The micro-CT also allows us to reconstruct 3-d structure of a
line drawn on the crumpled sheets of paper. The Hurst exponent for the root
mean square displacement along the line is estimated as $H\approx 0.9$ for the
length scale shorter than the scale of the radius of gyration, beyond which
the line structure becomes more random with $H\sim 0.5$.

###### pacs:

##  I Introduction

Crumpling a sheet of paper is the easiest way to make effective shock
absorbing buffer. A hand crumpled paper ball is very light with typically more
than 80% of its volume being empty, still shows strong resistance against
compression. These properties make it ideal spacer for box packing. Origin of
these properties is the large stretching energy in comparison with the bending
energy for a thin paper sheet. As a result, upon crumpling a sheet, Gaussian
curvature remains close to zero everywhere except for at singular points of
developable cone structures[1, 2, 3, 4]. This imposes stringent constraint on
the way how the paper sheet crumples, thus produces strong resistance against
compression even if much of the space is still empty[5, 6].

In spite of their complex structure, the balls of crumpled sheet have been
known to show simple scaling laws[7, 8, 9]. The radius of the ball $R$ follows
the scaling relation with the size of the original sheet $L$ as

$R\sim L^{\alpha},$ (1)

and with the force $F$ applied to make the crumpled ball as

$R\sim F^{-\delta}.$ (2)

The exponents have been estimated as $0.80\lesssim\alpha\lesssim 0.95$ for
aluminium foil and paper, and $\delta\approx 0.2$ for aluminium foil [7, 8, 9,
6, 5, 10, 11]. The small value of $\delta$ corresponds to the fact that the
crumpled sheets resists strongly against compression.

These two power law relations suggest the fractal structure in the object, and
may be combined into the single relation[12, 5]

${R\over h}\propto\left({L\over h}\right)^{\alpha}\left({F\over
Yh}\right)^{-\delta},$ (3)

where $Y$ and $h$ are the 2-d Young modulus and the sheet thickness,
respectively. The scaling relations (1) and (2), however, represent different
aspects of the structure, namely, the former implies the scaling in the
structure among the crumpled balls with different sizes, and the latter
suggests self-similarity of internal structure of each crumpled ball[13]111 It
should be noted that the definition of the applied force $F$ and the radius of
crumpled ball $R$ depend on an experimental protocol. Due to the plasticity
and the relaxation involved during and after crumpling, there is certain
subtleties in what these values really mean, but we will not go into this
problem, simply assuming that such ambiguity does not affect the scaling
relations we will study in the present work.

The internal structure of the crumpled paper ball has been studied by
examining the crease networks on an unfolded sheets[15, 16, 17, 18], the cross
sections obtained by cutting the balls in half[19, 11, 18], or the sequences
of holes made by a needle piercing through the balls[13] as well as numerical
simulations[12, 20, 21, 22]. These are indirect way of observing the internal
structure.

The X-ray micro computed tomography (micro-CT) is computer tomography with the
high resolution of the order of 100 $\mu$m, and makes it possible to study the
internal structure of crumpled sheets without either unfolding or destroying
them. It has been used to study the crumpled balls of aluminium foil to
examine the density distribution, the curvatures of the sheets, and the
fractal dimensions of the structure[23, 24, 25]. In this paper, we use micro-
CT to examine the internal structure of crumpled sheets of paper, cellophane,
and aluminium foils to determine scaling properties of the structure.
Experimental procedure is described in Sec.II, the scaling analysis for the
observed quantities is provided in Sec.III, experimental results are presented
to obtain several exponents in Sec. IV, and the discussions are given in Sec.
V.

##  II Experimental procedure

|  Tracing paper |  Cellophane |  Aluminium foil
---|---|---|---
Manufacturer |  Kokuyo Co., Ltd. |  Toyo Co. |  Shimojima Co., Ltd.
Thickness [mm] |  0.0385$\pm$ 0.0008 |  0.0203$\pm$0.0002 |  0.0115$\pm$0.0004
Density [$\rm g/m^{2}$] |  42.7$\pm$0.5 |  30.0$\pm$0.7 |  27.4$\pm$0.6
Table 1: Sample specifications.

Square sheets of tracing paper with the side length $L=$ 50, 100, 200, and 300
mm are hand crumpled without any specific protocol. Before being scanned, the
crumpled balls are left for 7 days under the environment with the temperature
$25\pm 2^{\circ}$C and the humidity $25\pm 5$% to allow them to settle, in
order to avoid structure relaxation during about 30 minutes of the scanning
time222 See Supplemental Material at [url] for the relaxation effects during
the 7-day waiting time in the sample preparation. . The X-ray micro-CT
(inspeXio SMX-100CT, Shimadzu Corporation, Kyoto, Japan) is operated at 40 kV
with 100 $\mu$A, and scans a cubic region of the space with the linear size
around $4$ cm at a resolution 50$\sim$150 $\mu{\rm m}$ with around $400^{3}$
voxels. We also scan hand crumpled cellophane sheets and aluminium foils of
the size $L=200$ mm following the same procedure described above except that,
in the case of aluminium foil, the samples are scanned on the day when they
are crumpled. The thicknesses and densities of the samples are listed in Table
1.

The micro-CT produces grey scale data for each slice of cross section; the
black and white binary data are generated by setting appropriate threshold,
and then the 3-d structures are reconstructed. Figure 1 shows examples of the
reconstructed 3-d structures from the CT images and the cross sections for
crumpled paper, cellophane sheet, and aluminium foil.

We also reconstruct the 3-d configurations of a line drawn on a crumpled paper
(Fig.2). A straight line is drawn using the ink that contains tungsten (Macky
gold, Zebra, Tokyo, Japan), which absorbs X-rays more efficiently than the
paper. Thus the line positions can be extracted from the CT images by setting
a higher threshold than that for the paper structure.

Figure 1: CT images of crumpled paper (a), cellophane sheet (b), and aluminium
foil (c). The upper images are reconstructed 3-d structures and the lower
images are their cross sections. Figure 2: A line on a crumpled paper of the
size $L=200$ mm. (a) schematic illustration of a line drawn on a paper with
the ink that contains tungsten, (b) an example of three dimensional image of a
crumpled paper with the line, and (c) the extracted 3-d image of the line.

##  III Observed Quantities and Scaling Analysis

Before we show our experimental results, we give some scaling analysis for the
physical quantities we observe.

### Three dimensional structure of crumpled sheets:

The mass fractal dimension $D_{M}$ is defined by the scaling relation between
the mass of the paper $M$ and the radius of the crumpled paper ball $R$ as

$M\sim R^{D_{M}}.$ (4)

Since the mass of the paper $M$ is proportional to its area, $M\propto L^{2}$,
the mass fractal dimension $D_{M}$ is related to the exponent $\alpha$ in
Eq.(1) as

$D_{M}=2/\alpha.$ (5)

On the other hand, the fractal dimension $d_{f}$ of the internal structure of
each crumpled ball is measured by the box counting method, using the relation

$N_{\rm box}\sim l^{-d_{f}},$ (6)

where $N_{\rm box}$ is the number of occupied boxes with the linear size $l$.

These two exponents correspond to two distinct features, i.e. the fractal
dimension $d_{f}$ defined by Eq.(6) describes the self-similarity of each
crumpled paper ball structure while the mass fractal dimension $D_{M}$
represents the similarity among crumpled paper balls with different size $L$.
However, the self-similar structure of each crumpled paper ball suggests the
self-similarity among those of different sizes, and Eq.(6) could be extended
as

$N_{\rm box}\sim{M\over h^{2}\sigma}\left({l\over h}\right)^{-d_{f}},$ (7)

where $h$ is the paper thickness and $\sigma$ is the area density of the
sheet. This leads to $d_{f}=D_{M}$ if $R$ is identified as the box size $l$
that corresponds to $N_{\rm box}=1$, i.e.

$1\sim{M\over h^{2}\sigma}\left({R\over h}\right)^{-d_{f}}.$ (8)

The structure factor $S(\bm{q})$ is the Fourier transform of the density
correlation function $g(\bm{r})$,

$S(\bm{q})=\iiint e^{-i\bm{q}\cdot\bm{r}}g(\bm{r})d\bm{r},$ (9)

where the density correlation is defined by

$g(\bm{r})=\iiint\big{<}\rho(\bm{r}^{\prime}+\bm{r})\rho(\bm{r}^{\prime})\big{>}d\bm{r}^{\prime}$
(10)

in terms of the density distribution $\rho(\bm{r})$. Here,
$\left<\cdots\right>$ means the ensemble average. If the structures of the
crumpled paper ball are self-similar with the fractal dimension $d_{f}$, then
the density correlation should be of the scaling form,

$g(r)\sim r^{d_{f}-3},$ (11)

then the structure factor is expected to be of the scaling form

$S(q)\sim q^{-d_{f}}.$ (12)

### Cross section of a crumpled sheet:

A 3-d CT image consists of hundreds of two dimensional slices of the density
distribution in the cross sections. We analyze the structure of the cross
section which contains the center of mass of the crumpled sheet. Suppose that
we take the center of mass of the crumpled sheet as the origin of the co-
ordinate, and consider the cross section by the $z=0$ plane. Let $\sigma_{\rm
cs}(\bm{r}_{\perp})$ denote the 2-d density distribution on the cross section
as a function of a position $\bm{r}_{\perp}=(x,y)$ on the $z=0$ plane,

$\sigma_{\rm cs}(\bm{r}_{\perp})=\int_{-h/2}^{h/2}\rho(\bm{r}_{\perp},z)dz.$
(13)

The density correlation on the cross section $g_{\rm cs}(\bm{r}_{\perp})$ is
defined by

$g_{\rm cs}(\bm{r}_{\perp})=\iint\big{<}\sigma_{\rm
cs}(\bm{r}^{\prime}_{\perp}+\bm{r}_{\perp})\sigma_{\rm
cs}(\bm{r}^{\prime}_{\perp})\big{>}d\bm{r}^{\prime}_{\perp},$ (14)

and the structure factor of the cross section $S_{\rm cs}(\bm{q}_{\perp})$ is
given by the 2-d Fourier transform

$S_{\rm cs}(\bm{q}_{\perp})=\iint
e^{-i\bm{q}_{\perp}\cdot\bm{r}_{\perp}}g_{\rm
cs}(\bm{r}_{\perp})d\bm{r}_{\perp},$ (15)

where $\bm{q}_{\perp}$ is the wave vector within the cross section plane. If
we assume the same form as Eq.(11) for $g_{\rm cs}$ as

$g_{\rm cs}(r_{\perp})\sim r_{\perp}^{d_{f}-3},$ (16)

then the structure factor would behaves as

$S_{\rm cs}(q_{\perp})\sim q_{\perp}^{-(d_{f}-1)},$ (17)

which simply shows that the fractal dimension for the cross section is
$d_{f}-1$.

### Straight line drawn on a crumpled paper:

The CT technique allows us to study the structure of a line drawn on a
crumpled paper. A straight line on a flat paper is deformed into a random
structure as the paper is crumpled. The configuration of the crumpled line can
be represented by the function

$\bm{r}_{\rm line}(s);\qquad 0\leq s\leq L,$ (18)

where $s$ is the distance along the line from one of the end. The root mean
square (RMS) distance $R_{\rm line}(s)$ from one of the end points is defined
by

$R_{\rm line}(s)\equiv\sqrt{\left<\big{(}\bm{r}_{\rm line}(s)-\bm{r}_{\rm
line}(0)\big{)}^{2}\right>}.$ (19)

If this shows the power law behavior with the Hurst exponent $H$,

$R_{\rm line}(s)\sim s^{H},$ (20)

then the scaling argument based on the self-similarity assumption leads to the
scaling law

$g_{\rm line}(r)\sim r^{-3+1/H}$ (21)

for the correlation of the line in 3-d space, and the scaling law

$S_{\rm line}(q)\sim q^{-1/H}$ (22)

for the 3-d structure factor of the line.

The size of the crumpled line $R_{\rm line}$ is expected to scale with the
size of the paper $L$ as

$R_{\rm line}\sim L^{\alpha_{\rm line}}.$ (23)

If the size of the line $R_{\rm line}$ should be of the same order with the
size of the crumpled ball $R$, and also with RMS of the end-to-end distance of
the line $R_{\rm line}(L)$ of Eq.(20), then the Hurst exponent should be
related to the mass fractal dimension as

$H=\alpha_{\rm line}=\alpha=2/D_{M}.$ (24)

##  IV Experimental Results

We analyze the 3-d structures of the crumpled sheet balls, the cross sections
of the balls, and the lines drawn on the paper.

### Three dimensional structure of crumpled sheets:

Figure 3(a) shows the averaged density distribution of crumpled paper as a
function of the distance from the center of mass $r$. Each line represents the
average distribution over the direction and around 10 samples, and the arrows
show the average values of the radii of gyration for the corresponding sizes
of the paper. One can see that the averaged density distribution inside the
ball is roughly uniform, but it is a slightly decreasing function of $r$ for
the crumpled balls of the smaller sheets $L=50$ and 100 mm, almost constant in
the range of $r\lesssim 12$ mm for that of $L=200$ mm, and slightly increasing
in $r\lesssim 16$ mm for that of $L=300$ mm. It is not clear how this tendency
extends to larger sheets. In Fig.3(b), the radii of gyration $R_{g}$ are
plotted against the paper size $L$ in the logarithmic scale. The data range is
less than one decade and not enough to give a precise value of the exponent,
but the plots are consistent with the power law behavior

$R_{g}\sim L^{\alpha}$ (25)

with the exponent $\alpha\approx 0.74$. This gives $D_{M}=2/\alpha\approx 2.7$
from Eq.(5).

Figure 3: Density distributions (a) and radii of gyration (b) for crumpled
paper sheets of the size $L=$ 50, 100, 200, and 300 mm. (a) Each curve
represents average over around 10 samples and the arrows show the average
values of the radii of gyration. The density distributions are normalized so
that the total mass should be proportional to $L^{2}$. (b) The grey circles
and the black circles represent the radius of gyration for each sample and
average values over them, respectively.

We estimate the fractal dimensions $d_{f}$ by the box counting method; the
number of occupied boxes $N_{\rm box}$ are plotted against the linear size
$l_{\rm box}$ of the box divided by $R_{g}$ in the logarithmic scale in
Fig.4(a) for paper and (b) for cellophane and aluminium foil. Each data point
is an average of about 10 samples. The data for different size $L$ or
different materials are shifted vertically by multiplying by the factor 2 to
avoid overlapping of the plots. The estimated $d_{f}$ for the crumpled paper
is 2.7 from Fig.4(a). This is consistent with $D_{M}$ estimated by $R_{g}$,
and suggests the self-similarity in the 3-d structure of the crumpled paper
ball as we have discussed. The fractal dimensions $d_{f}$ for both the
cellophane sheet and aluminium foil are estimated as 2.8 from Fig.4(b).

The structure factors $S(q)$ for the reconstructed 3-d structures from the CT
data are plotted in the logarithmic scale for the paper of sizes $L=50\sim
300$ mm in Fig.4(c), and for the paper sheets, the cellophane sheets, and the
aluminium foils of the size $L=200$ mm in Fig.4(d). Each data point represents
averaged value of about 10 samples. These structure factors $S(q)$ show the
power law behavior

$S(q)\sim q^{-\beta}.$ (26)

The apparent value of the exponent $\beta$ for the crumpled paper in Fig.4(c)
increases with the paper size $L$ and the fitted value for the largest paper
size $L=300$ mm is $\beta\approx 2.5$. The exponents $\beta$ for the
cellophane sheets and the aluminium foils are estimated from the plots for
$L=200$ mm in Fig.4(d) as $\beta\approx 2.6$. These values of $\beta$
estimated by $S(q)$ are slightly smaller than those of $d_{f}$ estimated by
the box counting method although they should coincide as

$\beta=d_{f}$ (27)

from Eq.(12) if the self-similarity holds.

Figure 4:  The box counting data and the averaged structure factors $S(q)$ for
the 3-d structures of crumpled paper, cellophane, and aluminium foil. The box
size $l_{\rm box}$ and the wave number $q$ are scaled by the radii of gyration
of each structure. $N_{\rm box}$ for different size $L$ (a) and different
materials (b) are shifted by the factor 2 to avoid overlapping.

### Cross section structure of crumpled sheets:

The structure of the cross section of the crumpled sheet is examined in the
same way. Figure 5 shows the box counting data and the 2-d structure factor
for the cross section of the crumpled sheets; The horizontal axes are scaled
by the 2-d radius of gyration

The fractal dimensions $d_{f,\rm cs}$ for the cross sections are estimated by
the box counting method, and we obtain $d_{f,\rm cs}\approx 1.8$ for the paper
of the size $L=50\sim 300$ mm, and $d_{f,\rm cs}\approx 1.8$ for cellophane
and aluminium of the size $L=200$ mm. The 2-d structure factor also shows the
power law behavior

$S_{2d}(q_{\perp})\sim q^{-\beta_{\rm cs}}$ (28)

with the exponent $\beta_{\rm cs}\approx 1.5$ for the paper and $\beta_{\rm
cs}\approx 1.6$ for the cellophane and the aluminium. Since the self-
similarity leads to the relation

$\beta_{\rm cs}=d_{f,\rm cs}=d_{f}-1$ (29)

from Eq.(17), the obtained values for $\beta_{\rm cs}$ are consistent with the
corresponding exponents for the 3-d structure.

Figure 5:  The box counting numbers $N_{\rm box}$ and the averaged structure
factors $S_{2d}(q)$ for the cross sections of crumpled paper sheets,
cellophane sheets, and aluminium foils. The box size $l_{\rm box}$ and the
wave number $q$ are scaled by the radii of gyration of each structure. $N_{\rm
box}$ for different size $L$ (a) and different materials (b) are shifted by
the factor 2 to avoid overlapping.

### Structure of a line on a crumpled paper:

Fig.6(a) shows a configuration of a line one a crumpled paper. The radii of
gyration $R_{g,\rm line}$ for the lines are plotted as a function of the paper
size $L$ in Fig.6(b) along with $R_{g}$ for the whole structure as have been
plotted in Fig.3(b); the values of $R_{g,\rm line}$ are somewhat smaller than
those of $R_{g}$, but the plot is consistent with the power law behavior with
the same exponents as it should be, i.e.

$R_{g,\rm line}\sim L^{\alpha_{\rm line}}.$ (30)

with $\alpha_{\rm line}\approx 0.74$. In Fig.6(c), RMS distance $R_{\rm
line}(s)$ defined by Eq.(19) is plotted as a function of $s$ in the
logarithmic scale to estimate the Hurst exponent $H$. The plot shows roughly
the power law behavior

$R_{\rm line}(s)\sim s^{H}$ (31)

with $H\approx 0.9$ for the range $R_{\rm line}(s)\lesssim R_{g,\rm line}$,
beyond which it saturates and seems to follow the power law with a smaller
exponent $H\approx 0.5$, but the range is too small to determine its behavior
with confidence. Finally, Fig.6(d) shows that the 3-d structure factor for the
line $S_{\rm line}(q)$ behaves as

$S_{\rm line}(q)\sim q^{-\beta_{\rm line}}\quad\mbox{with }\beta_{\rm
line}\approx 1.1.$ (32)

The results by Eqs.(31) and (32) are consistent with the relation (22), but
not with the relation (24).

Figure 6: The structure of a line drawn on a paper. (a) An example of a three dimensional configuration of a line on a crumpled paper of the size $L=50$ mm. (b) Radius of gyration v.s. the paper size $L$. (c) Averaged root mean square displacement at the position $s$ from one of the ends along the line. The arrows indicate the $R_{\rm line,g}$ for each size of the paper. (d) Averaged structure factors $S(q)$. The wave number $q$ is scaled by $R_{g,\rm line}$ for each size of the paper.  | 3-d structure | cross section | line
---|---|---|---
|  $\alpha$ |  $\beta$ |  $d_{f}$ |  $\beta_{\rm cs}$ |  $d_{f,\rm cs}$ |  $\alpha_{\rm line}$ |  $H$ |  $\beta_{\rm line}$
Paper ($L=50\sim 300$ mm) |  0.74 |  2.5 |  2.7 |  1.5 |  1.8  |  0.74 |  0.9 |  1.1
Cellophane ($L=200$ mm) |  $-$ |  2.6 |  2.8 |  1.6 |  1.8 |  $-$ |  $-$ |  $-$
Aluminium foil ($L=200$ mm) |  $-$ |  2.6 |  2.8 |  1.6 |  1.8 |  $-$ |  $-$ |  $-$
Table 2: Obtained exponents for the crumpled paper, cellophane, and aluminium
foils.

##  V Discussions

We have estimated several exponents which describe the scaling behaviors of
the structure of crumpled sheets; The results are tabulated in Table 2.
Considering the range of the data points and the data fluctuations, error bars
for each exponent would be around $\pm 0.1$.

As is described in Sec.II, the paper and cellophane samples are scanned 7 days
after they are crumpled. This is to avoid the structural relaxation
immediately after crumpling and to obtain steady values of measurement. One
might wonder if the 7-day waiting time is enough to obtain steady values,
especially when logarithmically slow relaxation has been observed up to three
weeks in similar systems of crumpled thin sheets[27, 16, 28]. In these works,
the slow relaxation is observed in the compaction height under a constant
force[27], the diameter of crumpled ball after the folding force is
withdrawn[16], and the stress under a constant compression[28]. We have
checked if a similar slow relaxation shows any significant effect on the
quantities we measure in the present work, but it turns out that the
relaxation effects are virtually invisible in the scaling exponents of the
structure beyond one hour after crumpling (See Supplemental Material).

As we have discussed in Sec.III, the exponents in Table 2 are related to the
two basic exponents: the mass fractal dimension $D_{M}$ and the fractal
dimension $d_{f}$. The former represents the scaling behavior among the
crumpled balls of the different sheet sizes while the latter describes the
self-similarity of the structure of each crumpled sheet. For the paper sheet
of the size $L=50\sim 300$ mm, the mass fractal dimension estimated from
$\alpha$ by Eq.(5) is $D_{M}\approx 2.7$. The fractal dimension for the
internal structure $d_{f}$ are estimated from the box counting and the
structure factor; The structure factor tends to give smaller values for the
fractal dimension as has been found also in [25], but overall data suggest
that the fractal dimension is in the range $2.5\lesssim d_{f}\lesssim 2.8$.
These estimates are consistent with

$D_{M}=d_{f}$ (33)

within the accuracy of our estimate, suggesting that the self-similarity in
the structure of each crumpled ball gives rise to the similarity among the
balls with different sizes. For the cellophane and the aluminium with the size
$L=$ 200 mm, the estimated fractal dimension for the structure $2.6\lesssim
d_{f}\lesssim 2.8$, which are slightly larger than that for the paper; We do
not have data to estimate $D_{M}$ for the cellophane and the aluminium.

These values for $D_{M}$ and $d_{f}$ are somewhat larger than the values
obtained for $D_{M}$ in previous works: 2.51 [9] and 2.1$\sim$2.5 [13] for
paper, and 2.5 [7, 8] and 2.3 [5] for aluminium foil. In these estimates, the
external diameter is used for the size of the ball $R$ while in the present
work the radius of gyration $R_{g}$ calculated from the density distribution
is used. This may lead to some difference in estimating $D_{M}$ especially
when the data range is not large enough although they should give the same
exponent in the limit of the infinite data range. Note that the fractal
dimensions obtained in the present work for cellophane and aluminium foil are
for the internal structure of each crumpled ball.

The unique piece of information that the micro-CT can provide is the structure
of a line drawn of the crumpled paper sheets. The estimated value of the Hurst
exponent $H\approx 0.9$ for the short length scale suggests that the line
configuration is quite ballistic for the length scale up to $R_{g}$, but it
eventually approaches the random walk for the longer scale. The value of the
Hurst exponent $H\approx 0.9$ is consistent with the relation (22) to the
exponent for the structure factor $\beta_{\rm line}\approx 1.1$, but not with
the relation (24) to $\alpha_{\rm line}\approx 0.74$ or the mass fractal
dimension $D_{M}\approx 2.7$. In other words, for the line on a crumpled
paper, the self-similarity of each line structure in the short length scale is
not consistent with the overall scaling upon changing the size $L$ in contrast
to the case of the whole structure of a crumpled paper, in which case
$D_{M}=d_{f}$, thus the self-similarity of the internal structure is
consistent with the global scaling. The existence of these two regimes for the
line structure may come from the layered structure of crumpled sheets; The
whole structure of crumpled sheets consists of random folding of a wrinkled
sheet, and typical scale of random folding is of order of $R_{g}$ while
wrinkling gives shorter length scale with $H\approx 0.9$.

Before concluding, let us discuss some of the previous works using CT
technique. Lin et al. [25, 24] examined the structure of crumpled aluminium
foils by CT. The aluminium foils of different radius $R_{0}=3\sim 10$ mm are
crumpled into the ball with the same final radius $R=1.5$ mm, i.e. different
compaction ratios. They estimate the fractal dimension by the box counting and
the correlation dimension as a function of the compaction ratio $R/R_{0}$[25].
Their estimates of the dimensions coincide fairly well with our estimates for
the aluminium foil by the box counting and the Fourier transform. They also
measured the correlation for the tangent vector and observed the layered
structure[24].

Cambou and Menon[23] also used CT to examine the internal structure of
crumpled aluminium foils. They obtained the mass distribution, the
distribution of the normal vector and the curvature radii, and found that they
are distributed quite uniformly. They also found the layered structure, but
again their orientation is distributed uniformly. These uniform distributions
might appear to contradict the fractal structure that has been found in the
present work as well as earlier studies[13, 25, 19, 18]. They are, however,
not contradicting because what they studied are averaged distributions of the
quantities; Spatial inhomogeneity of fractal structure varies from a sample to
another, thus does not likely show in the averaged distribution.

In the present work, we did not examine the scaling relation with the applied
force, Eq.(2). Its exponent $\delta$ should represent how the crumpling energy
increases as a paper sheet is crumpled into a smaller ball, thus should come
from the self-similarity of the internal structure, although we do not know
yet how it is related with other exponents.

###### Acknowledgements.

This work is partially supported by JSPS KAKENHI Grant Number JP20K03882.

## References

  * Lobkovsky  _et al._ [1995] A. Lobkovsky, S. Gentges, H. Li, D. Morse, and T. A. Witten, Science 270, 1482 (1995).
  * Cerda and Mahadevan [1998] E. Cerda and L. Mahadevan, Phys. Rev. Lett. 80, 2358 (1998).
  * Cerda  _et al._ [1999] E. Cerda, S. Chaieb, F. Melo, and L. Mahadevan, Nature 401, 46 (1999).
  * Witten [2007] T. A. Witten, Rev. Mod. Phys. 79, 643 (2007).
  * Balankin  _et al._ [2007a] A. S. Balankin, I. C. Silva, O. A. Martínez, and O. S. Huerta, Phys. Rev. E 75, 051117 (2007a).
  * Lin  _et al._ [2008] Y. C. Lin, Y. L. Wang, Y. Liu, and T. M. Hong, Phys. Rev. Lett. 101, 125504 (2008).
  * Kantor  _et al._ [1986] Y. Kantor, M. Kardar, and D. R. Nelson, Phys. Rev. Lett. 57, 791 (1986).
  * Kantor  _et al._ [1987] Y. Kantor, M. Kardar, and D. R. Nelson, Phys. Rev. A 35, 3056 (1987).
  * Gomes [1987] M. A. Gomes, J. Phys. A: Math. Gen 20, L283 (1987).
  * Habibi  _et al._ [2017] M. Habibi, M. Adda-Bedia, and D. Bonn, Soft Matter 13, 4029 (2017).
  * Deboeuf  _et al._ [2013] S. Deboeuf, E. Katzav, A. Boudaoud, D. Bonn, and M. Adda-Bedia, Phys. Rev. Lett. 110, 104301 (2013).
  * Vliegenthart and Gompper [2006] G. A. Vliegenthart and G. Gompper, Nature Materials 5, 216 (2006).
  * Balankin  _et al._ [2007b] A. S. Balankin, R. C. M. de Oca, and D. S. Ochoa, Phys. Rev. E 76, 032101 (2007b).
  * Note [1] It should be noted that the definition of the applied force $F$ and the radius of crumpled ball $R$ depend on an experimental protocol. Due to the plasticity and the relaxation involved during and after crumpling, there is certain subtleties in what these values really mean, but we will not go into this problem, simply assuming that such ambiguity does not affect the scaling relations we will study in the present work.
  * Blair and Kudrolli [2005] D. L. Blair and A. Kudrolli, Phys. Rev. Lett. 94, 166107 (2005).
  * Balankin  _et al._ [2006] A. S. Balankin, O. S. Huerta, R. Cortes Montes de Oca, D. S. Ochoa, J. Martínez Trinidad, and M. A. Mendoza, Phys. Rev. E 74, 061602 (2006).
  * Andresen  _et al._ [2007] C. A. Andresen, A. Hansen, and J. Schmittbuhl, Phys. Rev. E 76, 026108 (2007).
  * Balankin  _et al._ [2013] A. S. Balankin, A. Horta Rangel, G. García Pérez, F. Gayosso Martinez, H. Sanchez Chavez, and C. L. Martínez-González, Phys. Rev. E 87, 052806 (2013).
  * Balankin  _et al._ [2010] A. S. Balankin, D. S. Ochoa, I. A. Miguel, J. P. n. Ortiz, and M. A. M. Cruz, Phys. Rev. E 81, 061126 (2010).
  * Tallinen  _et al._ [2008] T. Tallinen, J. A. Åström, and J. Timonen, Phys. Rev. Lett. 101, 106101 (2008).
  * Tallinen  _et al._ [2009] T. Tallinen, J. A. Åström, and J. Timonen, Nature Materials 8, 25 (2009).
  * Liou  _et al._ [2014] S.-F. Liou, C.-C. Lo, M.-H. Chou, P.-Y. Hsiao, and T.-M. Hong, Phys. Rev. E 89, 022404 (2014).
  * Cambou and Menon [2011] A. D. Cambou and N. Menon, Proceedings of the National Academy of Sciences 108, 14741 (2011).
  * Lin  _et al._ [2009a] Y.-C. Lin, J.-M. Sun, J.-H. Hsiao, Y. Hwu, C. L. Wang, and T.-M. Hong, Phys. Rev. Lett. 103, 263902 (2009a).
  * Lin  _et al._ [2009b] Y.-C. Lin, J.-M. Sun, H. W. Yang, Y. Hwu, C. L. Wang, and T.-M. Hong, Phys. Rev. E 80, 066114 (2009b).
  * Note [2] See Supplemental Material at [url] for the relaxation effects during the 7-day waiting time in the sample preparation.
  * Matan  _et al._ [2002] K. Matan, R. B. Williams, T. A. Witten, and S. R. Nagel, Phys. Rev. Lett. 88, 076101 (2002).
  * Lahini  _et al._ [2017] Y. Lahini, O. Gottesman, A. Amir, and S. M. Rubinstein, Phys. Rev. Lett. 118, 085501 (2017).

Supplemental Material to
“Fold analysis of crumpled sheet using micro computed tomography”

Yumino Hayase, Hitoshi Aonuma, Satoshi Takahara,

Takahiro Sakaue, Shun’ichi Kaneko, and Hiizu Nakanishi

()

> Relaxation effects on paper samples after crumpling are examined for some of
> the scaling exponents that we report in the main text. It turns out that the
> effects are negligible already at one hour after crumpling in comparison
> with the overall accuracy of the measurement.

As being described in the text, the crumpled paper and cellophane balls are
left for 7 days after crumpling before being CT-scanned in order to allow them
to settle. Some may wonder if the 7-day waiting time is enough because it has
been observed that crumpled paper balls undergo slow relaxation logarithmic in
time over more than three weeks. In order to examine how our results may or
may not depend on the waiting time, we measure the time evolution of some of
the quantities we report in the main text.

The quantities we examined in this supplemental material are (i) the radius of
gyration $R_{g}$, (ii) the power spectrum of the density distribution $S(q)$,
and (iii) the box counting data for the fractal dimension. Five sheets of
square paper with the size $L=200$ mm are crumpled by hands, and are scanned
by micro-CT at one hour, 3 days, 7 days, and 14 days after being crumpled. The
samples are kept under fairly constant condition controlled at the temperature
$24\pm 1^{\circ}$ C and the humidity $40\pm 10\%$.

Figure S1:  Reconstructed images of five crumpled balls at one hour, 3 days, 7
days, and 14 days after being crumpled.

Before we present the measured quantities, we show the reconstructed images of
the samples upon each scan in Fig.S1. The sample orientation is not controlled
at the scanning, thus the lines of sight for these images are adjusted on the
computer by 3d viewer after each image is reconstructed. One may notice some
relaxation of the configuration, but they do not change much after one hour.

The calculated radii of gyration are listed in Table S1. They increase
slightly over time, but the change in the radius for each sample after one
hour is less than 1 mm; it is smaller than variation among samples, and should
not have any effect on the estimate of the exponent $\alpha\approx 0.74$ in
Fig.3(b) of the main text. The last data of Sample 5 is smaller than the
earlier ones. We do not understand how this happened, but a certain relaxation
in the density inhomogeneity inside the ball could yield such a fluctuation of
$R_{g}$.

time Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 1 hour 12.31 11.48 11.58
12.04 12.44 3 days 12.52 11.52 11.67 12.16 12.71 7 days 12.83 11.76 11.99
12.45 13.21 14 days 13.11 11.97 12.28 12.85 12.49

Table S1:  The radii of gyration $R_{g}$ [mm] of each sample at 1 hour, 3
days, 7 days, and 14 days after being crumpled. They are calculated from
binarized data of the images.

Figure S2 shows the evolution of the power spectra $S(q)$ of the density
distributions and the box counting data $N_{\rm box}$ v.s. $l_{\rm box}$ for
the fractal dimension; the plus marks show the data for each sample and the
red solid circles show the average over the 5 samples. The lines are the
scaling fits by $S(q)\sim q^{-\beta}$ and $N_{\rm box}\sim l_{\rm
box}^{-d_{f}}$ with $\beta=2.5$ and $d_{f}=2.7$. The data for different scan
times are shifted by the factor $10^{-2}$. One may find small changes over
time at the both ends of $q$ or $l_{\rm box}$, but the data are quite stable
in the middle range, where the exponents are estimated, therefore, the
estimates for the exponent $\beta$ and $d_{f}$ do not depend on the waiting
time for all the data we examine.

Figure S2: The evolution of the power spectra $S(q)$ and the box counting data
$N_{\rm box}$ v.s. $l_{\rm box}$. The data for different times are shifted by
the factor $10^{-2}$.


11institutetext: State Key Laboratory of Networking and Switching Technology,
Beijing University of Posts and Telecommunications 22institutetext: McGill
University 33institutetext: Department of Computer Science, City University of
Hong Kong
44institutetext: South China University of Technology
44email<EMAIL_ADDRESS>44email<EMAIL_ADDRESS>
# Distinctive Image Captioning via CLIP Guided Group Optimization

Youyuan Zhang 22 Jiuniu Wang 33 Hao Wu 44 Wenjia Xu Corresponding author.11

###### Abstract

Image captioning models are usually trained according to human annotated
ground-truth captions, which could generate accurate but generic captions. In
this paper, we focus on generating distinctive captions that can distinguish
the target image from other similar images. To evaluate the distinctiveness of
captions, we introduce a series of metrics that use large-scale vision-
language pre-training model CLIP to quantify the distinctiveness. To further
improve the distinctiveness of captioning models, we propose a simple and
effective training strategy that trains the model by comparing target image
with similar image group and optimizing the group embedding gap. Extensive
experiments are conducted on various baseline models to demonstrate the wide
applicability of our strategy and the consistency of metric results with human
evaluation. By comparing the performance of our best model with existing
state-of-the-art models, we claim that our model achieves new state-of-the-art
towards distinctiveness objective.

###### Keywords:

Distinctive Image Captioning, CLIP, Similar Image Group, Group Embedding Gap

##  1 Introduction

The task of image captioning involves interpreting the semantic information of
images into natural language sentences. It is considered as a fundamental task
in both computer vision and natural language processing. Image captioning is
applied in various scenario such as assisting visually impaired people [23],
search engine [12], medical imaging [41], etc.

General requirements of image captioning include two aspects: fluency, as the
sentence should be well-formed as natural language; descriptiveness, as the
sentence should accurately describe salient information of the image. In this
paper, we focus on another important aspect: distinctiveness, which aims to
generate captions with sufficient details and distinguish the target image
from other similar ones [37]. Currently, most of the existing models only use
human annotated ground-truth captions as labels. These models naturally focus
on the similarity of generated captions and ground-truth captions. As a
result, the generated captions are usually too generic because they are
trained to be close to all ground-truth captions. Some pioneers [33, 21]
introduce diversity and discriminability objective that operate on semantic
embeddings of images to generate more diverse or discriminative captions.
However, as discussed in [37], models following these objectives generate
grammarly different sentences with similar meanings, which does not follow the
requirement of distinctiveness.

Some of the restrictions of traditional training strategies that prevent
models from being distinctive include: 1) Only traditional language quality
metrics, such as CIDEr [35], are used for training; 2) Ground-truth captions
are annotated by human which does not necessarily include much details other
than salient objects. Therefore, optimizing towards ground-truth captions
through language quality metrics naturally makes captioning models focus on
salient information and ignore distinctive details in images. To overcome the
intrinsic shortcomings of traditional training strategy, we hereby propose a
group based optimization strategy that uses CLIP [28] to guide distinctive
training. We propose a series of reference-free metric based on CLIP to
evaluate the relevance and distinctiveness of captions and images.
Furthermore, we introduce similar group which is formed by the similar images
of an target image. The similar group can be seen as hard negatives from which
the model explicitly learns to distinguish detailed features. During training,
our group based optimization takes the similarity of generated captions with
target images as positive reward and takes the similarity with similar images
as negative reward. The similarity is obtained from CLIP thus is not affected
by the drawbacks of human annotations. Our proposed optimization strategy is
“plug and play” which can be applied on any sequence prediction model.
Experimental results show that our proposed strategy achieves outstanding
performance on a variety of existing models.

To summarize, the contributions in this paper are three fold: 1) We propose a
series of reference-free metrics to quantify the distinctiveness of image
captioning. 2) We propose a group based optimization strategy that exploits
the new metrics as reward to guide distinctive training. 3) We conduct
extensive experiments on various state-of-the-art captioning models and
demonstrate the effectiveness of our strategy.

##  2 Related Work

###  2.1 Image Captioning

Deep learning based captioning models generally employ encoder-decoder
framework. Early attempts encode images using CNN [34, 14] based feature
extractor, and output captions through RNN [7] and LSTM [15] based decoder
[36, 16, 22, 24]. Some works use object-level features extracted by Faster-
RCNN [30] and improves the performance. Recent works introduce attention
mechanism to encourage cross-modal interaction between the two modalities and
mitigate the long-term dependency problem in LSTM [6]. For example, [2]
applies bottom-up attention at object-level features and map salient visual
features to output words via top-down attention. [26] applies X-Linear
attention to leverage higher-order intra and inter-modal interactions.

###  2.2 Objectives for Image Captioning

Sequence prediction models are typically trained with maximum likelihood
estimation (MLE) objective by cross-entropy loss. However, as discussed in
[29], this method causes the problem of exposure bias, which accumulates the
error of caption generation at test time. To address this problem, [4]
introduces scheduled sampling to reduce the bias. [29, 31] treat sequence
generation as reinforcement learning (RL) problem. In specific, [31] proposes
self-critical sequence training that optimizes the policy gradient of the
model parameters using REINFORCE with baseline. However, as pointed out in
[10, 9, 39] such approaches usually result in over generic captions.
Therefore, different objectives, such as diversity and discriminability, are
proposed to enrich the expressive capacity of captions. In specific, Div-Cap
[39] propose to measure diversity by analyzing latent semantic space of image
encoding. G-GAN [9] use conditional generative adversarial network (CGAN) [25]
to improve diversity of captions. Disc-Cap [21] trains a retrieval model to
guide discriminative captioning. In terms of distinctiveness objective,
VisPara-Cap [20] employs a two-stage LSTM model and generates two captions,
where the second caption is pharaphrased from the first caption and is more
distinctive. CiderBtw-Cap [37] re-weights the ground-truth captions by
comparing their distinctiveness and trains the model using a weighted loss.
Gdis-Cap [38] designs a group attention module that re-weights the attention
features of images to highlight the distinctive features. CLIP-Cap [5]
directly use CLIP [28] to compute relevance of image-caption pairs and
generates more distinctive captions. Compared with these methods, our proposed
method integrates the advantages of the following: 1) We do not rely on a
specific model and therefore our method is more widely applicable. 2) We use
reference-free reward to avoid the drawbacks of human annotation. 3) We use
similar image group to better guide distinctive training via comparing.

###  2.3 Metrics for distinctive image captioning

Traditional metrics, such as CIDEr [35], SPICE [1], BLEU [27], ROUGE [18] and
MENTEOR [3], are used to measure the similarity between generated captions and
ground-truth captions. In terms of distinctiveness metrics, [40] designs
SPICE-U and introduces a notion of uniqueness over concepts in captions. [37]
proposes CIDErBtw metric which reflects the distinctiveness by measuring the
similarity of the target captions and captions of similar images. [38]
proposes DisWordRate metric which reflects the percentage of distinctive words
in captions.

##  3 Methodology

Figure 1: Pipeline of group optimization using weighted group embedding gap
reward.

The goal of an image captioning model is to generate a caption $c_{0}$ for the
target image $I_{0}$. In this paper, we focus on distinctive image captioning,
which could not only describe the target image $I_{0}$ correctly but also
include as many details as possible to distinguish $I_{0}$ from other similar
images $\\{I_{1},...,I_{K}\\}$, where $K$ denotes the number of similar
images.

Following the goal of generating distinctive captions, we first propose the
R@K-CLIP metric which uses the language-vision pre-training model CLIP [28] to
quantify the distinctiveness of target caption. We then propose a series of
metrics called Group Embedding Gap (GEG) including GEG-Avg and GEG-Min that
compare the similarity between images and captions, and use these metrics for
both training and evaluation. We design a weighted GEG reward which is used in
self-critical sequence training (SCST) to guide distinctive training. Note
that the GEG training strategy is “plug and play” and can be used on any
sequence prediction models.

###  3.1 Similar Image Group

We start by introducing the construction process of the similar image groups
$\\{I_{1},...,I_{K}\\}$. In each split of the training, validation and test
dataset, we construct the similar image group within the same split. The $K$
similar images are retrieved according to the semantic similarity between
their ground truth captions and the target image $I_{0}$ by image-to-text
retrieval.

Different from [37] which uses VSE++ [11] pre-trained model, we use CLIP
(Contrastive Language-Image Pre-Training) [28] trained on larger dataset to
perform image-to-text retrieval. CLIP is a neural network trained on a 400
million of image-text pairs collected from a variety of publicly available
sources on the Internet. CLIP contains an image encoder $E_{i}(I)$ and a text
encoder $E_{t}(c)$, which takes $I$ and $c$ as the input image and text, and
output their embeddings respectively. In this paper, we use the ViT-B/32
architecture, of which the image encoder $E_{i}(\cdot)$ is a vision
transformer with 12 transformer layers. The architecture of this vision
transformer has minor change compared with the original version as an layer
normalization is added to the combined patch and positional embeddings before
the transformer. The text encoder $E_{t}(\cdot)$ first encode the raw text
using a lower-cased byte paired encoding (BPE) with 49,152 vocab size and then
fed them to a Transformer also with 12 transformer layers. Experimental result
in section 4.2 demonstrates that CLIP has better image retrieval performance
than VSE++.The similarity of image $I$ and caption $c$ is defined as the inner
product of their encoding representations:

$\displaystyle f_{\scalebox{0.5}{CLIP}}(I,c)$
$\displaystyle=\frac{E_{i}(I)^{T}E_{t}(c)}{\left\lVert{E_{i}(I)}\right\rVert\left\lVert{E_{t}(c)}.\right\rVert}$
(1)

Note that CLIP, as well as other cross-modal retrieval models are not trained
on image-image pairs. Therefore, to obtain the semantic similarity between
images, we define the similarity score of target images $I_{0}$ and candidate
similar image $I_{i}$ through the ground-truth captions of $I_{i}$ as
following:

$\displaystyle s(I_{0},I_{i})$
$\displaystyle=\max_{n\in\\{1,2,\dots,N\\}}f_{\scalebox{0.5}{CLIP}}(I_{0},c_{i}^{n})\,,$
(2)

where $N$ is number of ground-truth captions for each image and $c_{i}^{n}\in
G_{t}(I_{i})$ is the $n$-th ground-truth caption of image $I_{i}$.

In practice, we first construct the set of similar captions
$\\{c_{1}^{\prime},c_{2}^{\prime},...,c_{N^{\prime}}^{\prime}\\}$, where
$N^{\prime}=N(K+1)$ and $K$ is the size of similar image group. In this way,
at least $K$ different images corresponding to the captions in the similar
caption set are included. Then we choose the top $K$ images to form the
similar image group $S_{im}(I_{0})$.

The similar image groups for all images in each split of the dataset are
generated offline. The training process uses similar image groups in both
forward and backward propagation. However, even though the evaluation of
distinctiveness of captions involves their similar image groups, the model is
unaware of the similar image groups and only takes target images as input in
the test set at inference time. Therefore, we claim that similar image groups
supervise the model to learn distinctive features of images in a contrastive
way.

###  3.2 Metrics

####  3.2.1 R@K-CLIP

The distinctiveness of image captioning is hard to quantify because it depends
on the dataset as well as the retrieval model. A general idea of evaluating
distinctiveness is that the distinct caption should be able to retrieve its
target image over a bunch of images via a text-image retrieval model. Previous
work [37, 38] use VSE++ to the retrieve images. In this paper, we choose CLIP
as the retrieval model because it is pre-trained on larger dataset thus more
reliable. User study in section 4.5 suggests that our metric for
distinctivness is highly consistent with human judgement.

In this paper, we fix the retrieval model to be CLIP ViT-B/32 and evaluate the
distinctiveness of the generated captions by computing the recall rate of
captions that retrieves their corresponding images, which is denoted as Recall
at K (R@K). In specific, we define our proposed retrieval metric R@K-CLIP as
the percentage of captions which retrieves their corresponding images within
the top $K$ images by CLIP ViT-B/32 model.

####  3.2.2 Group Embedding Gap

We propose the Group Embedding Gap (GEG) metrics that can be used for both
training and evaluating a distinctive captioning model. These metrics are
group specified, which means they depend on the choice of similar image group
for each target image. Intuitively, a distinctive caption $C$ should be close
to the target image $I$ while be distant to other similar images $S_{im}(I)$.
To this end, the GEG metrics measure the distinctiveness of generated caption
$c$ for target image $I$ by subtracting the similarity of $c$ and $S_{im}(I)$
from the similarity of $c$ and $I$. The difference is GEG-Avg ($G_{avg}$)
computes the average similarity of $c$ and every similar images in $S_{im}(I)$
while GEG-Min ($G_{min}$) focuses on the most similar one in $S_{im}(I)$:

$\displaystyle G_{avg}(I,c)$
$\displaystyle=f_{\scalebox{0.5}{CLIP}}(I,c)-\frac{1}{\left\lVert
S_{im}(I)\right\lVert}\sum\limits_{I^{\prime}\in
S_{im}(I)}f_{\scalebox{0.5}{CLIP}}(I^{\prime},c)\,,$ (3) $\displaystyle
G_{min}(I,c)$ $\displaystyle=f_{\scalebox{0.5}{CLIP}}(I,c)-\max_{I^{\prime}\in
S_{im}(I)}f_{\scalebox{0.5}{CLIP}}(I^{\prime},c)\,,$ (4)

where the CLIP model $f_{\scalebox{0.5}{CLIP}}$ is used to measure the
similarity between captions and images.

###  3.3 Group Embedding Gap Reward

Usually the training schedule for caption models involves both maximum
likelihood estimation (MLE) training and self-critical sequence training
(SCST) [31]. The purpose of SCST is to optimize a non-differentiable metric,
such as CIDEr [35]. SCST treats the generation of captions as a reinforcement
learning problem, where the expected reward of non-differentiable metric is
optimized by evaluating policy gradient.

Following the sketch of SCST, we compute the policy gradient as follows:

$\displaystyle\bigtriangledown L(\theta)=-\mathbb{E}_{c\sim
P(c|I;\theta)}[r(c)-r(c_{greedy})\bigtriangledown_{\theta}\log
P(c|I;\theta)]\,,$ (5)

where $P(c|I;\theta)$ is the probability distribution of the model generating
caption $c$ for target image $I$ with model parameter $\theta$. $c_{greedy}$
is the caption generated via greedy decoding, i.e. choosing the word with
highest probability at every time step. The expectation of the gradient of
loss can be estimated by sampling $c$ using beam search. Therefore, for each
sampled caption $c_{beam}$,

$\displaystyle\bigtriangledown
L(\theta)=-(r(c_{beam})-r(c_{greedy}))\bigtriangledown_{\theta}\log
P(c_{beam}|I;\theta)\,.$ (6)

$r(c)$ originally take the CIDEr score of caption $c$ as reward. Here we add
the group embedding gap metric as a new component and re-weight the original
CIDEr score reward and group embedding gap reward to get the final reward for
sampled caption $c$:

$\displaystyle r(c)=\alpha\text{CIDEr}(c,\hat{c})+\beta G(I,c)\,,$ (7)

where $G$ represents the choice of group embedding gap metric of either
$G_{avg}$ or $G_{min}$. $\alpha$ and $\beta$ are hyper-parameters to control
weights of the two rewards.

##  4 Experiments

In this section, we first introduce our experiment settings. Then we validate
the reliability of the R@K-CLIP metrics. Next we evaluate the improvement of
our group based optimization strategy as for distinctiveness on a variety of
captioning models, and compare the best model with existing state-of-the-art.
Finally, user study and qualitative results are provided to show the
consistency of metric based results and the human evaluation.

###  4.1 Implementation Details

####  4.1.1 Dataset

All our experiments are performed on MSCOCO dataset [19] with Karpathy split
[16], which consists of a training set with 113,287 images, validation set
with 5,000 images, and test set with 5,000 images. Each image is annotated
with 5 captions. In this paper, all experiments results are reported on the
test set of Karpathy split.

####  4.1.2 Models

Following [17], we evaluate our optimization strategy on seven different
baseline model architectures: LSTM-A3 [44], Attention [42], Up-Down [2], GCN-
LSTM [43], Transformer [32], Meshed-Memory [8] and X-LAN [26].

Two types of features, global features and top-down features are used in these
models. Global features are extracted from Resnet-101 [14] and top-down
features are spatial features extracted from Faster-RCNN [30]. For each image,
the global feature is a vector of dimension 2,048 and top-down feature is a
matrix of shape $N_{RoI}\times 1,024$ where $N_{RoI}$ is the number of regions
of interest. In specific, global features are used by LSTM-A3. Top-down
features are used by Attention, Up-down, GCN-LSTM, Transformer, Meshed-Memory
and X-LAN.

Each model is trained using three methods: 1) standard MLE objective with
cross-entropy loss, denoted as “model”; 2) SCST optimization, which is first
trained with MLE objective then optimized by SCST by CIDEr reward, denoted as
“model+SCST”; 3) GEG optimization, which uses SCST to optimize the weighted
group embedding gap reward based on the SCST model, and is denoted as
“model+SCST+GEG”.

####  4.1.3 Training Details

We apply GEG reward training on each model after SCST optimization. For GEG
reward training, we collect $K=5$ images to form the similar image group and
set $\alpha=0.1$ and $\beta=10$. Maintaining a non-zero $\alpha$ is crucial
because CLIP is not trained with language modeling objective. Ablation study
in section 4.4 shows that training using only GEG reward results in grammarly
incorrect captions. We use Adam optimizer with base learning rate $r=5\times
10^{-5}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\epsilon=10^{-8}$. Each
model is trained with GEG reward for 10 epochs.

####  4.1.4 Metrics

We use two groups of metrics. The first group includes language quality
metrics, i.e., CIDEr, BLEU@1, BLEU@2, BLEU@3, BLEU@4, METEOR, ROUGE-L and
SPICE, to evaluate the similarity of generated captions and ground-truth
captions. The second group includes distinctiveness metrics of retrieval
metrics, i.e., R@1-CLIP, R@5-CLIP, R@10-CLIP and GEG metrics GEG-Avg, GEG-Min.

We report the image retrieval performance of CLIP ViT-B/32 on MSCOCO test set
in Table 1. The results suggest that CLIP ViT-B/32 has generally better
performance than VSE++ because of higher R@1 and R@10 and slightly lower R@5.

Model | R@1 | R@5 | R@10 |
---|---|---|---|---
VSE++ [11] | 24.1 | 52.8 | 66.2 |
CLIP [28] | 30.4 | 50.0 | 66.9 |
Table 1: Image Retrieval performance of different models on MS COCO Karpathy
test split.

###  4.2 Main Results

We report the main results of applying different training strategies on seven
baseline captioning models in Table 2. The seven models are ranked by CIDEr
score of “model+SCST” from low to high. In specific, we consider CIDEr as the
main language quality metric to measure the similarity of generated caption
and ground-truth captions. The remaining language quality metrics are
positively correlated to CIDEr. We also consider R@K-CLIP metrics as the main
distinctiveness metrics, and GEG metrics are positively correlated to
R@K-CLIP.

By observing the results in Table 2, we have the following observations.
Firstly, by comparing the models trained by MLE or SCST from top to bottom,
the results suggest that models with higher language quality metrics tend to
have higher distinctiveness metrics. For instance, compared to Up-Down [2]
getting CIDEr value 113.1 and R@1-CLIP at 16.08, GCN-LSTM [43] achieves higher
CIDEr at 116.3 and R@1-CLIP at 17.46. This correlation implies the consistency
of the two types of metrics. When no extra supervision or objective is
provided, models which can better mimic human annotations describes more
distinctive details of the image. Then, by comparing “model+SCST+GEG” with the
baseline “model+SCST”, the results suggest that models trained by GEG reward
generate much more distinctive captions. For instance, compared to X-LAN+SCST
getting R@K-CLIP rates at 21.3, 44.74 and 55.74 repectively, X-LAN+SCST+GEG
achieves higher R@K-CLIP rates at 28.12, 50.3, 67.18 respectively. This
improvement demonstrates the effectiveness of our proposed methods. Although
the language metrics get lower, for instance, in the X-LAN group,
X-LAN+SCST+GEG has lower CIDEr (121.7) than X-LAN+SCST (130.0), we argue that
this is because human annotated ground-truth captions only consider the
consistency with target images and are not distinctive in many cases. We
provide case study in section 4.6 to demonstrate that the decrease in language
metrics does not negatively affect caption quality and on the contrary
generates better captions than both baseline captions and ground-truth
captions

Model | CIDEr | R@1 | R@5 | R@10 | B@1 | B@2 | B@3 | B@4 | M | R | S | GEG-Avg | GEG-Min
---|---|---|---|---|---|---|---|---|---|---|---|---|---
LSTM-A3 [44] | 107.7 | 11.52 | 30.88 | 42.34 | 75.3 | 59.0 | 45.4 | 35.0 | 26.7 | 55.6 | 19.7 | 0.0296 | -0.0064
LSTM-A3+SCST | 117.0 | 10.98 | 29.38 | 41.34 | 77.9 | 61.5 | 46.7 | 35.0 | 27.1 | 56.3 | 20.5 | 0.0288 | -0.0073
LSTM-A3+SCST+GEG | 114.3 | 12.98 | 32.86 | 45.7 | 76.9 | 60.4 | 45.6 | 33.9 | 26.9 | 55.8 | 20.6 | 0.0330 | -0.0046
Attention [42] | 113.0 | 14.74 | 36.4 | 48.6 | 76.4 | 60.6 | 46.9 | 36.1 | 27.6 | 56.6 | 20.4 | 0.0344 | -0.0015
Attention+SCST | 123.1 | 14.8 | 35.5 | 47.04 | 79.4 | 63.5 | 48.9 | 37.1 | 27.9 | 57.6 | 21.3 | 0.0338 | -0.0026
Attention+SCST+GEG | 116.6 | 17.68 | 40.94 | 52.84 | 77.1 | 60.8 | 45.9 | 34.3 | 27.4 | 56.4 | 21.1 | 0.0406 | 0.0026
Up-Down [2] | 113.1 | 16.08 | 37.4 | 49.18 | 76.3 | 60.3 | 46.6 | 36.0 | 27.6 | 56.6 | 20.7 | 0.0351 | -0.0008
Up-Down+SCST | 124.7 | 15.46 | 36.2 | 49.28 | 80.1 | 64.3 | 49.7 | 37.7 | 28.0 | 58.0 | 21.5 | 0.0349 | -0.0014
Up-Down+SCST+GEG | 114.2 | 19.78 | 42.7 | 55.68 | 76.24 | 60.15 | 45.46 | 33.85 | 27.41 | 56.24 | 21.17 | 0.0424 | 0.0048
GCN-LSTM [43] | 116.3 | 17.46 | 40.02 | 52.68 | 76.8 | 61.1 | 47.6 | 36.9 | 28.2 | 57.2 | 21.2 | 0.0373 | 0.0011
GCN-LSTM+SCST | 127.2 | 17.22 | 39.46 | 52.16 | 80.2 | 64.7 | 50.3 | 38.5 | 28.5 | 58.4 | 22.1 | 0.0375 | 0.0015
GCN-LSTM+SCST+GEG | 112.6 | 20.66 | 45.26 | 58.16 | 76.30 | 60.04 | 45.50 | 33.74 | 26.92 | 55.76 | 20.83 | 0.0441 | 0.0063
Transformer [32] | 116.6 | 18.76 | 42.26 | 54.16 | 76.4 | 60.3 | 46.5 | 35.8 | 28.2 | 56.7 | 21.3 | 0.0391 | 0.0028
Transformer+SCST | 130.0 | 19.6 | 42.72 | 55.62 | 80.5 | 65.4 | 51.1 | 39.2 | 29.1 | 58.7 | 23.0 | 0.0403 | 0.0033
Transformer+SCST+GEG | 123.2 | 25.06 | 50.8 | 62.96 | 77.66 | 61.83 | 47.57 | 35.79 | 28.58 | 57.36 | 22.84 | 0.0485 | 0.0102
Meshed-Memory [8] | 116.0 | 18.66 | 41.12 | 52.92 | 76.3 | 60.2 | 46.4 | 35.6 | 28.1 | 56.5 | 21.2 | 0.0383 | 0.0024
Meshed-Memory+SCST | 131.1 | 19.64 | 42.94 | 55.86 | 80.7 | 65.5 | 51.4 | 39.6 | 29.2 | 58.9 | 22.9 | 0.0402 | 0.0034
Meshed-Memory+SCST+GEG | 124,4 | 25.44 | 50.88 | 63.7 | 78.15 | 62.33 | 48.17 | 36.48 | 28.74 | 57.48 | 22.91 | 0.0484 | 0.0102
X-LAN [26] | 120.7 | 20.26 | 43.88 | 55.4 | 77.5 | 61.9 | 48.3 | 37.5 | 28.6 | 57.6 | 21.9 | 0.0402 | 0.0036
X-LAN+SCST | 130.0 | 21.3 | 44.74 | 57.74 | 80.4 | 65.2 | 51.0 | 39.2 | 29.4 | 59.0 | 23.2 | 0.0416 | 0.0045
X-LAN+SCST+GEG | 121.7 | 28.12 | 50.3 | 67.18 | 76.23 | 60.55 | 46.37 | 34.82 | 28.66 | 56.96 | 22.84 | 0.0517 | 0.0127
Stack-Cap [13] | 120.4 | 13.96 | 34.36 | 45.78 | - | - | 47.9 | 36.1 | 27.4 | 56.9 | 20.9 | 0.0339 | -0.0019
Disc-Cap [21] | 120.1 | 11.24 | 29.74 | 41.86 | - | - | 48.5 | 36.1 | 27.7 | 57.8 | 21.4 | 0.0313 | -0.0042
CIDErBtw-Cap [37] | 127.9 | 19.16 | 41.98 | 54.54 | - | - | 51.0 | 38.5 | 29.1 | 58.2 | 23.0 | 0.0390 | 0.0025
Gdis-Cap [38] | 127.5 | 17.62 | 38.94 | 51.68 | - | - | 50.0 | 38.1 | - | - | - | - | -

Table 2: The main results of applying different training strategy, i.e., the
MLE optimization (denoted as “model”), the SCST optimization (denoted as
“model + SCST”), and the GEG optimization (denoted as “+ GEG”), on seven
baseline models. Here R@K represents R@K-CLIP and B@K represents BLEU@K. M, R
and S represent METEOR, ROUGE-L and SPICE respectively.

###  4.3 Comparison with State-of-the-art

We compare our model with best performance X-LAN+SCST+GEG with three state-of-
the-art captioning models which also focus on distinctiveness object at the
bottom of Table 2. We compare distinctiveness of each model based on R@K-CLIP
and GEG metrics. Compared with Stack-Cap [13] and Disc-Cap [21], our model
shows better performance in both language quality metrics and distinctiveness
metrics. For instance, comparing to Stack-Cap [13] which has CIDEr at 120.4
and R@1-CLIP at 13.96, our X-LAN+SCST+GEG has higher CIDEr at 121.7 and higher
R@1-CLIP at 28.12. Compared with CIDErBtw-Cap [37] and Gdis-Cap [38], our
model has lower CIDEr score but still much better performance in
distinctiveness . Recall that decrease in CIDEr score does not imply the
reduction of language quality. Therefore, we claim that our model achieves new
state-of-the-art towards distinctive captioning.

###  4.4 Ablation Study

In order to further illustrate the effectiveness of our training strategy. We
provide an ablation study including five experiments as shown in Table 3. We
compare our proposed standard SCST+GEG strategy with four other strategies on
X-LAN model. X-LAN+SCST is the same baseline model trained using SCST as in
Table 2. X-LAN+SCST++ is the baseline model trained for 10 more epochs so
X-LAN+SCST++ and X-LAN+SCST+GEG are trained for the same number of epochs.
Instead of using weighted GEG reward, X-LAN+SCST+ER only takes the CLIP
embedding similarity of generated caption and target image as reward. In this
case, the embedding reward does not include the negative average similarity of
generated caption and similar images. X-LAN+SCST+GEG-sole sets $\alpha=0$
which removes CIDEr reward and only takes GEG reward.

By analysing Table 3, we have the following observations. Firstly, the results
of comparing X-LAN+SCST++ and X-LAN+SCST shows that further training has
little improvement on both CIDEr metric and distinctiveness metric.
Introducing the possitive similarity of generated caption and the target image
as reward (i.e., X-LAN+SCST+ER) already helps the model in improving the
distinctivess. For instance, X-LAN+SCST+ER improves the R@1-CLIP score from
21.8 (of X-LAN+SCST) to 25.86. Comparison between X-LAN+SCST+GEG and
X-LAN+SCST+ER shows the effectiveness of group based learning. By negatively
rewarding the similarity of generated caption and similar images, the
X-LAN+SCST+GEG model generates captions that are better in both language
quality metric (e.g., improves CIDEr from 120.7 to 121.7) and distinctive
metric (e.g., promoting R@1-CLIP from 25.86 to 28.13). X-LAN+SCST+GEG-sole
model has unexpectedly high distinctiveness metric without the CIDEr score
constraint. However, as discussed in section 4.1.3, CLIP is not trained with
language quality objective. Training with only GEG reward causes frequent
grammar mistakes of word disorder and repentance. An example is “a woman
eating a white dessert cake with lit candles with lit candles in a restaurant
with lit candles with”, which is completely unreadable.

Model | R@1-CLIP | R@5-CLIP | R@10-CLIP | CIDEr
---|---|---|---|---
X-LAN+SCST | 21.3 | 44.74 | 57.74 | 130.0
X-LAN+SCST++ | 21.8 | 44.24 | 57.98 | 130.5
X-LAN+SCST+ER | 25.86 | 52.38 | 64.88 | 120.7
X-LAN+SCST+GEG (Full model) | 28.13 | 50.3 | 67.18 | 121.7
X-LAN+SCST+GEG-sole | 32.82 | 61.3 | 72.8 | 17.4
Table 3: Comparison between different training strategies Model | R@1-User | R@5-User | R@10-User
---|---|---|---
X-LAN+SCST | 21.9 | 46.3 | 65.4
X-LAN+SCST+GEG | 31.7 | 55.7 | 71.5
Table 4: User-evaluated image retrieval Criterion | Win(%) | Tie(%) | Loss(%)
---|---|---|---
Accuracy | 16.7 | 80.3 | 3.0
Distinctiveness | 44.5 | 47.2 | 8.3
Table 5: Comparison on accuracy and distinctiveness. The rates of win, tie,
loss are related to X-LAN+SCST+GEG model.

---
X-LAN+SCST | A man sitting on a motorcycle in the street.
X-LAN+SCST+GEG | A man sitting on a motorcycle in front of a brick building.
Groud-Truth Captions | A couple of men that are on some motorcycles.
A cop riding on the back of a black motorcycle.
A man on a motorcycle in a parade.
A motorcycle rider dressed in leather drives a motorcycle down a crowded
street.
A man on a motorcycle stopped on a city street.
X-LAN+SCST | A vase filled with flowers on a table.
X-LAN+SCST+GEG | A glass vase filled with pink roses on a table.
Groud-Truth Captions | Creative centerpiece floral arrangement at an outdoor event.
A wedding centerpiece made of flowers and various other plants.
A vase of flowers sitting on an outdoor table.
A vase filled with flowers on top of a table.
A floral arrangement inside a clear cylinder shaped vase.
X-LAN+SCST | A plate with a sandwich and a bowl of soup.
X-LAN+SCST+GEG | A sandwich with meat and tomatoes on a white plate next to a bowl of soup.
Groud-Truth Captions | A plate with an open sandwich next to a little bowl of soup and a wine glass.
A plate topped with an open face sandwich net to a glass of wine.
There is a plate of food and a bowl of soup.
A plate with a sandwich, a bowl of soup and a wine glass.
A plate of food that has an open faced sandwich on it.
X-LAN+SCST | A bathroom with a toilet and a sink.
X-LAN+SCST+GEG |  A woman standing next to a toilet in a store.
Groud-Truth Captions | A display of a toilet and sink in a store.
Toilet and sink on display in a furniture store.
A toilet with a sink on the back is displayed.
A store filled with bathroom equipment and accessories.
A white toilet with a brown lid and a sink behind it.
X-LAN+SCST | A yellow train is traveling down the tracks.
X-LAN+SCST+GEG | A yellow train traveling down the tracks next to a traffic light.
Groud-Truth Captions | A train is near a track switching station.
A train on a track near many trees.
A train crossing with a train coming down the railroad tracks.
The train is slowing its speed heading into the station.
A yellow light is next to some train tracks.

Table 6: Example captions from the baseline model and our model with similar
images. The leftmost images with red borders are the target images. Red words
show the distinctive details in the target images which are not mentioned in
either baseline model captions or ground-truth captions

###  4.5 User Study

In order to fairly evaluate the improvement of our model and verify the
consistency between our proposed metrics and human perspective, we conducted
two user studies.

In the first experiment, each user is given a random group of images and a
caption. The caption is generated from either the baseline X-LAN+SCST model or
X-LAN+SCST+GEG model. The group of images contains an target image
corresponding to the caption and 20 similar images. The user is asked to list
in order the 10 most relevant images. In this way, we obtain the R@K-User
metric.

In the second experiment, each user is asked to give preference of two
captions generated by the above two models following two criteria: accuracy,
which caption describes the target image more accurately; distinctiveness,
which caption is more informative about the target image. The user can choose
win, loss or tie of the X-LAN+SCST+GEG model.

We received 1,200 responses in total. The results are shown in Table 4 and
Table 5. Compared with baseline model X-LAN+SCST, our model increases the
image retrieval rate R@1-User, R@5-User and R@10-User by 9.8, 9.4 and 6.1, and
surpasses the baseline model in both accuracy and distinctiveness. Both
observations show that human evaluation is consistent with metric results in
section 4.2.

###  4.6 Qualitative Results

The results shown in Table 6 illustrates the improvement of distinctiveness of
our model compared with baseline model. Due to the space limit, we only show
five groups of results and analyze the first two groups.

In each group of the images, the leftmost image is the target image and
followed by five similar images. We compare the generated captions from the
baseline model X-LAN+SCST and our model X-LAN+SCST+GEG. In both cases, the
baseline model generates accurate but generic captions that describe the
salient object in the target image. Contrastively, our model describes more
details that distinguish the target image from potential similar images. In
the first case, our model not only captures the main object, i.e. “man” and
“motorcycle”, but also notice the “brick building” in the background. In the
second case, our model extends the generic description of “vase” and “flowers”
to more specific description “glass vase” and “pink roses”. Recall that during
inference, our model is unaware of the similar images in test set. In this
case, after distinctive training, our model is able to capture the details
that distinguish the target image from potential similar images, which
demonstrate the effectiveness of our training strategy.

The two cases also explain the decrease in language quality metrics, e.g.
CIDEr. In the first case, none of the ground-truth captions mentions “brick
building”. In the second case, none of the ground-truth captions mentions
“glass vase” and “pink roses”. However, these ignored details are crucial to
distinguish the target images. Training with only language quality objective
results in generic captions possibly because the model is stuck in the minimum
which is close to every ground-truth captions. After distinctive training, our
model can discover more details than human annotated captions, which is
considered as the main reason of the decrease in CIDEr metric.

##  5 Conclusions

In this paper, we focus on the distinctiveness of image captioning. We propose
the R@K-CLIP metrics and GEG metrics to evaluate distinctiveness of captioning
models. In order to improve distinctiveness, we design a simple and effective
strategy which optimize on similar image group to guide distinctive training.
In this process, the large-scale vision-language pre-trained model CLIP is
used to measure the similarity of image-caption pairs. We conduct extensive
experiments on various existing models and demonstrate the wide applicability
and effectiveness of the group based optimization strategy in term of
improving distinctiveness. Finally, we provide qualitative results and user
study to show the consistency of the metric-side improvement with human
judgement.

## References

  * [1] Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: European conference on computer vision. pp. 382–398. Springer (2016)
  * [2] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)
  * [3] Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
  * [4] Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems 28 (2015)
  * [5] Cho, J., Yoon, S., Kale, A., Dernoncourt, F., Bui, T., Bansal, M.: Fine-grained image captioning with clip reward. arXiv preprint arXiv:2205.13115 (2022)
  * [6] Cho, K., Courville, A., Bengio, Y.: Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia 17(11), 1875–1886 (2015)
  * [7] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  * [8] Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10578–10587 (2020)
  * [9] Dai, B., Fidler, S., Urtasun, R., Lin, D.: Towards diverse and natural image descriptions via a conditional gan. In: Proceedings of the IEEE international conference on computer vision. pp. 2970–2979 (2017)
  * [10] Dai, B., Lin, D.: Contrastive learning for image captioning. Advances in Neural Information Processing Systems 30 (2017)
  * [11] Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017)
  * [12] Frankel, C., Swain, M.J., Athitsos, V.: Webseer: An image search engine for the world wide web. Tech. rep., Technical Report 96-14, University of Chicago, Computer Science Department (1996)
  * [13] Gu, J., Cai, J., Wang, G., Chen, T.: Stack-captioning: Coarse-to-fine learning for image captioning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [16] Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3128–3137 (2015)
  * [17] Li, Y., Pan, Y., Chen, J., Yao, T., Mei, T.: X-modaler: A versatile and high-performance codebase for cross-modal analytics. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 3799–3802 (2021)
  * [18] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)
  * [19] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [20] Liu, L., Tang, J., Wan, X., Guo, Z.: Generating diverse and descriptive image captions using visual paraphrases. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4240–4249 (2019)
  * [21] Luo, R., Price, B., Cohen, S., Shakhnarovich, G.: Discriminability objective for training descriptive captions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6964–6974 (2018)
  * [22] Ma, L., Lu, Z., Shang, L., Li, H.: Multimodal convolutional neural networks for matching image and sentence. In: Proceedings of the IEEE international conference on computer vision. pp. 2623–2631 (2015)
  * [23] Makav, B., Kılıç, V.: A new image captioning approach for visually impaired people. In: 2019 11th International Conference on Electrical and Electronics Engineering (ELECO). pp. 945–949. IEEE (2019)
  * [24] Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 (2014)
  * [25] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [26] Pan, Y., Yao, T., Li, Y., Mei, T.: X-linear attention networks for image captioning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10971–10980 (2020)
  * [27] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)
  * [28] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [29] Ranzato, M., Chopra, S., Auli, M., Zaremba, W.: Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 (2015)
  * [30] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
  * [31] Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7008–7024 (2017)
  * [32] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565 (2018)
  * [33] Shetty, R., Rohrbach, M., Anne Hendricks, L., Fritz, M., Schiele, B.: Speaking the same language: Matching machine to human captions by adversarial training. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4135–4144 (2017)
  * [34] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [35] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015)
  * [36] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2015)
  * [37] Wang, J., Xu, W., Wang, Q., Chan, A.B.: Compare and reweight: Distinctive image captioning using similar images sets. In: European Conference on Computer Vision. pp. 370–386. Springer (2020)
  * [38] Wang, J., Xu, W., Wang, Q., Chan, A.B.: Group-based distinctive image captioning with memory attention. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 5020–5028 (2021)
  * [39] Wang, Q., Chan, A.B.: Describing like humans: on diversity in image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4195–4203 (2019)
  * [40] Wang, Z., Feng, B., Narasimhan, K., Russakovsky, O.: Towards unique and informative captioning of images. In: European Conference on Computer Vision. pp. 629–644. Springer (2020)
  * [41] Xiong, Y., Du, B., Yan, P.: Reinforced transformer for medical image captioning. In: International Workshop on Machine Learning in Medical Imaging. pp. 673–680. Springer (2019)
  * [42] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057. PMLR (2015)
  * [43] Yao, T., Pan, Y., Li, Y., Mei, T.: Exploring visual relationship for image captioning. In: Proceedings of the European conference on computer vision (ECCV). pp. 684–699 (2018)
  * [44] Yao, T., Pan, Y., Li, Y., Qiu, Z., Mei, T.: Boosting image captioning with attributes. In: Proceedings of the IEEE international conference on computer vision. pp. 4894–4902 (2017)