# Optimization of Quantum-dot Qubit Fabrication via Machine Learning

Antonio B. Mei<EMAIL_ADDRESS>Ivan Milosavljevic  Amanda L. Simpson  Valerie
A. Smetanka  Colin P. Feeney  Shay M. Seguin  Sieu D. Ha  Wonill Ha  Matthew
D. Reed  HRL Laboratories, LLC, 3011 Malibu Canyon Road, Malibu, California
90265, USA

###### Abstract

Precise nanofabrication represents a critical challenge to developing
semiconductor quantum-dot qubits for practical quantum computation. Here, we
design and train a convolutional neural network to interpret in-line scanning
electron micrographs and quantify qualitative features affecting device
functionality. The high-throughput strategy is exemplified by optimizing a
model lithographic process within a five-dimensional design space and by
demonstrating a new approach to address lithographic proximity effects. The
present results emphasize the benefits of machine learning for developing
robust processes, shortening development cycles, and enforcing quality control
during qubit fabrication.

Semiconductor quantum-dot qubits offer an enticing path towards the
realization of scalable quantum information processing systems.Ladd and
Carroll (2018) Variants based on tensile-strained isotopically pure silicon
(800 ppm 29Si) quantum wells feature modest device footprints ($\approx$0.1
$\mu$m2/qubit), long qubit lifetimes ($>$100 ms)Veldhorst  _et al._ (2014);
Kawakami  _et al._ (2014); Eng  _et al._ (2015), long qdubit dephasing times
($>$2 $\mu$s),Eng  _et al._ (2015) fast control ($<$10 ns)Andrews  _et al._
(2019), and rapidly improving two-qubit gate fidelities Zajac  _et al._
(2018); Watson  _et al._ (2018). Despite this appeal, an inherent difficulty
currently limiting progress is meeting the tight fabrication tolerances
required by the technology.Tahan (2020); Sze (1981) 111The relatively large
effective mass of electrons in Si ($m_{t}*=0.19m_{e}$ versus, e.g.,
$m^{*}=0.067m_{e}$ in GaAs) results in weaker tunneling amplitudes, requiring,
in turn, closer quantum dot separations ($\approx$ 150 nm vs. 300 nm in GaAs).
$m_{e}$ is the electron rest mass in vacuum. Uncontrolled fabrication
processes can impact qubit functionality and performance, leading to both
potentially egregious failures (e.g., shorts or opens) as well as more subtle
effects (e.g., poorly controlled exchange couplingReed  _et al._ (2015) or
unreliable electron loadingThorbeck and Zimmerman (2015)).

Figure 1:  Challenges encountered in the fabrication of quantum-dot qubits.
(a) and (b) SEM micrographs of reference devices with well-defined screening
S, plunger P, exchange X, tunnel T, sensor Z, measure M, and bath B gates. The
images in panels (a) and (b) are acquired after pattering screening and
quantum-dot gates, respectively. (c)-(l) Critical failures encountered during
fabrication include (c-d) underexposed S gate, (e-f) overexposed S gate, (g)
missing X and P gates, (h) missing X gates, (i) merged X and P gates, (j)
merged leads, (k) particulate contamination, and (l) collapsed patterns. Scale
bars measure 500 nm.

In this Letter, we present a strategy based on machine learning analysis of
scanning electron microscopy (SEM) images to optimize processes and enforce
quality control during the fabrication of enhancement-mode quantum-dot qubits
on Si1-xGex heteroepitaxial foundations. The approach is showcased using
devices comprised of a linear array of six quantum dots and which feature a
screening gate architectureZajac  _et al._ (2015) to confine electrons in
buried silicon quantum wells.Schaffler (1997) Quantum information is encoded
in superpositions of electron spin states, which are manipulatedPetta  _et
al._ (2005) through pairwise interactions involving exchange gates.Reed  _et
al._ (2015); Andrews  _et al._ (2019) Qubit state readout is performed with
the aid of Coulomb blockade and spin-to-charge conversion.Elzerman  _et al._
(2004); Blumoff and et al. (2020)

Device fabrication involves subtractively defining gates and leads using
negative-tone electron-beam lithography. After depositing gate materials,
hydrogen silsesquioxane (HSQ) films are spun on with thicknesses
$20<d$HSQ$<80$ nm and immediately capped with aluminum layers with thicknesses
$d$Al between 0 (no cap) and $20$ nm. Wafers are then stored in vacuum for a
controlled time $t$HSQ spanning $10$ to $120$ hr before being exposed in an
$100$-keV Raith EBPG5200 e-beam writer with doses $0.4<D<2.8$ mC/cm2.
Development is carried out in MF312 (i.e., tetramethylammonium hydroxide
diluted to 4.9% in water) for $2<t$MF312 $<3$ min. This process is performed
twice: first to define the screening gate and second to pattern the quantum-
dot gates and leads.

The lithography are inspected after development using secondary-electron SEM
to identify nominally successful outcomes as well as various failure modes.
222Micrographs are collected using a Hitachi S9380 scanning electron
microscope equipped with a Schottky emitter. The primary electron beam is
operated at an accelerating voltage of 500 V and a beam current of 7 pA.
Figures 1(a) and 1(b) show representative examples of well-formed devices
after patterning screening and quantum-dot gates, respectively. Defective
devices with malformed screening gates are shown in Figs. 1(c)-(f), with the
latter (former) set of panels corresponding to underexposed (overexposed)
patterns. Figures 1(g)-(i) depict malformed quantum-dot gate morphologies,
ranging from entirely absent to completely merged. An example of a device with
overexposed leads is presented in Fig. 1(j). Devices lost to particulate
contamination and collapsed resist structures are shown in Figs. 1(k) and
1(l), respectively. The qualitative and nonexhaustive failure modes
illustrated by Figs. 1(c)-(l) pose challenges to improving qubit yield since
they are difficult to address with traditional metrology.

Figure 2:  Leveraging machine learning in quantum-dot qubit fabrication for
root-cause analysis, process optimization, and quality control. SEM
micrographs collected in-line are analyzed using a convolutional neural
network consisting of feature extraction and classification subnetworks. (a)
Features extraction occurs across fourteen blocks, each levering depth-wise
separable operations with independent convolution, batch normalization, and
activation layers. Outputs generated by each spatial (depth-wise) layer in
block $B_{1}$ are shown as examples. Similar outputs produced by spatial
convolutions in blocks $B_{1}$, $B_{2}$, $B_{9}$, and $B_{14}$ are provided in
(b). (c) Classification is carried out on mutually exclusive sub-classes using
on regularized three-layer-deep densely-connected networks. (d) Training and
validation accuracy converge above 90% for training epochs $\gtrsim$ 30\.

Our approach to optimizing and monitoring quantum-dot qubit devices in semi-
production environments where individual wafers contain thousands of devices
has been to design a convolutional neural network which is compatible with
high-throughput image interpretation and classification. We formulateAbadi
_et al._ (2016) the network as a two-part system comprised of feature
extraction and classification.

Feature extraction (Fig. 2(a)) is based on a depth-wise separable network
architectureHoward  _et al._ (2017) in which spatial and spectral operations
are factored to achieve high classification accuracy with modest computational
resources.Sandler  _et al._ (2018) The network is assembled as a fourteen-
block structure ($B_{1}$ through $B_{14}$), each with factored operations
involving independent convolutions, batch normalization,Ioffe and Szegedy
(2015) and activationRamachandran  _et al._ (2017). Learning is transferred in
the form of over three million parameters pre-trained using public domain
imagesDeng  _et al._ (2009). Example outputs from the set of spatial
operations within block $B_{1}$ are provided in Fig. 2(a) for the reference
image shown in Fig. 1(a). This particular set of operations highlights right
edges within images. Other features are emphasized in other channels, as the
complete set of $B_{1}$ depth-wise convolution outputs in Fig. 2(b)
demonstrates. Each extraction block $B_{i}$ typically doubles channels and
halves image resolution such that by $B_{14}$, the output adopts the form of a
8$\times$8$\times$1024-dimensional tensor. A consequence of these operations
is that information becomes increasingly abstract and distributed across a
growing number of channels (see Fig. 2(b)). That the final pooledLin  _et al._
(2013) 1024-dimensional vectors serves as a robust foundation for
classification is confirmed by inspecting a griddedJonker and Volgenant (1987)
two-dimensional embedded projectionMaaten (2013) of randomly sampled training
images organized by structural similarities.

Classification is handled by a three-layer-deep 1024-neuron-wide ReLU-
activatedRamachandran  _et al._ (2017) densely connected network with over two
million adjustable parameters (Fig. 2(c)). To control over-fitting,
dropoutHinton  _et al._ (2012); Srivastava  _et al._ (2014) and $L_{1}$
regularization333$L_{1}$ regularization promotes sparse weights, offering a
combination of built-in feature selection and reduced computational cost,
versus $L_{2}$ regularization. are employed.Goodfellow  _et al._ (2016);
Nabian and Meidani (2018) In addition, mutual exclusivity among classes is
enforced via separate but similar classification sub-networks. Node weights
are optimized by teaching the system to recognize thousands of manually
labeled SEM images, including Figs. 1(a)-(l). Training is monitored by re-
purposing one fifth of the training data set for validation. By 30 training
epochs, both training and validation accuracy exceed 90% (Fig. 2(d)),
indicating that despite the large number of tunable parameters, regularization
successfully preserves predictive power. The resulting network thus
constitutes a robust, high-throughput approach to monitoring device quality.

Figure 3:  Quantum-dot qubit device yield in a highly interdependent five-
dimensional lithographic design space. The panels show percentage of yielding
devices $f$ as a function of (a)-(d) exposure dose $D$, (e)-(g) resist
thickness $d$HSQ, (h)-(i) exposure delay $t$HSQ, and (j) aluminum cap
thickness $d$Al. The last row of panels comprise sets of curves for which
development time $t$MF312 = 2, 3, and 4 min. For the earlier rows, other
experimental factors are varied in accordance with the legend. The
lithographic process is maximally stable for $0.5\lesssim D\lesssim 1.3$
mC/cm2, $d$HSQ $\lesssim 50$ nm, $t$HSQ $\gtrsim 48$ hr, $d$Al $\gtrsim 20$
nm, and $t$MF312 $\gtrsim 3$ min.

The automated classification capability provided by the neural network affords
opportunities to understand and optimize processes in high-dimensional design
spaces. We demonstrate this by investigating a model process involving
lithography with HSQ. While HSQ has been successfully employed as an etch-
resistant negative-tone resist to pattern nanostructures with $\lesssim 10$ nm
resolution,Henschel  _et al._ (2003); Grigorescu  _et al._ (2007); Yang  _et
al._ (2009); G  _et al._ (2011) it suffers from a number of outstanding
challenges. Chief among these is an extreme sensitivity to exposure and
development conditions.Delft (2002); Clark  _et al._ (2006); Chen  _et al._
(2006)

To characterize HSQ and establish a robust lithography process suitable for
quantum-dot qubit fabrication, we explore a five-dimensional design space
spanned by exposure dose $D$, HSQ film thickness $d$HSQ, aluminum cap
thickness $d$Al, exposure delay $t$HSQ, and development time $t$MF312.
Lithographic quality, as determined via our neural network following screening
gate patterning, is correlated with experimental variables using a logistical
regression. The analysis models over five thousand unique design points to
identify, with an accuracy of 92%, a response surface which optimally
describes device yield as a function of the predictors. Yield curves obtained
by interpolating one-dimensional sections through the five-dimensional
manifold are plotted in Figs. 3(a)-(j). 444Curves are omitted from panels for
which interactions between variables were not investigated. The dashed line in
each panel represent reference yield curves centered about the design point
$D$ = 1.0 mC/cm-2, $d$HSQ = 50 nm, $d$Al = 15 nm, $t$HSQ = 48 hrs, and
$t$MF312 = 2 min. The additional solid and dotted lines are yield curves where
one of the predictors has been skewed according to the figure legend.

The reference profile in Fig. 3(a) shows the evolution of network-classified
lithography yield $f$ as a function of exposure dose $D$. The percentage of
correctly exposed devices increases rapidly from zero to a peak of $f\approx
75\%$ at $D\approx$ 1.0 mC/cm2, before decreasing asymptotically back to zero.
The lower threshold dose $D_{l}$, defined as the dose above which device yield
exceeds 50%, is found to be $\approx$ $0.6$ mC/cm2, in agreement with values
assessed previously from profilometry measurements.Yang  _et al._ (2009);
Westly  _et al._ (2011); G  _et al._ (2011) The upper dose limit, defined
analogously, is $2.0$ mC/cm2, yielding a dose exposure window $\Delta D\equiv
D_{u}-D_{l}$ = 1.4 mC/cm2.

Also plotted in Fig. 3(a) are yield curves corresponding to resist thickness
values $d$HSQ $=$ 20 and 80 nm. The three curves combined indicate that not
only does increasing the resist thickness monotonically reduce both lower
$D_{l}$ and upper $D_{u}$ exposure dose limits, but $D_{u}$ decreases faster
than $D_{l}$, leading to a restricted span of dose values over which properly
defined patterns are obtained (i.e., a narrower dose latitude $\Delta D$).
This unorthodox behavior, which corresponds to a thickness-dependent exposure
sensitivity, stems from a combination of factors including self-limiting
development processesYang  _et al._ (2009); Lee  _et al._ (2009) and
collateral exposure from secondary electrons generated within the resistG  _et
al._ (2011). Since spun-on resist preferentially accumulates in trenches and
tapers near step edges,Yu  _et al._ (2011) thickness-induced sensitization can
critically impact lithographic yield on uneven surfaces. In extreme cases,
patterns in trenches merge before features on mesas are defined (e.g., Fig.
1(i)).

HSQ is an intrinsically unstable compound.Grove (2005) The instability stems
from highly mobile and reactive hydrogen species, which reside on the vertices
of the cuboid silsesquioxane structure.Loboda and Toskey (1998) Exposure to
energy in the form of heat, electrons, or ultra-violet photons triggers the
release of hydrogen as radicals. The radicals diffuse catalyzing
redistribution reactions which cross-link neighboring molecules into base-
insoluble hydrogen-enriched SiOx oligomers.Olynick  _et al._ (2006); Sidorkin
_et al._ (2009); Olynick  _et al._ (2010) Ambient conditions are
insufficiently inert to completely impede the reactions,Grove (2005) resulting
in limited shelf lifeDelft (2002) and transient exposure behaviorsClark  _et
al._ (2006).

Figure 3(b) shows the impact delaying exposure by $t$HSQ = $24$, $48$, and
$120$ hrs has on the percentage of well-formed devices $f$ for doses $D$
between $0.4$ and $2.8$ mC/cm2. The upper dose limit above which half of the
patterns are overexposed increases with $t$HSQ from $D_{u}$ = $1.8$ ($24$) to
$2.5$ mC/cm2 ($t$HSQ = $120$ hr) as an increasing fraction of neighboring
features begin bridging together (e.g., Figs. 1(e), 1(f), 1(i), and 1(j)).
Improved device yield at longer $t$HSQ is consistent with published reports of
lithographic contrast being higher for HSQ films for which exposure was
delayed.Henschel  _et al._ (2003); Clark  _et al._ (2006)

At a near-optimal exposure dose $D=1.0$ mC/cm2, device yield is lowest and
most susceptible to varying with time when the resist is thick ($d$HSQ
$\gtrsim$ $80$ nm, Fig. 3(e)) and uncapped ($d$Al $=0$ nm, Fig. 3(h)). The
former observation is consistent with the cumulative and diffusive nature of
hydrogen-based catalysis. The latter finding highlights the effectiveness of
aluminum barriers in allowing the film to stabilize in an equilibrated state
through the suppression of hydrogen out diffusion.Siew  _et al._ (2000); Zeng
_et al._ (2000)

Collectively, Figures 3(a)-(j) establish that device yield is maximized for
$0.5\lesssim D\lesssim 1.3$ mC/cm2, $d$HSQ $\lesssim 50$ nm, $t$HSQ $\gtrsim
48$ hr, $d$Al $\gtrsim 20$ nm, and $t$MF312 $\gtrsim 3$ min. Aluminum capping
films not only boost maximum yield from $f\approx 75\%$ at $d$Al = $0$ nm (no
cap) to $f=100\%$ with for $d$Al = 20 nm, but also widen the process window
from $\Delta D\approx 0.5$ to $1.5$ mC/cm2 over the same range (see Fig.
3(c)). Aggressive development utilizing concentrated hydroxides and longer
processing times also extend the upper dose bounds $D_{u}$ by dissolving
collaterally exposed resist (see Fig. 3(d)).

Figure 4:  Machine learning approach to proximity effect correction in
electron-beam lithography. (a) Model describing the change in onset exposure
dose $D_{l}$ with local pattern density $\rho$ and back-to-forward electron
scattering energy ratio $\eta$. Percentage of underexposed devices as a
function of exposure dose $D$ and filling fraction $\rho$ (b) before and (c)
after accounting for proximity effects.

Even with optimized exposure and development procedures, a complication that
arises when carrying out electron-beam lithography is that regions with sparse
features will generally require higher exposure doses than areas where
features are denser (e.g., Fig.1(j)).Chang (1975) Conventionally, the effect
stems from an energy point spread function (PSF) which extends many beam
widths away from the exposure site. Approaches to address proximity effects to
date have relied predominately on measuring linewidths of dedicated monitor
structures.Murai (1992); Lopez  _et al._ (2018) We now demonstrate an
alternative approach that leverages our machine learning algorithm to identify
onset exposure doses as a function of local pattern densities naturally
occurring in the leads of our devices. 555Leads in our devices are spaced at a
pitch of $\sim$ 70 nm and exhibit widths which vary between $\sim$ 10–40 nm.

The evolution of onset exposure dose $D_{l}$ with local pattern density $\rho$
is traditionally described by a series-expanded Gaussian convolution
model,Bickford  _et al._ (2014)

$D_{l}(\rho,\eta)=A\frac{1+\eta}{1+2\rho\eta},$ (1)

in which $\eta$ is the energy ratio of backscattered and forward scattered
electrons and $A$ sets the baseline dose at half filling, i.e.
$D_{l}(\rho=50\%)$. A graphical representation of Eq. 1 is provided in Fig.
4(a) as constant-$\rho$ plots of $D_{l}(\rho,\eta)/A$ versus $\eta$. For
$\eta\lesssim$ 0.1, all curves are approximately equal to unity, indicating
that threshold doses are independent of pattern density. As $\eta$ increases
between $0.1\lesssim\eta\lesssim 10$, the curves fan out with $D_{l}$
increasing (decreasing) for sparse (dense) patterns. In the limit of
arbitrarily large $\eta$, $D_{l}$ approaches $A/(2\rho)$ asymptotically.

Figure 4(b) shows $f_{u}$ the percentage of underexposed devices as a function
of dose $D$ and local pattern density $\rho$. Experimental onset doses
$D_{l}$, defined as the highest value for which $f_{u}<50\%$, decay rapidly
with $\rho$ from $1.30$ mC/cm2 at $\rho=0.19$ to $0.60$ mC/cm2 for
$\rho=0.55$. $\eta$, which controls the rate of decay, is determined
graphically from Fig. 4(a) and rigorously from robust nonlinear regression
analyses ($R^{2}$ $\gtrsim$ 0.99) to be $\eta\simeq 5$. For comparison, Monte
Carlo simulations of electron-sample interactions yield $\eta\approx
0.1$;GenISys (2020) for 500-nm-thick ZEP 666ZEP is a common positive-tone
electron-beam lithography resist based on methyl styrene and chloromethyl
acrylate copolymer. and polymethyl methacrylate, $\eta$ ranges between 0.6 and
1.1, depending on the substrate material.Brown  The significantly larger value
of $\eta$ observed here reflects the complex nature of proximity effects in
HSQ, which involve not only the diffusion and scattering of electrons, but
also hydrogen radicals, which modulate cross-linking energy thresholds.Westly
_et al._ (2011)

A plot of underexposed device percentages $f_{u}(D,\rho)$ after accounting for
proximity effects is provided in Figure 4(c). The results are obtained by
locally modulating base doses within patterns with a spatially varying dose
multiplier that depends on the output of convolutions between computer-aided
designs and an $\eta$-corrected energy PSF. In distinct contrast to the
results obtained for uncorrected exposures (i.e., Fig. 4(b)), the revised
procedure produces threshold doses that are constant at $D_{l}\approx 0.7$
mC/cm2 and independent of pattern density $\rho$, thus enabling a span of
dissimilar geometries to be patterned concurrently.

In summary, we have designed and trained a convolutional neural network to
quantify qualitative failure modes impacting quantum-dot qubit fabrication
yield. The algorithm has been leveraged to understand and optimize a highly-
correlated five-dimensional lithographic process space. A novel strategy for
addressing electron beam proximity effects has also been successfully
demonstrated. The present results highlight the effectiveness of machine
learning in guiding root-cause analysis, process development, and quality
control of complex quantum devices.

##  I Acknowledgments

We thank John R. Lowell, Reed W. Andrews, and Matthew G. Borselli for valuable
discussions. We gratefully acknowledge technical support of HRL Laboratories
Microfabrication Technology Laboratory engineers, especially Jack A. Crowell,
Larken E. Cumberland, Chuong V. Dao, Joshua M. Doria, Elias A. Flores, Vinh S.
Ho, Rosalinda M. Ring, Golam Sabbir, Reggie D. Salinas, Mariano J. Taboada.
This research was carried out with financial support from The Boeing Company.

## References

  * Ladd and Carroll (2018) T. D. Ladd and M. S. Carroll, Encyclopedia of Modern Optics (Second Edition) 1 (2018), 10.1016/B978-0-12-803581-8.09736-8.
  * Veldhorst  _et al._ (2014) M. Veldhorst, J. C. C. Hwang, C. H. Yang, A. W. Leenstra, B. d. Ronde, J. P. Dehollain, J. T. Muhonen, F. E. Hudson, K. M. Itoh, A. Morello, and A. S. Dzurak, Nature Nanotechnology 9, 981 (2014), 1407.1950 .
  * Kawakami  _et al._ (2014) E. Kawakami, P. Scarlino, D. R. Ward, F. R. Braakman, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, and L. M. K. Vandersypen, Nature Nanotechnology 9, 666 (2014), 1404.5402 .
  * Eng  _et al._ (2015) K. Eng, T. D. Ladd, A. Smith, M. G. Borselli, A. A. Kiselev, B. H. Fong, K. S. Holabird, T. M. Hazard, B. Huang, P. W. Deelman, I. Milosavljevic, A. E. Schmitz, R. S. Ross, M. F. Gyure, and A. T. Hunter, Science Advances 1, e1500214 (2015).
  * Andrews  _et al._ (2019) R. W. Andrews, C. Jones, M. D. Reed, A. M. Jones, S. D. Ha, M. P. Jura, J. Kerckhoff, M. Levendorf, S. Meenehan, S. T. Merkel, A. Smith, B. Sun, A. J. Weinstein, M. T. Rakher, T. D. Ladd, and M. G. Borselli, Nature Nanotechnology 14, 747 (2019).
  * Zajac  _et al._ (2018) D. M. Zajac, A. J. Sigillito, M. Russ, F. Borjans, J. M. Taylor, G. Burkard, and J. R. Petta, Science 359, 439 (2018), https://science.sciencemag.org/content/359/6374/439.full.pdf .
  * Watson  _et al._ (2018) T. F. Watson, S. G. J. Philips, E. Kawakami, D. R. Ward, P. Scarlino, M. Veldhorst, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, and L. M. K. Vandersypen, Nature 555, 633 (2018).
  * Tahan (2020) C. Tahan, arXiv (2020), 2001.08251 .
  * Sze (1981) S. Sze,  _Physics of Semiconductor Devices_, Wiley-Interscience publication (Wiley, 1981).
  * Note (1) The relatively large effective mass of electrons in Si ($m_{t}*=0.19m_{e}$ versus, e.g., $m^{*}=0.067m_{e}$ in GaAs) results in weaker tunneling amplitudes, requiring, in turn, closer quantum dot separations ($\approx$ 150 nm vs. 300 nm in GaAs). $m_{e}$ is the electron rest mass in vacuum.
  * Reed  _et al._ (2015) M. D. Reed, B. M. Maune, R. W. Andrews, M. G. Borselli, K. Eng, M. P. Jura, A. A. Kiselev, T. D. Ladd, S. T. Merkel, I. Milosavljevic, E. J. Pritchett, M. T. Rakher, R. S. Ross, A. E. Schmitz, A. Smith, J. A. Wright, M. F. Gyure, and A. T. Hunter, Physical Review Letters 116, 110402 (2015).
  * Thorbeck and Zimmerman (2015) T. Thorbeck and N. M. Zimmerman, AIP Advances 5, 087107 (2015), https://doi.org/10.1063/1.4928320 .
  * Zajac  _et al._ (2015) D. M. Zajac, T. M. Hazard, X. Mi, K. Wang, and J. R. Petta, arXiv 106, 223507 (2015), 1502.01624 .
  * Schaffler (1997) F. Schaffler, Semiconductor Science and Technology 12, 1515 (1997).
  * Petta  _et al._ (2005) J. R. Petta, A. C. Johnson, J. M. Taylor, E. A. Laird, A. Yacoby, M. D. Lukin, C. M. Marcus, M. P. Hanson, and A. C. Gossard, Science 309, 2180 (2005).
  * Elzerman  _et al._ (2004) J. M. Elzerman, R. Hanson, L. H. Willems van Beveren, B. Witkamp, L. M. K. Vandersypen, and L. P. Kouwenhoven, Nature 430, 431 (2004).
  * Blumoff and et al. (2020) J. Z. Blumoff and et al., In Preparation (2020).
  * Note (2) Micrographs are collected using a Hitachi S9380 scanning electron microscope equipped with a Schottky emitter. The primary electron beam is operated at an accelerating voltage of 500 V and a beam current of 7 pA.
  * Abadi  _et al._ (2016) M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zhang, arXiv (2016), 1605.08695 .
  * Howard  _et al._ (2017) A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, arXiv (2017), 1704.04861 .
  * Sandler  _et al._ (2018) M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, arXiv (2018), 1801.04381 .
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy, arXiv (2015), 1502.03167 .
  * Ramachandran  _et al._ (2017) P. Ramachandran, B. Zoph, and Q. V. Le, arXiv (2017), 1710.05941 .
  * Deng  _et al._ (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, in  _2009 IEEE Conference on Computer Vision and Pattern Recognition_ (2009) pp. 248–255.
  * Lin  _et al._ (2013) M. Lin, Q. Chen, and S. Yan, arXiv (2013), 1312.4400 .
  * Jonker and Volgenant (1987) R. Jonker and A. Volgenant, Computing 38, 325 (1987).
  * Maaten (2013) L. v. d. Maaten, arXiv (2013), 1301.3342 .
  * Hinton  _et al._ (2012) G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, arXiv (2012), 1207.0580 .
  * Srivastava  _et al._ (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Journal of Machine Learning Research 15, 1929 (2014).
  * Note (3) $L_{1}$ regularization promotes sparse weights, offering a combination of built-in feature selection and reduced computational cost, versus $L_{2}$ regularization.
  * Goodfellow  _et al._ (2016) I. Goodfellow, Y. Bengio, and A. Courville,  _Deep Learning_ (MIT Press, 2016).
  * Nabian and Meidani (2018) M. A. Nabian and H. Meidani, arXiv 20 (2018), 10.1115/1.4044507, 1810.05547 .
  * Henschel  _et al._ (2003) W. Henschel, Y. M. Georgiev, and H. Kurz, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 21, 2018 (2003).
  * Grigorescu  _et al._ (2007) A. E. Grigorescu, M. C. v. d. Krogt, and C. W. Hagen,  , 65194A (2007).
  * Yang  _et al._ (2009) J. K. W. Yang, B. Cord, H. Duan, K. K. Berggren, J. Klingfus, S.-W. Nam, K.-B. Kim, and M. J. Rooks, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 27, 2622 (2009).
  * G  _et al._ (2011) A. T. K. G, S. N. Piramanayagam, M. Ranjbar, R. Sbiaa, and T. C. Chong, Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena 29, 011035 (2011).
  * Delft (2002) F. C. M. J. M. v. Delft, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 20, 2932 (2002).
  * Clark  _et al._ (2006) N. Clark, A. Vanderslice, R. Grove, and R. R. Krchnavek, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 24, 3073 (2006).
  * Chen  _et al._ (2006) Y. Chen, H. Yang, and Z. Cui, Microelectronic Engineering 83, 1119 (2006).
  * Note (4) Curves are omitted from panels for which interactions between variables were not investigated.
  * Westly  _et al._ (2011) D. A. Westly, D. M. Tennant, Y. Aida, H. Ohki, and T. Ohkubo, Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena 29, 06FJ02 (2011).
  * Lee  _et al._ (2009) H.-S. Lee, J.-S. Wi, S.-W. Nam, H.-M. Kim, and K.-B. Kim, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 27, 188 (2009).
  * Yu  _et al._ (2011) Z. Yu, J. Hwu, Y. Liu, G. Gauzner, K. Lee, and D. Kuo, Journal of Applied Physics 110, 014303 (2011).
  * Grove (2005) R. Grove,  _A patterning process utilizing a nanoimprint lithography for fabrication of planar perpendicular patterned magnetic media_ , Ph.D. thesis (2005).
  * Loboda and Toskey (1998) M. J. Loboda and G. A. Toskey, Solid State Technology (1998).
  * Olynick  _et al._ (2006) D. L. Olynick, J. A. Liddle, A. V. Tivanski, M. K. Gilles, T. Tyliszczak, F. Salmassi, K. Liang, and S. R. Leone, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 24, 3048 (2006).
  * Sidorkin  _et al._ (2009) V. A. Sidorkin, P. F. A. Alkemade, H. W. M. Salemink, R. Schmits, and E. v. d. Drift, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 27, 2503 (2009).
  * Olynick  _et al._ (2010) D. L. Olynick, B. Cord, A. Schipotinin, D. F. Ogletree, and P. J. Schuck, Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena 28, 581 (2010).
  * Siew  _et al._ (2000) Y. K. Siew, G. Sarkar, X. Hu, J. Hui, A. See, and C. T. Chua, Journal of The Electrochemical Society 147, 335 (2000).
  * Zeng  _et al._ (2000) Y. Zeng, S. W. Russell, A. J. McKerrow, L. Chen, and T. L. Alford, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 18, 221 (2000).
  * Chang (1975) T. H. P. Chang, Journal of Vacuum Science and Technology 12, 1271 (1975).
  * Murai (1992) F. Murai, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures 10, 3072 (1992).
  * Lopez  _et al._ (2018) G. G. Lopez, G. d. Villafranca, M. Azadi, M. G. Metzler, K. Lister, M. Labella, C. Eichfeld, N. Belic, and U. Hofmann, Journal of Vacuum Science & Technology B 36, 06JA05 (2018).
  * Note (5) Leads in our devices are spaced at a pitch of $\sim$ 70 nm and exhibit widths which vary between $\sim$ 10–40 nm.
  * Bickford  _et al._ (2014) J. R. Bickford, G. Lopez, N. Belic, and U. Hofmann, Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena 32, 06F511 (2014).
  * GenISys (2020) GenISys, “TRACER,” (2020).
  * Note (6) ZEP is a common positive-tone electron-beam lithography resist based on methyl styrene and chloromethyl acrylate copolymer.
  * (58) D. Brown, Georgia Institute of Technology. Unpublished .


# Pragmatic Issue-Sensitive Image Captioning

Allen Nie1
&Reuben Cohn-Gordon2
1Department of Computer Science 2Department of Linguistics
Stanford University
{anie, reubencg<EMAIL_ADDRESS>
&Christopher Potts1,2

###### Abstract

Image captioning systems need to produce texts that are not only _true_ but
also _relevant_ in that they are properly aligned with the current issues. For
instance, in a newspaper article about a sports event, a caption that not only
identifies the player in a picture but also comments on their ethnicity could
create unwanted reader reactions. To address this, we propose Issue-Sensitive
Image Captioning (ISIC). In ISIC, the captioner is given a target image and an
_issue_ , which is a set of images partitioned in a way that specifies what
information is relevant. For the sports article, we could construct a
partition that places images into equivalence classes based on player
position. To model this task, we use an extension of the Rational Speech Acts
model. Our extension is built on top of state-of-the-art pretrained neural
image captioners and explicitly uses image partitions to control caption
generation. In both automatic and human evaluations, we show that these models
generate captions that are descriptive and issue-sensitive. Finally, we show
how ISIC can complement and enrich the related task of Visual Question
Answering.

##  1 Introduction

Image captioning systems have improved dramatically over the last few years
(Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Hendricks et al., 2016;
Rennie et al., 2017; Anderson et al., 2018), creating new opportunities to
design systems that are not just accurate, but also produce descriptions that
include relevant, characterizing aspects of their inputs. Many of these
efforts are guided by the insight that high-quality captions are implicitly
shaped by the communicative goal of identifying the target image up to some
level of granularity (Vedantam et al., 2017; Mao et al., 2016; Luo et al.,
2018; Cohn-Gordon et al., 2018).

Figure 1: Examples highlighting the power of an issue-sensitive image
captioner. Four images are partitioned in two ways, each capturing different
issues by grouping them into equivalence classes. The first row contrasts the
brown and grey color of the bird, and the second contrasts the existence of
white eyebrows. The target image is the same in both cases, but the partition
leads to different captions that key into the structure of the input issue.

In this paper, we seek to more tightly control the information that a
pretrained captioner includes in its output texts. Our focus is on generating
captions that are _relevant_ to the current issues. To see how important this
can be, consider a newspaper article covering the action in a sports event. In
this context, a caption that not only identified the player in a picture but
also commented on their ethnicity could create unwanted reactions in readers,
as it would convey to them that such information was somehow deemed relevant
by the newspaper. On the other hand, in an article about diversity in
athletics, that same caption might seem entirely appropriate.

To push captioners to produce more relevant texts, we propose the task of
Issue-Sensitive Image Captioning (ISIC). In ISIC, the captioner’s inputs are
image/issue pairs, where an _issue_ is a set of images partitioned in a way
that specifies what information is relevant. In our first example above, we
might define a partition that grouped players into equivalence classes based
on their team positions, abstracting away from other facts about them. For the
second example, we might choose a more fine-grained partition based on
position and demographic features. Given such inputs, the objective of the
captioner is to produce a text that both accurately and uniquely describes the
cell of the partition containing the target image. Figure 1 illustrates with
examples from our own models and experiments.

In defining the task this way, we are inspired by Visual Question Answering
(VQA; Antol et al. 2015), but ISIC differs from VQA in two crucial respects.
First, we seek full image captions rather than direct answers. Second, our
question inputs are not texts, but rather issues in the semantic sense:
partitions on subsets of the available images. The ISIC module reasons about
the cells in these partitions as alternatives to the target image, and our
notion of relevance is defined in these terms. Nonetheless, VQA and ISIC
complement each other: issues (as partitions) can be automatically derived
from available image captioning and VQA datasets (Section 6), opening up new
avenues for VQA as well.

Our models are built on top of pretrained image captioners with no need for
additional training or fine-tuning. This is achieved by extending those models
according to the Rational Speech Acts model (RSA; Frank and Goodman 2012;
Goodman and Stuhlmüller 2013). RSA has been applied successfully to many NLP
tasks (Section 2.3). Our key modeling innovation lies in building issues into
these models. In this, we are inspired by linguistic work on question-
sensitive RSA (Goodman and Lassiter, 2015; Hawkins and Goodman, 2019).

Our central experiments are with the Caltech-UC San Diego-Bird dataset (CUB;
Welinder et al. 2010). This dataset contains extensive attribute annotations
that allow us to study the effects of our models in precise ways. Using CUB,
we provide quantitative evidence that our RSA-based models generate captions
that both richly describe the target image and achieve the desired kinds of
issue-sensitivity. We complement these automatic evaluation with a human
evaluation in which participants judged our models to be significantly more
issue-sensitive than standard image captioners. Finally, we show how to apply
our methods to larger image captioning and VQA datasets that require more
heuristic methods for defining issues. These experiments begin to suggest the
potential value of issue-sensitivity in other domains that involve
controllable text generation. We share code for reproducibility and future
development at https://github.com/windweller/Pragmatic-ISIC.

##  2 Related Work

###  2.1 Neural Image Captioning

The task of image captioning crosses the usual boundary between computer
vision and NLP; a good captioner needs to recognize coherent parts of the
image and describe them in fluent text. Karpathy and Fei-Fei (2015) and
Vinyals et al. (2015) showed that large-capacity neural networks can get
traction on this difficult problem. Much subsequent work has built on this
insight, focusing on two aspects. The first is improving image feature quality
by using object-based features Anderson et al. (2018). The second is improving
text generation quality by adopting techniques from reinforcement learning to
directly optimize for the evaluation metric Rennie et al. (2017). Our work
rests on these innovations – our base image captioning systems are those of
Hendricks et al. (2016) and Rennie et al. (2017), which motivate and employ
these central advancements.

There is existing work that proposes methods for controlling image caption
generation with attributes. In general, these approaches involve models in
which the attributes are part of the input, which requires a dataset with
attributes collected beforehand. For instance, Mathews et al. (2015) collected
a small dataset with sentiment annotation for each caption, Shuster et al.
(2019) collected captions with personality traits, and Gan et al. (2017) with
styles (such as humorous and romantic). The final metrics center around
whether the human-generated caption was reproduced, or around other subjective
ratings. By contrast, our method does not require an annotated dataset for
training, and we measure the success of a model by whether it has resolved the
issue under discussion.

###  2.2 Visual Question Answering

In VQA, the model is given an image and a natural language question about that
image, and the goal is to produce a natural language answer to the question
that is true of the image (Antol et al., 2015; Goyal et al., 2017). This is a
controllable form of (partial) image captioning. However, in its current form,
VQA tends not to elicit linguistically complex texts; the majority of VQA
answers are single words, and so VQA can often be cast as classification
rather than sequence generation. Our goal, in contrast, is to produce
linguistically complex, highly descriptive captions. Our task additionally
differs from VQA in that it produces a caption in response to an _issue_ ,
i.e., a partition of images, rather than a natural language question. In
Section 4.2 and Section 6, we describe how VQA and ISIC can complement each
other.

$\Bigg{\\{}$$\Big{\\{}$$\Big{\\}}$$\Big{\\{}$$\Big{\\}}$$\Big{\\{}$$\Big{\\}}$$\Bigg{\\}}$“A
red
square”$\Bigg{\\{}$$\Big{\\{}$$\Big{\\}}$$\Big{\\{}$$\Big{\\}}$$\Big{\\{}$$\Big{\\}}$$\Bigg{\\}}$“A
small square”TargetCaptionIssue

Figure 2: Two idealized examples highlighting the desired behavior for ISIC. A
single set of images is partitioned in two ways. The top row groups them by
color and shape, whereas the bottom row groups them by size and shape. A
successful system for ISIC should key into these differences: for the same
target image, its captions should reflect the partition structure and identify
which cell the target belongs to, as in our examples. A caption like “A
square” would be inferior in both contexts because it doesn’t convey which
cell the target image belongs to.

###  2.3 The Rational Speech Acts Model

The Rational Speech Acts model (RSA) was developed by Frank and Goodman (2012)
with important precedents from Lewis (1969), Jäger (2007), Franke (2009), and
Golland et al. (2010). RSA defines nested probabilistic speaker and listener
agents that reason about each other in communication to enrich the basic
semantics of their language. The model has been applied to a wide variety of
diverse linguistic phenomena. Since RSA is a probabilistic model of
communication, it is amenable for incorporation into many modern NLP
architectures. A growing body of literature shows that adding RSA components
to NLP architectures can help them to capture important aspects of context
dependence in language, including referential description generation (Monroe
and Potts, 2015; Andreas and Klein, 2016; Monroe et al., 2017), instruction
following (Fried et al., 2018), collaborative problem solving (Tellex et al.,
2014), and translation (Cohn-Gordon and Goodman, 2019).

Broadly speaking, there are two kinds of approaches to incorporating RSA into
NLP systems. One class performs end-to-end learning of the RSA agents (Monroe
and Potts, 2015; Mao et al., 2016; White et al., 2020). The other uses a
pretrained system and applies RSA at the decoding stage (Andreas and Klein,
2016; Vedantam et al., 2017; Monroe et al., 2017; Fried et al., 2018). We
adopt this second approach, as it highlights the ways in which one can imbue a
wide range of existing systems with new capabilities.

###  2.4 Issue-Sensitivity in Language

Our extension of RSA centers on what we call _issues_. In this, we build on a
long tradition of linguistic research on the ways in which language use is
shaped by the issues (often called Questions Under Discussion) that the
discourse participants regard as relevant (Groenendijk and Stokhof, 1984;
Ginzburg, 1996; Roberts, 1996). Issues in this sense can be reconstructed in
many ways. We follow Lewis (1988) and many others in casting an issue as a
partition on a space of states into cells. Each cell represents a possible
resolution of the issue. These ideas are brought into RSA by Goodman and
Lassiter (2015) and Hawkins and Goodman (2019). We translate those ideas into
the models for ISIC (Section 4), where an issue takes the form of a partition
over a set of natural images.

##  3 Task Formulation

In standard image captioning, the input $\mathbf{i}$ is an image drawn from a
set of images $\mathcal{I}$, and the output $\mathbf{w}$ is a sequence of
tokens $[w_{1},\ldots,w_{n}]$ such that each $w_{i}\in\mathcal{V}$, where
$\mathcal{V}$ is the vocabulary.

In ISIC, we extend standard image captioning by redefining the inputs as pairs
$(\mathbf{C},\mathbf{i})$, where $\mathbf{C}$ is a partition111A set of sets
$\mathbf{X}$ is a partition of a set $X$ iff $u\cap v=\varnothing$ for all
$u,v\in\mathbf{X}$ and $\bigcup_{u\in\mathbf{X}}=X$. on a subset of elements
of $\mathcal{I}$ and $\mathbf{i}\in\bigcup_{u\in\mathbf{C}}$. We refer to the
partitions $\mathbf{C}$ as _issues_ , for the reasons discussed in Section
2.4. The goal of ISIC is as follows: given input $(\mathbf{C},\mathbf{i})$,
produce a caption $\mathbf{w}$ that provides a true resolution of $\mathbf{C}$
for $\mathbf{i}$, which reduces to $\mathbf{w}$ identifying the cell of
$\mathbf{C}$ that contains $\mathbf{i}$, as discussed in Section 2.4. Figure 2
presents an idealized example. (Figure 1 is a real example involving CUB and
an ISIC captioner; see also Figure 3 and Figure 4 below.)

In principle, we could try to learn this kind of issue sensitivity directly
from a dataset of examples $((\mathbf{C},\mathbf{i}),\mathbf{w})$. We do think
such dataset could be collected, as discussed briefly in Section 7. However,
such datasets would be very large (each image needs to collect $|\mathbf{C}|$
number of captions), and our primary modeling goal is to show that such
datasets need not be created. The issue-sensitive pragmatic model we introduce
next can realize the goal of ISIC without training data of this kind.

##  4 Models

###  4.1 Neural Pragmatic Agents

The models we employ for ISIC define a hierarchy of increasingly sophisticated
speaker and listener agents, in ways that mirror ideas from Gricean pragmatics
(Grice, 1975) about how meaning can arise when agents reason about each other
in both production and comprehension (see also Lewis 1969).

Our base agent is a speaker $S_{0}(\mathbf{w}\mid\mathbf{i})$. In linguistic
and psychological models, this agent is often defined by a hand-built
semantics. In contrast, our $S_{0}$ is a trained neural image captioning
system. As such, they are learned from data, with no need to hand-specify a
semantic grammar or the like.

The pragmatic listener $L_{1}(\mathbf{i}\mid\mathbf{w})$ defines a
distribution over states $\mathbf{i}$ given a message $\mathbf{w}$. The
distribution is defined by applying Bayes’ rule to the $S_{0}$ agent:

$L_{1}(\mathbf{i}\mid\mathbf{w})=\frac{S_{0}(\mathbf{w}\mid\mathbf{i})P(\mathbf{i})}{\sum_{\mathbf{i}^{\prime}\in\mathcal{I}}S_{0}(\mathbf{w}\mid\mathbf{i}^{\prime})P(\mathbf{i}^{\prime})}$
(1)

where $P(\mathbf{i})$ is a prior over states $\mathbf{i}$ (always flat in our
work). This agent is pragmatic in the sense that it reasons about another
agent, showing behaviors that align with the Gricean notion of _conversational
implicature_ (Goodman and Frank, 2016).

We can then define a pragmatic speaker using a utility function $U_{1}$, in
turn defined in terms of $L_{1}$:

$\displaystyle U_{1}(\mathbf{i},\mathbf{w})$ $\displaystyle=\log
L_{1}(\mathbf{i}\mid\mathbf{w})$ (2) $\displaystyle
S_{1}(\mathbf{w}\mid\mathbf{i})$ $\displaystyle=\frac{\exp\left(\alpha
U_{1}(\mathbf{i},\mathbf{w})-\mathit{cost}(\mathbf{w})\right)}{\sum_{\mathbf{w}^{\prime}}\exp\left(\alpha
U_{1}(\mathbf{i},\mathbf{w}^{\prime})-\mathit{cost}(\mathbf{w}^{\prime})\right)}$
(3)

Here, $\alpha$ is a parameter defining how heavily $S_{1}$ is influenced by
$L_{1}$. The term $\mathit{cost}(\mathbf{w})$ is a cost function on messages.
In other work, this is often specified by hand to capture analysts’ intuitions
about complexity or markedness. In contrast, our version is entirely data-
driven: we specify $\mathit{cost}(\mathbf{w})$ as
${-}\log(S_{0}(\mathbf{w}\mid\mathbf{i}))$.

###  4.2 Issue-Sensitive Speaker Agents

The agent in (3) has been widely explored and shown to deliver a powerful
notion of context dependence (Andreas and Klein, 2016; Monroe et al., 2017).
However, it is insensitive to the issues $\mathbf{C}$ that characterize ISIC.
To make this connection, we extend (3) with a term for these issues:

$\displaystyle U_{1}^{\mathbf{C}}(\mathbf{i},\mathbf{w},\mathbf{C})$
$\displaystyle=\log\left(\sum_{\mathbf{i}^{\prime}\in\mathcal{I}}\delta_{[\mathbf{C}(\mathbf{i})=\mathbf{C}(\mathbf{i}^{\prime})]}L_{1}(\mathbf{i}^{\prime}\mid\mathbf{w})\right)$
(4) $\displaystyle S_{1}^{\mathbf{C}}(\mathbf{w}\mid\mathbf{i},\mathbf{C})$
$\displaystyle\propto\exp\left(\alpha
U_{1}^{\mathbf{C}}(\mathbf{i},\mathbf{w},\mathbf{C})-\mathit{cost}(\mathbf{w})\right)$
(5)

where $\delta_{[\mathbf{C}(\mathbf{i})=\mathbf{C}(\mathbf{i}^{\prime})]}$ is a
partition function, returning $1$ if $\mathbf{i}$ and $\mathbf{i}^{\prime}$
are in the same cell in $\mathbf{C}$, else $0$. This is based on a similar
model of Kao et al. (2014). We use $\mathbf{C}(\mathbf{i})$ to denote the cell
to which image $\mathbf{i}$ belongs under $\mathbf{C}$ (a slight abuse of
notation, since $\mathbf{C}$ is a set of sets).

The construction of the partitions $\mathbf{C}$ is deliberately left open at
this point. In some settings, the set of images $\mathcal{I}$ will have
metadata that allows us to construct these directly. For example, in the CUB
dataset, we can use the attributes to define intuitive partitions directly –
e.g., the partition that groups images into equivalence classes based on the
beak color of the birds they contain. The function can also be parameterized
by a full VQA model $\mathcal{A}$. For a given question text $\mathbf{q}$ and
image $\mathbf{i}$, $\mathcal{A}$ defines a map from $(\mathbf{q},\mathbf{i})$
to answers $\mathbf{a}$, and so we can partition a subset of $\mathcal{I}$
based on equivalence classes defined by these answers $\mathbf{a}$.

###  4.3 Penalizing Misleading Captions

The agent in (5) is issue-sensitive in that it favors messages that resolve
the issue $\mathbf{C}$. However, it does not include a pressure against hyper-
specificity; rather, it just encodes the goal of identifying partition cells.
This poses two potential problems.

The first can be illustrated using the top row of Figure 2. All else being
equal, our agent (5) would treat “‘A red square” and “A small red square” as
equally good captions, even though the second includes information that is
intuitively gratuitous given the issue. This might seem innocent here, but it
can raise concerns in real environments, as we discussed in Section 1 in
connection with our newspaper article examples.

The second problem relates to the data-driven nature of the systems we are
developing: in being hyper-specific, we observed that they often mentioned
properties not true of the target but rather only true of members of their
equivalence classes. For example, in Figure 2, the target could get
incorrectly described with “A large red square” because of the other member of
its cell.

We propose to address both these issues with a second utility term $U_{2}$:

$U_{2}(\mathbf{w},\mathbf{i},\mathbf{C})=H(L_{1}(\mathbf{i}^{\prime}\mid\mathbf{w})\cdot\delta_{[\mathbf{C}(\mathbf{i})=\mathbf{C}(\mathbf{i}^{\prime})]})$
(6)

where $H$ is the information-theoretic entropy. This encodes a pressure to
choose utterances which result in the $L_{0}$ spreading probability mass as
evenly as possible over the images in the target image cell. This discourages
very specific descriptions of any particular image in the target cell, thereby
solving both of the problems we identified above.

We refer to this agent as $S_{1}^{\mathbf{C}+H}$. Its full specification is as
follows:

$S_{1}^{\mathbf{C}+H}(\mathbf{w}\mid\mathbf{i},\mathbf{C})\propto\\\
\exp\left(\alpha\left((1-\beta)U_{1}+\beta
U_{2}\right)-\mathit{cost}(\mathbf{w})\right)$ (7)

where $\beta\in[0,1]$ is a hyperparameter that allows us to weight these two
utilities differently.

###  4.4 Reasoning about Alternative Captions

A pressing issue which arises when computing probabilities using (3), (5), and
(7) is that the normalization constant includes a sum over all possible
captions $\mathbf{w}^{\prime}$. In the present setting, the set of possible
captions is infinite (or at least exponentially large in the maximum caption
length), making this computation intractable.

There are two solutions to this intractability proposed in the literature: one
is to use $S_{0}$ to sample a small subset of captions from the full space,
which then remains fixed throughout the computation (Andreas and Klein, 2016;
Monroe et al., 2017). The drawback of this approach is that the diversity of
captions that the $S_{1}$ can produce is restricted by the $S_{0}$. Since our
goal is to generate captions which may vary considerably depending on the
issue, this is a serious limitation.

The other approach is to alter the model so that the RSA reasoning takes place
greedily during the generation of each successive word, word piece, or letter
in the caption, so that the possible “utterances” at each step are drawn from
a relatively small set of options to avoid exponential increase in search
space (Cohn-Gordon et al., 2018). We opt for this _incremental_ formulation
and provide the full details on this model in Appendix A.

##  5 CUB Experiments

###  5.1 Preliminaries

#### Dataset

The Caltech UC San Diego-Bird (CUB) dataset contains 11,788 images for 200
species of North American birds Welinder et al. (2010). Each image contains a
single bird and is annotated with fine-grained information about the visual
appearance of that bird, using a system of 312 attributes (all of them binary)
devised by ornithologists. The attributes have a _property::value_ structure,
as in has_wing_color::brown, and are arranged hierarchically from high-level
descriptors (e.g., _bill_) to very specific low-level attributes (e.g., _belly
pattern_). Appendix B provides a detailed example.

Reed et al. (2016) annotated each image in CUB with five captions. These
captions were generated by crowdworkers who did not have access to the
attribute annotations, and thus they vary widely in their alignment with the
CUB annotations.

#### Constructing CUB Partitions

CUB is ideal for testing our issue-sensitive captioning method because we can
produce partitions directly from the attributes. For example,
has_wing_color::brown induces a binary partition into birds with brown wings
and birds with non-brown wings, and has_wing_color alone induces a partition
that groups birds into equivalence classes based on their wing-color values.
We selected the 17 most frequently appearing attributes, which creates 17
equivalence classes to serve as our issues.

Figure 3: Captions for CUB. The left-hand cell in the ‘Issues’ column contains
the target image, and right-hand cell is the union of the distractor cells.
The ‘Base Caption’ texts are those produced by our $S_{0}$ model, and the
‘Issue-sensitive Caption’ texts were produced by $S_{1}^{\mathbf{C}+H}$.

#### Base Captioning System

We trained a model released by Hendricks et al. (2016) with the same data-
split scheme, where we have 4,000 images for training, 1,994 images for
validation, and 5,794 images for testing. The model is a two-layer long short-
term memory model (Hochreiter and Schmidhuber, 1997) with 1000-dimensional
hidden size and 1000-dimensional word embeddings. We trained for 50 epochs
with a batch size of 128 and learning rate 1$e{-}$3\. The final CIDEr score
for our model is 0.52 on the test split. We use greedy decoding to generate
our captions.

#### Feature-in-Text Classifier

In order to examine the effectiveness of our issue-sensitive captioning
models, we need to be able to identify whether the generated caption contains
information regarding the issue. Even though each CUB image has a complete
list of features for its bird, we must map these features to descriptions in
informal text. For this, we require a text classifier. Unfortunately, it is
not possible to train an effective classifier on the CUB dataset itself. As we
noted above, the caption authors did not have access to the CUB attribute
values, and so their captions tend to mention very different information than
is encoded in those attributes. Furthermore, even if we did collect entirely
new captions with proper attribute alignment, the extreme label imbalances in
the data would remain a challenge for learning.

To remedy this, we use a sliding window text classifier. First, we identify
keywords that can describe body parts (e.g. “head”, “malar”, “cheekpatch”) and
extract their positions in the text. Second, we look for keywords related to
aspects (e.g., “striped”, “speckled”); if these occur before a body-part word,
we infer that they modify the body part. Thus, for example, if “scarlet and
pink head” is in the caption, then we infer that it resolves an issue about
the color of the bird’s head.

This classifier is an important assessment tool for us, so it needs to be
independently validated. We meet this need using our human study in Section
5.4, which shows that our classifier is extremely accurate and, more
importantly, not biased towards our issue-sensitive models.

###  5.2 Evaluating Attribute Coverage

We begin by assessing the extent to which our issue-sensitive pragmatic models
produce captions that are more richly descriptive of the target image than a
base neural captioner $S_{0}$ and its simple pragmatic variant $S_{1}$. For
CUB, we can simply count how many attributes the caption specifies according
to our feature-in-text classifier. More precisely, for each image and each
model, we generate captions under all resolvable issues, concatenate those
captions, and then use the feature-in-text classifier to obtain a list of
attributes, which we can then compare to the ground truth for the image as
given by the CUB dataset.

For $S_{0}$ and $S_{1}$, the captions do not vary by issue, whereas our
expectation is that they do vary for $S_{1}^{\mathbf{C}}$ and
$S_{1}^{\mathbf{C}+H}$. To further contextualize the performance of the issue-
sensitive agents, we additionally define a model $S_{0}$ Avg that takes as
inputs the average of all the features from all the images in the current
partition, and otherwise works just like $S_{0}$. This introduces a rough form
of issue-sensitivity, allowing us to quantify the value of the more refined
approach defined by $S_{1}^{\mathbf{C}}$ and $S_{1}^{\mathbf{C}+H}$. Appendix
D provides full details on how these models were optimized.

Table 1 reports on this evaluation. Precision for all models is very high; the
underlying attributes in CUB are very comprehensive, so all high-quality
captioners are likely to do well by this metric. In contrast, the recall
scores vary substantially, and they clearly favor the issue-sensitive models,
revealing them to be substantially more descriptive than $S_{0}$ and $S_{1}$.
Figure 3 provides examples that highlight these contrasts: whereas the $S_{0}$
caption is descriptive, it simply doesn’t include a number of attributes that
we can successfully coax out of an issue-sensitive model by varying the issue.

The results also show that $S_{1}^{\mathbf{C}}$ and $S_{1}^{\mathbf{C}+H}$
provide value beyond simply averaging image features in $\mathbf{C}$, as they
both outperform $S_{0}$ Avg. However, it is noteworthy that even the rough
notion of issue-sensitivity embodied by $S_{0}$ Avg seems beneficial.

Table 2 summarizes attribute coverage at the level of individual categories,
for our four primary models. We see that the issue-sensitive models are clear
winners. However, the entropy term in $S_{1}^{\mathbf{C}+H}$ seems to help for
some categories but not others, suggesting underlying variation in the
categories themselves.

| Precision | Recall | F1
---|---|---|---
$S_{0}$ | 96.1 | 16.6 | 28.3
$S_{0}$ Avg | 94.0 | 32.7 | 48.5
$S_{1}$ | 93.9 | 29.6 | 45.0
$S_{1}^{\mathbf{C}}$ | 94.8 | 44.3 | 60.4
$S_{1}^{\mathbf{C}+H}$ | 94.7 | 50.8 | 66.2
Table 1: Attribute coverage results.

###  5.3 Evaluating Issue Alignment

Our previous evaluation shows that varying the issue has a positive effect on
the captions generated by our issue-sensitive models, but it does not assess
whether these captions resolve individual issues in an intuitive way. We now
report on an assessment that quantifies issue-sensitivity in this sense.

The question posed by this method is as follows: for a given issue
$\mathbf{C}$, does the produced caption precisely resolve $\mathbf{C}$? We can
divide this into two sub-questions. First, does the caption resolve
$\mathbf{C}$, which is a notion of _recall_. Second, does the caption avoid
addressing issues that are distinct from $\mathbf{C}$, which is a notion of
_precision_. The recall pressure is arguably more important, but the precision
one can be seen as assessing how often the caption avoids irrelevant and
potentially distracting information, as discussed in Section 4.3.

Table 3 reports on this issue-sensitive evaluation, with $F_{1}$ giving the
usual harmonic mean between our versions of precision and recall. Overall, the
scores reveal that this is a very challenging problem, which traces to the
fine-grained issues that CUB supports. Our $S_{1}^{\mathbf{C}+H}$ agent is
nonetheless definitively the best, especially for recall.

Issues | $S_{0}$ | $S_{1}$ | $S_{1}^{\mathbf{C}}$ | $S_{1}^{\mathbf{C}+H}$
---|---|---|---|---
wing pattern | 3.9 | 22.1 | 28.7 | 14.9
belly pattern | 7.6 | 19.4 | 32.0 | 39.8
breast pattern | 7.6 | 17.0 | 30.8 | 35.2
nape color | 6.0 | 18.1 | 31.2 | 49.2
upper tail color | 1.3 | 24.5 | 30.6 | 22.6
under tail color | 1.5 | 26.3 | 33.1 | 22.9
back color | 5.1 | 24.9 | 32.6 | 66.7
leg color | 6.2 | 51.0 | 45.6 | 7.4
throat color | 16.4 | 46.2 | 66.0 | 69.8
crown color | 49.5 | 50.4 | 77.4 | 90.4
bill shape | 47.7 | 43.7 | 71.1 | 91.8
eye color | 24.4 | 54.1 | 61.3 | 53.3
wing color | 39.2 | 70.5 | 82.7 | 77.4
bill color | 38.7 | 64.1 | 80.4 | 74.2
breast color | 42.8 | 61.1 | 77.4 | 90.4
belly color | 56.6 | 65.7 | 81.3 | 93.3
bill length | 60.3 | 55.8 | 84.3 | 95.5
Table 2: F1 scores for each body part. The issue-sensitive models are superior for all categories except ‘leg color’. | Precision | Recall | $F_{1}$
---|---|---|---
$S_{0}$ | 10.5 | 21.1 | 15.5
$S_{0}$ Avg | 12.1 | 29.0 | 17.0
$S_{1}$ | 11.2 | 21.7 | 14.8
$S_{1}^{\mathbf{C}}$ | 18.7 | 42.5 | 25.9
$S_{1}^{\mathbf{C}+H}$ | 16.6 | 46.6 | 24.5
Table 3: Issue alignment results.

###  5.4 Human Evaluation

We conducted a human evaluation of our models primarily to assess their issue
sensitivity, but also to validate the classifier we used in the previous
automatic evaluations.

#### Study Design

We randomly sampled 110 images. For each, we have five conditions: our four
RSA agents and the human captions from CUB. The items were arranged in a Latin
Square design to ensure that no participant saw two captions for the same
image and every participant saw an even distribution of conditions. We
recruited 105 participants using Mechanical Turk. Each item was completed by
exactly one participant. We received 1,365 responses in total.

Participants were presented with a question text and a caption and asked to
use the caption to select an answer to the question or indicate that the
caption did not provide an answer. (No images were shown, of course, to ensure
that only the caption was used.) For additional details, see Appendix C.

Figure 4: Captions for MS COCO with issues determined by VQA 2.0. The left-
hand cell in the ‘Issues’ column contains the target image, and right-hand
cell is the union of the distractor cells. The ‘Base Caption’ texts are those
produced by our $S_{0}$ model, and the ‘Issue-sensitive Caption’ texts were
produced by $S_{1}^{\mathbf{C}+H}$.

#### Issue Sensitivity

Caption Source | Percentage | Size
---|---|---
$S_{0}$ | 20.9 | 273
$S_{1}$ | 24.5 | 273
$S_{1}^{\mathbf{C}}$ | 42.1 | 273
$S_{1}^{\mathbf{C}+H}$ | 44.0 | 273
Human | 33.3 | 273
Table 4: Percentage of captions that contain an answer to the question
according to our human evaluation. By Fisher’s exact test, both issue-
sensitive models ($S_{1}^{\mathbf{C}}$, $S_{1}^{\mathbf{C}+H}$) are different
from issue-insensitive models ($S_{0}$, $S_{1}$) and humans ($p<0.05$), but we
do not have evidence for a difference between the issue-insensitive models.

Table 4 shows the percentage of captions that participants were able to use to
answer the questions posed. The pragmatic models are clearly superior. (The
human captions are not upper-bounds, since they were not created relative to
issues and so cannot vary by issue.)

Caption Source | Accuracy | Size
---|---|---
$S_{0}$ | 94.5 | 273
$S_{1}$ | 90.1 | 273
$S_{1}^{\mathbf{C}}$ | 92.7 | 273
$S_{1}^{\mathbf{C}+H}$ | 94.9 | 273
Human | 90.5 | 273
Table 5: Classifier accuracy according to our human evaluation.

#### Classifier Fairness

We can also use our human evaluation to assess the fairness of the feature-in-
text classifier (Section 5.1) that we used for our automatic evaluations. To
do this, we say that the classifier is correct for an example $x$ if it agrees
with the human response for $x$. Table 5 presents these results. Not only are
accuracy values very high, but they are similar for $S_{0}$ and
$S_{1}^{\mathbf{C}+H}$.

##  6 MS COCO and VQA 2.0

The annotations in the CUB dataset allow us to generate nuanced issues that
are tightly connected to the content of the images. It is rare to have this
level of detail in an image dataset, so it is important to show that our
method is applicable to less controlled, broader coverage datasets as well. As
a first step in this direction, we now show how to apply our method using the
VQA 2.0 dataset Goyal et al. (2017), which extends MS COCO Lin et al. (2014)
with the question and answer annotations needed for VQA. While MS COCO does
have instance-level annotations, they are mostly general category labels, so
the attribute-dependent method we used for CUB isn’t effective here. However,
VQA offers a benefit: one can now control captions by generating issues from
questions.

#### Dataset

MS COCO contains 328k images that are annotated with instance-level
information. The images are mostly everyday objects and scenes. A subset of
them (204,721 examples) are annotated with whole image captions. Antol et al.
(2015) built on this resource to create a VQA dataset, and Goyal et al. (2017)
further extended that work to create VQA 2.0, which reduces certain linguistic
biases that made aspects of the initial VQA task artificially easy. VQA 2.0
provides 1,105,904 question annotations for all the images from MS COCO.

#### Constructing Partitions

To generate issues, we rely on the ground-truth questions and answers in the
VQA 2.0 dataset. Here, each image is already mapped to a list of questions and
corresponding answers. Given an MS COCO image and a VQA question, we identify
all images associated with that question by exact string match and then
partition these images into cells according to their ground-truth answers.
Exactly the same procedure could be run using a trained VQA model rather than
the ground-truth annotations in VQA 2.0.

#### Base Captioning System

We use a pretrained state-of-the-art Transformer model with self-critical
sequence training Rennie et al. (2017). This has 6 Transformer layers with a
2048-dimensional hidden states, 512-dimensional input embeddings, and 8
attention heads at each layer. We use image features extracted by Anderson et
al. (2018). The model achieves a CIDEr score of 1.29 for the test split. We
use beam search (with beam size 5) to generate our captions.

#### Example Captions

We show some examples for MS COCO in Figure 4. We chose these to highlight the
potential of our model as well as remaining challenges. In datasets like this,
the captioning model must reason about a large number of diverse issues, from
objects and their attributes to more abstract concepts like types of food,
sports positions, and relative distances (“How far can the man ride the
bike?”; answer: “Far”). Our model does key into some abstract issues (e.g.,
“black and white photo” in row 2 of Figure 4), but more work needs to be done.
Figure 4 also suggests shortcomings concerning over-informativity (e.g., the
mention of a tub in response to an issue about toilets).

##  7 Conclusion

We defined the task of Issue-Sensitive Image Captioning (ISIC) and developed a
Bayesian pragmatic model that allows us to address this task successfully
using existing datasets and pretrained image captioning systems. We see two
natural extensions of this approach that might be explored.

First, the method we proposed can be used as a method for assessing the
quality of the underlying caption model. Using a dataset with issue
annotations, if the model trained over the plain captions is more issue-
sensitive, then it is better at decomposing the content of an image by its
objects and abstract concepts.

Second, one could extend our notion of issue-sensitivity to other domains. As
we saw in Section 6, questions (as texts) naturally give rise to issues in our
sense where the domain is sufficiently structured, so these ideas might find
applicability in the context of question answering and other areas of
controllable natural language generation.

## Acknowledgement

We thank the EMNLP referees for their constructive reviews. We also thank John
Hewitt, Qi Peng, and Yuhao Zhang for feedback on an early draft of this paper.
And special thanks to our Mechanical Turk participants for their crucial
contributions.

## References

  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6077–6086.
  * Andreas and Klein (2016) Jacob Andreas and Dan Klein. 2016.  Reasoning about pragmatics with neural listeners and speakers.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1173–1182, Austin, Texas. Association for Computational Linguistics.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2425–2433.
  * Cohn-Gordon and Goodman (2019) Reuben Cohn-Gordon and Noah Goodman. 2019.  Lost in machine translation: A method to reduce meaning loss.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 437–441, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Cohn-Gordon et al. (2018) Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018.  Pragmatically informative image captioning with character-level inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics.
  * Frank and Goodman (2012) Michael C. Frank and Noah D. Goodman. 2012.  Predicting pragmatic reasoning in language games.  _Science_ , 336(6084):998.
  * Franke (2009) Michael Franke. 2009.  _Signal to Act: Game Theory in Pragmatics_.  ILLC Dissertation Series. Institute for Logic, Language and Computation, University of Amsterdam.
  * Fried et al. (2018) Daniel Fried, Jacob Andreas, and Dan Klein. 2018.  Unified pragmatic models for generating and following instructions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gan et al. (2017) Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017.  Stylenet: Generating attractive visual captions with styles.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3137–3146.
  * Ginzburg (1996) Jonathan Ginzburg. 1996.  Dynamics and the semantics of dialogue.  In Jerry Seligman, editor, _Language, Logic, and Computation_ , volume 1, pages 221–237. CSLI, Stanford, CA.
  * Golland et al. (2010) Dave Golland, Percy Liang, and Dan Klein. 2010.  A game-theoretic approach to generating spatial descriptions.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 410–419, Cambridge, MA. Association for Computational Linguistics.
  * Goodman and Frank (2016) Noah D. Goodman and Michael C. Frank. 2016.  Pragmatic language interpretation as probabilistic inference.  _Trends in Cognitive Sciences_ , 20(11):818–829.
  * Goodman and Lassiter (2015) Noah D. Goodman and Daniel Lassiter. 2015.  Probabilistic semantics and pragmatics.  In Shalom Lappin and Chris Fox, editors, _The Handbook of Contemporary Semantic Theory_ , 2 edition. Wiley-Blackwell, Oxford.
  * Goodman and Stuhlmüller (2013) Noah D. Goodman and Andreas Stuhlmüller. 2013.  Knowledge and implicature: Modeling language understanding as social cognition.  _Topics in Cognitive Science_ , 5(1):173–184.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6904–6913.
  * Grice (1975) H. Paul Grice. 1975.  Logic and conversation.  In Peter Cole and Jerry Morgan, editors, _Syntax and Semantics_ , volume 3: Speech Acts, pages 43–58. Academic Press, New York.
  * Groenendijk and Stokhof (1984) Jeroen Groenendijk and Martin Stokhof. 1984.  _Studies in the Semantics of Questions and the Pragmatics of Answers_.  Ph.D. thesis, University of Amsterdam.
  * Hawkins and Goodman (2019) Robert X. D. Hawkins and Noah D. Goodman. 2019.  Why do you ask? The informational dynamics of questions and answers.  PsyArXiv, 6.
  * Hendricks et al. (2016) Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016.  Generating visual explanations.  In _European Conference on Computer Vision_ , pages 3–19. Springer.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Jäger (2007) Gerhard Jäger. 2007.  Game dynamics connects semantics and pragmatics.  In Ahti-Veikko Pietarinen, editor, _Game Theory and Linguistic Meaning_ , pages 89–102. Elsevier, Amsterdam.
  * Kao et al. (2014) Justine T Kao, Jean Y Wu, Leon Bergen, and Noah D Goodman. 2014.  Nonliteral understanding of number words.  _Proceedings of the National Academy of Sciences_ , 111(33):12002–12007.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3128–3137.
  * Lewis (1969) David Lewis. 1969.  _Convention_.  Harvard University Press, Cambridge, MA.
  * Lewis (1988) David Lewis. 1988.  Relevant implication.  _Theoria_ , 54(3):161–174.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Luo et al. (2018) Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018.  Discriminability objective for training descriptive captions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6964–6974.
  * Mao et al. (2016) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016.  Generation and comprehension of unambiguous object descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 11–20. IEEE.
  * Mathews et al. (2015) Alexander Mathews, Lexing Xie, and Xuming He. 2015.  Senticap: Generating image descriptions with sentiments.  _arXiv preprint arXiv:1510.01431_.
  * Monroe et al. (2017) Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017.  Colors in context: A pragmatic neural model for grounded language understanding.  _Transactions of the Association for Computational Linguistics_ , 5:325–338.
  * Monroe and Potts (2015) Will Monroe and Christopher Potts. 2015.  Learning in the Rational Speech Acts model.  In _Proceedings of 20th Amsterdam Colloquium_ , Amsterdam. ILLC.
  * Reed et al. (2016) Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016.  Learning deep representations of fine-grained visual descriptions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 49–58.
  * Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7008–7024.
  * Roberts (1996) Craige Roberts. 1996.  Information structure: Towards an integrated formal theory of pragmatics.  In Jae Hak Yoon and Andreas Kathol, editors, _OSU Working Papers in Linguistics_ , volume 49: Papers in Semantics, pages 91–136. The Ohio State University Department of Linguistics, Columbus, OH.
  * Shuster et al. (2019) Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019\.  Engaging image captioning via personality.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 12516–12526.
  * Tellex et al. (2014) Stefanie Tellex, Ross A. Knepper, Adrian Li, Thomas M. Howard, Daniela Rus, and Nicholas Roy. 2014.  Asking for help using inverse semantics.  In _Proceedings of Robotics: Science and Systems_.
  * Vedantam et al. (2017) Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017\.  Context-aware captions from context-agnostic supervision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 251–260.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3156–3164.
  * Welinder et al. (2010) Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. 2010.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology.
  * White et al. (2020) Julia White, Jesse Mu, and Noah D. Goodman. 2020.  Learning to refer informatively by amortizing pragmatic reasoning.  In _Proceedings of the Cognitive Science Society_.

##  Appendix A Incremental Pragmatic Reasoning

The normalization terms of $S_{1}$, $S_{1}^{\mathbf{C}}$, and
$S_{1}^{\mathbf{C}+H}$ all require a sum over all messages, rendering them
intractable to compute.

We now describe a variant of the $S_{1}$ which performs pragmatic reasoning
incrementally. This method extends in an obvious fashion to
$S_{1}^{\mathbf{C}}$ and $S_{1}^{\mathbf{C}+H}$.

We begin by noting that a neural captioning model, at decoding time, generates
a caption $\mathbf{w}$ one segment at a time (depending on the architecture,
this segment may be a word, word piece, or character). We write
$\mathbf{w}=(w_{1}\ldots w_{n})$, where $w_{i}$ is the $i$th segment.

Concretely, a trained neural image captioner can be specified as a
distribution over the subsequent segment given the image and previous words,
which we write as $S_{0}(w_{n+1}\mid\mathbf{i},[w_{1}\cdots w_{n}])$. This
allows us to define incremental versions of $L_{1}$ and $S_{1}$, as follows:

$L_{1}(\mathbf{i}\mid w_{n+1},[w_{1}\ldots w_{n}])\propto\\\
S_{0}(w_{n+1}\mid\mathbf{i},[w_{1}\ldots w_{n}])P(\mathbf{i})$ (8)
$U_{1}(\mathbf{i},w_{n+1},[w_{1},w_{n}])=\\\ \log L_{1}(\mathbf{i}\mid
w_{n+1},[w_{1}\ldots w_{n}])$ (9) $S_{1}(w_{n+1}\mid\mathbf{i},[w_{1}\ldots
w_{n}])\propto\\\ \exp(\alpha
U_{1}(\mathbf{i},w_{n+1},[w_{1},w_{n}])-\mathit{cost}(w_{n+1}))$ (10)

Here, we define the cost as the negative log-likelihood of the $S_{0}$
producing $w_{n+1}$ given the image $i$ and previous segments $[w_{1}\ldots
w_{n}]$. We can then obtain a caption-level model, which we term
$S_{1}^{\mathit{INC}}$ by contrast to the $S_{1}$ defined in (3):

$\displaystyle
S_{1}^{\mathit{INC}}(\mathbf{w}\mid\mathbf{i})=\prod_{i=1}^{n}S_{1}(w_{i}\mid[w_{1}\ldots
w_{i-1}],\mathbf{i})$ (11)

$S_{1}^{\mathit{INC}}(\mathbf{w}\mid\mathbf{i})$ then serves as a tractable
approximation of the caption-level $S_{1}(\mathbf{w}\mid\mathbf{i})$, and the
same approach is easily extended to $S_{1}^{\mathbf{C}}$ and
$S_{1}^{\mathbf{C}+H}$.

##  Appendix B CUB Attribute Annotation Example

Figure A1: A Carolina Wren from CUB. There can be multiple aspects per body
part. Some general descriptors (e.g., _size_) do not have fine-grained
aspects.

##  Appendix C Human Study Design

Our study involved 110 randomly sampled images from CUB. For each, we have
five conditions: original human caption, $S_{0}$ (base image caption model),
$S_{1}$, $S_{1}^{\mathbf{C}}$, and $S_{1}^{\mathbf{C}+H}$. The items were
arranged in a Latin Square design to ensure that no participant saw two
captions for the same image and every participant saw an even distribution of
conditions. We recruited 105 participants using Mechanical Turk. Each
participant completed 13 items, and each item was completed by exactly one
participant. We received a total of 1,365 responses. No participants or
responses were excluded from our analyses.

In an instruction phase, participants were shown labeled images of birds to
help with specialized terminology and concepts in the CUB domain. Before
beginning the study, they were shown three examples of our task (Figure A2).
These examples were chosen to familiarize participants with the underlying
semantics of the captions. For example, if caption gives an ambiguous generic
description like “the bird has a white body”, it does not provide enough
information to answer questions about the color of specific body parts, even
though it does mention a color.

Figure A2: Example phase of the MTurk study

Following the example phase, we included a short trial phase of two example
items, shown in Figure A3. We required participants to complete this trial
before they started the study itself. We provided feedback immediately
(“Wrong” or “’Correct’) after they made selections in this phase.

Figure A3: Trial phase of the MTurk study

Finally, an example item is given in Figure A4. Each annotation is structured
in terms of a question (we rephrase issues in CUB as a question:
has_wing_color is rephrased as What is the wing color?). Since the CUB
attribute annotations consist of a property::value structure, we take the
values associated with the property as our answer options.

Figure A4: An annotation item from the MTurk study. In this example, the
correct answer is “curved (up or down)”.

##  Appendix D Optimization Details

#### Computing infrastructure

Our experiment on CUB and MSCOCO is conducted on an NVIDIA TITAN X (Pascal)
with 12196 MB graphic memory.

#### Computing time

Since we do not re-train our model, we report the inference time for our
algorithm. Running our $S_{1}^{\mathbf{C}+H}$ on CUB test examples (5,794
images) takes about 40–50 minutes on a single GPU with specs listed above. For
MS COCO, it takes substantially longer (about 1 minute per image) due to the
more complex Transformer base image captioning model.

#### Hyperparameters

We did not conduct a hyperparameter search. We manually set hyperparameters
for our RSA-based models. The hyperparameters include rationality $\alpha$,
entropy penalty $\beta$, and number of examples in a partition cell. The
hyperparameters are chosen by small scale trial-and-error on validation data.
We looked at the generated captions for 4 or 5 validation images of each
model, and we decreased or increased our hyperparameters so that the generated
captions for these images were coherent, grammatical, and issue-sensitive
(when applicable). In Table A1, we report the hyperparameters we used for each
RSA model.

Model | Cell Size | $\alpha$ | $\beta$
---|---|---|---
$S_{1}$ | 40 | 3 | —
$S_{1}^{\mathbf{C}}$ | 40 | 10 | —
$S_{1}^{\mathbf{C}+H}$ | 40 | 10 | 0.4
Table A1: Hyperparameter for each RSA model.

#### Validation performance

We report the performance on the validation set in Figure A2 and Figure A3,
which contains 1,994 images. Even though the absolute numbers are slightly
different from our result from the test set, the general trend still holds:
issue-sensitive models significantly outperform issue-insensitive models.

| Precision | Recall | F1
---|---|---|---
$S_{0}$ | 95.6 | 16.1 | 27.5
$S_{1}$ | 93.6 | 27.8 | 42.9
$S_{1}^{\mathbf{C}}$ | 94.8 | 41.6 | 57.8
$S_{1}^{\mathbf{C}+H}$ | 94.4 | 43.7 | 59.7
Table A2: Attribute coverage results for validation set. | Precision | Recall | $F_{1}$
---|---|---|---
$S_{0}$ | 10.8 | 28.1 | 15.6
$S_{1}$ | 11.0 | 20.8 | 14.4
$S_{1}^{\mathbf{C}}$ | 18.5 | 41.0 | 25.5
$S_{1}^{\mathbf{C}+H}$ | 14.8 | 42.4 | 22.0
Table A3: Issue alignment results for validation set.

##  Appendix E More Examples in CUB

We randomly sampled two test-set images (Figure A5 amd Figure A6) to show
qualitatively how well our issue-sensitive caption model does compared to
other models. To increase readability in these figures, we gloss issues as
question texts.

Image ID: 121.Grasshopper Sparrow - Grasshopper Sparrow 0078 116052.jpg
---
Human caption: Small bird black tan and white feathers with a speck of yellow
above beak.
$S_{0}$ caption: This bird has a white belly and breast with a speckled
appearance on its back.
What is the eye/eyering/eyebrow color?
$S_{1}$ caption: A small tan and tan patterend bird with light belly.
$S_{1}^{\mathbf{C}}$ caption: This is a littl brown bird with black stripes on
the crown and superciliary.
$S_{1}^{\mathbf{C}+H}$ caption: This is a bird with a white belly and a black
and white spotted back.
What is the beak color?
$S_{1}$ caption: This bird has a white belly with dark spots spots and a short
pointy beak.
$S_{1}^{\mathbf{C}}$ caption: This is a white and grey bird with an orange
eyebrow and orange feet.
$S_{1}^{\mathbf{C}+H}$ caption: This is a small bird with a white belly and a
brown back.
What is the breast color?
$S_{1}$ caption: A tan and tan sparrow connects to a white belly with tints of
tan.
$S_{1}^{\mathbf{C}}$ caption: A small bird with a white belly and throat and a
spotted brown back and head.
$S_{1}^{\mathbf{C}+H}$ caption: This bird has a white belly and breast with a
brown crown and short pointy bill.
What is the belly color?
$S_{1}$ caption: A small light brown and white bird with dark eyes and a short
red-tipped bill.
$S_{1}^{\mathbf{C}}$ caption: This bird has a white belly and breast with a
speckled appearance elsewhere.
$S_{1}^{\mathbf{C}+H}$ caption: A small bird with a white belly and breast and
a light brown crown and nape.
What is the beak length?
$S_{1}$ caption: A small round bird with multicolored tan and tan feathers.
$S_{1}^{\mathbf{C}}$ caption: This is a brown and tan speckled bird with a
small beak and long tail feathers.
$S_{1}^{\mathbf{C}+H}$ caption: This bird has a speckled belly and breast with
a short pointy bill.
Figure A5: Issue-sensitive and Issue-insensitive captions for an image of a
Grasshopper Sparrow. To increase readability in these figures, we gloss issues
as question texts. Image ID: 030.Fish Crow - Fish Crow 0073 25977.jpg
---
Human caption: A large all black bird with a fluffy throat and thick round
beak.
$S_{0}$ caption: This bird is completely black with a long blunt bill.
What is the crown color?
$S_{1}$ caption: An all black crow black shiny black bill legs feet and body.
$S_{1}^{\mathbf{C}}$ caption: This bird is all black with a long hooked bill
and a think wingspan.
$S_{1}^{\mathbf{C}+H}$ caption: This is a mostly completely solid color bird
with a black crown and a pointed bill.
What is the beak color?
$S_{1}$ caption: An all black crow black shiny black beak legs legs to shiny
crow.
$S_{1}^{\mathbf{C}}$ caption: An all black crow with strong thick downward
downward curved black beak and black legs.
$S_{1}^{\mathbf{C}+H}$ caption: This is a small pointy bird with a medium
sized beak and is mostly black with a short beak.
What is the breast color?
$S_{1}$ caption: An all black crow black bill legs feet and body.
$S_{1}^{\mathbf{C}}$ caption: This stoutly black bird has strong legs and a
long black beak.
$S_{1}^{\mathbf{C}+H}$ caption: This bird has a short black bill a white
throat and a dark brown crown.
What is the belly color?
$S_{1}$ caption: An all jet black shiny black beak.
$S_{1}^{\mathbf{C}}$ caption: A solid black crow with strong claws and a
trinagular jet-black belly.
$S_{1}^{\mathbf{C}+H}$ caption: This bird has a short bill a white belly and a
a black crown.
Figure A6: Issue-sensitive and Issue-insensitive captions for an image of a
Fish Crow. To increase readability in these figures, we gloss issues as
question texts.