# Sign switch of Gaussian bending modulus for microemulsions; a self-
consistent field analysis exploring scale invariant curvature energies

Ramanathan Varadharajan<EMAIL_ADDRESS>Frans A. M. Leermakers
<EMAIL_ADDRESS>Physical Chemistry and Soft Matter, Wageningen
University & Research Center, Stippeneng 4, 6708 WE Wageningen, The
Netherlands.

###### Abstract

Bending rigidities of tensionless balanced liquid-liquid interfaces as
occurring in microemulsions are predicted using self-consistent field theory
for molecularly inhomogeneous systems. Considering geometries with scale
invariant curvature energies gives unambiguous bending rigidities for systems
with fixed chemical potentials: The minimal surface Im3m cubic phase is used
to find the Gaussian bending rigidity, $\bar{\kappa}$, and a torus with
Willmore energy $W=2\pi^{2}$ allows for direct evaluation of the mean bending
modulus, $\kappa$. Consistent with this, the spherical droplet gives access to
$2\kappa+\bar{\kappa}$. We observe that $\bar{\kappa}$ tends to be negative
for strong segregation and positive for weak segregation; a finding which is
instrumental for understanding phase transitions from a lamellar to a sponge-
like microemulsion. Invariably, $\kappa$ remains positive and increases with
increasing strength of segregation.

††preprint:

Interfaces characterized by dense surfactant packings, such as microemulsions
de Gennes and Taupin (1982); Guha  _et al._ (2017); Labrador  _et al._ (2017)
and biological membranes Tanford (1980) that are found naturally or
manipulated artificially to be in a state of near-zero tension, have extensive
areas. Often such interfaces feature a spontaneous curvature that manifests in
spherical or cylindrical (swollen) micelles Schacht  _et al._ (1996); Gompper
_et al._ (1995). When a system is tensionless and precisely balanced—typical
for single component bilayers and expected for the middle-phase
microemulsions—the interface’s spontaneous curvature vanishes Safran (1994)
and ultra-low interfacial energies can be achieved Safran  _et al._ (1986);
Scriven (1976). Here, the elastic moduli, mean ($\kappa$) and Gaussian
($\bar{\kappa}$) bending rigidities, control the interface fluctuations and
topology, respectively. Such systems show a first-order phase transition from
lamellar to sponge-like phases, e.g., upon an increase of the temperature for
nonionic systems, and a change of the salinity for ionic systems Safran  _et
al._ (1986); Cates  _et al._ (1988); Labrador  _et al._ (2017). A pre-eminent
challenge is to predict, from a molecular model for such interfaces, a means
to induce a sign change in the $\bar{\kappa}$ from negative to positive; this
signals the loss of stability of the lamellar, $L_{\alpha}$, oil-surfactant-
water ordering in favor of a phase with saddles, $L_{3}$ or sponge-like.
Another long-standing problem is understanding the relation between surfactant
chain architecture and corresponding bending rigidities Hellweg and Langevin
(1998); Kurtisovski  _et al._ (2007).

Earlier theoretical methods Szleifer  _et al._ (1988); Würger (2000),
experiments Gradzielski  _et al._ (1996, 1997); Safinya  _et al._ (1989), and
simulations Rekvig  _et al._ (2004); Thakkar  _et al._ (2011) that attempted
to link bending rigidities to molecular properties did not provide information
on $\bar{\kappa}$; moreover, the results for $\kappa$ were not consistent with
each other. Therefore uncertainties prevail and these persist also because
internal checks for presented rigidities are rarely provided. As a result,
there exists no accepted molecular level theory that convincingly links
molecular characteristics to both mechanical parameters of the interfaces
($\kappa$ and $\bar{\kappa}$). Notably, the missing information for
$\bar{\kappa}$ is remarkable as its magnitude and, in particular, its sign are
fundamental to the understanding of microemulsions.

The primary obstacle in establishing a molecular model for determining bending
rigidities is the requirement of curving the interface at fixed chemical
potentials. In this letter, we propose an elegant protocol with internal
checks to find these rigidities. We consider interfaces with scale invariant
curvature energies and illustrate the protocol for tensionless, balanced
liquid-liquid (L/L) interfaces. In line with experimental findings, we report
the existence of a sign switch for $\bar{\kappa}$ which triggers a phase
transition from planar to sponge-like phases in middle-phase microemulsions.
We focus on the role of the interaction parameter which in strong segregation
has a large value and for weak segregation a small value; further, we
elaborate on the role of the molecular weights of the solvents and that of the
amphiphile.

Experiments, simulations, and calculations Szleifer  _et al._ (1988); Würger
(2000); Gradzielski  _et al._ (1996, 1997); Safinya  _et al._ (1989); Rekvig
_et al._ (2004); Thakkar  _et al._ (2011) reviewed above have major
disadvantages and ambiguities because the systems featured too many
complications. We examine a tensionless balanced interface which still is
highly relevant to middle-phase microemulsion systems wherein oil and water
are separated by a surfactant film with extensive areas and often a complex
interface topology. Our focus on tensionless (interfacial tension $\gamma=0$)
balanced (spontaneous curvature $J_{0}=0$) L/L interface avoids the
complications of a finite Laplace pressure (i.e. $\Delta P_{L}=0$) when
imposing some interfacial curvature. Such a model is readily implemented in
the Scheutjens-Fleer Self-Consistent Field theory (SF-SCF) for molecularly
inhomogeneous systems. We can consider this idealized system in three
different geometries with scale-invariant curvature energies. The latter is
essential, as it allows for an analysis in the grand canonical ensemble
($\mu,V,T$), which opens a convincing route to estimate the rigidities: (i) A
spherically curved droplet with $\Delta P_{L}=0$ is used to find
$2\kappa+\bar{\kappa}$; (ii) A minimal Im3m surface (by construction has
$\Delta P_{L}=0$) is used to find $\bar{\kappa}$; (iii) A minimal torus
interface is used to find $\kappa$ also for conditions that $\Delta P_{L}=0$.

We note that the route to obtain rigidities of balanced tensionless L/L
interfaces shows similarities but also important differences from the
symmetric freely dispersed lipid bilayers Leermakers (2013). For bilayers, we
could use the Im3m cubic phase and the spherical vesicle to find
$\bar{\kappa}$ and $2\kappa+\bar{\kappa}$, respectively. The cylindrically
curved vesicle could be used to obtain $\kappa$ in two ways: (i) As the number
of lipids per unit area is found to be a constant (i.e., not a function of the
radius $R$ of the cylindrical vesicle), $\kappa$ was found from the excess
Helmholtz energy per unit length, $F^{\sigma}_{c}$, i.e.,
$\kappa=RF^{\sigma}_{c}/\pi$; (ii) Realizing that the grand potential of the
cylindrical vesicle per unit length $\Omega_{c}$ is split up equally into
bending energy and stretching energy, $\kappa$ is also found from (half) the
grand potential density per unit length, i.e. $\kappa=R\Omega_{c}/(2\pi)$.
However, for the tensionless balanced L/L interface, curved in cylindrical
geometry with $\Delta P_{L}=0$, $\kappa$ can neither be computed from the
Helmholtz energy per unit length, nor from the grand potential per unit
length, as there is neither a conservation of the number of surfactant per
unit area nor a conservation of the chemical potentials of the molecules of
the system, cf. Figs. (1c) and (1d) shown below. Importantly, in the L/L
interface, we do not find a coincidental equal splitting of curvature and
tension energies.

Following Helfrich, by expanding the interfacial tension ($\gamma$) in mean
curvature ($J=1/{R_{1}}+1/{R_{2}}$) and Gaussian curvature
($K={1}/{R_{1}R_{2}}$), with $R_{1}$, $R_{2}$ being principle radii of
curvature as

$\gamma(J,K)-\gamma(0,0)=-\kappa J_{0}J+\frac{1}{2}\kappa
J^{2}+\bar{\kappa}K,$ (1)

we identify $\gamma(J,K)$ as the appropriate characteristic function that
carries the bending information for curving the interface at constant chemical
potentials Helfrich (1973). This expansion is the starting point for our
analysis of interfacial equilibrium properties, as it appears, refer Fig.
(1d), for the balanced L/L interface, we find curved interfaces that exist at
chemical potentials equal to that of the ground state (tensionless balanced
planar interfaces) not only for the surfactant but also for the two solvents.
Note, that in this system, both $\gamma(0,0)=0$ and $J_{0}=0$, Eqn. (1)
simplifies to $\gamma(J,K)=\frac{1}{2}\kappa J^{2}+\bar{\kappa}K$.

Within SF-SCF framework, extremizing the mean field free energy for a
molecularly inhomogeneous system provides both structural and accurate
thermodynamic information Scheutjens and Fleer (1979); Fleer  _et al._ (1993);
Leermakers (2013); Kik  _et al._ (2010). We have implemented a coarse-grained
molecular model in which there are two types of spherically symmetric segments
A and B. These segments are used in two solvents, each with length $n$,
$A_{n}$ and $B_{n}$ forming the two liquid phases $\alpha$ and $\beta$,
respectively, and in a diblock copolymer composed of blocks of equal length
$N$, $A_{N}B_{N}$. This approach requires molecular partition functions, which
are evaluated within a lattice considering the molecules as freely jointed
chains. Accordingly, segments fit on lattice sites. The lattice sites are
organized as homogeneously curved- or planar layers. Driven by the segregation
between the segments, an interface develops on which the lattice geometry
imposes the curvature. Segment density gradients can only develop in the
direction perpendicular to such interface, as a mean field approximation is
implemented in lattice layers ‘parallel’ to the interface. In the absence of
density gradients, the model is equivalent to the Flory-Huggins theory. There
is just one Flory-Huggins interaction parameter ($\chi$) between monomers A
and B. We choose a value slightly above the critical point of the binary
solvent ($\chi^{\rm cr}=2/n$). Below the minimum value used for $n$ is $4$,
and the interaction is chosen between $\chi=0.52$ \- $0.68$ (for more details
on method and model refer supplemental material sup ).

Figure 1: (a) Volume fraction distribution and (b) Lateral pressure
distribution (in $k_{B}T/b^{3}$) in a planar tensionless interface. (c) Area
per surfactant molecule ($\Gamma$) [in units $b^{2}$] and (d) Chemical
potential of surfactant ($\mu\equiv\mu_{s}$) [in units of $k_{B}T$] as a
function of interface curvature as indicated for systems with $\Delta
P_{L}=0$. Surfactant A30B30, solvents, A4 and B4. $\chi=0.6$

Volume fraction profiles, $\varphi(z)$, and the lateral pressure distribution,
$p(z)=-\omega(z)$, with $\omega$ as the grand potential density, are presented
for the default planar tensionless L/L interface, in Figs. (1a) and (1b).
Here, $z\equiv z/b$ is the dimensionless normal coordinate. In Fig. (1a) we
see that the two liquids give a Van der Waals-like profile and the accumulated
copolymers have their blocks on corresponding sides of the interface. The
pressure profile $p(z)$, see Fig. (1b), has a negative excursion at the
interface due to the contribution from the L/L interface and positive ‘wings’
on either side of the interface due to the overlap of copolymers in a brush-
like configuration. From earlier work Oversteegen  _et al._ (1999), we know
that $\gamma=-\sum_{z}p(z)$ and that the second moment of the pressure
distribution with respect to the Gibbs plane ($R^{g}$) provides a direct
estimate of $\bar{\kappa}=\sum-(z-R^{g})^{2}p(z)$. The latter relation proved
useful for the evaluating $\bar{\kappa}$ of lipid bilayers, and presents a
strong test for alternative, more elaborate routes to obtain the same
quantity.

Similar as for lipid bilayers, an independent alternative route for evaluating
$\bar{\kappa}$ makes use of three-gradient SCF computation as shown in Fig.
(2a), where on all six faces of the elementary box, Neumann boundary
conditions apply; the elementary box is 1/8th of a unit cell of an Im3m phase,
8 unit cells are shown in Fig. (2b). When equal amounts of A and B are present
in the system, the interface splits the volume into two identical sub-volumes
(phase $\alpha$ and $\beta$) while $J=0$ along the surface and $\Delta
P_{L}=0$. As soon as the copolymers are added, such that the chemical
potential of all molecular species is equal to the corresponding values of the
planar tensionless system, we lie within an $(\mu,V,T)$-ensemble; thus, the
grand potential, $\Omega=F-\sum_{j}\mu_{j}n_{j}$, is the characteristic
function, and $\Omega=\bar{\kappa}\int_{M}KdA$. Using Gauss-Bonnet theorem for
compact, boundary-less Riemann manifold, the integral of curvature over the
area can be evaluated as $-8\pi$ Chern (1944); Fenchel (1940). Thus, the grand
potential for the unit cell, $\Omega=-8\pi\bar{\kappa}$. Hence, from the
scale-invariant grand potential $\Omega$ directly follows $\bar{\kappa}$. The
result is consistent with the second moment over the pressure profile (see
Tab. (1) in supplemental material sup ).

Figure 2: Volume fraction distribution of $\alpha-$phase from 3D SCF
calculation of interface modeled as Im3m cubic phase. $1/8$ of a unit cell is
shown in (a) 8 unit cells are shown for visualization in (b). Schematic
illustration of an interface in torus shape is shown in (c). Volume fraction
distribution of $\alpha-$phase from 2D SCF calculation of minimal torus in a
cylindrical lattice is shown in (d). The molecular model is similar as in Fig.
1. Color scale from blue to red is $0.2-0.8$ for all contours.

The procedures to evaluate $\kappa$ are more involved. In Figs. (1c) and (1d),
we have presented typical results for spherically and cylindrically curved
interfaces when $\Delta P_{L}=0$ as a result of the adsorption of the
copolymers. In Fig. (1c) we show the area per copolymer at the interface
(inverse of the adsorbed amount) and in Fig. (1d) the corresponding chemical
potentials as a function of the curvature $J$. In Fig. (1d) we notice that the
chemical potentials remain constant upon bending in case of spherical
curvature. This means that in this geometry bending is performed in the
$(\mu,V,T)$-ensemble. The reason why the system can maintain its chemical
potentials upon bending of the interface is traced to the known fact that
integrating Eqn. (1) over the area,
$\Omega=\int_{M}\gamma(J,K)dA=4\pi(2\kappa+\bar{\kappa})$, is a constant
irrespective of the size of the spherical droplet showing scale invariance.

Now an indirect route is available to compute $\kappa$, namely from combining
the total curvature energy from the spherical droplet with the Gaussian
bending modulus $\bar{\kappa}$ found above. Ideally, we would like to validate
this indirect route with a direct estimate.

Again, as in the cylindrical geometry, neither the adsorbed amount of
surfactant, cf. Fig. (1c), nor the corresponding chemical potential, cf. Fig.
(1d), is conserved, and we cannot use this geometry to obtain $\kappa$. A
direct route to evaluate the mean bending modulus is still possible using a
system that features a minimal torus, as illustrated in Fig. (2c). Within SF-
SCF this is realized using a two-gradient ($r,z$) cylindrical lattice. A
typical result is presented in Fig. (2d) as a density contour plot in the
$(r,z)$ cross-section. From Gauss-Bonnet theorem, as the torus has genus $g=1$
the integral $\int_{M}KdA$ vanishes. Moreover, the so-called Willmore energy
of the torus has contribution only from mean curvature,
$W=\frac{1}{4}\int_{M}J^{2}dA$.

In 1965, T.J. Willmore conjectured that the Willmore energy ($W$) of a smooth
torus immersed in 3D space is always greater than or equal to $2\pi^{2}$
Willmore (1965). This conjecture was proved by Marques and Neves in 2012
Marques and Neves (2014). The Willmore energy reaches its minimum when the
radius of revolution is $\sqrt{2}$ times the radius of the generating circle,
as shown in Fig. (2d). By integrating the Helfrich equation for the toroidal
configuration with minimal Willmore energy, we obtain the grand potential for
torus as, $\Omega_{t}=\frac{1}{2}\kappa\int_{M}J^{2}dA=2\kappa
W=4\pi^{2}\kappa$.

Now the protocol boils down to generating this minimal torus in SF-SCF while
adding the copolymer such that $\Delta P_{L}=0$. It occurs that in this case,
the system converges with all its chemical potentials equal to that of the
planar tensionless interface and lies within the ($\mu,V,T$)-ensemble. Similar
as in the droplet case this result is traced to the scale invariance, in this
case of the minimal Willmore energy. Its grand potential gives a direct
estimate of $\kappa={\Omega_{t}}/{4\pi^{2}}$.

The values found for $\kappa$ by the direct and indirect routes are congruent,
proving that there is complete consistency in obtaining the bending
rigidities, using scale-invariant surfaces, for tensionless balanced L/L
interfaces sup . Our protocol is available at https://wp.me/p7KmNt-9C as a
open source software package.

As we have established the molecular link for bending rigidities, we now
present the chain length dependence of the bending rigidities for the regime
where $N>n$ in in Fig. (3a) and (3b), and for $N\approx n$ in Fig. (3c) and
(3d). The trends for $N>n$ support the results from simulations Rekvig  _et
al._ (2004); Thakkar  _et al._ (2011); Smit  _et al._ (1990). While it would
be alluring to conclude that bending rigidities have a linear dependence on
the chain length of the surfactants, results in regime $n\approx N$ contradict
this observation and the dependences are clearly nonlinear, see Figs. (3c) and
(3d). It is observed that the dependence of rigidities on surfactant chain
length is strongly influenced by solvent chain length and interaction
parameter between monomeric units, an important effect which has not been
addressed in previous works Hellweg and Langevin (1998); Kurtisovski  _et al._
(2007); Szleifer  _et al._ (1988); Würger (2000); Gradzielski  _et al._ (1996,
1997); Safinya  _et al._ (1989); Rekvig  _et al._ (2004); Thakkar  _et al._
(2011). A thorough analysis of the magnitude of $\kappa$ is, however, beyond
the scope of the present letter and will be presented elsewhere.

Figure 3: Chain length dependence of bending rigidities [in units of
$k_{B}T$]. (a) and (b) $N>n$ Regime: Chain length of bulk phases fixed (A4,
B4), surfactant chain length is varied (ANBN, where $20<N<50$;
$0.1<\Delta\chi<0.2$). (c) and (d) $N\approx n$ Regime: Chain length of bulk
phases fixed (A20, B20), surfactant chain length is varied (ANBN, where
$16<N<20$; $0.3<\Delta\chi<0.4$).

Moving from the regime where the solvent length is smaller compared to the
surfactant block length, $n<N$, to the regime where the solvent length is
comparable to that of the surfactant, we observe that $\bar{\kappa}$ is of
opposite sign, cf. Figs. (3a) and Fig. (3c). Such a sign switch is of
exceptional interest, as it addresses a topological phase transition in
microemulsions that can be achieved in two ways: (1) by tuning the interaction
parameter for fixed solvent and surfactant lengths and (2) by tuning solvent
length for fixed surfactant length and $\Delta\chi=\chi-2/n$.

In Fig. (4a) the dependence of both $\kappa$ and $\bar{\kappa}$ are shown for
surfactant block length of $N=20$ as a function of a measure of closeness to
the critical point of the binary solvent $\Delta\chi$. The Gaussian bending
modulus, $\bar{\kappa}$, switches from negative to positive when moved towards
weak segregation; this transition occurs earlier in higher $n$ (dashed line)
for fixed $N$.

A similar effect can also be achieved by tuning the surfactant chain length
for a given solvent chain length and interaction energy ($\Delta\chi$).
Experimentally, one can reach weak segregation by the addition of a suitable
co-solvent which diminishes the difference between the two primary solvents.

A summary of results, obtained by tuning $N$, is presented as a ‘phase
diagram’ in Fig. (4b). The two governing parameters, i.e., block length of the
copolymer and the chain length of the solvent are on the $x-$ and $y-$axis,
respectively. The interaction parameter is chosen as $\Delta\chi$. By tuning
the surfactant length for given interactions ($\Delta\chi$) and solvent chain
length, $\kappa$ increases and $\bar{\kappa}$ decreases monotonically, also
showing a sign switch at the solid lines.

These results imply the tendency of the interface to remain planar on average
when $n<<N$. This result is contrasted with the situation when the length of
the solvent molecules is increased to be similar, $n\approx N$, or even larger
than that of the copolymer, $n>N$; $\bar{\kappa}$ becomes positive in this
regime while $\kappa$ is small but positive. For large $n$, we have
$\bar{\kappa}>0;\ 0<\kappa<1$. These features are consistent with a sponge
phase (Winsor III) Gompper  _et al._ (1995) which grows in importance with
reducing $\chi$. Stable but very flexible, and strongly fluctuating lamellar
phases (as $\bar{\kappa}<0$ and $0<\kappa<1$) are observed as $N$ is increased
for fixed $n$, whereas for very large $N$, $\kappa$ is $>1$, and we enter a
region where the fluctuations of the interface are weak, crossing the dashed
lines, as shown in Fig. (4b).

Figure 4: (a) Gaussian bending modulus (blue axis) $\bar{\kappa}$ [in units of
$k_{B}T$] and mean bending modulus (red axis) $\kappa$ [in units of $k_{B}T$]
as a function of $\Delta\chi$ ($\Delta\chi=\chi-2/n$). Surfactants are modeled
as $A_{N}B_{N}$. $\alpha-$phase is modeled as $A_{n}$ and $\beta-$phase is
modeled as $B_{n}$. [solid line: $n=4$, dashed line: $n=6$] (b) Phase diagram
in $n$ and $N$ coordinates for fixed $\Delta\chi$ as indicated. The sign and
magnitude [in units of $k_{B}T$] of the rigidities are indicated. Note that
within the mean field model $n$, $N$ are related to the radii of gyration in
the bulk as $R_{g}=b\sqrt{n/6}$) and $R_{G}=b\sqrt{2N/6}$, for the solvent and
copolymer respectively, where $b$ is bond length.

We have linked molecular characteristics to bending rigidities for surfactant-
covered L/L interfaces. Using surfaces with scale-invariant curvature energies
is embellished as an elegant route to determine $\kappa$ and $\bar{\kappa}$
unambiguously; this route cautiously exploits the tensionless state of the
interfaces and avoids the linear term in curvature. Large deviations from
these constraints imply the loss of the microemulsion middle-phase in favor of
emulsions with oil-in-water or water-in-oil droplets; however, understanding
the effects of small deviations is vital, as it is a prerequisite for any
detailed comparison with experiments. The current analysis provides a natural
starting/reference point to generalize for molecular asymmetry, spontaneous
curvature and finite tension of the interfaces.

This work is part of an Industrial Partnership Programme, ‘Shell/NWO
Computational Sciences for Energy Research (CSER-16)’, of the Foundation for
Fundamental Research on Matter (FOM), which is part of the Netherlands
Organisation for Scientific Research (NWO). Project number: 15CSER26. We are
indebted to T. E. Kodger and J. M. Clough for proofreading our letter.

## References

  * de Gennes and Taupin (1982) P. G. de Gennes and C. Taupin, J. Phys. Chem. 86, 2294 (1982).
  * Guha  _et al._ (2017) I. F. Guha, S. Anand, and K. K. Varanasi, Nature Communications 8, 1371 (2017).
  * Labrador  _et al._ (2017) A. Labrador, A. Seddon, A. Squires, C. Dicko, C. Pfrang, E. Cabrera-Martinez, K. Rastogi, N. Cowieson, and T. Plivelic, Nature Communications 8, 1724 (2017).
  * Tanford (1980) C. Tanford,  _The Hydrophobic Effect: Formation of Micelles and Biological Membranes_ (J. Wiley., 1980).
  * Schacht  _et al._ (1996) S. Schacht, Q. Huo, I. G. Voigt-Martin, G. D. Stucky, and F. Schuth, Science 273, 768 (1996).
  * Gompper  _et al._ (1995) G. Gompper, M. Schick, and S. Milner, “Self-assembling amphiphilic systems,”  (1995).
  * Safran (1994) S. A. Safran,  _Statistical thermodynamics of surfaces, interfaces, and membranes_, Vol. 90 (Perseus Books, 1994).
  * Safran  _et al._ (1986) S. Safran, D. Roux, M. Cates, and D. Andelman, Phys. Rev. Lett. 57, 491 (1986).
  * Scriven (1976) L. Scriven, Nature (London) 263, 123 (1976).
  * Cates  _et al._ (1988) M. Cates, D. Roux, D. Andelman, S. Milner, and S. Safran, EPL (Europhys. Lett.) 5, 733 (1988).
  * Hellweg and Langevin (1998) T. Hellweg and D. Langevin, Phys. Rev. E 57, 6825 (1998).
  * Kurtisovski  _et al._ (2007) E. Kurtisovski, N. Taulier, R. Ober, M. Waks, and W. Urbach, Phys. Rev. Lett. 98, 258103 (2007).
  * Szleifer  _et al._ (1988) I. Szleifer, D. Kramer, A. Ben-Shaul, D. Roux, and W. M. Gelbart, Phys. Rev. Lett. 60, 1966 (1988).
  * Würger (2000) A. Würger, Phys. Rev. Lett. 85, 337 (2000).
  * Gradzielski  _et al._ (1996) M. Gradzielski, D. Langevin, and B. Farago, Phys. Rev. E 53, 3900 (1996).
  * Gradzielski  _et al._ (1997) M. Gradzielski, D. Langevin, T. Sottmann, and R. Strey, J. Chem. Phys. 106, 8232 (1997).
  * Safinya  _et al._ (1989) C. R. Safinya, E. B. Sirota, D. Roux, and G. S. Smith, Phys. Rev. Lett. 62, 1134 (1989).
  * Rekvig  _et al._ (2004) L. Rekvig, B. Hafskjold, and B. Smit, Phys. Rev. Lett. 92, 116101 (2004).
  * Thakkar  _et al._ (2011) F. M. Thakkar, P. K. Maiti, V. Kumaran, and K. G. Ayappa, Soft Matter 7, 3963 (2011).
  * Leermakers (2013) F. A. M. Leermakers, J. Chem. Phys. 138, 04B610 (2013).
  * Helfrich (1973) W. Helfrich, Z. Naturforsch. C 28, 693 (1973).
  * Scheutjens and Fleer (1979) J. M. H. M. Scheutjens and G. J. Fleer, J. phys. Chem 83, 1619 (1979).
  * Fleer  _et al._ (1993) G. Fleer, M. A. C. Stuart, J. M. H. M. Scheutjens, T. Cosgrove, and B. Vincent,  _Polymers at interfaces_ (Springer Science & Business Media, 1993).
  * Kik  _et al._ (2010) R. A. Kik, F. A. M. Leermakers, and J. M. Kleijn, Phys. Rev. E 81, 021915 (2010).
  * (25) See Supplemental Material http://link.aps.org/ for details about method and additional figures. .
  * Oversteegen  _et al._ (1999) S. M. Oversteegen, P. A. Barneveld, J. V. Male, F. A. M. Leermakers, _et al._ , Physical Chemistry Chemical Physics 1, 4987 (1999).
  * Chern (1944) S. S. Chern, Annals of mathematics , 747 (1944).
  * Fenchel (1940) W. Fenchel, Journal of the London Mathematical Society 1, 15 (1940).
  * Willmore (1965) T. J. Willmore, An. Sti. Univ. “Al. I. Cuza” Iasi Sect. I a Mat.(NS) B 11, 493 (1965).
  * Marques and Neves (2014) F. C. Marques and A. Neves, Annals of Mathematics 179, 683 (2014).
  * Smit  _et al._ (1990) B. Smit, P. A. J. Hilbers, K. Esselink, L. A. M. Rupert, N. M. Van Os, and A. G. Schlijper, Nature (London) 348, 624 (1990).


# Plausible May Not Be Faithful: Probing Object Hallucination in Vision-
Language Pre-training

Wenliang Dai1, Zihan Liu ${}^{\text{ }2}$, Ziwei Ji1, Dan Su1, Pascale Fung1
1The Hong Kong University of Science and Technology
2NVIDIA
<EMAIL_ADDRESS><EMAIL_ADDRESS>∗ Work done during PhD at
HKUST.

###### Abstract

Large-scale vision-language pre-trained (VLP) models are prone to hallucinate
non-existent visual objects when generating text based on visual information.
In this paper, we exhaustively probe the object hallucination problem from
three aspects. First, we examine various state-of-the-art VLP models, showing
that models achieving better scores on standard metrics (e.g., BLEU-4, CIDEr)
could hallucinate objects more frequently. Second, we investigate how
different types of visual features in VLP influence hallucination, including
region-based, grid-based, and patch-based. Surprisingly, we find that patch-
based features perform the best and smaller patch resolution yields a non-
trivial reduction in object hallucination. Third, we decouple various VLP
objectives and demonstrate their effectiveness in alleviating object
hallucination. Based on that, we propose a new pretraining loss, object masked
language modeling, to further reduce object hallucination. We evaluate models
on both COCO (in-domain) and NoCaps (out-of-domain) datasets with our improved
CHAIR metric. Furthermore, we investigate the effects of various text decoding
strategies and image augmentation methods on object hallucination.

†Work in progress.

##  1 Introduction

Thanks to the advancement of large pre-trained Language Models (LMs) and
Vision-Language Pre-training (VLP) methods, models are able to achieve
surprisingly good performance in vision-conditioned text generation, e.g.,
image captioning. However, large LMs are found to often generate unfaithful or
nonsensical texts given the source input (Ji et al., 2022), which is called
hallucination. This problem is also inherited to VLP models (Alayrac et al.,
2022), as they often generate fluent and seems likely sentences if we only see
the text, but wrong when includes the visual inputs. One major type of
hallucination in VL is known as object hallucination (Rohrbach et al., 2018),
where models generate non-existent or inaccurate objects from the input image.
Object hallucination in VLP models essentially limit their performance and
raise safety concerns for industrial applications. For example, in biomedical
image captioning (Pavlopoulos et al., 2019), object hallucination reduces the
accuracy of diagnosis and may lead to severe consequences to the patient.
Despite the limitations and potential risks caused by the object
hallucination, this problem in VLP models has not been studied in contemporary
works yet.

To narrow down the aforementioned research gap, we systematically investigate
four fundamental research questions about object hallucination: 1) how much do
modern VLP models hallucinate? 2) how do different forms of image encoding
affect object hallucination? 3) what are the effects of various VLP objectives
on object hallucination? and 4) how to alleviate object hallucination based on
our findings?

To evaluate object hallucination, we adopt and improve upon the CHAIR metric,
Caption Hallucination Assessment with Image Relevance, proposed by Rohrbach et
al. (2018). In addition to the in-domain COCO dataset, we extend the
evaluation with NoCaps to further assess the faithfulness of generated
captions in the out-of-domain scenario.

For our first question, we examine recently proposed VLP models, showing that
they still hallucinate frequently, especially on out-of-domain images even if
they have been pre-trained on millions of image-text pairs. Interestingly,
models achieving better scores on previous standard metrics (e.g., BLEU-4,
CIDEr) could hallucinate more often. Additionally, we discover that the widely
adopted optimization method SCST (Rennie et al., 2017) leads to more severe
hallucination problem. Second, we investigate how different types of image
encoding in VLP influence hallucination, including region-based, grid-based,
and patch-based. Surprisingly, we find that patch-based features perform the
best and smaller patch resolution yields a non-trivial reduction in object
hallucination. Third, we decouple common VLP objectives, demonstrating that
discriminative losses (e.g., cross-modal contrastive, matching, and their
variants) do not mitigate object hallucination by learning global multimodal
representations. For generative losses, they indeed reduce hallucination while
different pre-training datasets lead to distinctive model behaviors. Finally,
besides the discoveries above, we propose a new VLP loss, namely object masked
language modeling, to further alleviate object hallucination by enhancing the
alignment between text tokens and visual objects during generation.

Our contributions are three-fold:

  * •

We systematically investigate state-of-the-art vision-language pre-trained
models on the object hallucination problem, showing that it is still far from
resolved and previous methods that improve standard metrics may reflect in
even worse hallucination.

  * •

We study the effects of different types of image encodings and decouple three
common VLP objectives to analyze which parts of modern VLP methods impact
object hallucination.

  * •

We propose a simple yet effective pre-training objective to mitigate object
hallucination, namely object masked language modeling. Experimental results
show that it reduces object hallucination by 17.4% without the need of new
dataset.

We believe our insightful findings will grease the way for building more
responsible and reliable VLP models. Code and evaluation setups will be
released.

##  2 Related Work

###  2.1 Hallucination in Deep Learning

Generally, the term hallucination denotes the appearance of undesirable output
that is unfaithful to the conditional input Maynez et al. (2020), even though
it may appear to be fluent or reasonable. In the multimodal field, the
hallucination phenomenon refers to the prediction of non-existent or incorrect
objects (e.g., in object detection or image captioning) and is called object
hallucination (Rohrbach et al., 2018; Biten et al., 2022). Despite the success
of deep learning models, they suffer the hallucination problem, which degrades
the performance and hinders practical applications Ji et al. (2022).

Many works have been proposed to mitigate hallucination in recent years. Nie
et al. (2019) applied data refinement with self-training to improve the
equivalence between the input and the paired text in the data-to-text
generation task. Xiao and Wang (2021) proposed the uncertainty-aware beam
search as an add-on technique to the original beam search, in both image
captioning and data-to-text generation. To reduce hallucination in dialog
systems, Shuster et al. (2021) introduced knowledge augmentation and Dziri et
al. (2021) presented a post-processing method to refine generated outputs. Su
et al. (2022) augment the generation model with fine-grained, answer-related
salient information predicted by a machine reading comprehension module, to
reduce hallucination in generative question answer task.

###  2.2 Vision-Language Pre-training

The research on vision-language pre-training (VLP) has progressed vastly in
recent years. Due to the demand for large-scale data, most VLP methods use
self-supervised pretraining objectives to utilize image-text pairs crawled
from the web. In the beginning, BERT (Devlin et al., 2019)-style VLP models
(Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020;
Shen et al., 2021) are trained to perform multimodal understanding tasks,
using objectives like image-text matching and masked language modeling. Later,
encoder-decoder architectures are introduced to additionally handle multimodal
generation tasks with a causal language modeling loss (Li et al., 2021b; Lin
et al., 2021; Cho et al., 2021; Ding et al., 2021; Li et al., 2022; Wang et
al., 2022). Another line of research uses a dual-stream architecture (Radford
et al., 2021; Jia et al., 2021; Zhai et al., 2021; Yao et al., 2021) with
separate image and text encoders aligned together through an image-text
contrastive loss. They improve the performance of various multimodal
downstream tasks by a large step.

Alayrac et al. (2022) show that fatal object hallucination can happen
naturally or be provoked by the adversarial prompting in modern VLP models.
However, in previous works, how different VLP strategies influence the
faithfulness of generated text given images has not been studied. Moreover,
the effects of using different types of image encoding are also unclear,
including region-based (Li et al., 2020b; Zhang et al., 2021; Hu et al.,
2021), grid-based (Wang et al., 2021), and patch-based (Kim et al., 2021; Li
et al., 2021a).

##  3 Evaluation Setup

In this section, we first introduce the CHAIR evaluation metric and our
proposed improvements to it in section 3.1. Then, in section 3.2, we discuss
the datasets used for evaluation and explain how to calculate CHAIR under
different settings.

###  3.1 Evaluation Metric

We use the CHAIR metric, Caption Hallucination Assessment with Image
Relevance, proposed by Rohrbach et al. (2018) to measure the object
hallucination. CHAIR calculates what proportion of object words generated are
actually in the image according to the ground truth. CHAIR has two variants:
CHAIRi (instance-level) and CHAIRs (sentence-level), which are formulated as
follows:

$\textrm{CHAIR}_{i}=\frac{\textrm{\\# \\{hallucinated objects\\}}}{\textrm{\\#
\\{all objects in ground truth\\}}},$ $\textrm{CHAIR}_{s}=\frac{\textrm{\\#
\\{hallucinated sentences\\}}}{\textrm{\\# \\{all sentences\\}}},$

where CHAIRi measures the proportion of hallucinated objects over all the
ground-truth objects (note that it calculates sample by sample and then
averages the score over all samples), and CHAIRs measures the proportion of
the hallucinated sentence (has at least one hallucinated object) over all
sentences.

We notice that the CHAIRi score will tend to be small when there are
substantial objects in the ground truth (the denominator becomes large) or
when the model tends to generate a small number of objects, leading to a
relatively small number of hallucinated objects (the numerator becomes small).
Therefore, we propose a modified version of $\text{CHAIR}_{i}$ as follows:

$\textrm{CHAIR}_{i}^{\prime}=\frac{\textrm{\\# \\{hallucinated
objects\\}}}{\textrm{\\# \\{all objects in prediction\\}}},$

where the denominator denotes the number of predicted objects. We can see that
$\text{CHAIR}_{i}^{\prime}$ score measures the proportion of hallucinated
objects in the generation, which will not be affected by the number of objects
in the ground truth. Compared to CHAIRi, $\text{CHAIR}_{i}^{\prime}$ can
better measure the likelihood of the model’s object hallucination. Without
further mentioning, CHAIRi represents our modified one in the following
sections.

Model |  CIDEr Optim (SCST) |  # Pretrain Image-Text Pairs | COCO Caption Karpathy Test | NoCaps Validation
---|---|---|---|---
Out-of-domain
B@4$\uparrow$ | C$\uparrow$ | M$\uparrow$ | S$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$ | C$\uparrow$ | S$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$
OSCAR Base∗ | ✗ | 6.5M | 34.4 | 117.6 | 29.1 | 21.9 | 7.1 | 13.0 | - | - | - | -
OSCAR Base∗ | ✓ | 6.5M | 39.6 | 134.2 | 29.8 | 23.5 | 7.2 | 13.5 | - | - | - | -
VinVL Base | ✗ | 6.5M | 38.2 | 129.3 | 30.3 | 23.6 | 5.3 | 10.0 | 83.1 | 10.8 | 12.1 | 21.2
VinVL Base | ✓ | 6.5M | 40.9 | 140.4 | 30.9 | 25.1 | 5.7 | 10.9 | 87.5 | 11.7 | 17.4 | 32.1
VinVL Large | ✗ | 6.5M | 38.5 | 130.8 | 30.4 | 23.4 | 5.5 | 10.5 | - | - | - | -
VinVL Large | ✓ | 6.5M | 41.0 | 140.9 | 31.1 | 25.2 | 5.6 | 10.6 | - | - | - | -
BLIP Base | ✗ | 129M | 39.7 | 133.3 | 31.0 | 23.8 | 4.9 | 8.9 | 112.1 | 14.2 | 6.6 | 10.5
BLIP Large | ✗ | 129M | 40.4 | 136.7 | 31.1 | 24.3 | 4.7 | 8.8 | 115.3 | 14.4 | 6.4 | 10.5
OFA Large | ✗ | 21M† | 41.7 | 140.5 | 31.2 | 24.2 | 4.7 | 8.9 | 103.2 | 13.3 | 6.4 | 10.2
OFA Large | ✓ | 21M† | 43.8 | 149.5 | 31.8 | 25.9 | 4.2 | 8.1 | 113.1 | 15.2 | 7.1 | 12.4

Table 1: Image captioning results of recent state-of-the-art VLP models (Li et
al., 2020b; Zhang et al., 2021; Li et al., 2022; Wang et al., 2022) on the
COCO Caption Karpathy test set and NoCaps validation set. Here, B@4, C, M, S,
and CH denote BLEU-4, CIDEr, METEOR, SPICE, and CHAIR, respectively. CIDEr
Optim indicates whether the SCST CIDEr optimization is used or not. All
results are generated by using their officially provided checkpoints and
hyper-parameters, * means the model is finetuned by us as the provided one is
broken. $\dagger$ denotes the model also uses unimodal data besides image-text
pairs.

###  3.2 Evaluation Datasets

##### COCO Caption.

The COCO Caption (Lin et al., 2014) is a large-scale and widely used dataset
for the training and evaluation of the image captioning task. We use the
Karpathy split (Karpathy and Fei-Fei, 2017), in which 82K, 5K, and 5K images
are in the train, validation, and test sets, respectively. Each image is
annotated with at least five ground truth captions.

To calculate CHAIR scores on this dataset, we follow the setting proposed in
Rohrbach et al. (2018). In practice, we first tokenize each sentence and then
singularize each word. Then, we use a list of synonyms from Lu et al. (2018)
to map fine-grained objects to the pre-defined 80 coarse-grained MSCOCO
objects categories (e.g., mapping “puppy”, “chihuahua”, “poodle” objects to
the “dog” object). The purpose of doing this mapping is to ensure that we do
not detect hallucinated objects by mistake. For example, when the ground-truth
caption only has the “puppy” object, the CHAIR metrics will consider the “dog”
object generated by models as the hallucinated object if we do not map the
previous object to the “dog” object.

##### NoCaps.

The NoCaps (Agrawal et al., 2019) dataset aims to evaluate models trained on
the training set of COCO Caption data to examine how well they generalize to a
much larger variety of visual concepts, i.e., unseen object categories. There
are 4,500 images in the validation set and 10,600 images in the test set. The
images are from the Open Images V4 (Kuznetsova et al., 2020) dataset, which
contains 600 object classes.

To calculate CHAIR scores on this dataset, we follow the setting used in COCO
Caption. Specifically, we map the fine-grained classes defined in NoCaps to
coarse-grained categories based on the hierarchical object
relationship111https://github.com/nocaps-org/image-feature-
extractors/blob/master/data/oi_categories.json to improve the effectiveness of
CHAIR metrics. Specifically, we only add two types of object categories to our
final object list: 1) the super-category that has sub-categories, and 2) the
object category that has neither super-category nor sub-categories.
Eventually, we construct a list of 139 coarse-grained object categories from
the 600 classes.

##  4 Object Hallucination in VLP Models

Benefitting from the vast advancement of various VLP methods, the performance
of image captioning has been improved by a large step. Generally, this
performance is measured by metrics like CIDEr (Vedantam et al., 2015), SPICE
(Anderson et al., 2016), METEOR (Banerjee and Lavie, 2005), and BLEU (Papineni
et al., 2002), which consider the semantic and syntactic similarity or n-gram-
based fluency between the model generated and ground truth captions. However,
the faithfulness of generated captions is neglected.

In this section, we provide a preliminary analysis of recently proposed VLP
models on the image captioning task to investigate and understand how much
they hallucinate when generating text conditioned on an image. The results are
shown in Table 1. Models are finetuned on the COCO Karpathy training set and
evaluated on both of the COCO Karpathy test set and the NoCaps validation set.

Figure 1: Comparison of image captioning examples generated by VinVLBase and
OFALarge with and without the SCST CIDEr optimization. Red color denotes the
object is hallucinated.

Overall, we observe two noteworthy insights. First, for all CHAIR scores, they
are not proportional to standard evaluation metrics. Although standard metrics
(e.g., the cosine similarity in CIDEr) could potentially penalize the wrong
object prediction, they do not directly reflect faithfulness. Captions can
still have good scores from standard metrics as long as they contain
sufficient accurate objects, even if hallucinated objects exist. For example,
VinVLLarge achieves higher CIDEr and BLEU-4 scores than VinVLBase, but its
CHAIR scores are also higher.

Figure 2: An overview of the model architecture for image captioning and
vision-language pretraining objectives, including image-text contrastive
(ITC), matching (ITC), and image-conditioned language modeling (LM).

Second, the Self-Critical Sequence Training (SCST) (Rennie et al., 2017) for
the CIDEr optimization method harms the faithfulness of generated captions.
SCST is a reinforcement learning algorithm that has been widely adopted as a
second-stage finetuning method after the standard cross-entropy optimization
for image captioning (Anderson et al., 2018; Zhou et al., 2020; Li et al.,
2020b; Zhang et al., 2021; Hu et al., 2021; Wang et al., 2022). It calculates
the reward based on the CIDEr score by sampling captions during training
without the need of another baseline. Although SCST can significantly boost
previous standard metric scores, it encourages models to hallucinate more
inaccurate objects in the captions. For example, applying SCST improves the
CIDEr score by 11.1 and BLEU-4 score by 2.7 for VinVLBase, yet it also
increases 0.9 CHAIRs score on the COCO dataset. Moreover, this problem becomes
more severe on out-of-domain images. For the VinVLBase model, there are 10.9%
more generated captions containing at least one hallucinated object after
using SCST. We speculate that the CIDEr-based optimization encourages models
to generate more n-grams with higher cosine similarity values to the ground
truth captions in the multimodal representation space, which can be plausible
but not faithful. We show a case study in Figure 1. After finetuned by SCST,
models will take a bigger risk to generate more detailed yet incorrect
information (e.g., in the second example in Figure 1, the sentence with
hallucination generates more detailed information “mirror”, which cannot be
found in the image). This will further amplify the object hallucination
problem on out-of-domain images as models may have lower confidence on
unfamiliar visual objects.

Although insightful, these preliminary results cannot reveal more detailed
reasons for the object hallucination, as different VLP models use different
architectures, pre-training datasets, pre-training objectives, etc. In follow-
up sections, we will study how various strategies influence hallucination and
how to mitigate it.

##  5 Probing Image Features and VLP Objectives

In this section, we investigate two fundamental factors of VLP models that can
potentially affect the degree of object hallucination: 1) the type of visual
features including region-based, grid-based, and patch-based; and 2) the
pretraining objectives in VLP. We first introduce the model architecture in
section 5.1. Then, we ablate and compare the effect of different visual
formats (section 5.2), analyze VLP objectives that intuitively could influence
object hallucination (section 5.3), and finally propose a simple yet effective
technique to mitigate this problem. Implementation details are included in
Appendix A.

###  5.1 Model Architecture

##### CLIP.

CLIP (Radford et al., 2021) is a dual-stream VLP model that consists of an
image encoder and a text encoder. It is pretrained on 400 million image-text
pairs data using a cross-modal contrastive loss. Specifically, CLIP explores
the image encoder with different sizes of two
architectures222https://github.com/openai/CLIP/blob/main/model-card.md,
including the ResNet (He et al., 2016) and the Vision Transformer (ViT)
(Dosovitskiy et al., 2021). The resulting image and text encoders are aligned
in the same multimodal feature space.

##### BERT.

BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) model pre-
trained on a large corpus by the masked language modeling (MLM) and sentence
permutation losses. It is shown to have excellent performance on various
downstream tasks after finetuning. Moreover, BERT can also handle generation
tasks when the self-attention layers are restricted to the left-to-right
direction to generate text auto-regressively. In this paper, we refer to this
variant as BertLM.

We design a flexible architecture that can plug in various visual encoders and
fit modern VLP objectives without introducing extra influencing factors. As
shown in Figure 2, the model consists of two parts, a visual encoder to encode
images and a text decoder to generate captions conditioned on the image
representations. We use two separate modules rather than a unified single-
stream model as it is convenient to alter the visual encoder while keeping the
text decoder the same. Specifically, for region-based image features, we
explore the Faster R-CNN object detector (Ren et al., 2015) with two different
backbones: the ResNet-101 used in BUTD (Anderson et al., 2018) and the
ResNeXt-152 (Xie et al., 2017) used by Zhang et al. (2021). They are both
pretrained on COCO (Lin et al., 2014) and Visual Genome (Krishna et al., 2016)
datasets for object detection. For the grid-based convolutional image features
and patch-based image features, we adpot the visual encoders from the CLIP
family, as all its variants are pretrained on the same visual data.

Visual Encoder | #Params |  | COCO
---
Karpathy Test
| NoCaps Val
---
Out-of-domain
C$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$ | C$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$
Region features |  |  |  |  |  |  |
BUTD-RN101 | 45M | 110.6 | 9.1 | 15.9 | 40.5 | 36.7 | 49.0
ResNeXt-152 | 60M | 115.9 | 7.1 | 12.9 | 45.1 | 30.5 | 41.1
Grid features |  |  |  |  |  |  |
RN50$\times$4 | 83M | 107.6 | 11.2 | 19.1 | 41.6 | 37.5 | 49.9
RN50$\times$16 | 160M | 111.6 | 9.0 | 15.8 | 47.5 | 33.1 | 45.2
RN50$\times$64 | 401M | 115.8 | 7.5 | 13.2 | 56.2 | 26.3 | 36.6
Patch features |  |  |  |  |  |  |
ViT-B/32 | 84M | 108.9 | 10.3 | 17.9 | 44.4 | 34.7 | 46.8
ViT-B/16 | 82M | 111.8 | 8.1 | 14.7 | 51.9 | 30.3 | 42.3
ViT-L/14 | 290M | 120.7 | 6.4 | 11.6 | 59.8 | 24.2 | 33.5

Table 2: Results of different types of visual encoders with the same BertLM
text decoder on the COCO Karpathy test set and NoCaps validation set (out-of-
domain).

###  5.2 Effects of Different Image Features

Recognizing visual objects correctly is crucial for avoiding object
hallucination. In Table 2, we compare the performance of different visual
encoders with the same text decoder on COCO (in-domain) and NoCaps (out-of-
domain) datasets.

Overall, patch-based visual encoders attain the best performance in terms of
object hallucination. Models with grid features hallucinate more frequently
when achieving comparable CIDEr scores to the other models. For example, on
COCO, RN50$\times$16 has similar CIDEr to ViT-B/16 but higher CHAIRs, which is
also observed between RN50$\times$64 and ResNeXt-152. We conjecture that the
inductive biases of the Convolutional Neural Network (CNN), such as locality
and translation invariance, weaken the connection of different characteristics
of a single object and thus lead to more hallucination. Oppositely, regional
or patch-level features are obtained by directly dividing images into
different parts and further encode them through positional embeddings. In
addition, we see that a smaller patch resolution helps to reduce object
hallucination without enlarging the model size.

For region-based visual encoders, although they achieve modest results on COCO
with relatively small model sizes, their performance of object hallucination
on out-of-domain images drops dramatically. One important reason is that the
output of such encoders only contains representations of detected visual
objects rather than the whole image, which may amplify detection errors as
there is much less context. Moreover, as the object detector is pretrained
separately from the whole model and fixed during finetuning, this gap could
also aggravate object hallucination on unseen images.

###  5.3 Effects of Different VLP Objectives

Based on the best performing ViT-L/14 baseline, we explore three commonly used
vision-language pre-training objectives and their variants that could possibly
affect object hallucination.

####  5.3.1 Pre-training Datasets

We explore two datasets for pre-training: 1) the VG Caption from the Visual
Genome (Krishna et al., 2016) dataset, which contains 10K images with half
overlapped with COCO; and 2) the more large-scale CC3M (Sharma et al., 2018)
dataset that contains 3 millions of image-text pairs.

VLP Objectives |  | COCO
---
Karpathy Test
| NoCaps Val
---
Out-of-domain
C$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$ | C$\uparrow$ | CHi$\downarrow$ | CHs$\downarrow$
(a) None | 120.7 | 6.4 | 11.6 | 59.8 | 24.2 | 33.5
Discriminative Objectives
CC3M |  |  |  |  |  |
(b) ITC | 120.5 | 6.5 | 11.7 | 59.9 | 24.4 | 33.8
(c) ITC${}_{\textit{Late}}$ | 121.2 | 6.2 | 11.3 | 60.5 | 23.8 | 32.9
(d) ITC${}_{\textit{Late}}$ \+ ITM | 121.0 | 6.3 | 11.5 | 60.2 | 23.9 | 33.1
(e) ITC${}_{\textit{Late}}$ \+ ITM${}_{\textit{Hard}}$ | 120.9 | 6.6 | 11.7 | 59.9 | 24.2 | 33.3
Generative Objectives
Visual Genome |  |  |  |  |  |
(f) LM | 120.3 | 5.5 | 9.8 | 62.8 | 9.0 | 13.9
(g) LM + ObjectMLM | 121.9 | 5.3 | 9.2 | 63.8 | 8.8 | 13.1
CC3M |  |  |  |  |  |
(h) LM | 122.3 | 6.0 | 10.9 | 92.1 | 8.3 | 14.5
(i) LM + ObjectMLM | 124.5 | 5.1 | 9.0 | 94.0 | 8.0 | 13.1
(c) + (i) | 125.1 | 4.9 | 8.8 | 94.5 | 7.9 | 12.5

Table 3: Comparison of the effects of different VLP objectives and their
combination on object hallucination.

####  5.3.2 Image-Text Contrastive (ITC) Loss

The cross-modal contrastive loss is shown to be fairly effective in
representation learning (Tian et al., 2020; Sigurdsson et al., 2020) and
vision-language pre-training (Radford et al., 2021; Li et al., 2021a, 2022).
It aligns the visual and textual representations into the same multimodal
feature space by shortening the distance between an image and a text if they
are paired, and enlarging if they are not.

Counter-intuitively, as shown in Table 3 (b), ITC does not have any
improvement on the faithfulness of generated captions. We speculate that it
only enhances model’s understanding on global-level representations rather
than object-level similarities. To verify, we test the ITC with a more fine-
grained token-level late interaction (ITC${}_{\textit{Late}}$) proposed by Yao
et al. (2021). As shown in Table 3 (c), ITC${}_{\textit{Late}}$ is more
effective than the original ITC and slightly reduce object hallucination. We
think this is benefit from the word-patch alignment ability enabled by
ITC${}_{\textit{Late}}$, as illustrated in Yao et al. (2021).

####  5.3.3 Image-Text Matching (ITM) Loss

ITM is a widely used objective in VLP (Li et al., 2020a; Chen et al., 2020;
Zhou et al., 2021). It is a binary classification task that aims to make the
model learn whether an image and a sentence are paired or not. Based on that,
ITM with hard negatives (ITM${}_{\textit{Hard}}$) is introduced to increase
the difficulty of the task, which is shown to be very effective (Kalantidis et
al., 2020; Robinson et al., 2021; Li et al., 2021b). We follow the ITM loss
proposed by Li et al. (2022), in which an in-batch negative example is sampled
either uniformly (normal) or from the similarity distribution of image-text
pairs computed by ITC (hard).

The results are exhibited in Table 3 (d) and (e). Similar to ITC, both ITM and
ITM${}_{\textit{Hard}}$ provide no enhancement on object hallucination.
Although the ITM${}_{\textit{Hard}}$ can be seen as an analogy to the object
hallucination problem (plausible but not correct) in a global and
discriminative way, it has negligible effect on the downstream generative
tasks.

Figure 3: Comparison of ground truth captions in COCO and Visual Genome
datasets for the same image. Figure 4: Comparison of generated captions with
or without the image-conditioned language modeling pretraining on the VG
dataset before finetuning.

####  5.3.4 Image-Conditioned Language Modeling

Various image-conditioned language modeling losses have been proposed in the
VLP research, in the form of masked language modeling (MLM) (Sun et al.,
2019b, a; Tan and Bansal, 2019; Su et al., 2020), text infilling (Dai et al.,
2022; Wang et al., 2022), prefix LM (Wang et al., 2021), and causal LM (Hu et
al., 2021; Li et al., 2022). This is one of the most crucial pre-training
losses to activate the cross-modal text generation ability for the model.

We first examine the causal LM loss, which is exactly the same objective as
the image captioning loss. Surprisingly, as shown in Table 3 (f), although
pretraining on VG does not improve previous standard metrics like CIDEr, it
helps to reduce object hallucination by a large margin when compared to (a).
There are two reasons behind this performance lift. First, as described in
Figure 3, for each image, VG contains more and shorter captions than COCO.
Each caption in VG only describes one specific aspect of the image, unlike the
global descriptions in COCO. Therefore, pre-training on VG and then finetuning
on COCO is a fine-to-coarse process to first accurately describe different
parts of an image and connect these clues together at a higher viewing point.
Second, due to the nature of the short length of VG captions, the model
becomes slightly more cautious. On average, after pre-training on VG, there
are 0.08 and 0.24 fewer objects generated in each caption on COCO and NoCaps,
respectively. Figure 4 illustrates VG’s effects on generated samples; the
model is more faithful but more likely to lack some details when it is not
confident.

For CC3M, we observe a leap in all metrics. It improves the general image
translation ability of the model, which can be seen as a large-scale data
augmentation. However, it is less effective than VG in terms of reducing
object hallucination.

Furthermore, inspired by the whole word masking (Cui et al., 2021) in MLM, we
propose a simple yet effective visual object MLM to mitigate object
hallucination. It replaces words appearing in the object categories (section
3.2) with the [MASK] token and train the model to recover them. This objective
enhances the model’s recognition ability when describing the spatial
relationship between objects, which is a common scenario that causes
hallucination frequently. Finally, we achieve the best performance when
combining ITC${}_{\textit{Late}}$, causal LM, and object MLM on our baselines,
which is comparable to state-of-the-art VLP models with regard to object
hallucination. More cases are included in Appendix B.

##  6 Conclusion

In this paper, we investigate the object hallucination problem in modern
vision-language pre-trained models. Particularly, we study this issue from
three aspects: 1) how much do contemporary VLP models hallucinate, and what
are the patterns of their hallucination; 2) the differences between commonly
used image encodings, including region-, grid-, and patch-based image features
in terms of object hallucination; and 3) the effects of various VLP objectives
and the way they affect object hallucination. We further propose a visual
object masked language modeling loss to mitigate object hallucination. We
believe our findings are beneficial for future work to build more reliable and
responsible cross-modal text generation systems.

## References

  * Agrawal et al. (2019) Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019.  nocaps: novel object captioning at scale.  _International Conference on Computer Vision_ , pages 8947–8956.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  _ArXiv_ , abs/2204.14198.
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation.  In _ECCV_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6077–6086.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _IEEvaluation@ACL_.
  * Biten et al. (2022) Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. 2022.  Let there be a clock on the beach: Reducing object hallucination in image captioning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 1381–1390.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Haochen Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  _ArXiv_ , abs/2102.02779.
  * Cui et al. (2021) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2021.  Pre-training with whole word masking for chinese bert.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3504–3514.
  * Dai et al. (2022) Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Enabling multimodal generation on CLIP via vision-language knowledge distillation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2383–2395, Dublin, Ireland. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Ding et al. (2021) Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021.  Cogview: Mastering text-to-image generation via transformers.  In _NeurIPS_.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ArXiv_ , abs/2010.11929.
  * Dziri et al. (2021) Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021.  Neural path hunter: Reducing hallucination in dialogue systems via path grounding.  _EMNLP_.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Hu et al. (2021) Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021.  Scaling up vision-language pre-training for image captioning.  _ArXiv_ , abs/2111.12233.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.  _arXiv preprint arXiv:2202.03629_.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_.
  * Kalantidis et al. (2020) Yannis Kalantidis, Mert Bulent Sariyildiz, No’e Pion, Philippe Weinzaepfel, and Diane Larlus. 2020.  Hard negative mixing for contrastive learning.  _ArXiv_ , abs/2010.01028.
  * Karpathy and Fei-Fei (2017) Andrej Karpathy and Li Fei-Fei. 2017.  Deep visual-semantic alignments for generating image descriptions.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:664–676.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _ICML_.
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2016.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123:32–73.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020.  The open images dataset v4.  _International Journal of Computer Vision_ , 128:1956–1981.
  * Li et al. (2020a) Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2020a.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _AAAI_.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  _ArXiv_ , abs/2201.12086.
  * Li et al. (2021a) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021a.  Align before fuse: Vision and language representation learning with momentum distillation.  In _NeurIPS_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _ArXiv_ , abs/1908.03557.
  * Li et al. (2021b) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021b.  Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.  _ArXiv_ , abs/2012.15409.
  * Li et al. (2020b) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020b.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _ECCV_.
  * Lin et al. (2021) Junyang Lin, Rui Men, An Yang, Chan Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, J. Zhang, Jianwei Zhang, Xu Zou, Zhikang Li, Xiao Qing Deng, Jie Liu, Jinbao Xue, Huiling Zhou, Jianxin Ma, Jin Yu, Yong Li, Wei Lin, Jingren Zhou, J ie Tang, and Hongxia Yang. 2021.  M6: A chinese multimodal pretrainer.  _ArXiv_ , abs/2103.00823.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _ECCV_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _ICLR_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_.
  * Lu et al. (2018) Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018.  Neural baby talk.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7219–7228.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919.
  * Nie et al. (2019) Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019.  A simple recipe towards reducing hallucination in neural surface realisation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2673–2679, Florence, Italy. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_.
  * Pavlopoulos et al. (2019) John Pavlopoulos, Vasiliki Kougia, and Ion Androutsopoulos. 2019.  A survey on biomedical image captioning.  In _Proceedings of the Second Workshop on Shortcomings in Vision and Language_ , pages 26–36, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:1137–1149.
  * Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1179–1195.
  * Robinson et al. (2021) Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021.  Contrastive learning with hard negative samples.  _ArXiv_ , abs/2010.04592.
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018.  Object hallucination in image captioning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4035–4045.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_.
  * Shen et al. (2021) Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2021.  How much can clip benefit vision-and-language tasks?  _ArXiv_ , abs/2107.06383.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  _EMNLP_.
  * Sigurdsson et al. (2020) Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh, Lucas Smaira, Mateusz Malinowski, João Carreira, Phil Blunsom, and Andrew Zisserman. 2020\.  Visual grounding in video for unsupervised word translation.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10847–10856.
  * Su et al. (2022) Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Read before generate! faithful long form question answering with machine reading.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 744–756.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  Vl-bert: Pre-training of generic visual-linguistic representations.  _ArXiv_ , abs/1908.08530.
  * Sun et al. (2019a) Chen Sun, Fabien Baradel, Kevin P. Murphy, and Cordelia Schmid. 2019a.  Contrastive bidirectional transformer for temporal representation learning.  _ArXiv_ , abs/1906.05743.
  * Sun et al. (2019b) Chen Sun, Austin Myers, Carl Vondrick, Kevin P. Murphy, and Cordelia Schmid. 2019b.  Videobert: A joint model for video and language representation learning.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7463–7472.
  * Tan and Bansal (2019) Hao Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _ArXiv_ , abs/1908.07490.
  * Tian et al. (2020) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020.  Contrastive multiview coding.  In _ECCV_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _ArXiv_ , abs/1706.03762.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4566–4575.
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  _ArXiv_ , abs/2202.03052.
  * Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021\.  Simvlm: Simple visual language model pretraining with weak supervision.  _ArXiv_ , abs/2108.10904.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On hallucination and predictive uncertainty in conditional language generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2734–2744.
  * Xie et al. (2017) Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017\.  Aggregated residual transformations for deep neural networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5987–5995.
  * Yao et al. (2021) Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021.  Filip: Fine-grained interactive language-image pre-training.  _ArXiv_ , abs/2111.07783.
  * Zhai et al. (2021) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2021.  Lit: Zero-shot transfer with locked-image text tuning.  _ArXiv_ , abs/2111.07991.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  Vinvl: Revisiting visual representations in vision-language models.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5575–5584.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and vqa.  _ArXiv_ , abs/1909.11059.
  * Zhou et al. (2021) Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021.  Uc2: Universal cross-lingual cross-modal vision-and-language pre-training.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4153–4163.

##  Appendix A Implementation Details

Our experiments are implemented in the PyTorch framework (Paszke et al.,
2019). For both pre-training and finetuning, we use 8 Nvidia V100 GPUs. For
the finetuning of various image encoders, we use a batch size of 512 and train
the models with the AdamW optimizer (Loshchilov and Hutter, 2019) for 10
epochs with a learning rate of $5e^{-5}$ and a weight decay of $1e^{-2}$. The
learning rate is decayed linearly after each epoch with a rate of 0.85. For
the pre-training of generative losses like causal LM and Object MLM, we keep
the same hyper-parameters. For ITC and ITM losses, we increase the batch size
to 1024.

##  Appendix B Additional Case Studies

Figure 5: More cases of generated captions from different models, where the
hallucinated objects are marked in red.