# Factorization theorems for exclusive heavy-quarkonium production

Geoffrey T. Bodwin  High Energy Physics Division, Argonne National Laboratory,
9700 South Cass Avenue, Argonne, Illinois 60439, USA  Xavier Garcia i Tormo
High Energy Physics Division, Argonne National Laboratory,
9700 South Cass Avenue, Argonne, Illinois 60439, USA  Jungil Lee  Department
of Physics, Korea University, Seoul 136-701, Korea

###### Abstract

We outline the proofs of the factorization theorems for exclusive two-body
charmonium production in $B$-meson decay and $e^{+}e^{-}$ annihilation to all
orders in perturbation theory in quantum chromodynamics. We find that
factorized expressions hold up to corrections of order $m_{c}/m_{b}$ in
$B$-meson decay and corrections of order $m_{c}^{2}/s$ in $e^{+}e^{-}$
annihilation, where $m_{c}$ is the charm-quark mass, $m_{b}$ is the bottom-
quark mass, and $\sqrt{s}$ is the $e^{+}e^{-}$ center-of-momentum energy.

###### pacs:

12.38.-t, 12.38.Bx, 14.40.Gx

††preprint: ANL-HEP-PR-08-28

The exclusive decays of $B$ mesons into a light meson plus a charmonium are
significant tools for understanding the weak interactions. For example, they
could provide new constraints on the Cabibbo-Kobayashi-Maskawa (CKM) matrix
and enhance our understanding of the origins of CP violation. However, the
effects of the strong interactions are significant in such processes and
present an obstacle to achieving reliable theoretical calculations of the
process rates. The exclusive production of double-charmonium in $e^{+}e^{-}$
annihilation provides an arena in which to explore the mechanisms of
charmonium production and the interplay between the perturbative and
nonperturbative regimes of quantum chromodynamics (QCD). However, measurements
of the cross sections for double-charmonium production by the Belle Abe:2004ww
and BABAR Aubert:2005tj  Collaborations have spurred a re-examination of the
theoretical predictions for those cross sections Bodwin:2007ga .

In theoretical computations of the rates for the exclusive decays of $B$
mesons into a light meson plus a charmonium and of the rates for the exclusive
production of double-charmonium in $e^{+}e^{-}$ annihilation, a crucial step
is the separation of the effects of the strong interactions into short-
distance, perturbatively calculable contributions and long-distance,
inherently nonperturbative contributions. Such separations are usually
embodied in factorization theorems. In the case of the exclusive decays of $B$
mesons into a light meson plus a charmonium, several factorization theorems
have been posited Beneke:2000ry ; Chay:2000xn ; Bobeth:2007sh . In the case of
the exclusive production of double-charmonium in $e^{+}e^{-}$ annihilation,
factorization conjectures have generally been formulated in terms of
nonrelativistic QCD (NRQCD) Bodwin:2007ga . In this letter, we outline the
proofs of the factorization theorems for these processes in QCD to all orders
in perturbation theory.

We wish to show that the amplitudes for these processes can be written in a
factorized form. The meaning of this statement, for the case of exclusive
decays of $B$ mesons into a light meson plus a charmonium, is that the decay
amplitude is decomposed into the sum of products of a $B$-meson-to-light-meson
form factor and an amplitude for a charm-quark-antiquark ($c\bar{c}$) pair to
be produced at short-distances in a color-singlet state and evolve into a
charmonium. The form factor contains a term that can be decomposed into a
convolution of a hard-scattering amplitude with $B$-meson and light-meson
light-cone amplitudes and a term that cannot be decomposed further. (These two
terms are analogous to the two terms in the factorization formula in Eq. (4)
of Ref. Beneke:2000ry  for the case of decays to two light mesons.) The
$c\bar{c}$-to-charmonium amplitudes can be further decomposed into a sum of
products of long-distance NRQCD matrix elements times short-distance
coefficients. We argue that this factorized form holds up to corrections of
relative order $m_{c}/\sqrt{s}$, where $s=m_{b}^{2}$. This result was
suggested previously in Ref. Beneke:2000ry . For the case of the exclusive
production of double-charmonium in $e^{+}e^{-}$ annihilation, factorization
means that the production amplitude decomposes into a sum of products of a
short-distance amplitude and two amplitudes for a $c\bar{c}$ pair to be
produced at short-distances in a color-singlet state and evolve into a
charmonium. Again, the $c\bar{c}$-to-charmonium amplitudes can be further
decomposed into a sum of products of long-distance NRQCD matrix elements times
short-distance coefficients. We argue that this factorized form holds up to
corrections of relative order $m_{c}^{2}/s$, where $\sqrt{s}$ is the center-
of-momentum (CM) energy of the $e^{+}e^{-}$ pair. Although our analyses are
for the specific cases of $B$ decay and $e^{+}e^{-}$ annihilation, the
techniques that we describe should apply to other exclusive quarkonium
production processes and may also shed light on factorization in inclusive
quarkonium production. We note that, because we consider exclusive two-body
decays, we avoid the issues raised in Ref. Nayak:2005rt  concerning light
particles co-moving with the charmonium and the issues raised in Ref.
Nayak:2007mb  concerning an additional heavy quark co-moving with the
charmonium.

We carry out our analyses in the $B$-meson rest frame and in the CM frame of
the $e^{+}e^{-}$ pair, choosing the three-momentum of the light meson or one
of the charmonia to be in the negative $3$ direction and choosing the three-
momentum of the other charmonium to be in the positive $3$ direction. For a
momentum $k$, we define light-cone momentum components $k^{\pm}=(k^{0}\pm
k^{3})/\\!\sqrt{2}$, $\bm{k}_{\perp}$. We model the $B$ meson as an on-shell
active bottom quark, which participates in the electroweak interaction, and an
on-shell spectator light antiquark, which does not participate in the
electroweak interaction. We take the quark and antiquark to be in a color-
singlet state. We take the bottom quark to have momentum $p_{b}$, with
$p_{b}^{0}=m_{b}$ and all other components of momentum zero. We take the
spectator quark to have momentum $p_{l}$, with
$p_{l\,\mu}\\!\sim\\!\Lambda_{\rm QCD}$, the QCD scale. We neglect
$\Lambda_{\rm QCD}$ and the difference between $m_{b}$ and the $B$-meson mass
in comparison with $m_{b}$. Similarly, we model the light meson as an on-shell
active light quark and an on-shell spectator light antiquark, with the quark
and antiquark in a color-singlet state. We take the active-quark momentum to
be $p_{k_{1}}\\!=\\!yp_{k}+q_{k}$ and the spectator-quark momentum to be
$p_{k_{2}}\\!\\!=\\!\\!(1\\!\\!-\\!\\!y)p_{k}-q_{k}$, where $p_{k}$ is the
light-meson momentum. $p_{k}^{-}\\!\sim\\!m_{b}$, $q_{k}^{-}=0$,
$p_{k\perp}=0$, $q_{k\perp}\sim\Lambda_{\rm QCD}$, and
$p_{k}^{+}\\!\sim\\!q_{k}^{+}\\!\sim\\!\Lambda_{\rm QCD}^{2}/m_{b}$. Finally,
we model a charmonium as an on-shell $c\bar{c}$ pair in a color-singlet state,
with the momentum of the $c$ ($\bar{c}$) equal to $p_{i}$ ($\bar{p}_{i}$). We
take $p_{i}=P_{i}/2+q_{i}$ and $\bar{p}_{i}=P_{i}/2-q_{i}$, where $P_{i}$ is
the charmonium momentum and $P_{i}\cdot q_{i}=0$. In the charmonium rest
frame, $q_{i}$ has only spatial components, whose magnitudes are of order
$m_{c}v$, where $v$ is the typical charm-quark velocity in the charmonium rest
frame ($v^{2}\approx 0.3$). In the $e^{+}e^{-}$ CM frame or $B$-meson rest
frame, $P_{1}^{+}\sim P_{2}^{-}\sim Q$, $P_{1\perp}=P_{2\perp}=0$, and
$P_{1}^{-}\sim P_{2}^{+}\sim m_{c}^{2}/Q$, where $Q=\sqrt{s}$ is the large
momentum scale. One can think of the on-shell amplitude that we use in our
model as the on-shell perturbative QCD amplitude that is matched to a soft-
collinear-effective-theory amplitude in the case of the light meson, a heavy-
quark-effective-theory amplitude in the case of the $B$ meson, and an NRQCD
amplitude in the case of the charmonia.

The next step is to identify the regions of loop momenta that give
contributions that are leading in powers of $Q$ when we dress the lowest-order
decay and production amplitudes in these models with additional gluons. These
leading momentum regions correspond to particular Feynman-diagram topologies
Sterman:1978bi ; Sterman:1978bj , which we describe below. The identification
of leading regions has been discussed in detail in Ref. Collins:1989gx . For a
loop momentum $k$, the leading regions are the hard region, in which
$k_{\mu}\sim Q$; the collinear-to-plus (minus) region, in which
$|k^{+}|\\!\gg\\!|k^{-}|$ ($|k^{-}|\\!\gg\\!|k^{+}|$),
$k^{+}k^{-}\\!\sim\\!\bm{k}_{\perp}^{2}\\!\ll\\!Q^{2}$, and the soft region,
in which $k^{+}\\!\sim\\!k^{-}\\!\sim\\!\\!|\bm{k}_{\perp}|\\!\ll\\!Q$. The
collinear regions correspond to the directions of the momenta of the final-
state light meson and charmonia. We note that, in the case of $B$-meson
decays, there is also a “semi-hard region” in which propagators are off shell
by an amount of order $m_{b}\Lambda_{\rm QCD}$. We treat this semi-hard region
as part of the hard region. The “Glauber” region is also leading in power
counting Bodwin:1981fv ; Bodwin:1984hc . In this region,
$|k^{+}|\\!\ll\\!|k_{\perp}|$, $|k^{-}|\\!\ll\\!|k_{\perp}|$,
$k_{\perp}^{2}\\!\ll\\!Q^{2}$. However, for the exclusive processes we are
considering here, the contours of integration of loop momenta are not pinched
in the Glauber region, and it is possible to deform them out of it on a
diagram-by-diagram basis. This is in contrast with the situation in, for
instance, the Drell-Yan process, for which such a diagram-by-diagram contour
deformation is not possible Bodwin:1984hc ; Collins:1985ue ; Collins:1989gx .
Therefore, we ignore the Glauber region in the remainder of our discussion.
Contributions from the hard (semi-hard) region involve propagator denominators
that are of order $s$ ($m_{b}\Lambda_{\rm QCD}$) and can be calculated in
perturbation theory. Integrations of gluon momenta over the soft region or a
collinear region that is associated with a massless particle encounter
singularities in propagator denominators that result in logarithmic
divergences. [The soft singularities are at $k_{\mu}=0$, and the collinear
singularities are at $k^{-}=0$, $\bm{k}_{\perp}=0$ ($k^{+}=0$,
$\bm{k}_{\perp}=0$) for the collinear-to-plus (to-minus) region.] Thus,
contributions from the soft region and the collinear regions that are
associated with a massless particle cannot be computed reliably in
perturbation theory. The essence of the proofs of the factorization theorems
that we consider here is to show that these contributions cancel or can be
absorbed into the $B$-meson-to-light-meson form factor or the nonperturbative
NRQCD matrix elements. When a gluon three-momentum becomes parallel to the
three-momentum of a charm quark or charm antiquark, the potential collinear
singularity is shielded by $m_{c}$. Hence, the contributions from such
collinear regions can be computed reliably in perturbation theory.
Nevertheless, it is important for the arguments that we will make below to
treat these regions separately from $H$. As we shall see, the contributions
from these regions cancel.

Figure 1: Leading regions for the decay of a $B$ meson to a light meson and a
charmonium.  Figure 2: Leading regions for double-charmonium production in
$e^{+}e^{-}$ annihilation. The wavy line represents the virtual photon.

Now let us specify the diagrammatic topologies that correspond to the leading
regions. We work in the Feynman gauge. There are two distinct topologies in
the case of $B$-meson decays: one in which the $B$-meson and light-meson
spectators participate in the hard interaction and another in which they do
not. These topologies are shown in Figs. 1(a) and 1(b), respectively. After
one has achieved factorization, the contribution from the first topology can
be decomposed into an expression that contains the convolution of a hard-
scattering amplitude with $B$-meson and light-meson light-cone amplitudes,
while the contribution from the second topology cannot be decomposed further.
The topology for the case of double-charmonium production is represented in
Fig. 2. The components of the topologies are as follows: a $B$ meson ($B$);
jet sub-diagrams for each of the collinear regions, corresponding to a
charmonium ($J_{1}$) and a light meson ($J_{2}$) in Fig. 1 and the two
charmonia ($J_{1}$ and $J_{2}$) in Fig. 2; a hard sub-diagram ($H$) that
includes the lowest-order annihilation or production process; and a soft sub-
diagram ($S$). In $H$, all propagator denominators are of order $s$ or
$m_{b}\Lambda_{\rm QCD}$. $J_{i}$ contains the active- and spectator-quark
lines for a given meson or charmonium, as well as gluons and loops involving
quarks and ghosts with momenta collinear to the meson or charmonium. $J_{i}$
attaches to $H$ through the active- and spectator-quark lines in the topology
of Fig. 1(a), through the active-quark lines in the topology of Fig. 1(b) and
through any number of gluons. $S$ includes gluons with soft momenta and loops
involving quarks and ghost with soft momenta. $S$ attaches to $J_{i}$ and to
the $B$-meson quark and antiquark lines through any number of soft gluon
lines. Recall that the $B$-meson spectator antiquark itself carries a soft
momentum. In Fig. 1(b), the light-antiquark-spectator line carries a momentum
with minus component of order $\Lambda_{\rm QCD}$. Such a momentum can arise
either through an endpoint contribution, in which
$p_{k_{2}}^{-}\sim\Lambda_{\rm QCD}$ Beneke:2000ry ; Chernyak:1990ag , or
through the soft-collinear-messenger mechanism Becher:2003qh , in which an
emitted gluon carries away most of $p_{k_{2}}$.

At this point we can outline the proof of factorization. Suppose that a gluon
from $J_{i}$ enters $H$. Then we can apply a collinear approximation to that
gluon Bodwin:1984hc ; Collins:1985ue ; Collins:1989gx . Specifically, we
replace $g_{\mu\nu}$ in the gluon-propagator numerator with
$k_{\mu}\bar{n}_{i\nu}/k\cdot\bar{n}_{i}$, where $k$ is the gluon momentum,
$\bar{n}_{1}$ ($\bar{n}_{2}$) is a unit light-like vector in the minus (plus)
direction. The index $\mu$ corresponds to the attachment of the gluon to $H$,
and the index $\nu$ corresponds to the attachment of the gluon to $J_{i}$. For
$k$ in the collinear region, the collinear approximation is valid at leading
order in $Q$. Having made the collinear approximation, one can use the factor
$k_{\mu}$ and the diagrammatic Ward identity (Feynman identity) for fermion
lines $k\cdot\gamma=[(q+k)\cdot\gamma+m]-[q\cdot\gamma+m]$ and its
generalizations to non-Abelian gauge theories to show that the gluons with
collinear momenta decouple from $H$ Bodwin:1984hc ; Collins:1985ue ;
Collins:1989gx . In general, the decoupled gluons attach to light-like eikonal
lines (path integrals of the gauge field) that connect to the point at which
an active parton enters $H$. However, in our case these eikonal lines cancel
because the light meson and the charmonia are color-singlet states. Now
suppose that a gluon from $S$ attaches to a jet sub-diagram at a line with
momentum $p$. We can apply a soft approximation to that gluon Collins:1981uk .
Specifically, we replace $g_{\mu\nu}$ in the gluon-propagator numerator with
$k_{\mu}p_{\nu}/k\cdot p$, where $k$ is the gluon momentum, the index $\mu$
corresponds to the attachment of the gluon to $J_{i}$ and the index $\nu$
corresponds to the attachment of the gluon to $S$. We would like to use the
factor $k_{\mu}$ in the soft approximation and the diagrammatic Ward
identities to decouple the soft gluons. However, in general, the momentum $p$
is different for every line to which the soft gluon attaches. That is, the
soft approximation, unlike the collinear approximation, is not independent of
the line to which the $\mu$ end of the gluon attaches. Nevertheless, we can
apply the same soft approximation to all lines in the collinear-to-light-meson
jet, since they are all proportional to the same light-like vector (up to
corrections of order $\Lambda_{\rm QCD}/m_{b}$). For the constituents of each
charmonium, we can also apply the same soft approximation up to corrections of
order $m_{c}/Q$. The reason for this is that, in going from the rest frame of
the charmonium to the $e^{+}e^{-}$ CM frame or $B$-meson rest frame, the
momenta of constituents of the charmonium undergo boosts that render all of
the momenta nearly parallel. When we explicitly consider those boosts, it
follows that, for the charmonium with momentum $P_{1}$ ($P_{2}$), the plus
(minus) components are boosted by a factor of order $Q/m_{c}$ and the minus
(plus) components are boosted by a factor of order $m_{c}/Q$, while the
transverse components are unchanged. Thus, all of the momenta of the
constituents of the charmonium with momentum $P_{1}$ ($P_{2}$) are dominated
by the plus (minus) component, up to corrections of order $m_{c}/Q$. This
result holds provided that all of the components are of approximately the same
size in the charmonium rest frame. This is the case for $P_{i}$, $q_{i}$, but
it is also true for other momenta that characterize the charmonium in its rest
frame, such as a typical potential-gluon momentum (which has spatial
components of order $m_{c}v$ and a temporal component of order $m_{c}v^{2}$)
or a typical rest-frame soft-gluon momentum (which has all components of order
$m_{c}v$). Therefore, we use a modified soft approximation for the
constituents of the charmonium in the jet $J_{i}$ in which we replace
$g_{\mu\nu}$ in the gluon-propagator numerator with $k_{\mu}n_{i\nu}/k\cdot
n_{i}$, where $n_{1}$ ($n_{2}$) is a vector with unit component in the plus
(minus) direction and all other components equal to zero. This modified soft
approximation differs from the standard soft approximation for each
constituent of a quarkonium by terms that are suppressed as $m_{c}/Q$. Hence,
it accounts for all soft (logarithmic) singularities, up to terms that are
suppressed as $m_{c}/Q$. We can use the modified soft approximation, plus the
diagrammatic Ward identities to decouple the gluons with soft momenta from the
charmonium jets, relying on the fact that the $c$ and $\bar{c}$ in each
charmonium are in a color-singlet state Collins:1981uk ; Collins:1985ue ;
Collins:1988ig . Once the gluons with momenta collinear to the jet have been
decoupled from $H$ and the gluons with soft momenta have been decoupled from
the charmonium jets, there remain gluons with momenta collinear to the jet
that begin and end within the jet. In the rest frame of the charmonium, these
gluons correspond to soft and potential gluons. The contributions from regions
of momentum of order $m_{c}v$ or less that arise from these gluons (including
infrared divergences) can be absorbed into nonperturbative NRQCD matrix
elements Bodwin:1994jh . The associated NRQCD short-distance coefficients then
contain only contributions involving momenta of order $m_{c}$. The NRQCD
short-distance coefficients couple to the production process only through the
active $c$ and $\bar{c}$ lines. Thus, we have arrived at the factorized form.

As we have mentioned, the modified soft approximation allows us to decouple
gluons with soft momentum from the charmonium jets, up to corrections that are
suppressed as $m_{c}/Q$. Thus, in general, one would expect corrections to the
factorized form to appear at order $m_{c}/Q$. However, in the case of double-
charmonium production in $e^{+}e^{-}$ annihilation, $S$ decouples from both
the collinear-to-plus charmonium jet and the collinear-to-minus charmonium
jet. Each decoupling holds up to corrections of order $m_{c}/Q$, and, so, the
overall decoupling holds up to corrections of order $m_{c}^{2}/s$. In
perturbation theory, the factorization-violating corrections may be enhanced
by logarithms of $s/m_{c}^{2}$. Furthermore, they are infrared divergent. In
reality, these infrared divergences are cut off by nonperturbative effects
associated with confinement. Our analysis does not determine the size of these
factorization-violating corrections: it shows only that they are proportional
to one or two powers of $m_{c}/Q$. The constant of proportionality might be
determined through experiment and/or lattice calculations.

At the lowest order in $\alpha_{s}$ and $v$, the decoupling of soft gluons
from each charmonium is exact. At the lowest order in $\alpha_{s}$, the
charmonium has only the $c$ and $\bar{c}$ as constituents. At the lowest order
in $v$, one sets $q_{i}=0$, and the $c$ and $\bar{c}$ momenta become equal.
Then, one can apply the same soft approximation for both the $c$ and $\bar{c}$
in the charmonium. Since, at the lowest order in $\alpha_{s}$, there is, at
most, one soft gluon, the soft decoupling in double-charmonium production is
exact if either charmonium is treated at the lowest order in $v$. An explicit
calculation of the one-loop corrections to $S$-wave charmonium production in
$B$-meson decays at the lowest order in $v$ Chay:2000xn  and an explicit
calculation of the one-loop corrections to $\sigma[e^{+}e^{-}\to
J/\psi+\chi_{cJ}]$, in which the $J/\psi$ is treated at lowest order in $v$
Zhang:2008gp , confirm our expectation that these corrections should be free
of infrared divergences. $P$-wave charmonium production in $B$-meson decays
was considered at the one-loop level in Ref. Song:2003yc  within the
factorization framework of Ref. Beneke:2000ry . When we use the light-cone
distribution amplitudes that are specified in Ref. Beneke:2000ry , we obtain
agreement with the calculations in Ref. Song:2003yc , which yield only
infrared divergences that are suppressed as $m_{c}^{2}/s$. As we have
discussed, we find more generally that the violations of factorization are
suppressed as $m_{c}/\sqrt{s}$. At higher orders in $v$, we expect the soft
cancellation for each charmonium to hold only up to corrections of order
$m_{c}/\sqrt{s}$, even in the case of $S$-wave charmonium production.
Similarly, in relative order $\alpha_{s}^{2}$ and beyond, we expect the soft
cancellation for each charmonium to hold only up to corrections of order
$m_{c}/\sqrt{s}$, even at the lowest order in $v$.

###### Acknowledgements.

We thank Kuang-Ta Chao, Jianwei Qiu, George Sterman, and Yu-Jie Zhang for
helpful discussions. The work of G.T.B. and X.G.T. was supported by the U.S.
Department of Energy, Division of High Energy Physics, under contract DE-
AC02-06CH11357. J.L. was supported by KOSEF under contract
R01-2008-000-10378-0.

## References

  * (1) K. Abe et al. [Belle Collaboration], Phys. Rev. D 70, 071102 (2004).
  * (2) B. Aubert et al. [BABAR Collaboration], Phys. Rev. D 72, 031101 (2005).
  * (3) For a discussion of the current status of the theoretical predictions that are based on the NRQCD factorization approach, see G. T. Bodwin, J. Lee, and C. Yu, Phys. Rev. D 77, 094018 (2008).
  * (4) M. Beneke et al., Nucl. Phys. B 591, 313 (2000).
  * (5) J. Chay and C. Kim, arXiv:hep-ph/0009244.
  * (6) C. Bobeth, B. Grinstein, and M. Savrov, Phys. Rev. D 77, 074007 (2008).
  * (7) G. C. Nayak, J. W. Qiu, and G. Sterman, Phys. Lett. B 613, 45 (2005); Phys. Rev. D 72, 114012 (2005).
  * (8) G. C. Nayak, J. W. Qiu, and G. Sterman, Phys. Rev. Lett. 99, 212001 (2007); Phys. Rev. D 77, 034022 (2008).
  * (9) G. Sterman, Phys. Rev. D 17, 2773 (1978).
  * (10) G. Sterman, Phys. Rev. D 17, 2789 (1978).
  * (11) J. C. Collins, D. E. Soper, and G. Sterman, Adv. Ser. Direct. High Energy Phys. 5, 1 (1988).
  * (12) G. T. Bodwin, S. J. Brodsky, and G. P. Lepage, Phys. Rev. Lett. 47, 1799 (1981).
  * (13) G. T. Bodwin, Phys. Rev. D 31, 2616 (1985) [Erratum-ibid. D 34, 3932 (1986)].
  * (14) J. C. Collins, D. E. Soper, and G. Sterman, Nucl. Phys. B 261, 104 (1985).
  * (15) V. L. Chernyak and I. R. Zhitnitsky, Nucl. Phys. B 345, 137 (1990).
  * (16) T. Becher, R. J. Hill, and M. Neubert, Phys. Rev. D 69, 054017 (2004).
  * (17) J. C. Collins and D. E. Soper, Nucl. Phys. B 193, 381 (1981) [Erratum-ibid. B 213, 545 (1983)].
  * (18) J. C. Collins, D. E. Soper, and G. Sterman, Nucl. Phys. B 308, 833 (1988).
  * (19) G. T. Bodwin, E. Braaten, and G. P. Lepage, Phys. Rev. D 51, 1125 (1995) [Erratum-ibid. D 55, 5853 (1997)].
  * (20) Y. J. Zhang, Y. Q. Ma, and K. T. Chao, arXiv:0802.3655 [hep-ph].
  * (21) Z. Z. Song et al., Phys. Rev. D 69, 054009 (2004).


# Understanding Information Storage and Transfer in Multi-modal Large Language
Models

Samyadeep Basu
University of Maryland
&Martin Grayson
Microsoft Research
Cecily Morrison
Microsoft Research
&Besmira Nushi
Microsoft Research
&Soheil Feizi
University of Maryland
&Daniela Massiceti
Microsoft Research Work done during a part-time internship at MSR;
Correspondence to<EMAIL_ADDRESS>
###### Abstract

Understanding the mechanisms of information storage and transfer in
Transformer-based models is important for driving model understanding
progress. Recent work has studied these mechanisms for Large Language Models
(LLMs), revealing insights on how information is stored in a model’s
parameters and how information flows to and from these parameters in response
to specific prompts. However, these studies have not yet been extended to
Multi-modal Large Language Models (MLLMs). Given their expanding capabilities
and real-world use, we start by studying one aspect of these models – how
MLLMs process information in a factual visual question answering task. We use
a constraint-based formulation which views a visual question as having a set
of visual or textual constraints that the model’s generated answer must
satisfy to be correct (e.g. What movie directed by _the director in this
photo_ has won a _Golden Globe_?). Under this setting, we contribute i) a
method that extends causal information tracing from pure language to the
multi-modal setting, and ii) _VQA-Constraints_ , a test-bed of 9.7K visual
questions annotated with constraints. We use these tools to study two open-
source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these
MLLMs rely on MLP and self-attention blocks in much earlier layers for
information storage, compared to LLMs whose mid-layer MLPs are more important.
We also show that a consistent small subset of visual tokens output by the
vision encoder are responsible for transferring information from the image to
these causal blocks. We validate these mechanisms by introducing MultEdit, a
model-editing algorithm that can correct errors and insert new long-tailed
information into MLLMs by targeting these causal blocks.

##  1 Introduction

Multi-modal Large Language Models (MLLMs) trained on both text and images are
rapidly moving from research into deployment and are being used by millions of
people. Yet, while there have been some advances in understanding how Large
Language Models (LLMs) work, much less has been done to understand MLLMs. This
paper begins to close this gap by studying how information is stored and
transferred in MLLMs. To do this through the lens of a factual Visual Question
Answering (VQA) task – a very common use case of MLLMs today [26, 35, 27].

MLLMs process factual information in two steps: information storage and
information transfer. Information storage refers to how facts from a pre-
training dataset are stored in a model’s parameters – its so-called
‘parametric’ memory. Information transfer describes how information from input
prompts are propagated through these storage locations to the model’s final
output. Understanding these mechanisms can have many benefits, including
ensuring models are factually grounded and informing better evaluation
protocols.

Figure 1:  MLLMs retrieve information from earlier internal layers compared to
their LLM counterparts. We find that very early MLP layers [1-4] have high
indirect estimation effects to outputs (i.e., they are causal) in LLaVa-7B,
whereas the middle MLP layers [4-7] are causal in LLaMA (Vicuna)-7B. For
LLaMA, a larger window size (e.g., 5) is also required to find causal sites,
compared to a window size of 1 for LLaVA-7B.

Extensive works have explored how LLMs store and transfer factual information
[12, 29, 24], however, this has not been studied for multi-modal inputs. For
example, it is suggested that auto-regressive Transformer-based LLMs store
factual information in their mid-layer MLP parameters [12, 24]. However, MLLMs
and LLMs are different: an MLLM involves an additional (continuous) image
input, alongside a (discrete) text prompt, and requires additional modules to
process it [26, 18, 19, 16, 7]. Typically, a vision encoder (e.g. CLIP) is
used to convert this image into either visual tokens via a projection layer
[18, 19, 11] or cross-attention layers [3, 7, 16, 36] which are then
integrated into the language encoder. These differences suggest that our
existing understanding of LLM information storage and transfer may not map
directly to MLLMs.

In this work, we use a factual VQA task to study the mechanisms of multi-modal
information storage and transfer. We use a constraint-based formulation which
views a visual question as having a set of either visual or textual
constraints (e.g. What movie directed by _this director_ has won a _Golden
Globe_?). The information retrieved by the model should satisfy these
constraints (e.g. _this director_ , _Golden Globe_) for the answer to be
factually correct. This formulation, therefore, offers a systematic way of
understanding a model’s behavior. Under this framework, we propose
MultiModalCausalTrace, which extends LLM causal tracing [12, 28, 33] to the
multi-modal setting, to understand information storage, as well as leverage
attention contribution methodologies [37, 8] to study information transfer in
MLLMs. We also introduce _VQA-Constraints_ , a new dataset of 9.7k factual
questions annotated with constraints, spanning natural images (from OK-VQA
[22], WikiMovies [37], and Known [12]). With these tools, we study how a
widely-used MLLM family processes multi-modal information, specifically LLaVa
[18, 19] and multi-modal Phi-2 [11]111Taken from the Bunny repository..

Our key findings are that MLLMs, in contrast to LLMs:  1) retrieve information
from earlier MLP layers (i.e. layers 1-4 vs layers 4-7 in a LLM) (see Fig. 1);
and  2) use less parametric memory (require a smaller window size to retrieve
this information),  when answering a multi-modal question. We also find that
information is transferred from a given image to these early MLP blocks
through a consistent subset of visual tokens (e.g. last $\sim$36 tokens from
LLaVa’s CLIP encoder), and that the self-attention blocks in the middle layers
are primarily responsible for shuttling this information to the last token.

Finally, we demonstrate that by editing these early causal MLPs, we can
correct errors and insert new factual information into an MLLM. Specifically,
we propose a new model-editing algorithm, MultEdit, which modifies the
projection matrix of the early causal MLPs with a closed-form update. We
empirically show MultEdit’s effectiveness on questions from _VQA-Constraints_
and Encyclopedic VQA [25].

In summary, our contributions are:

  1. 1.

A novel multi-modal causal tracing methodology that can be used to study
information storage from image and text inputs in MLLMs.

  2. 2.

A new dataset, _VQA-Constraints_ , of 9.7K factual visual questions about
natural images annotated with constraints to support future research in these
directions.

  3. 3.

A suite of novel insights on the mechanisms underlying multi-modal information
storage and retrieval in MLLMs.

  4. 4.

A model-editing method, MultEdit, which we demonstrate can precisely correct
erroneous information and insert new long-tailed information in an MLLM.

Figure 2:  We introduce MultiModalCausalTrace, a causal tracing method to
understand information storage in MLLMs. A clean model is corrupted by
replacing the question’s constraint with an incorrect one for the given image
(e.g. “This place” –> “Paris city" for an image of “Vinson Massif”). The
activations of windows of layers are then iteratively copied from the clean to
the corrupted model until the corrupted model restores its output probability
to match the clean model’s.

##  2 Related Works

Multimodal Large Language Models.. We consider a MLLM to be a model that takes
an image and text as input, and generates a text output [2]. Over the last
year, such models have made tremendous advances in tasks like VQA and image
captioning, including BLIP [15], BLIP-2 [16], Instruct-BLIP [7], LLaVA [18,
19], Flamingo [3] and multi-modal Phi-2 (from the Bunny repo) [11]. These
MLLMs can broadly be categorized into two families based on how their visual
information is integrated into the language model: (i) by embedding the vision
encoder’s output into each layer of the language model with a cross-attention
layer (e.g., Flamingo, BLIP) or, (ii) by mapping the vision encoder’s output
into “visual tokens” in the language model’s input space (i.e. alongside the
text tokens) via a projection layer (e.g., LLaVA, Bunny). Both families are
widely used, however, the projection layer family has recently shown stronger
performance on popular benchmark [18, 19, 11]. We, therefore, focus our study
of information storage and transfer on this model family.

Interpretability of MLLMs. A well-established arm of model interpretability
examines the relationship between a model’s performance and its internals. A
range of recent works have studied the internal mechanisms of information
storage [23, 34, 24] and transfer [9, 37] in LLMs. However, to the best of our
knowledge, only a few works [30, 31] have studied the interpretability of
MLLMs, with none specifically investigating the relationship between a model’s
outputs and its internal states. [30], for example, designs an interactive
interface to visualize the attention maps in an MLLM, while [31] explores the
shortcomings of the CLIP vision encoder in MLLMs. Neither consider the
influence of both vision _and_ text inputs on model internals or offer causal
insights, as our work does. Our model editing approach which targets the
projection layer MLLM family, is complemented by [6], who propose baselines
for inserting information into the cross-attention layer MLLM family.

##  3 A Constraint-Based Framework for Studying Information Storage and
Transfer in MLLMs

In this section, we describe the constraint-based formulation we use to study
information storage and transfer in MLLMs. Under this framing, we introduce
MultiModalCausalTrace, a novel causal information tracing technique which we
use to study multi-modal information storage. We also describe how we use
attention contributions [8] to study multi-modal information transfer.
Finally, we describe _VQA-Constraints_ , a new test-bed of visual questions
annotated with constraints.

Figure 3:  Information to answer a visual question with a single constraint is
mainly retrieved from early MLP and self-attention layers in MLLMs.
MultiModalCausalTrace obtains high indirect estimation effect values in
LLaVa’s early MLP and self-attention blocks corresponding to the visual
constraint, across all 3 datasets in _VQA-Constraints_. This suggests these
layers are causally important for information storage. The causal traces
emerge with a window size of 3 (see results with a window size of 1 in
Appendix C).

###  3.1 A Multi-modal Constraint-based Framework

Prior works have used a constraint satisfaction framework to study LLMs [37,
20, 14] as they offer a systematic way of studying model behavior. This
framing defines a constraint as a set of words in the question. The model must
retrieve information that satisfies this constraint from its parametric memory
in order to generate the correct answer. For example, for the question “What
city is the _Space Needle_ in?", the model must retrieve information relevant
to the constraint “Space Needle". In the multi-modal setting, we consider
these textual constraints as well as introduce visual constraints. We define a
visual constraint to be a set of words in the question which refers to an
entity in the image. The model must similarly retrieve information about this
entity to generate the correct answer. For example, given an image of
Christopher Nolan and the question “Name a movie directed by this director in
2006?”, the visual constraint is “this director” (and the text constraint is
“2006”). We refer to a question involving both a visual and text constraint as
a multi-constraint question, and ones with only a visual constraint as a
single-constraint question.

We use this framework to study the widely-used “projection layer” MLLMs
family. These models are composed of a visual encoder $f_{\theta}$, a large
language model $g_{\phi}$ and a projection head $p_{\gamma}$. The projection
head $p_{\gamma}$ is responsible for mapping the output of the visual encoder
into the input space of the language model, as so-called “visual tokens”.
Given an image-text pair denoted as (x, y), the language model processes them
as $g_{\phi}(p_{\gamma}(f_{\theta}(\textbf{x})),h(t(y)))$, where
$p_{\gamma}(f_{\theta}(\textbf{x}))=\\{\textbf{v}_{i}\\}_{i=1}^{N}$ is the set
of visual token embeddings and $t(y)=\\{t_{i}\\}_{i=1}^{M}$ is the tokenized
text inputs for the language model. These text tokens are processed by an
embedding layer $h$ to obtain text token embeddings as
$\\{e_{i}\\}_{i=1}^{M}\hskip 2.84544pt\in\mathbb{R}^{d}$. Because we are
interested in studying the outputs of specific layers in response to specific
tokens, we use $g_{\phi}(.)_{k,\ell}$ to denote the output layer embedding
corresponding to the kth token position and the layer $\ell$. We denote the
output of a MLP layer as $g_{\phi}(.)_{k,\ell_{mlp}}$ and the output of a
self-attention block as $g_{\phi}(.)_{k,\ell_{attn}}$.

###  3.2 MultiModalCausalTrace: Studying Information Storage in MLLMs

Causal tracing, derived from the causality literature [28], can be combined
with a constraint-based framework to gain causal insights on how information
propagates through a model with respect to specific constraint tokens. This
has been used to identify where information is stored in LLMs [23, 24] and
text-to-image generative models [4]. The central idea of causal tracing is to
corrupt a clean model by perturbing the input prompt. The activations of a
small subset of layers are then iteratively copied from the clean model to the
corrupted model, until the corrupted model restores its output probability to
match the clean model’s. In this way, we can identify which layers are used to
retrieve information relevant to the constraints in the prompt (i.e. causally
relate the input to the output).

In LLMs, the model is corrupted by adding a small amount of Gaussian noise to
the embeddings of the textual constraint tokens (usually less than 5) [23]. In
MLLMs, however, noise must be added to a much larger number of token
embeddings – those of the visual tokens (e.g., 576 in LLaVa and 729 in multi-
modal Phi-2) from the projection head, _and_ the textual tokens of the visual
constraint (e.g. “ _this director_ ”). Our experiments show that this large
noise injection makes it difficult to revert the MLLM to a clean state and
recover relevant causal traces (see Fig. 7).

We, therefore, introduce MultiModalCausalTrace (see Fig. 2) to address this.
Rather than adding Gaussian noise to the embeddings, we instead corrupt the
visual constraint token IDs by replacing them with token IDs from a separate
word or phrase, such that the visual information is ignored. We illustrate
this through an example. (1) Clean Model: Given an image x of the Space
Needle, and the question $y$, “Which city is this building located in?”, we
compute the probability of the model’s output $O$ (e.g., “Seattle”) as
$\mathcal{P}_{clean}(O)$. (2) Corrupted Model: We substitute the visual
constraint with an alternative such that the question does not require
information from the image to be answered (e.g., “this building” is replaced
with “Taj Mahal”). For a multi-constraint question which also has a textual
constraint, we can either add Gaussian noise to the textual constraint tokens’
embeddings (since there are only a few) or we can similarly replace the token
IDs. After all replacements, we ensure that the question still makes semantic
sense. We then measure the probability of original output $O$ as
$\mathcal{P}_{corr}(O)$. With the right corruption, $\mathcal{P}_{corr}(O)$ is
expected to be low. (3) Restored Model: We then iteratively copy layer
activations $g_{\phi}(.)_{k,\ell_{mlp}}$ and
$g_{\phi}(.)_{k,\ell_{attn}}\hskip 2.84544pt\forall k\in[1,M+N]$ from the
clean model to the corrupted model, for each layer in turn. For a given layer
$\ell$ and token position $k$, we denote the restored probability as
$\mathcal{P}_{restored}(O)_{k,\ell}$. After a layer is copied, we observe if
$\mathcal{P}_{restored}(O)_{k,\ell}$ is high – indicating that layer $\ell$
has a strong causal association to the output $O$. In some cases, no layers
have a causal association. We note that this copy operation can be performed
over a window of layers $\\{\ell_{i}\\}_{i=1}^{W}$ at a time, where $W$ is the
window size. A window size of 1 copies only one layer at a time. Similar to
[23], we track the indirect estimation effect for a layer $\ell$ as
$\mathcal{P}_{restored}(O)_{k,\ell}-\mathcal{P}_{corr}(O)$ and use it as a
metric to track causal states. Intuitively, this measures the difference in
the probability of $O$ under the corrupted model and when a layer $\ell$ is
restored to its original clean state. A high value indicates that the copied
layer/s can restore the model to its original clean state (i.e. the layer is
causal).

Figure 4:  Information to answer a visual question with a visual and textual
constraint is retrieved from early _and_ middle MLP and self-attention layers
in MLLMs. This suggests that meeting multiple constraint requires more
parametric memory compared to single constraints. We show that
MultiModalCausalTrace obtains high indirect estimation effect values in the
early and middle layers in LLaVa’s on the OK-VQA dataset in _VQA-Constraints_
(see multi-constraint results from the Movies dataset in Appendix F).

###  3.3 Studying Information Transfer in MLLMs with Attention Contributions

MultiModalCausalTrace enables us to identify the specific layers a model
retrieves information from in order to answering a visual question. A second
component of understanding how MLLMs process factual information is
understanding how input prompts are propagated through these storage locations
to the model’s final output. For this, we use attention contributions [8, 37]
which compute how much one set of input tokens influences a set of output
tokens during the self-attention operation. Specifically, we use this to track
(i) how information is transferred from the visual tokens to the causal
layers, and (ii) from these layers to the final token, where the final
probabilities are computed.

Defining Attention Contributions. The attention operation in a Transformer
[32] consists of the query, value, key and output weight matrices:
$W_{q}^{\ell},W_{v}^{\ell},W_{k}^{\ell},W_{o}^{\ell}$. Each of these are
divided into $H$ heads as
$W_{q}^{\ell,h},W_{v}^{\ell,h},W_{k}^{\ell,h}\in\mathbb{R}^{d\times
d_{h}},W_{o}^{\ell,h}\in\mathbb{R}^{d_{h}\times d}$, where $d$ is the
dimension of the internal token embeddings and $d_{h}$ is the dimensionality
of the token embedding for a particular attention head $h$. We define the
attention contribution from a token $j$ to token $i$ in layer $\ell$ as
follows:

$a_{ij}^{\ell}=\sum_{h=1}^{H}A_{ij}^{l,h}(g_{\phi}(.)_{j,\ell-1}W_{v}^{\ell,h})W_{o}^{\ell,h}$
(1)

where the attention matrix for layer $\ell$ and head $h$ is defined as:

$A^{\ell,h}=softmax(\frac{(g_{\phi}(.)_{1:M+N,\ell-1}W_{q}^{\ell,h})(g_{\phi}(.)_{1:M+N,\ell-1}W_{k}^{\ell,h})^{T}}{\sqrt{d/H}})$
(2)

where $g_{\phi}(.)_{1:(M+N),\ell-1}\in\mathbb{R}^{(M+N)\times d}$ and
$A^{\ell,h}\in\mathbb{R}^{(M+N)\times(M+N)}$. For understanding information
transfer from the visual tokens to the causal layers, we use $j\in[1,M]$ and
$i=c_{\text{constraint}}$, where $c_{\text{constraint}}$ corresponds to the
last token in the visual constraint. For understanding the information
transfer to the last token, we set $j=c_{\text{constraint}}$ and
$i=c_{\text{last}}$, where $c_{\text{constraint}}$ corresponds to visual
constraint’s last token and $c_{\text{last}}$ corresponds to the last token in
the question.

###  3.4 VQA-Constraints: A Constraint Annotated Test-Bed for VQA

Alongside the above tools, we also introduce a new test-bed called VQA-
Constraints to enable our analyses. The test-bed consists of $9.7K$ natural
images paired with factual questions, where each question is annotated with
visual and textual constraints (see Sec. 3.1). Specifically, we source image-
question pairs from the following datasets i) OK-VQA [21] which covers general
knowledge questions, ii) Movies [37] which includes questions about movie
directors and awards from Wikipedia, and iii) Known [23] which covers
questions about countries, famous people, and places. The questions from the
Movies and Known datasets are not originally multi-modal, so we modify them to
refer to images which we source from Bing. We provide more details on the
dataset construction and the constraint annotation in the Appendix B.

We leverage GPT-4 [27] to annotate the textual and visual constraints in the
visual questions in VQA-Constraints. We first manually annotate 100 examples
from each dataset with constraints. We provide this as context to GPT-4 and
prompt it (see Appendix B) to annotate the constraints in new questions from
Multimodal Known and OK-VQA. Due to the templated prompts in Multimodal Movies
(e.g., “Name a movie directed by this director? ”), the constraints are
constant (“this director”) across all examples and does not require external
annotations.

##  4 Key Findings in how MLLMs Store and Transfer Information

Using the tools presented in Sec. 3.1, we present our key findings in how
MLLMs retrieve information from internal layers and how this information is
transferred across the model.

###  4.1 Finding 1: Early MLPs and self-attention layers are causal

We find that information required to answer a visual question is mainly
retrieved from the early-layer MLP and self-attention blocks of a MLLM. This
is confirmed by the high indirect estimation effects which
MultiModalCausalTrace assigns to the early layers in both LLaVa (see Fig. 3,
Fig. 8) and multi-modal Phi-2 (see Appendix E) across the three datasets in
VQA-Constraints.

This contrasts earlier results for LLMs which have been shown to retrieve
information from mid-layer MLPs to answer factual questions [23, 24]. To
obtain a fairer comparison, we apply MultiModalCausalTrace to LLaMA and LLaVa
which use the same language backbone. We run both on the same set of questions
– LLaMA on the Known dataset [23], and LLaVa on our multi-modal version of
Known which modifies the questions to refer to an image (see Sec. 3.4). In
Fig. 1, we show that the MLPs in the first 4 layers are causal for LLaVa while
the MLPs in layers 4-7 are causal for LLaMa. We also find that causal traces
can be extracted from LLaVA with a minimum window size of 1, while LLaMa
requires a minimum window size of 5 to obtain any significant causal traces.

Although we find a smaller window size of 1 to provide relevant causal traces
(see Fig. 8), we find that a slightly larger window size of 3 results in
consistent causal traces across all the questions. In Fig. 3, we report the
results using a window size of 3, where we find that the early MLPs as well as
self-attention layers are used to retrieve relevant knowledge to answer a
visual question. We note that this observation is consistent for all the
datasets in VQA-Constraints.

For multi-constraint questions that consist of a visual and textual
constraint, information corresponding to the textual constraint is retrieved
from a broader set of layers – _both_ the early and mid-layer MLP and self-
attention blocks (see Fig. 4). We also find that a larger window size (at
least of 6) is required to obtain any causal traces. This suggests that more
parametric memory is required to meet both a visual and textual constraint in
a given question.

Figure 5:  The late visual tokens are primarily responsible for transferring
information from the image to the early causal layers, via the first self-
attention layer. We visualize attention contributions (see Eq.(1)) from the
visual tokens to the visual constraint token averaged across the three
datasets in VQA-Constraints.

###  4.2 Finding 2: Only a subset of visual tokens are involved in
transferring information from the image to the early causal MLP layers.

We also find that the late visual tokens transfer information from the image
to the early causal MLP layers (where information is mainly stored) via the
first self-attention layer. We show this in Fig. 5 by visualizing the
attention values in the early self-attention layers between each visual tokens
and the final token in the visual constraint using the method described in
Sec. 3.3. We see that the values are the highest in LLaVa’s first self-
attention layer (which occurs just before the first causal MLP layer),
specifically for the last subset of visual tokens (indexes 540-576 out of 576
in total). We hypothesize that these tokens may be summarizing image
information that is relevant to the given question before it is transferred
and then stored in the MLPs, however we leave this to future study. We note
that this pattern holds across all three datasets from VQA-Constraints.

###  4.3 Finding 3: Mid-layer self-attention layers are involved in
transferring information from the early causal layers to the question’s final
token

Rather counter-intuitively, we find that even though information is stored in
the early layers, the self-attention blocks in the middle layers (rather than
layers immediately after) are responsible for propagating this information to
the question’s final token. The model’s answer is sampled at this point, hence
the information present here likely influences the ultimate generation. We see
this in Fig. 15 and Fig. 16, which plots the attention contribution values
between the last visual constraint token and the last token in the question
across all the layers, using the methodology in Sec. 3.3. For LLaVa, the self-
attention blocks in layers 16-17 are most active. This behaviour is similar in
LLMs which also use mid-layer self-attention blocks to transfer information
from (mid-layer) stored locations to the last token position.

###  4.4 Finding 4: Mid-layer self-attention contributions can be used to
predict whether a MLLM will generate a correct answer, but model confidence is
a more reliable predictor

When a MLLM generates a correct answer, we observe that the self-attention
contributions in its middle layers are higher to when it generates an
incorrect answer (see Fig. 15). For LLaVa, this is specifically the attention
contributions between the last constraint token and the last prompt token in
the 16th and 17th layer. This holds potential to detect when a model will
answer correctly without running a full inference pass, therefore enabling
“early” failure mode detection. To investigate this, we compute the scalar
average of the attention contributions from these two layers as
$\frac{a_{c_{\text{last}},c_{\text{constraint}}}^{16}+a_{c_{\text{last}},c_{\text{constraint}}}^{17}}{2}$
and find that it can classify a correctly generated answer with an AUROC of
0.63 on the Known dataset in _VQA-Constraints_ (see Fig. 16). We find,
however, that the model’s confidence – the probability of the generated output
token at the final layer – is a slightly stronger predictor, with an AUROC of
0.76 (see Fig. 17). This parallel’s previous work in LLMs [37] which used the
attention contributions from _all_ the LLM’s layers (via a linear model) to
predict the generated answer’s correctness. In comparison, our results suggest
that a subset of MLLM’s middle layers alone can be leveraged as a coarse
“early” failure mode detector.

Takeaway. MLLMs behave differently in terms of information retrieval from
their parametric memory however quite similarly to LLMs in terms of
information transfer to the final token.

Figure 6:  MultEdit can correct error cases (left) and insert long-tailed
factual knowledge (right) by editing the early causal MLP layers in an MLLM.
We use the average probability of the correct token $O^{*}$ as the metric for
Editing Efficacy and Generalization, and VQA-Accuracy for Specificity (higher
the better for all).

##  5 Correcting and Inserting Long-Tailed Information in MLLMs

Previous works have shown that counterfactual information can be inserted into
LLMs by editing their causal layers [23, 24]. In this section, we verify if a
similar approach can be used to edit a MLLM. Specifically, we introduce
MultEdit, which applies a closed-form update to the early causal MLP layers we
identified in Sec. 4. We show that our approach can effectively both (i) fix
erroneous answers, and (ii) insert new long-tailed information in LLaVa in a
VQA task.

###  5.1 MultEdit

Given an image-question $(\textbf{x},y)$, we denote a MLLM’s generated answer
as $O$. MultEdit updates a few parameters in the model such that it generates
a new answer $O^{*}$. Similar to [23, 13], we update the $W_{proj}^{\ell}$
matrix at specific causal MLP layers such that the probability
$\mathcal{P}(O^{*})$ increases. Specifically, we view $W_{proj}^{\ell}$ as a
linear-associated memory (where the matrix’s input are treated as keys and its
output as values) and uses a closed-form update to map the original keys to
new (correct) values. Below, we outline the process for acquiring both the
keys and values.

Obtaining keys. Let $\\{v_{i}\\}_{i=1}^{N}$ be the visual token embeddings and
$\\{e_{i}\\}_{i=1}^{M}$ the text token embeddings. Given a causal layer $\ell$
and the last token of a constraint $c$, we refer to the input of the layer’s
$W^{\ell}_{proj}$ matrix as its keys. Specifically, we define the key
$k_{c,\ell}$ to be the input embedding to the $W_{proj}^{\ell}$ matrix
corresponding to the last token of the constraint. This can be obtained with a
simple forward pass with the visual and text token embeddings.

Obtaining values. Given a causal layer $\ell$, we refer to the output of the
layer’s $W^{\ell}_{proj}$ matrix as its values. Specifically, we define
$z_{c,\ell}$ as the output embedding corresponding to the last token of the
constraint. We optimize $z_{c,\ell}$ such that the probability of the correct
answer $\mathcal{P}(O^{*})$ increases as:

$z_{c,\ell}^{*}=\arg\min_{z_{c,\ell}}\mathcal{L}(z_{c,\ell})$ (3)

where $\mathcal{L}(z_{c,\ell})$ is the standard next-token prediction loss
used to train LLMs:

$\mathcal{L}(z_{c,\ell})=-\log\mathcal{P}(O^{*}|v_{1}...v_{N}e_{1}....e_{M})$
(4)

MultEdit modifies the $W_{proj}^{\ell}$ matrix such that the old keys
$k_{c,\ell}$ are mapped to the new optimized values $z_{c,\ell}^{*}$ which
increase $\mathcal{P}(O^{*})$. We define MultEdit’s editing objective as:

$W^{\ell*}_{proj}=\arg\min_{W^{\ell}_{proj}}{\|W^{\ell}_{proj}k_{c,\ell}-z_{c,\ell}^{*}\|_{2}^{2}}+{\lambda\|W^{\ell}_{proj}-W_{proj}^{\ell^{\prime}}\|_{2}^{2}}$
(5)

The second regularization term ensures that $W^{\ell^{\prime}}_{proj}$, the
weights before the edit, do not deviate too much from $W^{\ell}_{proj}$. This
helps to preserve performance on unrelated VQA pairs.

###  5.2 Experimental details

We experimentally validate MultEdit in two real-world model-editing
applications:

Fixing incorrect answers to common questions.  We test MultEdit on a set of
$\sim$450 visual questions which LLaVa answers incorrectly (detected using
incorrect VQA accuracy) from the multi-modal Known dataset in _VQA-
Constraints_. These are generally questions about well-known places, persons
and brands and companies.

Inserting long-tailed VQA knowledge. We test MultEdit on a set of visual
questions from the Encyclopedia-VQA dataset [25]. These query fine-grained
knowledge about rare landmarks around the world, which MLLMs have been shown
to struggle on [25].

For both settings, we compare MultEdit to the fine-tuning baselines:(i) fine-
tuning from [23] which fine-tunes all the layers using the language modeling
objective, and (ii) fine-tuning with constraints from [38] which fine-tunes
the layers in a language model with a constraint on the weights to ensure
local loss continuity.

We measure the success of each edit operation using the following metrics: (i)
Editing Efficacy, which uses $\mathcal{P}(O^{*})$ to measure the edited
model’s ability to generate the correct answer for image-question
($\textbf{x},y$). (ii) Generalization, which uses $\mathcal{P}(O^{*})$ to
measure the edited model’s ability to generate the correct answer for the
question $y$ paraphrased using a language model (see Appendix for details),
and (iii) Specificity, which uses VQA accuracy [1] to measure the edited
model’s performance on unrelated VQA questions. We consider unrelated
questions to be those from the OK-VQA and Movies datasets in VQA-Constraints
(see further details in Appendix G).

###  5.3 Results

Overall, our results show that updating the projection matrix using MultEdit
at just a single early (causal) MLP layer can be a very effective approach for
both correcting incorrect answers and inserting new knowledge in MLLMs.

Fixing incorrect answers. In Fig. 6 (left), we show that MultEdit is able to
successfully fix the generations for all questions with wrong answers. In
editing efficacy, the average probability of the correct answer improves from
0.07 to 0.82 after the edit. We also observe strong generalization, with a
probability of 0.76 for the correct answer even when the question is
paraphrased. Although we see a small drop of $1.5\%$ accuracy on unrelated
image-question pairs, we note that MultEdit outperforms fine-tuning with and
without constraints on all the metrics.

Inserting long-tailed information. In Fig. 6 (right), we show that MultEdit is
able to reliably insert new long-tailed knowledge in the model with an editing
efficacy of 0.83. We see similar strong generalization when paraphrasing
questions with an efficacy of 0.79. Similar to above, MultEdit incurs a small
drop of 1.4$\%$ on unrelated questions but is less affected than other
methods.

In Fig. 14, we also provide ablations showing that editing the early causal
MLP layers leads to better editing efficacies than editing the middle or the
later MLP layers.

##  6 Conclusion

Our paper takes a closer look at how MLLMs process multi-modal information. We
contribute a novel multi-modal causing tracing methodology and test-bed, _VQA-
Constraints_ , as well as a range of novel insights on how MLLMs retrieve and
transfer information. We also introduce a novel model-editing algorithm,
MultEdit, which can effectively fix errors or introduce long-tailed knowledge
in MLLMs using a simple closed-form update which targets the early causal
MLPs. Overall, this work deepens our scientific understanding of recent MLLM
architectures, and enables future work in this direction.

## Acknowledgements

This project was supported in part through a part-time internship at Microsoft
Research and in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP
award N00014-22-1-2271, ARO’s Early Career Program Award 310902-00001,
HR00112090132 (DARPA/ RED), HR001119S0026 (DARPA/ GARD), Army Grant No.
W911NF2120076, the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute
for Trustworthy AI in Law and Society, TRAILS), an Amazon Research Award and
an award from Capital One. We thank Hamid Palangi for proofreading the draft
and providing feedback.

## References

  * Agrawal et al. [2016] A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Batra, and D. Parikh.  VQA: Visual Question Answering, 2016.
  * Agrawal et al. [2023] A. Agrawal, I. Kajić, E. Bugliarello, E. Davoodi, A. Gergely, P. Blunsom, and A. Nematzadeh.  Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization, 2023.
  * Alayrac et al. [2022] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, and M. Reynolds.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 23716–23736, 2022.
  * Basu et al. [2023] S. Basu, N. Zhao, V. Morariu, S. Feizi, and V. Manjunatha.  Localizing and Editing Knowledge in Text-to-Image Generative Models, 2023.
  * Basu et al. [2024] S. Basu, K. Rezaei, P. Kattakinda, R. Rossi, C. Zhao, V. Morariu, V. Manjunatha, and S. Feizi.  On Mechanistic Knowledge Localization in Text-to-Image Generative Models, 2024.
  * Cheng et al. [2024] S. Cheng, B. Tian, Q. Liu, X. Chen, Y. Wang, H. Chen, and N. Zhang.  Can We Edit Multimodal Large Language Models?, 2024.
  * Dai et al. [2024] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi.  InstructBLIP: Towards general-purpose vision-language models with instruction tuning.  In _Advances in Neural Information Processing Systems_ , volume 36, 2024.
  * Elhage et al. [2021] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 2021.
  * Geva et al. [2023] M. Geva, J. Bastings, K. Filippova, and A. Globerson.  Dissecting Recall of Factual Associations in Auto-Regressive Language Models, 2023.
  * Hase et al. [2023] P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun.  Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models, 2023.
  * He et al. [2024] M. He, Y. Liu, B. Wu, J. Yuan, Y. Wang, T. Huang, and B. Zhao.  Efficient Multimodal Learning from Data-centric Perspective.  _arXiv preprint arXiv:2402.11530_ , 2 2024.
  * Kevin Meng and David Bau and Alex Andonian and Yonatan Belinkov [2022] Kevin Meng and David Bau and Alex Andonian and Yonatan Belinkov.  Locating and editing factual associations in GPT.  _Advances in Neural Information Processing Systems_ , 35:17359–17372, 2022.
  * Kohonen [1972] T. Kohonen.  Correlation Matrix Memories.  _IEEE Transactions on Computers_ , C-21:353–359, 1972.
  * Lawless et al. [2023] C. Lawless, J. Schoeffer, L. Le, K. Rowan, S. Sen, C. S. Hill, J. Suh, and B. Sarrafzadeh.  "I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming.  _arXiv preprint arXiv:2312.06908_ , 2023.
  * Li et al. [2022] J. Li, D. Li, C. Xiong, and S. Hoi.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, 2022.
  * Li et al. [2023a] J. Li, D. Li, S. Savarese, and S. Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International Conference on Machine Learning_ , pages 19730–19742. PMLR, 2023a.  ISBN 2640-3498.
  * Li et al. [2023b] Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. T. Lee.  Textbooks Are All You Need II: phi-1.5 technical report, 2023b.
  * Liu et al. [2023] H. Liu, C. Li, Y. Li, and Y. J. Lee.  Improved Baselines with Visual Instruction Tuning.  _arXiv preprint arXiv:2310.03744_ , 10 2023.
  * Liu et al. [2024] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Lu et al. [2023] A. Lu, H. Zhang, Y. Zhang, X. Wang, and D. Yang.  Bounding the capabilities of large language models in open text generation with prompt constraints.  _arXiv preprint arXiv:2302.09185_ , 2023.
  * Marino et al. [2019a] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  _CoRR_ , abs/1906.00067, 2019a.  URL http://arxiv.org/abs/1906.00067.
  * Marino et al. [2019b] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3195–3204, 2019b.
  * Meng et al. [2023a] K. Meng, D. Bau, A. Andonian, and Y. Belinkov.  Locating and editing factual associations in gpt, 2023a.
  * Meng et al. [2023b] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau.  Mass-Editing Memory in a Transformer, 2023b.
  * Mensink et al. [2023] T. Mensink, J. Uijlings, L. Castrejon, A. Goel, F. Cadar, H. Zhou, F. Sha, A. Araujo, and V. Ferrari.  Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories.  In _VQA_ , pages 3113–3124, 2023.
  * OpenAI [2023] OpenAI.  Gpt-4v(ision) system card, 9 2023.
  * OpenAI et al. [2024] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph.  GPT-4 Technical Report, 2024.
  * Pearl [2001] J. Pearl.  Direct and Indirect Effects.  In _Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence_ , UAI’01, page 411–420, 2001.  ISBN 1558608001.
  * Sharma et al. [2024] A. S. Sharma, D. Atkinson, and D. Bau.  Locating and Editing Factual Associations in Mamba.  _arXiv preprint arXiv:2404.03646_ , 2024.
  * Stan et al. [2024] G. B. M. Stan, R. Y. Rohekar, Y. Gurwicz, M. L. Olson, A. Bhiwandiwalla, E. Aflalo, C. Wu, N. Duan, S.-Y. Tseng, and V. Lal.  LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models, 2024.
  * Tong et al. [2024] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie.  Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, 2024.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention Is All You Need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Vig et al. [2020] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. M. Shieber.  Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias.  _CoRR_ , abs/2004.12265, 2020.  URL https://arxiv.org/abs/2004.12265.
  * Wang et al. [2022] K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt.  Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, 2022.
  * Yang et al. [2023] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang.  The dawn of LMMs: Preliminary explorations with GPT-4V (ision).  _arXiv preprint arXiv:2309.17421_ , 9(1):1, 2023.
  * Yu et al. [2022] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu.  CoCa: Contrastive Captioners are Image-Text Foundation Models.  _arXiv preprint arXiv:2205.01917_ , 5 2022.
  * Yuksekgonul et al. [2023] M. Yuksekgonul, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi.  Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models.  In _International Conference on Learning Representations_ , 9 2023.
  * Zhu et al. [2020] C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, D. Li, F. X. Yu, and S. Kumar.  Modifying memories in transformer models.  _CoRR_ , abs/2012.00363, 2020.  URL https://arxiv.org/abs/2012.00363.

##  Appendix A Limitations and Ethical Considerations

The findings of our work are limited to factual questions on natural images
with short-form answers. Future work should investigate other types of
questions (e.g. subjective), image domains (e.g. charts), and longer-form
answers. Our work also does not map visual tokens to specific concepts in the
input image – another important research direction. Our editing method,
MultEdit, while effective for correcting and adding new information, could
also be used to add false or harmful information into a MLLM. Work is needed
on how to reliably detect misinformation in multi-modal settings.

##  Appendix B VQA-Constraints Details

In Sec. 3.1, we introduce the constraint framework for the task of factual
VQA. In particular, there are two types of constraints: (i) Visual Constraint:
A set of words in a question which refer to an entity in an image; (ii) Text
Constraint: A set of words in a question which together with the visual
constraint are responsible for answering a question. In our VQA-Constraints
dataset there are two types of questions: (i) Single Constraint Questions:
These questions comprise of only the visual constraint; (ii) Multi-Constraint
Questions: These questions comprise of both the visual as well as the text
constraint. In our paper, we primarily focus on the Single Constraint
questions, therefore a majority of questions in our test bed of VQA-
Constraints consist of only the visual constraints. Our VQA dataset VQA-
Constraints which is annotated with constraints comprise of the following
three parts:

OK-VQA: We annotate the questions in the test-set of OK-VQA with the
constraints. In total, this part consists of 5k questions. We use the same
images as those used in the original OK-VQA test-set.

Multimodal Movies: We use the text-only WikiMovies dataset from [37] and
download images using the Bing API using the name of the directors. Our team
filters and validates that the downloaded images are of the director itself
and there’s no noise in the downloaded images. We use a set of 1.5k questions
in the final dataset.

Multimodal Known: For understanding knowledge storage in language models, [23]
use a probe dataset (known.json) consisting of 1.2k questions. We first
replace the subject in the question with a constraint and then download images
from Bing API. Our team validates that the downloaded images are correct and
uses it to create the Multimodal Known dataset.

Multi-Constraint Questions: The multi-constraint questions in VQA-Constraints
are created from OK-VQA and Multimodal Movies. In particular, there are 150
multi-constraint questions for OK-VQA and 500 multi-constraint questions from
Multimodal Movies. Given that the primary focus of our paper is on Single
Constraint questions, this set of Multi-Constraint questions is relatively
small.

In total, VQA-Constraints consist of 9.7K VQA questions with Single
Constraints and 650 Multi-Constraint VQA questions which we use for
interpretability analysis.

| Description of Questions in VQA-Constraints |  |  |
---|---|---|---|---
Dataset | Description | Example 1 | Example 2 | Example 3 |  |  |  |
OK-VQA | Single Constraint | What sport can you use this for? | Why might someone go to this place? | What flavor is this pastry? |  |  |  |
OK-VQA | Single Constraint | Which airlines is this insignia? | What fruits come from these trees? | What is this animal mostly known for? |  |  |  |
Multimodal Known | Single Constraint |  This person is a citizen of? |  This newspaper is written in? | The capital of this city is in? |  |  |  |
Multimodal Known | Single Constraint |  This venue is owned by? |  This building is located in which city? |  This show debuted on? |  |  |  |
Multimodal Movies | Single Constraint | Name a movie directed by this director? | Name a movie directed by this director? | Name a movie directed by this director? |  |  |  |
Multimodal Movies | Single Constraint | Name a movie directed by this director? | Name a movie directed by this director? | Name a movie directed by this director? |  |  |  |
OK-VQA | Multi-Constraint | Which is the most famous type of this food in India | How does the population of this city compare to Mumbai?" | What is the relationship between this person and Venus Williams? |  |  |  |
Multimodal Movies | Multi-Constraint | Name a movie directed by this director in year 2006? | Name a movie directed by this director in year 2010? | Name a movie directed by this director in year 1994? |  |  |  |

Table 1:  Description of the different VQA questions in VQA-Constraints. The
constraints are marked in italic in the columns.

Annotating and Validating the Visual Constraints. To automatically annotate
the constraints in the visual questions from VQA-Constraints, we use a strong
language model such as GPT-4. In particular, we annotate only the Multimodal
Known and OK-VQA sub-parts of the VQA-Constraints dataset. First from each
dataset, we annotate 100 examples which we use as in-context examples to the
language model to annotate new questions. Given these annotations, our team
then verified if the annotation is correct and if incorrect, modified the
constraint annotation to make it correct. In total, the visual constraints are
annotated for 9.7k VQA questions.

##  Appendix C Standard Causal Tracing Does Not Recover Causal States

In Fig. 7, we find that the standard causal tracing approach, which adds
Gaussian noise to the span of visual tokens and the constraint in the text
does not recover any causal states, even with a large window size of 10. This
is true for recovering both the MLP as well as the self-attention causal
layers.

Figure 7: Standard Causal Tracing: We find that using the standard causal
tracing procedure from [23] is not able to recover causal states. Averaged
across all the datapoints in Multimodal Known and Multimodal Movies from VQA-
Constraints. A window size of 10 is used.

##  Appendix D MultiModalCausalTrace \- Qualitative Plots

In this section, we present various qualitative plots corresponding to
MultiModalCausalTrace showing that our method is able to recover causal states
in both the MLP as well as self-attention layers very early on in the model.
We note that even using a small window size of 1 is sufficient to recover
causal states. We highlight that earlier works [23, 24] show that for language
models, a larger window size of 10 is required and also the middle layers are
used to recover the relevant knowledge. However, we find that in the presence
of visual prompts, relevant knowledge is retrieved from the early layers.

Figure 8: MultiModalCausalTrace with Window Size 1: Early MLP layers are
causal. Qualitative tracing results for examples with single constraint
questions (containing only a visual constraint).  Figure 9:
MultiModalCausalTrace with Window Size 1: Early self-attention layers are
weakly causal compared to causal MLPs. Qualitative tracing results for
examples with single constraint questions (containing only a visual
constraint).

##  Appendix E Multimodal Phi-2 Tracing Results

We use MultiModalCausalTrace on multimodal phi-2, where the language model is
phi-2 [17], a small language model capable of obtaining strong performances
and often close to large language models for certain language tasks. Similar
to our earlier observations for LLaVa, we find that even in multimodal phi-2,
the early MLP and self-attention layers are causal. This shows the
generalizability of our results for latest multimodal language models. We also
note that a small window size of 1, is sufficient to recover causal states in
multimodal language models.

Figure 10: MultiModalCausalTrace with Window Size 1: Early MLP layers are
causal for Multimodal Phi-2, similar to LLaVa-7B. Qualitative tracing results
for examples with single constraint questions (containing only a visual
constraint).  Figure 11: MultiModalCausalTrace with Window Size 1: Early Self-
Attention layers are also causal for Multimodal Phi-2, similar to LLaVa-7B.
Qualitative tracing results for examples with single constraint questions
(containing only a visual constraint).

##  Appendix F Multi-constraint Questions Plots

In this section, we present qualitative plots for the multi-constraint
questions from OK-VQA. These questions consist of a visual constraint as well
as a text-constraint which in conjunction are used to answer a given question.
Given that we earlier observe early causal layers corresponding to the visual
constraint – in this section, we show results corresponding to the text-
constraint. We have two primary observations: (i) A larger window size (at
least a size of 6) is required to recover causal states, which highlights that
a set of internal layers are used to retrieve relevant information; (ii) The
causal layers span the early as well as the middle layers. This is true for
both the MLP as well as self-attention layers.

Figure 12: MultiModalCausalTrace with Window Size 6: Early and mid MLP layers
are causal. Qualitative tracing results for examples with multiple constraint
questions (containing a visual constraint and a text constraint). Here
MultiModalCausalTrace is performed at the text-constraint position. For multi-
constraint questions, a larger window size is needed to recover any causal
states.  Figure 13: MultiModalCausalTrace with Window Size 6: Early and mid
self-attention layers are causal for a majority of the visual questions.
Qualitative tracing results for examples with multiple constraint questions
(containing a visual constraint and a text constraint). Here
MultiModalCausalTrace is performed at the text-constraint position. For multi-
constraint questions, a larger window size is needed to recover any causal
states.

##  Appendix G Model Editing Formulation and Further Results

As described in Sec. 5.1, MultEdit modifies the $W_{proj}^{\ell}$ matrix such
that the old keys $k_{c,\ell}$ are mapped to the new optimized values
$z_{c,\ell}^{*}$ such that the new values can drive the model towards
increasing $\mathcal{P}(O^{*})$. We define the editing objective in MultEdit
as below:

$W^{\ell*}_{proj}=\arg\min_{W^{\ell}_{proj}}\underbrace{\|W^{\ell}_{proj}k_{c,\ell}-z_{c,\ell}^{*}\|_{2}^{2}}_{term1}+\underbrace{\lambda\|W^{\ell}_{proj}-W_{proj}^{\ell^{\prime}}\|_{2}^{2}}_{term2}$
(6)

where $W^{\ell^{\prime}}_{proj}$ denotes the weights of the $W^{\ell}_{proj}$
matrix before the edit optimization. The second term acts as a regularization
which ensures that $W^{\ell^{\prime}}_{proj}$ does not deviate too much from
$W^{\ell}_{proj}$.

From Eq. 6, we can observe that it can be solved using a simple closed form
update as follows:

$W^{\ell*}_{proj}=(\lambda
W_{proj}^{\ell^{\prime}}+z_{c,\ell}^{*}k_{c,\ell}^{T})(\lambda
I+k_{c,\ell}k_{c,\ell}^{T})^{-1}$ (7)

Given access to the keys and values, this closed-form optimization can be
solved on a CPU. We note that this closed-form update for model editing has
been recently used for text-to-image models [4, 5]. However to the best of our
knowledge, ours is the first work to use such an update for MLLMs.

Hyperparameters. We use a learning rate of 0.1 to optimize for the values
using Adam Optimizer. For the regularization factor $\lambda$, we use 0.01
after a grid search. Amongst the set of early causal layers between 0-5, we
find editing Layer 2 to result in the best editing efficacy.

In Fig. 14, we show that editing the early causal MLP layers

Figure 14: Editing early causal layers leads to better editing efficacy,
though editing non-causal layers also leads to non-trivial editing efficacy.
Similar results for language models have been highlighted in [10]. The editing
is performed at the position of the last token of the visual constraint.

is better than the middle or the later layers. However, we find that editing
the middle or the later layers also lead to a non-trivial editing accuracy,
though it’s lesser in terms of effectiveness than editing the early causal
layers. Recent work [10] also observe this behaviour of non-trivial editing
accuracy at locations which are not causal layers.

##  Appendix H Information Transfer and Early Failure Mode Detection -
Qualitative Plots

In this section, we first show qualitative in Fig. 15 that the attention
contributions from the constraint tokens to the last token is significantly
more for correct answers than for incorrect answers. This is quantitatively
highlighted and shown in Fig. 16 where we find that Layer 16 and Layer 17 to
be the main orchestrator of this difference. A recent work [37] use attention
contributions as a signal to detect hallucinations in language models. They
show that the attention contribution values when used together with a linear
model can be used to predict the correctness of the answer. In fact, they find
this metric to be close to the confidence metric (which is the probability of
the generated token). However, [37] necessitates the use of all self-attention
layers which reduces it to be used as an “early” failure mode detection
metric. For multimodal language models, instead of using all the self-
attention layers, we use the average attention contributions from Layer 16 and
Layer 17 as an “early” failure mode detection metric. From Fig. 17, we find
that the average attention contributions indeed lead to a non-trivial AUROC of
0.63, but it lags behind the confidence metric which obtains AUROC of 0.76.
Our early results show that using the model internals can be used to “early”
detect if the model is going towards a failure mode, but it lags behind the
confidence although that requires the full forward pass.

Figure 15:  Correct answers have stronger attention contributions from the
constraint tokens (X-axis) to the last token, when compared to incorrect
answers. These qualitative examples are from the Multimodal Known dataset in
VQA-Constraints. Figure 16:  Correct answers have stronger attention
contributions from the constraint tokens (X-axis) to the last token, when
compared to incorrect answers on an average. Computed on the Multimodal Known
dataset in VQA-Constraints. In particular the layer 16 and layer 17 are the
distinguishing layers which have higher attention contributions in the correct
answers.  Figure 17:  Attention Contribution Metric although lags behind
Confidence Metric, it can be used as a coarse “early” failure mode detection
metric. Dataset used: Multimodal Known from VQA-Constraints. We use the
average attention contributions from Layer 16 and Layer 17 for computing the
failure mode metric.

##  Appendix I Editing Dataset

The editing dataset consists of two parts: (i) VQA questions from Multimodal
Known on which the LLaVa-7B gives incorrect answers; and (ii) VQA questions
from Encyclopedia-VQA which tests the long-tailed knowledge of the model. The
long-tailed questions are primarily about landmarks. Below we present some
qualitative examples on the long-tailed VQA questions:

Figure 18:  Qualitative Examples from the long-tailed knowledge editing
subset. The questions are sourced from the questions involving landmarks in
Encyclopedia-VQA.

##  Appendix J Compute

All our experiments are performed on Nvidia-A6000 and A5000 GPUs.