# Decomposition of symmetric separable states and ground state energy of
bosonic systems

Stephan Weis

###### Abstract.

We prove that every symmetric separable state admits a convex decomposition
into symmetric pure product states. While the result is not new in itself,
here we focus on convex geometry. We discuss the decomposition in the context
of numerical ranges and ground state problems of infinite bosonic systems.

###### Key words and phrases:

quantum state, symmetric state, antisymmetric state, separable state,
entanglement, numerical range, ground state energy, bosonic system

###### 2010 Mathematics Subject Classification:

15A69, 15A60, 47L07, 81V70, 81P40

##  1\. Introduction

Bosons are quantum mechanical particles described by density operators acting
on the symmetric tensor product of a one-particle Hilbert space [7]. A classic
result by Størmer [26] shows the $k$-particle marginal of an infinite bosonic
system is a convex combination of symmetric pure product states. This result
is known as the quantum de Finetti theorem in statistical mechanics [20, 30]
and in quantum information theory [11, 19]. Besides the de Finetti theorem,
the interrelation between symmetry and correlation is a more general topic in
quantum information theory [12, 4, 23].

The mentioned symmetric pure product states are separable states, that is to
say, they are convex combinations of product states. We prove the converse
statement: every symmetric separable state admits a convex decomposition into
symmetric pure product states. We are aware that the assertion is well known,
see [16, 27, 5] and [17, Lemma 7.29]. Our proof stresses the convex geometry
of the problem, namely that the set of symmetric states is a face of the set
of all quantum states. The claim then follows because all one-particle
marginals of a symmetric state are identical. Upon a sign change, our proof
shows every antisymmetric state is entangled.

The motivation behind this work is to examine linear images
$\Pi_{\mathrm{sym}}$ of the set of symmetric pure product states. As per the
quantum de Finetti theorem, the ground state energy of a local energy operator
of an infinite bosonic system is the distance of the origin from a supporting
hyperplane to $\Pi_{\mathrm{sym}}$. Studying the convex hull
$\operatorname{conv}(\Pi_{\mathrm{sym}})$ is therefore pivotal to
understanding ground state phenomena of bosonic systems [29, 8, 9].

Quantum information theory uses numerical ranges to represent linear images of
subsets of quantum states [25, 13, 9, 21]. We show the set
$\operatorname{conv}(\Pi_{\mathrm{sym}})$ is a linear image of the set of
symmetric separable states, a face of the set of separable states. This would
enable studies of $\operatorname{conv}(\Pi_{\mathrm{sym}})$ by taking
advantage of methods used with the set of separable states [2, 18, 6, 3].

In Section 2 we recall that the set of quantum states supported on a subspace
is a face of the set of all quantum states. Section 3 contains the main proofs
and a question regarding the number of terms in the decomposition of a
symmetric separable state. In Section 4 we discuss numerical ranges, which we
connect to the ground state energy of an infinite bosonic system in Section 5.

##  2\. Decomposition of states supported on subspaces

We recall aspects of the convex geometry of the set of quantum states and
point out that every state that contributes to a convex decomposition of a
state supported on a subspace is itself supported on the same subspace.

Let $\mathcal{H}$ be a finite-dimensional Hilbert space,
$\mathcal{B}(\mathcal{H})$ the Banach algebra of linear operators on
$\mathcal{H}$, and $\mathfrak{S}(\mathcal{H})$ the compact, convex set of
density operators on $\mathcal{H}$, that is to say, positive operators with
trace one, also known as states or quantum states. If
$\mathcal{H}=\mathbb{C}^{n}$ for some $n\in\mathbb{N}$, we identify
$\mathcal{B}(\mathcal{H})$ with the set $M_{n}$ of $n$-by-$n$ matrices, and
$\mathfrak{S}(\mathcal{H})$ with the compact, convex set of positive semi-
definite matrices of trace one.

A projector on $\mathcal{H}$ is an idempotent, self-adjoint operator
$P\in\mathcal{B}(\mathcal{H})$; this means that $P=P^{2}=P^{\ast}$. The set of
projectors on $\mathcal{H}$ is partially ordered by the relation $P\preceq Q$,
which signifies that $Q-P$ is a positive operator. It is well known that the
mapping that sends a projector $P$ on $\mathcal{H}$ to the image
$\operatorname{im}(P)=\\{P(\varphi)\mid\varphi\in\mathcal{H}\\}$ is a lattice
isomorphism from the set of projectors on $\mathcal{H}$ to the set of
subspaces of $\mathcal{H}$ partially ordered by inclusion.

A subset $F$ of a convex set $C$ is a _face_ of $C$ if $F$ is convex and
whenever $x\in F$ and $x=(1-\lambda)y+\lambda z$ for some $\lambda\in(0,1)$
and $y,z\in C$, then $y$ and $z$ are also in $F$. A point $x\in C$ is called
an extreme point of $C$ if $\\{x\\}$ is a face of $C$. Let
$\mathcal{K}\subset\mathcal{H}$ be a subspace. We say an operator
$A\in\mathcal{B}(\mathcal{H})$ is _supported_ on $\mathcal{K}$ if the images
$\operatorname{im}(A)=\\{A(\varphi):\varphi\in\mathcal{H}\\}$ of $A$ and
$\operatorname{im}(A^{\ast})=\\{A^{\ast}(\varphi):\varphi\in\mathcal{H}\\}$ of
the adjoint $A^{\ast}$ are included in $\mathcal{K}$. Let

(1) $\mathfrak{S}(\mathcal{H},\mathcal{K})$

denote the set of states on $\mathcal{H}$ that are supported on $\mathcal{K}$.
It is well known that the mapping that sends a subspace $\mathcal{K}$ of
$\mathcal{H}$ to the set $\mathfrak{S}(\mathcal{H},\mathcal{K})$ is a lattice
isomorphism from the set of subspaces of $\mathcal{H}$ to the set of faces of
$\mathfrak{S}(\mathcal{H})$ partially ordered by inclusion [1]. All faces of
the set of quantum states $\mathfrak{S}(\mathcal{H})$ are compact, convex
sets.

In particular, the set of extreme points of the set of quantum states
$\mathfrak{S}(\mathcal{H})$, also called pure states on $\mathcal{H}$, is the
set of projectors of rank one,

(2)
$\operatorname{ext}(\mathfrak{S}(\mathcal{H}))=\\{\ket{\varphi}\\!\\!\bra{\varphi}\colon\varphi\in\mathcal{H},\braket{\varphi}{\varphi}=1\\}.$

In quantum information theory, the projector onto the span of a unit vector
$\varphi\in\mathbb{C}^{n}$ is denoted by $\ket{\varphi}\\!\\!\bra{\varphi}$;
both $\varphi$ and $\ket{\varphi}\\!\\!\bra{\varphi}$ are referred to as pure
states.

Recall that the _relative interior_ of a convex subset $C$ of a Euclidean
vector space is the interior of $C$ with respect to the topology of the affine
hull of $C$.

######  Lemma 1.

Let $\mathcal{K}\subset\mathcal{H}$ be a subspace and let
$\rho\in\mathfrak{S}(\mathcal{H},\mathcal{K})$ be a state supported on
$\mathcal{K}$. Let $\lambda_{1},\dots,\lambda_{d}>0$ be strictly positive real
numbers that sum up to one and let $\rho=\sum_{i=1}^{d}\lambda_{i}\rho_{i}$ be
a convex combination of states
$\rho_{1},\dots,\rho_{d}\in\mathfrak{S}(\mathcal{H})$. Then each state
$\rho_{i}$ is supported on $\mathcal{K}$. If
$\rho_{i}=\ket{\varphi_{i}}\\!\\!\bra{\varphi_{i}}$ is the pure state
corresponding to a unit vector $\varphi_{i}\in\mathcal{H}$, then $\varphi_{i}$
lies in $\mathcal{K}$ for $i=1,\dots,d$.

###### Proof.

By Theorem 6.9 of [24], the state $\rho$ belongs to the relative interior of
the convex hull $C=\operatorname{conv}(\\{\rho_{1},\dots,\rho_{d}\\})$ of the
points $\rho_{1},\dots,\rho_{d}$. Since $\rho$ lies in the face
$\mathfrak{S}(\mathcal{H},\mathcal{K})$, Theorem 18.1 of [24] shows that the
whole convex set $C$ is included in $\mathfrak{S}(\mathcal{H},\mathcal{K})$.
This proves the first statement; the second statement follows as the image of
$\ket{\varphi_{i}}\\!\\!\bra{\varphi_{i}}$ is spanned by $\varphi_{i}$. ∎

The spectral projections of a state are supported on the same subspace as the
state.

######  Remark 1.

Every state $\rho\in\mathfrak{S}(\mathcal{H},\mathcal{K})$ is the convex
combination of at most $\dim_{\mathbb{C}}(\mathcal{K})$ pure states supported
on $\mathcal{K}$. It suffices to write $\rho$ as a convex combination of its
spectral projections. Lemma 1 shows the spectral projections are supported on
$\mathcal{K}$.

##  3\. Decomposition of symmetric separable states

We show that every symmetric separable state is a convex combination of
symmetric pure product states. The same proof shows all antisymmetric states
are entangled.

We label the units of a many-particle system by a finite set $\nu$. Let
$\mathcal{H}$ be a finite-dimensional Hilbert space, the one-particle Hilbert
space. The Hilbert space of the subsystem $\mu$ is the tensor product
$\mathcal{H}^{\otimes\mu}=\bigotimes_{i\in\mu}\mathcal{H}$, the identity
operator of which we denote by $\mathds{1}_{\mu}$, for all subsets
$\mu\subset\nu$. We replace $\nu$ with its cardinality $|\nu|$ if convenient,
for example to write symmetric product states
$\sigma^{|\nu|}=\sigma^{\nu}=\bigotimes_{i\in\nu}\sigma$ where
$\sigma\in\mathfrak{S}(\mathcal{H})$ is a state on the one-particle Hilbert
space.

Writing the complement as $\overline{\mu}=\nu\setminus\mu$, we define the
partial trace over the subsystem $\overline{\mu}$ as the linear map
$\operatorname{Tr}_{\overline{\mu}}:\mathcal{B}(\mathcal{H}^{\otimes\nu})\to\mathcal{B}(\mathcal{H}^{\otimes\mu})$,
where $\operatorname{Tr}_{\overline{\mu}}(A)$ is characterized by the
conditions

$\textstyle\operatorname{Tr}\big{(}\operatorname{Tr}_{\overline{\mu}}(A)B\big{)}=\operatorname{Tr}\big{(}A(B\otimes\mathds{1}_{\overline{\mu}})\big{)},\qquad
B\in\mathcal{B}(\mathcal{H}^{\otimes\mu}),$

for all operators $A\in\mathcal{B}(\mathcal{H}^{\otimes\nu})$. If $\rho$ is a
state on $\mathcal{H}^{\otimes\nu}$, then
$\operatorname{Tr}_{\overline{\mu}}(\rho)$ is a state on
$\mathcal{H}^{\otimes\mu}$, called the marginal of $\rho$ in the subsystem
$\mu$.

A state $\rho\in\mathfrak{S}(\mathcal{H}^{\otimes\nu})$ is a product state
(with respect to $\mathcal{H}$ and $\nu$) if it can be written in the form
$\rho=\bigotimes_{i\in\nu}\rho_{i}$, where
$\rho_{i}\in\mathfrak{S}(\mathcal{H})$ for all $i\in\nu$. Using pure states
(2) we define the set of pure product states by

(3)
$\mathfrak{S}_{\mathrm{pure},\mathrm{prod}}(\mathcal{H},\nu)=\Big{\\{}\bigotimes_{i\in\nu}\rho_{i}\mid\rho_{i}\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\;\forall
i\in\nu\Big{\\}}.$

A state $\rho\in\mathfrak{S}(\mathcal{H}^{\otimes\nu})$ is separable if $\rho$
is a convex combination of product states. Otherwise, $\rho$ is entangled. We
denote the set of separable states by

(4) $\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu).$

The possibility to decompose each factor of a product state into pure states
and the distributive law show $\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu)$
is the convex hull of
$\mathfrak{S}_{\mathrm{pure},\mathrm{prod}}(\mathcal{H},\nu)$. In fact,
$\mathfrak{S}_{\mathrm{pure},\mathrm{prod}}(\mathcal{H},\nu)$ is the set of
extreme points of $\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu)$ as every pure
product state is an extreme point of $\mathfrak{S}(\mathcal{H}^{\otimes\nu})$
and hence of $\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu)$.

The symmetric group $S_{\nu}$ of the finite set $\nu$ acts by linear
automorphisms on the many-particle Hilbert space $\mathcal{H}^{\otimes\nu}$ as
per the law

$\sigma\Big{(}\bigotimes_{i\in\nu}\varphi_{i}\Big{)}=\bigotimes_{i\in\nu}\varphi_{\sigma^{-1}(i)},\qquad\varphi_{i}\in\mathcal{H}\;\forall
i\in\nu,$

for all permutations $\sigma\in S_{\nu}$. The $\nu$-fold _symmetric tensor
product_ of the Hilbert space $\mathcal{H}$ is defined by

(5)
$\mathcal{H}_{\mathrm{sym}}^{\otimes\nu}=\Big{\\{}\varphi\in\mathcal{H}^{\otimes\nu}\mid\sigma(\varphi)=\varphi\;\forall\sigma\in
S_{\nu}\Big{\\}}.$

It is easy to write down a basis for the symmetric tensor product.

######  Remark 2.

Let $\ket{0},\ket{1},\dots,\ket{n-1}$ be an orthonormal basis of
$\mathcal{H}$. Let $T=(T_{0},\dots,T_{n-1})$ be an $n$-tuple of non-negative
integers that sum up to the cardinality $|\nu|$ of $\nu$ and define

$\varphi_{T}=c_{T}\sum_{\sigma\in S_{\nu}}\sigma\Big{(}\ket{\underbrace{0\dots
0}_{\text{$T_{0}$}}\underbrace{1\dots
1}_{\text{$T_{1}$}}\dots\underbrace{(n-1)\dots(n-1)}_{\text{$T_{n-1}$}}}\Big{)},$

where $c_{T}=(|\nu|!\prod_{i=0}^{n-1}T_{i}!)^{-1/2}$ is a normalization
constant. The orthonormal set $\\{\varphi_{T}\\}$ is a basis of the symmetric
tensor product $\mathcal{H}_{\mathrm{sym}}^{\otimes\nu}$, because
$\\{\varphi_{T}\\}$ spans the image of the projector
$\mathcal{H}^{\otimes\nu}\to\mathcal{H}^{\otimes\nu}$ defined by
$\tfrac{1}{|\nu|!}\sum_{\sigma\in S_{\nu}}\sigma$, see [14, Section 4.10]. For
example, the set of Dicke states [22] with $k$ excitations,

$\varphi_{(|\nu|-k,k)}=(|\nu|!(|\nu|-k)!k!)^{-1/2}\sum_{\sigma\in
S_{\nu}}\sigma\Big{(}\ket{\underbrace{0\dots
0}_{\text{$|\nu|-k$}}\underbrace{1\dots 1}_{\text{$k$}}}\Big{)},\quad
k=0,\dots,|\nu|,$

is an orthonormal basis of the $\nu$-fold symmetric tensor product of
$\mathbb{C}^{2}$.

######  Definition 1 (Symmetric States).

A state $\rho\in\mathfrak{S}(\mathcal{H}^{\otimes\nu})$ is symmetric (with
respect to $\mathcal{H}$ and $\nu$) if $\rho$ is supported on the symmetric
tensor product $\mathcal{H}_{\mathrm{sym}}^{\otimes\nu}$ introduced in
equation (5). Using the notation from equation (1) and (4), we denote the set
of symmetric separable states by

(6)
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)=\mathfrak{S}(\mathcal{H}^{\otimes\nu},\mathcal{H}_{\mathrm{sym}}^{\otimes\nu})\cap\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu).$

Every convex combination of symmetric pure product states
$\sigma^{\otimes\nu}$ is a symmetric separable state. Here we prove the
converse statement that every symmetric separable state is a convex
combination of symmetric pure product states.

######  Theorem 1.

The set of symmetric separable states is the convex hull of the set of
symmetric pure product states,

(7)
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)=\operatorname{conv}\big{(}\\{\sigma^{\otimes\nu}\mid\sigma\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\\}\big{)}.$

The symmetric pure product states are the extreme points of
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$,

(8)
$\operatorname{ext}\big{(}\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)\big{)}=\big{\\{}\sigma^{\otimes\nu}\mid\sigma\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\big{\\}}.$

###### Proof.

In order to prove (7) it suffices to show that every symmetric separable state
$\rho$ is a convex combination of symmetric pure product states. Being a
separable state, $\rho$ admits a convex decomposition
$\rho=\sum_{i=1}^{d}\lambda_{i}\ket{\varphi_{i}}\\!\\!\bra{\varphi_{i}}$ with
positive coefficients $\lambda_{1},\dots,\lambda_{d}>0$ into pure product
states

$\textstyle\varphi_{i}=\bigotimes_{j\in\nu}\psi_{i,j}\in\mathcal{H}^{\otimes\nu},$

where $\psi_{i,j}\in\mathcal{H}$ is a pure state for all $i=1,\ldots,d$ and
$j\in\nu$. Since $\rho$ is supported on the symmetric tensor product
$\mathcal{H}_{\mathrm{sym}}^{\otimes\nu}$, Lemma 1 shows $\varphi_{i}$ lies in
the symmetric tensor product $\mathcal{H}_{\mathrm{sym}}^{\otimes\nu}$ for all
$i=1,\ldots,d$. This implies the one-particle marginals of $\varphi_{i}$ are
identical to a pure state $\psi_{i}\in\mathcal{H}$,

$\ket{\psi_{i}}\\!\\!\bra{\psi_{i}}=\ket{\psi_{i,j}}\\!\\!\bra{\psi_{i,j}}=\operatorname{Tr}_{\overline{\\{j\\}}}(\ket{\varphi_{i}}\\!\\!\bra{\varphi_{i}}),\qquad
j\in\nu,$

for all $i=1,\ldots,d$, which shows

$\textstyle\rho=\sum_{i=1}^{d}\lambda_{i}\ket{\varphi_{i}}\\!\\!\bra{\varphi_{i}}=\sum_{i=1}^{d}\lambda_{i}(\ket{\psi_{i}}\\!\\!\bra{\psi_{i}})^{\otimes\nu}$

and proves the claim.

The inclusion “$\subset$” in equation (7) implies that every extreme point of
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$ must be a
symmetric pure product state. Conversely, every symmetric pure product state
is an extreme point of $\mathfrak{S}(\mathcal{H}^{\otimes\nu})$ and hence of
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$, which yields
equation (8). ∎

######  Question 1.

How many symmetric pure product states are required in the decomposition of a
symmetric separable state $\rho$? The rank of $\rho$ may not be an upper bound
on $d$ (as in Remark 1) because the spectral projections of $\rho$ may be
entangled. Examples of separable two-qubit states of rank three that require
four pure product states in the decomposition are described in [28]. How is
the situation in the symmetric setting?

######  Remark 3.

A similar decomposition as in Theorem 1 exists for the class of completely
symmetric states, which is important in entanglement theory [10, 23]. These
states are convex combination of real symmetric pure product states.

The $\nu$-fold _antisymmetric tensor product_ of the Hilbert space
$\mathcal{H}$ is defined by

$\Big{\\{}\varphi\in\mathcal{H}^{\otimes\nu}\mid\sigma(\varphi)=\operatorname{sgn}(\sigma)\varphi\;\forall\sigma\in
S_{\nu}\Big{\\}},$

where $\operatorname{sgn}(\sigma)$ is the sign of a permutation $\sigma\in
S_{\nu}$, which is $+1$ if $\sigma$ is a composition of an even number of
transpositions, and $-1$ else. A state on $\mathcal{H}^{\otimes\nu}$ is
_antisymmetric_ if it is supported on the antisymmetric tensor product.

######  Theorem 2.

If $\nu$ has at least two elements, then every antisymmetric state is
entangled.

###### Proof.

The same reasoning as in Theorem 1 shows every antisymmetric separable state
is a convex combination of symmetric pure product states. Hence, the existence
of an antisymmetric separable state would lead to a contradiction. ∎

##  4\. Joint Numerical Ranges

Numerical ranges, originally a topic of matrix theory, have become
indispensable tools in quantum information theory [25, 13, 9, 21] to study
linear images of certain subsets of quantum states. We show two notions of
symmetric numerical ranges to be equal.

Let $A_{1},\dots,A_{m}\in\mathcal{B}(\mathcal{H}^{\otimes\nu})$ be a sequence
of hermitian operators. Using the sets of pure states (2), pure product states
(3), and separable states (4), we define the numerical ranges [13, 9]

$\displaystyle\Lambda$ $\displaystyle=\left\\{\operatorname{Tr}(\rho
A_{i})_{i=1}^{m}\mid\rho\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}^{\otimes\nu}))\right\\},$
(joint numerical range) $\displaystyle\Pi$
$\displaystyle=\left\\{\operatorname{Tr}(\rho
A_{i})_{i=1}^{m}\mid\rho\in\mathfrak{S}_{\mathrm{pure},\mathrm{prod}}(\mathcal{H},\nu)\right\\},$
(  –"– product numerical range) $\displaystyle\Theta$
$\displaystyle=\left\\{\operatorname{Tr}(\rho
A_{i})_{i=1}^{m}\mid\rho\in\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu)\right\\}.$
(  –"– separable  –"– –"– )

Symmetric numerical ranges have been studied, too [9]. Using symmetric pure
product states, we define the set

(9)
$\Pi_{\mathrm{sym}}=\left\\{\operatorname{Tr}(\sigma^{\otimes\nu}A_{i})_{i=1}^{m}\mid\sigma\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\right\\},$

which we call _joint symmetric product numerical range_. Employing the set of
symmetric separable states (6), we define the set

$\Theta_{\mathrm{sym}}=\left\\{\operatorname{Tr}(\rho
A_{i})_{i=1}^{m}\mid\rho\in\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)\right\\},$

which we call _joint symmetric separable numerical range_.

Recalling equation (3), we observe the set of separable states is the convex
hull of the set of pure product states; therefore
$\Theta=\operatorname{conv}(\Pi)$. The analogue is true for symmetric
numerical ranges.

######  Corollary 1.

The joint symmetric separable numerical range $\Theta_{\mathrm{sym}}$ is the
convex hull of the joint symmetric product numerical range
$\Pi_{\mathrm{sym}}$.

###### Proof.

Theorem 1 proves the set of symmetric separable states
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$ is the convex hull
of the set
$\\{\sigma^{\otimes\nu}\mid\sigma\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\\}$
of symmetric pure product states. The claim follows as $\Theta_{\mathrm{sym}}$
is the image of the former set and $\Pi_{\mathrm{sym}}$ is the image of the
latter set under the same linear map. ∎

Corollary 1 clarifies a relationship between two numerical ranges.

######  Remark 4.

Chen et. al [9] have defined the symmetric version of the joint separable
numerical range $\Theta$ to be the convex hull
$\operatorname{conv}(\Pi_{\mathrm{sym}})$ of the joint symmetric product
numerical range $\Pi_{\mathrm{sym}}$. This brings to mind the equation
$\Theta=\operatorname{conv}(\Pi)$. The analogous equation
$\Theta_{\mathrm{sym}}=\operatorname{conv}(\Pi_{\mathrm{sym}})$ shows the set
$\operatorname{conv}(\Pi_{\mathrm{sym}})$ is a linear image of the set of
symmetric separable states
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$, thereby
corroborating $\operatorname{conv}(\Pi_{\mathrm{sym}})$ is a natural choice
for a symmetric version of $\Theta$.

Corollary 1 offers a new approach to study the set
$\operatorname{conv}(\Pi_{\mathrm{sym}})$.

######  Remark 5.

The identity $\operatorname{conv}(\Pi_{\mathrm{sym}})=\Theta_{\mathrm{sym}}$
allows us to study the set $\operatorname{conv}(\Pi_{\mathrm{sym}})$ _via_ the
set of symmetric separable states
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$, the preimage of
the numerical range $\Theta_{\mathrm{sym}}$. The convex set
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$ is a face of the
set of separable states $\mathfrak{S}_{\mathrm{sep}}(\mathcal{H},\nu)$, for
which algebraic tools are available [2, 18, 6, 3]. Studying linear images of
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$ in the context of
statistical mechanics is interesting already for two qubits [8, 9], where
every separable state has a positive partial transpose [15].

##  5\. The Quantum de Finetti Theorem

This section highlights the role the joint symmetric separable numerical range
$\Theta_{\mathrm{sym}}$ plays in the statistical mechanics of bosonic systems.

Symmetric states on the many-particle Hilbert space $\mathcal{H}^{\otimes\nu}$
are called bosonic states in physics [7, 20, 9]. Due to the symmetry, the
marginals of a symmetric state
$\rho\in\mathfrak{S}(\mathcal{H}^{\otimes\nu},\mathcal{H}_{\mathrm{sym}}^{\otimes\nu})$
are themselves symmetric states. The marginals of $\rho$ depend on the
subsystems only through their sizes,

$\operatorname{Tr}_{\overline{\mu}}(\rho)=\operatorname{Tr}_{\overline{\eta}}(\rho)\qquad\text{for
all $\mu,\eta\subset\nu,|\mu|=|\eta|$.}$

Thus, one studies the set of $k$-particle $N$-representable density operators

$\mathcal{P}_{N}^{(k)}=\left\\{\operatorname{Tr}_{\overline{\\{1,\ldots,k\\}}}(\rho)\mid\rho\in\mathfrak{S}(\mathcal{H}^{\otimes
N},\mathcal{H}_{\mathrm{sym}}^{\otimes
N})\right\\}\subset\mathfrak{S}(\mathcal{H}^{\otimes
k},\mathcal{H}_{\mathrm{sym}}^{\otimes k})$

with reference to the total number $N=|\nu|$ of units and the size $k$ of the
subsystem. According to a classic result by Størmer [26], also known as the
quantum de Finetti theorem [20, 9, 30], we have

$\bigcap_{N\geq
k}\mathcal{P}_{N}^{(k)}=\operatorname{conv}\big{(}\\{\sigma^{\otimes
k}\mid\sigma\in\operatorname{ext}(\mathfrak{S}(\mathcal{H}))\\}\big{)},\qquad
k\in\mathbb{N}.$

In other words, the set of all possible $k$-particle marginals of an infinite
bosonic system is the convex hull of the set of symmetric pure product states.
By Theorem 1,

(10) $\bigcap_{N\geq
k}\mathcal{P}_{N}^{(k)}=\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},k),\qquad
k\in\mathbb{N},$

where
$\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},k)=\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},\nu)$
is the set of symmetric separable states on a system $\nu$ of size $|\nu|=k$.

Quantum phase transitions are associated with abrupt changes of the ground
state energy of an energy operator

$H(x)=x_{1}H_{1}+x_{2}H_{2}+\dots+x_{m}H_{m}$

on $\mathcal{H}^{\otimes N}$ under smooth changes of the parameter
$x=(x_{1},\dots,x_{k})\in\mathbb{R}^{m}$ for large system sizes $N\to\infty$.
In many cases the energy operators $H_{1},\dots,H_{m}$ are $k$-local
Hamiltonians for some fixed $k\in\mathbb{N}$. This means

$\textstyle
H_{i}=\sum_{\mu\subset\nu,|\mu|=k}H_{i,\mu}\otimes\mathds{1}_{\overline{\mu}},\qquad
i=1,\dots,m,$

where $H_{i,\mu}\in\mathcal{B}(\mathcal{H}^{\otimes\mu})$ interacts with at
most $|\mu|=k$ units of the total system $\nu=\\{1,\ldots,N\\}$. The ground
state energy of a bosonic system with energy operator $H(x)$ is

$\textstyle\min_{\rho\in\mathfrak{S}(\mathcal{H}^{\otimes
N},\mathcal{H}_{\mathrm{sym}}^{\otimes
N})}\operatorname{Tr}(H(x)\rho)=\min_{\sigma\in\mathcal{P}_{N}^{(k)}}\operatorname{Tr}(A(x)\sigma),$

where $A(x)=\sum_{i=1}^{m}x_{i}A_{i}$ and

$\textstyle
A_{i}=\sum_{\mu\subset\nu,|\mu|=k}H_{i,\mu}\in\mathcal{B}(\mathcal{H}^{\otimes
k}),\qquad i=1,\dots,m.$

For this sum to be well-defined, one may assume each operator $H_{i,\mu}$ is
supported on the symmetric tensor product
$\mathcal{H}_{\mathrm{sym}}^{\otimes\mu}$ or replace $H_{i,\mu}$ with its
compression onto $\mathcal{H}_{\mathrm{sym}}^{\otimes\mu}$. Provided the
convergence of the energy operators $A_{i}$ is guaranteed in the thermodynamic
limit $N\to\infty$, Størmer’s result in the version of equation (10) shows the
ground state energy of $H(x)$ is

$\min_{\sigma\in\mathfrak{S}_{\mathrm{sym},\mathrm{sep}}(\mathcal{H},k)}\operatorname{Tr}(A(x)\sigma)=\min_{y\in\Theta_{\mathrm{sym}}}x_{1}y_{1}+x_{2}y_{2}+\dots+x_{m}y_{m},$

where $\Theta_{\mathrm{sym}}$ is the joint symmetric separable numerical range
(9) with respect to the operators
$A_{1},\dots,A_{m}\in\mathcal{B}(\mathcal{H}^{\otimes k})$.

In other words, the ground state energy of $H(x)$ is the distance of the
origin from the supporting hyperplane to $\Theta_{\mathrm{sym}}$ with inner
normal vector $x$. This rationale strongly motivates to study the geometry of
the numerical range $\Theta_{\mathrm{sym}}$. The key-feature of a ruled
surface on the boundary of $\Theta_{\mathrm{sym}}$ is an expression of a phase
transition [29, 8, 9] and deserves a thorough investigation.

Acknowledgements. I thank Karol Życzkowski for discussions on numerical ranges
and their use in quantum mechanics. I thank Lin Chen, who kindly pointed out
to me the difference between symmetric and completely symmetric states. I
thank an anonymous referee, who made me aware of the literature [16, 17, 27,
5].

## References

  * [1] Alfsen, E. M. and F. W. Shultz, State Spaces of Operator Algebras: Basic Theory, Orientations, and C*-Products, Boston, Birkh user, 2001.
ISBN: 978-1-4612-6634-1, DOI: 10.1007/978-1-4612-0147-2

  * [2] Alfsen, E. and F. Shultz, Unique decompositions, faces, and automorphisms of separable states, Journal of Mathematical Physics 51, 052201 (2010).
DOI: 10.1063/1.3399808

  * [3] Aubrun, G. and S. J. Szarek, Alice and Bob Meet Banach: The Interface of Asymptotic Geometric Analysis and Quantum Information Theory, Providence, AMS, 2017. ISBN: 978-1-4704-3468-7
  * [4] Augusiak, R., J. Tura, J. Samsonowicz, and M. Lewenstein, Entangled symmetric states of N qubits with all positive partial transpositions, Phys. Rev. A 86, 042316 (2012). DOI: 10.1103/PhysRevA.86.042316
  * [5] Bohnet-Waldraff, F., D. Braun, and O. Giraud, _Partial transpose criteria for symmetric states_ , Physical Review A 94, 042343 (2016). DOI: 10.1103/PhysRevA.94.042343
  * [6] Bengtsson, I. and K. Życzkowski, Geometry of Quantum States: An Introduction to Quantum Entanglement, 2nd edition, Cambridge, Cambridge University Press, 2017. ISBN: 978-1-139-20701-0, DOI: 10.1017/9781139207010
  * [7] Messiah, A., Quantum Mechanics, Mineola, Dover Publications, 1999.
ISBN: 978-0-486-78455-7

  * [8] Chen, J.-Y., Z. Ji, Z.-X. Liu, X. Qi, N. Yu, B. Zeng, and D. Zhou, Physical origins of ruled surfaces on the reduced density matrices geometry, Science China Physics, Mechanics & Astronomy 60, 020311 (2017). DOI: 10.1007/s11433-016-0396-2
  * [9] Chen, J., C. Guo, Z. Ji, Y.-T. Poon, N. Yu, B. Zeng, and J. Zhou, Joint product numerical range and geometry of reduced density matrices, Science China Physics, Mechanics & Astronomy 60, 020312 (2017). DOI: 10.1007/s11433-016-0404-5
  * [10] Chen, L., D. Chu, L. Qian, and Y. Shen, Separability of completely symmetric states in a multipartite system, Phys. Rev. A 99, 032312 (2019).
DOI: 10.1103/PhysRevA.99.032312

  * [11] Christandl, M., R. König, and R. Renner, Postselection technique for quantum channels with applications to quantum cryptography, Phys. Rev. Lett. 102, 020504 (2009).
DOI: 10.1103/PhysRevLett.102.020504

  * [12] Christandl, M., N. Schuch, and A. Winter, Entanglement of the antisymmetric state, Communications in Mathematical Physics 311, 397–422 (2012).
DOI: 10.1007/s00220-012-1446-7

  * [13] Gawron, P., Z. Puchała, J. A. Miszczak, Ł. Skowronek, and K. Życzkowski, Restricted numerical range: A versatile tool in the theory of quantum information, Journal of Mathematical Physics 51, 102204 (2010). DOI: 10.1063/1.3496901
  * [14] Greub, W., Multilinear Algebra, New York, Springer, 1978.
ISBN: 978-0-387-90284-5, DOI: 10.1007/978-1-4613-9425-9

  * [15] Horodecki, M., P. Horodecki, and R. Horodecki, Separability of mixed states: Necessary and sufficient conditions, Physics Letters A 223, 1–8 (1996).
DOI: 10.1016/S0375-9601(96)00706-2

  * [16] Korbicz, J. K., J. I. Cirac, and M. Lewenstein, _Spin squeezing inequalities and entanglement of n qubit states_ , Phys. Rev. Lett. 95, 120502 (2005).
DOI: 10.1103/PhysRevLett.95.120502

  * [17] Kraus, B., _Entanglement Properties of Quantum States and Quantum Operations_ , PhD thesis, Innsbruck, 2003.
  * [18] Kye, S.-H., Facial structures for various notions of positivity and applications to the theory of entanglement, Rev. Math. Phys. 25, 1330002 (2013).
DOI: 10.1142/S0129055X13300021

  * [19] Lancien, C. and A. Winter, Flexible constrained de Finetti reductions and applications, Journal of Mathematical Physics 58, 092203 (2017). DOI: 10.1063/1.5003633
  * [20] Lewin, M., P. T. Nam, and N. Rougerie, Derivation of Hartree’s theory for generic mean-field Bose systems, Advances in Mathematics 254, 570–621 (2014).
DOI: 10.1016/j.aim.2013.12.010

  * [21] Li, C.-K., Y.-T. Poon, and Y.-S. Wang, Joint numerical ranges and commutativity of matrices. arXiv:2002.02768 [math.FA]
  * [22] Meill, A. and D. A. Meyer, Symmetric three-qubit-state invariants, Phys. Rev. A 96, 062310 (2017). DOI: 10.1103/PhysRevA.96.062310
  * [23] Qian, L. and D. Chu, Decomposition of completely symmetric states, Quantum Inf Process 18, 208 (2019). DOI: 10.1007/s11128-019-2318-2
  * [24] Rockafellar, R. T., Convex Analysis, Princeton, Princeton University Press, 1970.
ISBN: 978-1-4008-7317-3, DOI: 10.1515/9781400873173

  * [25] Schulte-Herbrüggen, T., G. Dirr, U. Helmke, and S. J. Glaser, The significance of the C-numerical range and the local C-numerical range in quantum control and quantum information, Linear and Multilinear Algebra 56, 3–26 (2008).
DOI: 10.1080/03081080701544114

  * [26] Størmer, E., Symmetric states of infinite tensor products of C*-algebras, Journal of Functional Analysis 3, 48–68 (1969). DOI: 10.1016/0022-1236(69)90050-0
  * [27] T th, G., and O. Gühne, _Entanglement and permutational symmetry_ , Phys. Rev. Lett. 102, 170503 (2009). DOI: 10.1103/PhysRevLett.102.170503
  * [28] Wu, S., The convex sum of product states for a separable state, Physics Letters A 321, 301–307 (2004). DOI: 10.1016/j.physleta.2003.12.046
  * [29] Zauner, V., D. Draxler, L. Vanderstraeten, J. Haegeman, and F. Verstraete, Symmetry breaking and the geometry of reduced density matrices, New Journal of Physics 18, 113033 (2016). DOI: 10.1088/1367-2630/18/11/113033
  * [30] Zeng, B., X. Chen, D.-L. Zhou, and X.-G. Wen, Quantum Information Meets Quantum Matter, New York, Springer, 2019. DOI: 10.1007/978-1-4939-9084-9

Stephan Weis

Berlin, Germany

e-mail<EMAIL_ADDRESS>


capbtabboxtable[][]

# ReVeaL: Retrieval-Augmented Visual-Language Pre-Training with
Multi-Source Multimodal Knowledge Memory

Ziniu Hu1 , Ahmet Iscen2, Chen Sun2, Zirui Wang2, Kai-Wei Chang1, Yizhou Sun1
Cordelia Schmid2, David A. Ross2, Alireza Fathi2
1University of California, Los Angeles, 2Google Research This work was done
when Ziniu was an intern at Google.

###### Abstract

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language
Model (ReVeaL) that learns to encode world knowledge into a large-scale
memory, and to retrieve from it to answer knowledge-intensive queries. ReVeaL
consists of four key components: the memory, the encoder, the retriever and
the generator. The large-scale memory encodes various sources of multimodal
world knowledge (_e.g_. image-text pairs, question answering pairs, knowledge
graph triplets, _etc_.) via a unified encoder. The retriever finds the most
relevant knowledge entries in the memory, and the generator fuses the
retrieved knowledge with the input query to produce the output. A key novelty
in our approach is that the memory, encoder, retriever and generator are all
pre-trained end-to-end on a massive amount of data. Furthermore, our approach
can use a diverse set of multimodal knowledge sources, which is shown to
result in significant gains. We show that ReVeaL achieves state-of-the-art
results on visual question answering and image captioning. The project page is
ReVeaL-CVPR.github.io.

Figure 1: We augment a visual-language model with the ability to retrieve
multiple knowledge entries from a diverse set of knowledge sources, which
helps generation. Both retriever and generator are trained jointly, end-to-
end, by optimizing a language modeling objective.

##  1 Introduction

Recent large-scale models such as T5 [33], GPT-3 [4], PaLM [9], CoCa [49],
Flamingo [2], BEIT-3 [43] and PaLI [7] have demonstrated the ability to store
substantial amounts of world knowledge, when scaled to tens of billions of
parameters and trained on vast text and image corpora. These models achieve
state-of-the-art results in downstream tasks such as image captioning, visual
question answering and open vocabulary recognition. Yet, these models have a
number of drawbacks: (i) they require massive scale, of parameters, data and
computation, and (ii) they need to be re-trained every time the world
knowledge is updated.

To address these issues, we adopt a different approach. Instead of statically
compiling world knowledge into model weights, we transform the knowledge into
a key-value memory through neural representation learning. Our model learns to
utilize the memory for answering knowledge-intensive queries. By decoupling
the knowledge memorization from reasoning, we enable our model to leverage
various external sources of knowledge (_e.g_., Wikipedia passages and images
[37], the WikiData knowledge graph [40], Web image-text pairs [5] and visual
question answering data [12]). This enables the model parameters to focus on
understanding the query and conducting reasoning, rather than being dedicated
to memorization.

Retrieval-augmented models have attracted a fair amount of attention in the
fields of NLP [14, 18] and computer vision [25, 13]. Typically, these models
often use a pre-existing single-modality backbone to encode and retrieve
information from the knowledge corpus. Such approaches do not leverage all
available modalities in the query and knowledge corpora, and hence they might
not find the information that is most helpful for generating the model output.
A key novelty in our approach is that we encode and store various sources of
multimodal world knowledge into a unified memory, which the retriever can
access via multimodal query encodings, to find the most relevant information
from across complementary sources. Our multimodal memory and retriever are
pre-trained end-to-end together together with the rest of the model, on a
massive amount of data and using diverse knowledge sources.

A key challenge of pre-training the multimodal retriever end-to-end is the
lack of direct supervision. There is no ground-truth indicating which
knowledge entries are most helpful for answering knowledge-intensive queries.
Some of the existing works in NLP [14, 23, 34] propose to acquire training
signal by assessing the usefulness of each retrieved knowledge entry
independently for helping language modelling. This approach is inefficient, as
it involves estimating hundreds of retrieved knowledge entries independently,
and also inaccurate as it discards the dependency between different knowledge
entries in the retrieval set. In contrast, we propose to get this training
signal while simultaneously considering multiple retrieved knowledge entries,
by introducing an attentive fusion layer that injects retrieval score into the
attention calculation procedure. This enables the retrieval module to be
differentiable and jointly pre-trained with the rest of the model.

In summary, our key contributions are as follows:

  * •

We are the first to propose an end-to-end pre-training paradigm that learns to
index into a large-scale memory to solve knowledge-intensive visual-language
tasks.

  * •

Our method can construct a large-scale memory by encoding various sources of
multimodal world knowledge, including Wikipedia passage, web images with alt-
text captions, and knowledge graph triplets.

  * •

ReVeaL achieves state-of-the-art performance on several knowledge-intensive
visual question answering and image captioning datasets. Notably on the OKVQA
benchmark, ReVeaL achieves a new state-of-the-art, 59.1$\%$ accuracy, while
using order of magnitude fewer parameters than previous works.

##  2 Related Work and Background

Knowledge-based Visual Question Answering. To evaluate a model’s ability to
comprehend multimodal world knowledge not easily inferred from input data,
several knowledge-based Visual Question Answering (VQA) datasets have been
introduced. KB-VQA [41] and FVQA [42] design questions that can be answered by
retrieving relevant triplets from domain-specific structured knowledge graphs.
OK-VQA [29] improves these datasets by necessitating the use of external
knowledge, which goes beyond what can be directly observed in the input
images. More recently, A-OKVQA [35] offers further improvements to OK-VQA by
exclusively selecting questions that demand both external knowledge and
commonsense reasoning about the image scenes. To tackle knowledge-based VQA
tasks, many approaches have been proposed to incorporate external knowledge
into visual-language models. One line of research uses explicit knowledge from
structured knowledge graphs [31, 11, 44, 15] or unstructured text corpora [28,
27, 46]. The key component for these works is the knowledge retriever. Some
works [31, 11, 28, 46] utilize off-the-shelf vision detection models to
generate image tags for knowledge retrieval, while others train the retrieval
model via distant supervision [27] or auxiliary tasks (_e.g_. entity linking)
[13]. Another research direction aims to incorporate implicit knowledge from
pre-trained Large Language Models, such as GPT-3 [4] or PaLM [9]. These
approaches utilize off-the-shelf image caption models to convert images into
text, feed them into a language model, and use the generated text output as
augmented knowledge [48, 13, 24]. Our work follows the first direction,
augmenting a vision-language model with an explicit knowledge retriever. The
main distinction is that we propose an end-to-end training framework to
jointly learn the answer generator and retriever, rather than using a fixed or
predefined knowledge retrieval.

#### End-to-End Training of Retrieval-Augmented Models.

Given the advantage of knowledge retrieval, a key question is how to get
learning signal to train the retrieval model. For tasks with annotated
retrieval ground-truth, retrieval training can be conducted via standard
contrastive learning [22]. However, most tasks do not provide clear
indications of which knowledge entries are relevant for generating answers. To
this end, a series of studies have investigated retrieval training using
supervision derived from downstream tasks. REALM [14] trains a single-document
retriever by concatenating each retrieved result with the query, to calculate
the final loss independently. A similar approach has been used by EMDR2 [34]
for multi-document retrieval training. FID-KD [17] proposes to use the
aggregated attention score calculated by the generator as a distillation
signal to train the retriever. Atlas [18] further introduces a perplexity
distillation loss and a leave-one-out variant. Our ReVeaL proposes to inject
the retrieval scores directly into an attentive fusion module, enabling to
train the retriever to directly optimize downstream tasks as well as pre-
training objectives.

##  3 Method

Figure 2:  The overall workflow of ReVeaL consists of four main steps: (a)
encode a multimodal input into a sequence of token embeddings and a summarized
query embedding; (b) encode each knowledge entry from different corpus into
unified key and value embedding pairs, where key is used to index the memory
and value contains full information of the knowledge; (c) retrieve top-K most
similar knowledge items from different knowledge sources, and return the pre-
computed in-memory value embeddings and re-encoded value; and (d) fuse the
top-K knowledge items via attentive knowledge fusion layer by injecting the
retrieval score as a prior during attention calculation. This facilitates
ReVeaL’s key novelty: the memory, encoder, retriever and the generator can be
jointly trained in an end-to-end manner.

We propose a Retrieval-Augmented Visual Language Model (ReVeaL), which learns
to use knowledge from different sources for solving knowledge-intensive tasks.
For both pre-training and fine-tuning, our goal is to learn the distribution
$P(y\mid x)$ to generate a textual output $y$ conditioned on a multimodal
input query $x$. ReVeaL contains four components: knowledge encoding, memory,
retrieval and generation. Given an input query $x$, we first retrieve $K$
possibly helpful entries $M=\\{m_{1},\cdots,m_{K}\\}$ from the memory corpora
$\mathcal{M}$. Each $m$ is a memory entry containing the encoded single key
embedding and a sequence of value embeddings (we will describe how to encode
knowledge items into memory entries in Sec. 3.2). With it, the retriever can
use embedding similarity to find relevant memory entries. We model this
retrieval process as sampling from distribution $p(M\mid x)$. Then, we
condition on both the retrieved set $M$ and the original input query $x$ to
generate the output $y$, modeled as $p(y\mid x,M)$. To obtain the overall
likelihood of generating $y$, we treat $M$ as a latent variable from the
entire memory $\tilde{\mathcal{M}}$ and marginalize over it yielding:

$\displaystyle p(y\mid
x)=\sum_{M\subset\tilde{\mathcal{M}}}\underbrace{p(M\mid
x)}_{retrieval}\cdot\underbrace{p(y\mid x,M)}_{generation}.$ (1)

However, this marginal probability involves an intractable summation over all
size-$K$ subsets of the memory corpora $\tilde{\mathcal{M}}$. We approximate
this instead by using the top-K entries in memory with the highest probability
under $p(M\mid x)$. This is reasonable if most of the unrelated memory entries
do not contribute to the generation. Note that we use an online memory that is
updated as the knowledge encoder is trained end-to-end with the rest of the
model.

Figure 2 illustrates the overall workflow of ReVeaL, and we describe each
component in this section. In particular, in Sec. 3.1 we describe how the
query is encoded. In Sec. 3.2 we go over how the multimodal knowledge memory
is constructed and updated during pre-training. Next, we describe how we
retrieve the memory entries that are most relevant to the input query in Sec.
3.3. Finally, in Sec. 3.4 we describe the generator that fuses the query and
retrieved knowledge and decodes them into the generated text.

###  3.1 Query Encoding

Figure 2 (a) depicts how the input image-text query is encoded. We use a base
visual-language encoder $b(\cdot)$ to turn the query input and each knowledge
item (with potentially different modalities _e.g_. text-only, image-only or
image-text pairs) into a sequence of embeddings (tokens). We adopt a Vision
Transformer (ViT) [10] to encode the images and we use a lower-layer111We
denote the last $l$ layers of a T5 encoder as ‘upper-layer’, and the remaining
ones including the token embedding layer as ‘lower-layer’. T5 encoder [33] to
encode the texts. We add a projection layer on top of the ViT model to map the
image tokens into the same space as the text tokens. We then concatenate the
two modalities together. We use an upper-layer T5 module as both the query
Head $\phi_{\texttt{Query}}(\cdot)$ and the key Head
$\phi_{\texttt{Key}}(\cdot)$ to compute the query embedding and memory keys.
We take the output of the first [CLS] tokens followed by a linear projection
and L2-normalization to summarize the input into a $d$-dimensional embedding.

###  3.2 Memory

Figure 2 (b) shows how memory is constructed and updated by encoding knowledge
items. Our approach differs from previous works primarily by leveraging a
diverse set of multimodal knowledge corpora (WikiData knowledge graph,
Wikimedia passages and images, Web image-text pairs). Throughout the paper, we
denote each corpus as $\mathcal{C}^{j}=\\{z^{j}_{1},\dots,z^{j}_{N}\\}$, in
which each $z^{j}_{i}\in\mathcal{C}^{j}$ is a knowledge item that could be an
image-text pair, text only, image only, or a knowledge graph triplet. We
denote the unified knowledge corpus as
$\tilde{\mathcal{C}}=\mathcal{C}^{1}\cup\mathcal{C}^{2}\dots\cup\mathcal{C}^{S}$
that combines $|\tilde{\mathcal{C}}|=S$ different knowledge corpora. We encode
the external knowledge corpora into a unified memory
$\tilde{\mathcal{M}}=[\mathcal{M}^{1},\dots,\mathcal{M}^{|\tilde{\mathcal{C}|}}]$.
Each knowledge item $z_{i}$ is encoded into a key/value pair
$m_{i}=(\texttt{Emb}_{\texttt{Key}}(z_{i}),\texttt{Emb}_{\texttt{Value}}(z_{i}))$
in memory. Each key
$\texttt{Emb}_{\texttt{Key}}(z)=\phi_{\texttt{Key}}\big{(}b(z)\big{)}\in\mathbb{R}^{d}$
is a $d$-dimensional embedding vector encoded via Key Head. Each value is a
sequence of token embeddings representing the full information of knowledge
item $z$. We follow a similar procedure as in [14] to precompute key/value
embeddings of knowledge items from different sources and index them in a
unified knowledge memory. We continuously re-compute the memory key/value
embeddings as the model parameters get updated during the pre-training phase.
We update the memory $\tilde{\mathcal{M}}$ asynchronously at every 1000
training steps.

Scaling Memory by Compression A naive solution for encoding the memory value
is to keep the whole sequence of tokens for each knowledge item. Then, the
generator could fuse the input query and the top-K retrieved memory values by
concatenating all their tokens together and feeding them into a Transformer
Encoder-Decoder pipeline [23]. This approach has two issues: (1) storing
hundreds of millions of knowledge items in memory is impractical given that
each memory value would consist of hundreds of tokens; (2) transformer encoder
has quadratic complexity with respect to the total number of tokens times $K$
for self-attention.

Therefore, we propose to use the Perceiver architecture [19] as the Value Head
to encode and compress knowledge items. The Perceiver model uses a transformer
decoder $\psi(\cdot)$ with learnable $c$-length latent embeddings to compress
the full token sequence into an arbitrary length $c$, such that
$\texttt{Emb}_{\texttt{Value}}(z)=\psi(b(z))\in\mathbb{R}^{c\times d}$ (In our
experiments we use $c=32$). This lets us retrieve top-K memory entries for K
as large as a hundred. To make the compressed embeddings generated by
Perceiver more expressive, we add two additional regularizations. The first
one is a disentangled regularization [16] that forces every two output tokens
to be linearly de-correlated
$\mathcal{L}_{\text{decor}}=\sum_{i,j=1}^{K}\Big{\lVert}\text{Covariance}\big{(}\psi\big{(}b(z_{i})\big{)},\psi\big{(}b(z_{j})\big{)}\big{)}\Big{\rVert}_{F}^{2}$,
and the second one is an alignment regularization that minimizes the distance
of L2-Norm between the query and compressed knowledge embedding:
$\mathcal{L}_{\text{align}}=\bigg{\lvert}1-\frac{\sum_{z}\lVert\psi\big{(}b(z)\big{)}\rVert_{2}}{\sum_{x}\lVert
b(x)\rVert_{2}}\bigg{\rvert}$.

###  3.3 Retriever

Figure 2 (c) shows ReVeaL’s retrieval procedure. Given the input query $x$,
the retriever’s task is to find top-K memory entries $M$ with the highest
probability $p(M\mid x)$ which we approximate as $p(M\mid x)=\prod_{m\in
M}p(m\mid x)$ by retrieving each entry independently. Note that we retrieve
from a large-scale unified memory
$\tilde{\mathcal{M}}=[\mathcal{M}^{1},\dots,\mathcal{M}^{|\tilde{\mathcal{C}|}}]$
that is constructed from a diverse set of knowledge sources. To help the query
to better choose the most appropriate knowledge sources, we learn a gating
function that models the probability of retrieving from each memory corpus.
With the corpus gating, for $m^{j}_{i}\in\mathcal{M}^{j}$ we re-weight
$p(m^{j}\mid x)$ by the computed corpus gating score:

$\displaystyle p(m^{j}_{i}\mid x)=p(\mathcal{M}^{j}\mid x)\cdot
p(m^{j}_{i}\mid x;\mathcal{M}^{j})$ (2)
$\displaystyle=Gate_{\mathcal{M}^{j}}(x)\cdot\frac{\exp\big{(}Rel(x,m^{j}_{i})/\tau\big{)}}{\sum_{m^{j}_{k}\in\mathcal{M}^{j}}\exp\big{(}Rel(x,m^{j}_{k})/\tau\big{)}}$
(3)

where
$Gate_{\mathcal{M}^{j}}(x)=\text{Softmax}(W\cdot\texttt{Emb}_{\texttt{Query}}(x)+b)[j]$
is a softmax gating that assigns a score to each memory corpus $M^{j}$, with
$W$ and $b$ as function parameters. $Rel(x,m^{j}_{i})$ models relevance score
between query $x$ and each memory entry via embedding dot product, such that
$Rel(x,m^{j}_{i})=\texttt{Emb}_{\texttt{Query}}(x)^{T}\cdot\texttt{Emb}_{\texttt{Key}}(z^{j}_{i})$.
where $z_{i}$ is the knowledge item corresponding to the memory entry $m_{i}$
and $\tau$ is the temperature parameter.

After identifying the top-K memory entries, the retriever passes the pre-
computed in-memory key and value embeddings to the generator. In the meantime,
to support end-to-end training of the encoders, we also re-encode a small
portion (i.e., 10$\%$) of the retrieved knowledge items $z_{i}$ from scratch.
In this way, the memory encoders could be updated with moderate computational
cost. We concatenate the re-encoded knowledge with in-memory ones to construct
the final top-K retrieved key/value embeddings.

Figure 3: Detailed procedure of attentive knowledge fusion module. We inject
retrieval probability as a prior to knowledge token embeddings, so the
retriever can receive gradients via back-propagating over
{self/cross}-attention part.

###  3.4 Generator

Figure 2 (d) shows how the query and the retrieved knowledge items are fused
to generate the output answer. All $K$ retrieved memory values are
concatenated with the query embedding, which is feasible due to the Perceiver
module utilized as the value head $\psi(\cdot)$, compressing each knowledge
item into a short sequence. We denote the concatenated query embedding and
memory values as
$X=[b(x),\psi(b(z_{1})),\dots,\psi(b(z_{K}))]\in\mathbb{R}^{(I+c\cdot K)\times
d}$, where $I$ is the number of tokens of the input query $x$ and $c$ is the
number of compressed tokens. To guide the generator towards attending to the
most important items in $X$ and facilitate backpropagation of gradients to the
retriever, we propose an attentive fusion module $f(\cdot)$ capable of
incorporating the retriever score as a prior for calculating cross-knowledge
attention. The detailed procedure is illustrated in Figure 3. We firstly
compute a latent soft attention mask over $X$ as
$\texttt{Mask}_{\texttt{att}}=[1,p(z_{1}|x),\dots,p(z_{K}|x)]$. Finally, we
pass the fused representation $f(X,\texttt{Mask}_{\texttt{att}})$ into a T5
decoder module $g(\cdot)$ to generate the textual output.

##  4 Generative Pre-Training

Knowledge Source | Corpus Size | Type of Text | Avg. Text Length
---|---|---|---
WIT [37] | 5,233,186 | Wikipedia Passage | 258
CC12M [5] | 10,009,901 | Alt-Text Caption | 37
VQA-V2 [12] | 123,287 | Question Answer | 111
WikiData [40] | 4,947,397 | Linearlized Triplets | 326
Table 1: Statistics of the knowledge sources used. Model Name | T5 Variant | Image Encoder | # params. | GFLOPs
---|---|---|---|---
ReVeaL-Base | T5-Base | ViT-B/16 | 0.4B | 120
ReVeaL-Large | T5-Large | ViT-L/16 | 1.4B | 528
ReVeaL | T5-Large | ViT-g/14 | 2.1B | 795
Table 2: Model configuration of different ReVeaL variants.

The existing VQA datasets are not large enough for training a complex multi-
component model like ours from scratch. Therefore, we pre-train our model on a
massive image-text corpus. In Sec. 4.1 we go over the details of our pre-
training data and objective. Then in Sec. 4.2 we introduce the various sources
of knowledge used in our experiments. Finally, in Sec. 4.3 we describe the
pre-training implementation details.

###  4.1 Pre-Training Objective

We pre-train our model on the Web-Image-Text dataset [51], a large-scale
corpus containing 3 billion image alt-text caption pairs collected from the
public Web. Since the dataset is noisy, we add a filter to remove data points
whose captions are shorter than 50 characters. This yields roughly 1.3 billion
image caption pairs for pre-training.

We denote the pre-training Web-Image-Text dataset [51] as $\mathcal{D}$. We
use the text generation objective used in Wang et al. [45]) to pre-train our
model on $\mathcal{D}$. Given an image-text example
$x=$($\texttt{img},\texttt{txt}$) from $\mathcal{D}$, we randomly sample a
prefix length $T_{p}$. We feed $x_{<T_{p}}$ that contains the text prefix and
image to the model as input and our objective is to generate $x_{\geq T_{p}}$
containing the rest of the text as output. The training goal is to condition
on $x_{<T_{p}}$ and autoregressively generate the remaining text sequence
$x_{\geq T_{p}}$:

$\displaystyle\mathcal{L}_{\text{PrefixLM}}=$
$\displaystyle-\mathbb{E}_{x\sim\mathcal{D}}\big{[}{\log p(x_{\geq T_{p}}\mid
x_{<T_{p}})}\big{]}$ (4) $\displaystyle=$
$\displaystyle-\mathbb{E}_{x\sim\mathcal{D}}\Big{[}\sum_{i\geq T_{p}}{\log
p(x_{i}\mid x_{<i})}\Big{]}.$

Warm Starting the Model In order to pre-train all components of our model end-
to-end, we need to warm start the retriever at a good state. Otherwise, if
starting with random weights, the retriever would often return irrelevant
memory items that would never generate useful training signals.

To avoid this cold-start problem, we propose to construct an initial retrieval
dataset with pseudo ground-truth knowledge to give the pre-training a
reasonable head start. We create a modified version of the Wikipedia-Image-
Text (WIT) [37] dataset for this purpose. Each image-caption pair in WIT also
comes with a corresponding Wikipedia passage (words surrounding the text). We
put together the surrounding passage with the query image and use it as the
pseudo ground-truth knowledge that corresponds to the input query. As the
passage provides rich information about the image and caption, it definitely
is useful for initializing the model. To avoid the model from relying on low-
level image features for retrieval, we apply random data augmentation to the
input query image. Given this modified dataset that contains pseudo retrieval
ground-truth, we train the query and memory key embeddings by optimizing the
following contrastive loss:

$\displaystyle\mathcal{L}_{contra}=-\text{logSoftmax}(\texttt{Emb}_{\texttt{Query}}(x)^{T}\texttt{Emb}_{\texttt{Key}}(\hat{z}))$

where $\hat{z}$ represents the pseudo ground-truth knowledge entry
corresponding to the input query $x$.

VQA Model Name | Knowledge Sources | Accuracy ($\%$) | Memory (GB)
---|---|---|---
MUTAN+AN [29] | Wikipedia + ConceptNet | 27.8 | -
ConceptBERT [11] | Wikipedia | 33.7 | -
KRISP [28] | Wikipedia + ConceptNet | 38.4 | -
Visual Retriever-Reader [27] | Google Search | 39.2 | -
MAVEx [46] | Wikipedia+ConceptNet+Google Images | 39.4 | -
KAT-Explicit [13] | Wikidata | 44.3 | 1.5
PICa-Base [48] | Frozen GPT-3 | 43.3 | 350
PICa-Full [48] | Frozen GPT-3 | 48.0 | 350
KAT [13] (Single) | Wikidata + Frozen GPT-3 | 53.1 | 1.5 + 352+ 500
KAT [13] (Ensemble) | Wikidata + Frozen GPT-3 | 54.4 | 4.6 + 352 + 500
ReVIVE [24] (Single) | Wikidata + Frozen GPT-3 | 56.6 | 1.5 + 354 + 500
ReVIVE [24] (Ensemble) | Wikidata+Frozen GPT-3 | 58.0 | 4.6 + 354 + 500
ReVeaL-Base | WIT + CC12M + Wikidata + VQA-2 | 55.2 | 0.8 + 7.5 + 744
ReVeaL-Large | WIT + CC12M + Wikidata + VQA-2 | 58.0 | 2.8 + 10 + 993
ReVeaL | WIT + CC12M + Wikidata + VQA-2 | 59.1 | 4.2 + 10 + 993
Table 3: Visual Question Answering results on OK-VQA, compared with existing
methods that use different knowledge sources. For the memory cost, we assume
all models use bfloat16. Green means on-device model parameters that are
learnable, Blue means on-device memory of frozen model parameters, and Red
means CPU/disk storage cost that are not involved in computation.

###  4.2 Knowledge Sources

We use the following four sources of knowledge in our experiments: Wikipedia-
Image-Text (WIT) [37] consists of the images in Wikipedia, as well as their
alt-text captions and contextualized text passages. Conceptual (CC12M) [5]
contains web images paired with alt-text captions. It includes many long-tail
entities. VQA-v2 [12] is a visual question answering dataset. We merge all
question-answer pairs per image into a single passage. WikiData [40] is a
structural knowledge graph encoding relations between Wikipedia entities. We
linearize all relational triplets per entity into a textual passage following
the procedure of [32]. We have listed the statistical details of these
knowledge sources in Table 1.

###  4.3 Implementation Details

Incorporating all the components introduced above, ReVeaL can be directly pre-
trained over large-scale image caption datasets after proper initialization.
As our model architecture is based on T5 and ViT, we use pre-trained ViT
checkpoints from [50] and pre-trained T5 checkpoints from [33] to initialize
the encoder parameters. The query head, key head and attentive fusion layers
are initialized from upper T5, while the base text encoder is initialized from
lower T5. The combination of these modules can be found in Table 2 for three
three model variants, ReVeaL-Base, ReVeaL-Large and ReVeaL, of which the
largest ReVeaL model has around 2 billion parameters.

Distributed Online Retrieval. Finding the top-k most-relevant knowledge
entries is a standard Maximum Inner Product Search (MIPS) problem. There are
approximate search algorithms [36, 8] that scale sub-linearly with the size of
the knowledge corpus $|C|$. We use TPU-KNN [8] to conduct distributed MIPS
search, by splitting and storing the memory embeddings across all training
devices. The query is synced to each device, which retrieves approximate top-K
results from its own memory. Then these results are combined to compute the
global top-K retrieved items.

Pre-Training Pipeline. We first train the multimodal retriever on our modified
version of the Wikipedia Image Text (WIT) dataset via $\mathcal{L}_{contra}$.
We use the Adafactor optimizer without momentum ($\beta_{1}=0$,
$\beta_{2}=0.999$), with weight decay of 0.001222The remaining experiments use
the same optimizer configuration., and with a peak learning rate of $6e4$, to
train for 10 epochs. We use this checkpoint to warm-start our generative pre-
training. We set the number of retrieved knowledge entries as $K=10$ during
pre-training, and use adafactor with a peak learning rate of $1e{-3}$ and
inverse squared root learning rate scheduler with 10,000 linear warm-up steps.
We use $\mathcal{L}_{PrefixLM}$ as the main objective, adding
$\mathcal{L}_{contra}$, $\mathcal{L}_{decor}$ and $\mathcal{L}_{align}$
weighted by $0.01$. We use a batch size of 4096 across 256 CloudTPUv4 chips
and train for about 5 days.

##  5 Experimental Results

VQA Model Name | Accuracy ($\%$)
---|---
ViLBERT [26] | 30.6
LXMERT [38] | 30.7
ClipCap [30] | 30.9
KRISP [28] | 33.7
GPV-2 [21] | 48.6
ReVeaL-Base | 50.4
ReVeaL-Large | 51.5
ReVeaL | 52.2
Table 4: Visual Question Answering results on A-OKVQA.

We evaluate our proposed method on knowledge-based VQA in Sec. 5.1 and image
captioning in Sec. 5.2. We then conduct ablation studies in Sec. 5.3 to
analyze the impact of each model component on overall performance.

###  5.1 Evaluating on Knowledge-Based VQA

One of the most knowledge intensive visual-language tasks is knowledge-based
visual question answering (VQA), exemplified by the OK-VQA [29] and A-OKVQA
[35] benchmarks. To finetune our pre-trained model on these VQA tasks, we use
the same generative objective where the model takes in an image question pair
as input and generates the text answer as output. There are a few differences
between the fine-tuning and the pre-training stages: 1) we set the number of
retrieved knowledge entries to $K=50$, so the model is able to retrieve
sufficient supporting evidence; 2) we freeze the whole base V-L encoder to
stabilize training; and 3) we use a batch size of 128, with the Adafactor
optimizier, a peak learning rate of 1e-4. We use the soft VQA accuracy metric
[3] to evaluate the model’s generated answer.

Our results on OKVQA and A-OKVQA datasets are shown in Table 3 and Table 4
respectively. For OKVQA, earlier attempts that incorporate a fixed knowledge
retriever report results that are below 45%. Recently a series of works
utilize large language models (_e.g_. GPT-3) as implicit knowledge sources,
which achieve much better performance with the trade-off of a huge
computational cost. ReVeaL achieves higher performance than those methods
without relying on such large language models333As shown in the last column of
Table 3, ReVeaL stores external knowledge as value embeddings on disk,
occupying 993GB of space. The key embeddings consume 10GB space and are kept
in TPU memory for fast lookup. On the other hand, KAT and REVIVE need to load
the entire 350GB GPT-3 model in the GPU/TPU memory. Furthermore, storing
WikiData on disk consumes 500GB of disk memory.. Compared with the previous
state-of-the-art, KAT and ReVIVE, which also utilizes T5-Large as a generator,
ReVeaL achieves accuracy of 59.1%, which is +6.0% higher than the single KAT
[13] model and +2.5% higher than ReVIVE [24].

On A-OKVQA, ReVeaL achieves 52.2% accuracy, which is +3.6% higher than the
previous best, GPV-2 [21]. We also show two examples of these datasets in
Figure 4. All these results show that, with proper end-to-end retrieval
training and a diverse set of knowledge sources, ReVeaL can learn to retrieve
meaningful knowledge entries, and achieve promising results without relying on
a large language model.

###  5.2 Evaluating on Image Captioning

We also evaluate ReVeaL on image captioning benchmarks: MSCOCO Captions [6]
and NoCaps [1]. We follow the evaluation protocol used in [49]. We directly
fine-tune our generator model on the MSCOCO training split via cross-entropy
generative objective. We measure our performance on the MSCOCO test split and
NoCaps val set with the CIDEr metric [39]. The results of these two datasets
are shown in Table 5. Note that ReVeaL achieves better results than strong
recent baselines such as SimVLM [45] and CoCa [49] on both benchmarks.
Notably, ReVeaL $-{\text{Large}}$ with 1.4B parameters outperforms the
2.1B-parameter CoCa model and is significantly better than 80B-parameter
Flamingo model [2].

Model Name | MSCOCO | NoCaps | # params.
---|---|---|---
Flamingo [2] | 138.1 | - | 80B
VinVL [52] | 140.9 | 105.1 | 0.4B
SimVLM [45] | 143.3 | 112.2 | 1.5B
CoCa [49] | 143.6 | 122.4 | 2.1B
ReVeaL-Base | 141.1 | 115.8 | 0.4B
ReVeaL-Large | 144.5 | 121.3 | 1.4B
ReVeaL | 145.4 | 123.0 | 2.1B
Table 5: Image Captioning results on MSCOCO (Karpathy-test split) and NoCaps
(val set). Evaluated using the CIDEr metric. Figure 4: VQA Examples. ReVeaL is
able to use knowledge from different sources to correctly answer the question.
We show more examples in Figure 1-3 of Supplementary Material, indicating that
our model can retrieve and use items from diverse knowledge sources to
correctly solve different input query.

###  5.3 Analyzing Effects of Key Model Components

In the following we study which design choices contribute most to the model’s
performance. We focus on three research questions: (1) Does utilizing multiple
knowledge sources enhance performance? (2) Does the proposed attentive fusion
surpass existing end-to-end retrieval training methods? (3) Can we add
knowledge by only updating the memory without modifying model parameters?

Analyzing multiple knowledge sources. A major distinction of ReVeaL compared
to previous retrieval-augmented approaches is its capacity to utilize a
diverse set of knowledge sources during inference. To assess the relative
importance of each data source and the efficacy of retrieving from various
corpora, we conduct two ablation studies: 1) Only-One-Left: employing a single
knowledge source to evaluate the outcomes; and 2) Leave-One-Out: excluding one
knowledge source from the complete set $\mathcal{C}$. These ablation studies
are executed using the ReVeaL ${}_{\text{Base}}$, evaluated on the OKVQA
validation set under the aforementioned conditions. As shown in Figure 6,
among the four knowledge sources utilized in this paper, WIT is the most
informative, with the highest accuracy when used in isolation (53.1The
remaining three corpora, CC12M, VQA-v2, and WikiData, do not offer the same
level of informativeness as WIT when utilized independently. However,
excluding any of these corpora from the complete dataset results in
performance decreases of 1.3%, 0.6%, and 1.1%, respectively. This observation
implies that these knowledge sources effectively complement one another,
contributing valuable information to enhance performance. To further
substantiate this hypothesis, we perform an additional experiment involving
pairs of knowledge sources, as illustrated in Figure 6. Notably, even when
paired with an informative knowledge source such as WIT, incorporating an
extra corpus consistently leads to performance improvements.

Figure 5: OKVQA Accuracy of ReVeaL using 1) Only-One-Left: only use a single
knowledge source; 2) Leave-One-Out: use all without this knowledge source.

Figure 6: OKVQA Accuracy of ReVeaL using all Pair of Knowledge Sources.
Results show that combining multiple sources could consistently improve
performance.

Analyzing different retrieval training methods. Another core component of
ReVeaL is the attentive fusion layer, which supports efficient joint training
of the retriever and generator. We investigate its performance compared to two
existing retrieval training method categories: 1) a frozen retriever based on
ALIGN [20] representations ; 2) end-to-end retrieval training methods
including Attention Distill [17], EMDR2 [47], and Perplexity Distill [18].

We use the pre-trained ReVeaL-Base model, fix the generator and randomly
initialize the retriever (query head and key head). We utilize our modified
version of WIT dataset with pseudo ground-truth retrieved labels as the
evaluation corpus. We evaluate retrieval performance by checking whether the
correct passage appears in top-10/100 results. For the ALIGN model, we
directly evaluate the retrieval results from the pre-trained checkpoint, while
for other models, we perform retrieval-augmented training on the WIT dataset.
To prevent the model from relying on image similarity for accurate results, we
only use text passages as knowledge entries and discard images. Subsequently,
we finetune the model on OKVQA and report its accuracy. The results are
presented in Table 6. We observe that directly using pre-trained encoder does
not perform well, even with a strong model like ALIGN. Moreover, among the
various end-to-end retrieval training approaches, our attentive fusion method
attains better accuracy in both retrieval and OKVQA tasks. Importantly, our
technique exhibits a computational cost (quantified by GFLOPs) comparable to
that of attention distillation, yet significantly lower than EMDR2 and
Perplexity distillation. This indicates that our proposed method is more
efficient and effective for pre-training retrieval-augmented visual-language
models.

Retrieval Method | Acc@10 | Acc@100 | OKVQA Acc. | GFLOPs
---|---|---|---|---
ALIGN [20] (fixed) | 0.638 | 0.793 | 44.7 | -
Attention Distill [17] | 0.674 | 0.835 | 45.9 | 119
EMDR2 [47] | 0.691 | 0.869 | 46.5 | 561
Perplexity Distill [18] | 0.704 | 0.886 | 46.7 | 561
Ours (Attentive Fusion) | 0.726 | 0.894 | 47.3 | 120
Table 6: Analysis of Retrieval Training Method: We train ReVeaL-Base (frozen
generator, only train randomly initialized retriever) to retrieve from the WIT
dataset (only text passage without image), and show the retrieval accuracy at
the first 10 or 100 results, as well as fine-tuned OKVQA accuracy. Figure 7:
Study of Knowledge Update. The blue curve shows result by removing certain
percentage of knowledge during both fine-tuning and inference stage. The
orange curve shows results by still first removing the knowledge, and then
adding the knowledge back during inference, which simulates the knowledge
update.

Analyzing Knowledge Modification. One advantage of utilizing knowledge memory
is that we could easily add or update knowledge entries without re-training
model’s parameters. To validate this, we conducted ablation studies in which
we removed a specific percentage of knowledge entries from the corpora and
assessed the performance of the ReVeaL-Base model on the OKVQA dataset.
Subsequently, we add the removed knowledge back into the corpora, allowing the
trained model to make predictions using the complete set of corpora. This
approach ensured that the removed knowledge was not seen by the model during
fine-tuning, enabling us to test its ability to accurately retrieve and
utilize that knowledge for problem-solving.

The results are illustrated in Figure 7, with the blue curves representing the
inference outcomes without the removed knowledge and the orange curve
depicting the results after adding the removed knowledge back. A notable
performance improvement was observed upon reintroducing the knowledge (orange
curve) compared to the outcomes with the removed knowledge (blue curve).
Specifically, for the model fine-tuned with only 10% of the knowledge, the
reintroduction of the removed knowledge resulted in an accuracy of 51.8 (+6.7
higher than when removed). This finding demonstrates that the ReVeaL model can
swiftly adapt to new knowledge by merely updating the memory, obviating the
need for re-training model parameters.

##  6 Conclusion

This paper presents an end-to-end Retrieval-augmented Visual Language model
(ReVeaL), which contains a knowledge retriever that learns to utilize a
diverse set of knowledge sources with different modality. The retriever is
trained jointly with the generator to return multiple knowledge entries. We
pre-train ReVeaL on a massive image-text corpus with four diverse knowledge
corpora, and achieves state-of-the-art results on knowledge-intensive visual
question answering and image caption tasks. In the future we’d explore the
ability of this model to be used for attribution, and applying it to broader
class of multimodal tasks.

## References

  * [1] Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee.  nocaps: novel object captioning at scale.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 8947–8956. IEEE, 2019.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  CoRR, abs/2204.14198, 2022.
  * [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh.  VQA: visual question answering.  In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society, 2015.
  * [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In CVPR, 2021.
  * [6] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO captions: Data collection and evaluation server.  ArXiv preprint, abs/1504.00325, 2015.
  * [7] Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  CoRR, abs/2209.06794, 2022.
  * [8] Felix Chern, Blake Hechtman, Andy Davis, Ruiqi Guo, David Majnemer, and Sanjiv Kumar.  TPU-KNN: K nearest neighbor search at peak flop/s.  CoRR, abs/2206.14286, 2022.
  * [9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  CoRR, abs/2204.02311, 2022.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [11] François Gardères, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lecue.  ConceptBert: Concept-aware representation for visual question answering.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489–498, Online, 2020. Association for Computational Linguistics.
  * [12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325–6334. IEEE Computer Society, 2017.
  * [13] Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, and Jianfeng Gao.  KAT: A knowledge augmented transformer for vision-and-language.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 956–968, Seattle, United States, 2022. Association for Computational Linguistics.
  * [14] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: retrieval-augmented language model pre-training.  ArXiv preprint, abs/2002.08909, 2020.
  * [15] Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun.  Empowering language models with knowledge graph reasoning for open-domain question answering.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9562–9581. Association for Computational Linguistics, 2022.
  * [16] Ziniu Hu, Zhe Zhao, Xinyang Yi, Tiansheng Yao, Lichan Hong, Yizhou Sun, and Ed H. Chi.  Improving multi-task generalization via regularizing spurious correlation.  CoRR, abs/2205.09797, 2022.
  * [17] Gautier Izacard and Edouard Grave.  Distilling knowledge from reader to retriever for question answering.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [18] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave.  Few-shot learning with retrieval augmented language models.  CoRR, abs/2208.03299, 2022.
  * [19] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira.  Perceiver: General perception with iterative attention.  In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4651–4664. PMLR, 2021.
  * [20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904–4916. PMLR, 2021.
  * [21] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi.  Webly supervised concept expansion for general purpose vision models.  In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, volume 13696 of Lecture Notes in Computer Science, pages 662–681. Springer, 2022.
  * [22] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, 2020. Association for Computational Linguistics.
  * [23] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [24] Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan.  REVIVE: regional visual representation matters in knowledge-based visual question answering.  CoRR, abs/2206.01201, 2022.
  * [25] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel.  Retrieval augmented classification for long-tail visual recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 6949–6959. IEEE, 2022.
  * [26] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13–23, 2019.
  * [27] Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral.  Weakly-supervised visual-retriever-reader for knowledge-based question answering.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6417–6431, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
  * [28] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach.  KRISP: integrating implicit and symbolic knowledge for open-domain knowledge-based VQA.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 14111–14121. Computer Vision Foundation / IEEE, 2021.
  * [29] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195–3204. Computer Vision Foundation / IEEE, 2019.
  * [30] Ron Mokady, Amir Hertz, and Amit H. Bermano.  Clipcap: CLIP prefix for image captioning.  ArXiv preprint, abs/2111.09734, 2021.
  * [31] Medhini Narasimhan, Svetlana Lazebnik, and Alexander G. Schwing.  Out of the box: Reasoning with graph convolution nets for factual visual question answering.  In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2659–2670, 2018.
  * [32] Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Sejr Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih.  Unified open-domain question answering with structured and unstructured knowledge.  ArXiv preprint, abs/2012.14610, 2020.
  * [33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21:140:1–140:67, 2020.
  * [34] Devendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer, and Dani Yogatama.  End-to-end training of multi-document reader and retriever for open-domain question answering.  In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 25968–25981, 2021.
  * [35] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-OKVQA: A benchmark for visual question answering using world knowledge.  CoRR, abs/2206.01718, 2022.
  * [36] Anshumali Shrivastava and Ping Li.  Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS).  In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2321–2329, 2014.
  * [37] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  WIT: wikipedia-based image text dataset for multimodal multilingual machine learning.  In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2443–2449. ACM, 2021.
  * [38] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, Hong Kong, China, 2019. Association for Computational Linguistics.
  * [39] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566–4575. IEEE Computer Society, 2015.
  * [40] Denny Vrandecic and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  Commun. ACM, 57(10):78–85, 2014.
  * [41] Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel.  Explicit knowledge-based reasoning for visual question answering.  In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1290–1296. ijcai.org, 2017.
  * [42] Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel.  FVQA: fact-based visual question answering.  IEEE Trans. Pattern Anal. Mach. Intell., 40(10):2413–2427, 2018\.
  * [43] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  CoRR, abs/2208.10442, 2022.
  * [44] Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec.  VQA-GNN: reasoning with multimodal semantic graph for visual question answering.  CoRR, abs/2205.11501, 2022.
  * [45] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [46] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi.  Multi-modal answer validation for knowledge-based VQA.  In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 2712–2721. AAAI Press, 2022.
  * [47] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin.  End-to-end open-domain question answering with BERTserini.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
  * [48] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.  An empirical study of GPT-3 for few-shot knowledge-based VQA.  ArXiv preprint, abs/2109.05014, 2021.
  * [49] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  CoRR, abs/2205.01917, 2022.
  * [50] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1204–1213. IEEE, 2022.
  * [51] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18102–18112. IEEE, 2022.
  * [52] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579–5588. Computer Vision Foundation / IEEE, 2021.