# Normally located polyhedra

Ivan Arzhantsev  Faculty of Computer Science, HSE University, Pokrovsky
Boulevard 11, Moscow, 109028 Russia<EMAIL_ADDRESS>
###### Abstract.

Lattice polyhedra $Q_{1}$ and $Q_{2}$ with the same tail cone are said to be
normally located if every lattice point in the Minkowski sum $Q_{1}+Q_{2}$ is
the sum of lattice points from $Q_{1}$ and $Q_{2}$, respectively. We prove
that if the normal fan of $Q_{1}$ refines the normal fan of $Q_{2}$, then
there is a positive integer $k$ such that for any positive integer $s$ the
polyhedra $skQ_{1}$ and $skQ_{2}$ are normally located. This result is based
on an interpretation of the problem in terms of graded algebras and earlier
results on surjectivity of the multiplicaiton map on homogeneous components.
Also we provide an example of two lattice triangles $P$ and $Q$ on the plane
such that for any positive integer $k$ the triangles $kP$ and $kQ$ are not
normally located.

###### Key words and phrases:

Polytope, polyhedron, lattice, Minkowski sum, graded algebra, toric variety

###### 2010 Mathematics Subject Classification:

Primary 11P21, 52B20; Secondary 14M25, 52B11

##  1\. Introduction

Let us consider the lattice ${\mathbb{Z}}^{d}$ and the rational vector space
${\mathbb{Q}}^{d}$ generated by ${\mathbb{Z}}^{d}$. By a _lattice polytope_ we
mean a convex polytope $P$ in ${\mathbb{Q}}^{d}$ with vertices in
${\mathbb{Z}}^{d}$. Let us assume additionally that the lattice points in $P$
generate the lattice ${\mathbb{Z}}^{d}$; this can be achieved by replacing
${\mathbb{Z}}^{d}$ with a proper sublattice.

It is easy to construct examples of two lattice polytopes $P$ and $Q$ such
that the Minkowski sum $P+Q$ contains a lattice point that is not a sum of
lattice points from $P$ and $Q$, respective; see, e.g., [8]. Moreover,
starting from dimension $3$ it may happen even when $P=Q$.

######  Definition 1.

A lattice polytope $P$ is called _normal_ if for every positive integer $s$
and every lattice point $z\in sP$ there are lattice points
$z_{1},\ldots,z_{s}\in P$ such that $z=z_{1}+\ldots+z_{s}$.

Normal polytopes play an important role in many areas of modern mathematics,
see [5] for a recent survey on this subject. In particular, such polytopes
define integrally closed graded monoid algebras and projectively normal
embeddings of projective toric varieties.

Let us recall that a lattice polytope $P$ is _smooth_ if the primitive edge
vectors at every vertex of $P$ form a basis of ${\mathbb{Z}}^{d}$. Smooth
polytopes correspond to projective embeddings of smooth toric varieties. Oda’s
question [5, Section 1] asks whether every smooth polytope is normal. This
question is open in all dimensions $\geqslant 3$.

The proof of the following theorem may be found in [3, Proposition 1.3.3] or
[4].

######  Theorem 1.

For every lattice polytope $P$ in ${\mathbb{Q}}^{d}$ the polytope $(d-1)P$ is
normal.

In particular, in dimension $2$ any lattice polygon is normal.

The aim of this paper is to generalize the property of normality to a pair of
polytopes.

######  Definition 2.

Lattice polytopes $P$ and $Q$ in ${\mathbb{Q}}^{d}$ are said to be _normally
located_ if for every lattice point $z\in P+Q$ there are lattice points
$z^{\prime}\in P$ and $z^{\prime\prime}\in Q$ such that
$z=z^{\prime}+z^{\prime\prime}$.

Oda’s Conjecture (see [8] or [6, Section 1]) states the following. Let $P$ and
$Q$ be lattice polytopes. Assume that $P$ is smooth and the normal fan of $Q$
refines that of $P$. Then the polytopes $P$ and $Q$ are normally located. This
is a generalization of Oda’s question on normality of a smooth polytope.

In [6, Theorem 1.1], it is shown that if $P$ and $Q$ are lattice polygons such
that the normal fan of $Q$ refines that of $P$, then $P$ and $Q$ are normally
located. In this paper we prove a version of Oda’s Conjecture (Theorem 3)
which generalizes [6, Theorem 1.1] to higher dimensions. The result is
obtained not only for polytopes, but also for polyhedra with the same tail
cone.

The paper is organized as follows. In Section 2 we give an explicit example of
two lattice triangles $P$ and $Q$ on the plane such that for any positive
integer $k$ the triangles $kP$ and $kQ$ are not normally located. This example
is intended to demonstrate that, unlike the normality property, the property
of normal location cannot always be achieved just by rescaling two given
polytopes.

In Section 3 we recall basic definitions and facts on polyhedra, their
Minkowski sums and normal fans. Also we consider polyhedra that appear as
fibers of a projection of the positive octant in a bigger lattice to a smaller
lattice. Section 4 is devoted to interpretations of the objects defined above
in terms of graded algebras and the multiplication map on homogeneous
components. We recall the results of [1] that allow to relate the property of
normal location (up to scalar) of a pair of polyhedra in fibers over two
points with the location of these points with respect to the so-called GIT-
fan. Finally, in Section 5 we explain how to realize a pair of polyhedra via
the fiber construction and prove our main result (Theorem 3).

##  2\. An example in dimension $2$

Let us show that, in contrast to Theorem 1, for two lattice triangles absence
of the property of normal location is not just a question of scale.

######  Proposition 1.

There are two lattice triangles $P$ and $Q$ in ${\mathbb{Q}}^{2}$ such that
for every positive integer $k$ the triangles $kP$ and $kQ$ are not normally
located.

###### Proof.

Take $P=\operatorname{Conv}(a_{1},a_{2},a_{3})$ and
$Q=\operatorname{Conv}(b_{1},b_{2},b_{3})$, where

$a_{1}=(165,0),\ a_{2}=(175,0),\ a_{3}=(0,385)\quad\text{and}\quad
b_{1}=(0,0),\ b_{2}=(35,0),\ b_{3}=(0,77).$

Equivalently, the triangle $P$ is given by inequalities

$7x+3y\geqslant 1155,\quad 11x+5y\leqslant 1925,\quad y\geqslant 0$

and $Q$ is given by

$11x+5y\leqslant 385,\quad x\geqslant 0,\quad y\geqslant 0.$

The Minkowski sum $P+Q$ is $\operatorname{Conv}(c_{1},c_{2},c_{3},c_{4})$,
where

$c_{1}=(165,0),\quad c_{2}=(210,0),\quad c_{3}=(0,385),\quad c_{4}=(0,462).$

The inequalites that determine $P+Q$ are

$7x+3y\geqslant 1155,\quad 11x+5y\leqslant 2310,\quad x\geqslant 0,\quad
y\geqslant 0.$

It is easy to check that for every positive integer $k$ the point
$s=(1,385k-2)$ is contained in $kP+kQ$. Assume that $s=p+q$, where $p\in
kP\cap{\mathbb{Z}}^{2}$ and $q\in kQ\cap{\mathbb{Z}}^{2}$. Then only two cases
are possible.

Case 1. Let $p=(1,a)$ and $q=(0,385k-2-a)$ for some non-negative integer $a$.
Since $p$ is contained in $P$, we have

$7+3a\geqslant 1155k\quad\text{and}\quad 11+5a\leqslant 1925k.$

These inequalities may be rewritten as

$a\geqslant 385k-2-\frac{1}{3}\quad\text{and}\quad a\leqslant
385k-2-\frac{1}{5}.$

Since $a$ is integer, we come to a contradiction.

Case 2. Let $p=(0,a)$ and $q=(1,385k-2-a)$ for some non-negative integer $a$.
Since $p$ is contained in $P$, we have

$3a\geqslant 1155k\quad\text{and}\quad 5a\leqslant 1925k,$

so

$a\geqslant 385k\quad\text{and}\quad a\leqslant 385k.$

We conclude that $a=385k$. Since $q$ lies in $Q$, we have

$385k-2-a=-2>0.$

These two contradictions complete the proof. ∎

##  3\. Generalities on polyhedra

By a _polyhedron_ in ${\mathbb{Q}}^{d}$ we mean the intersection of finitely
many closed affine half spaces. A _lattice polyhedron_ is a polyhedron in
${\mathbb{Q}}^{d}$ whose vertices are in ${\mathbb{Z}}^{d}$. Note that a
polytope can be defined as a bounded polyhedron in ${\mathbb{Q}}^{d}$. For a
polyhedron $Q$ we define its _relative interior_ as the set obtained by
removing all proper faces from $Q$. Let us denote the relative interior of $Q$
by $Q^{\circ}$. Further, a _cone_ in ${\mathbb{Q}}^{d}$ is the intersection of
finitely many closed linear half spaces. A cone is _pointed_ if it contains no
line. If a cone has dimension at least $2$, it is pointed if and only if it is
generated by its one-dimensional faces.

The set of all polyhedra in ${\mathbb{Q}}^{d}$ comes with a natural structure
of a commutative semigroup: one defines the Minkowski sum of two polyhedra
$Q_{1}$ and $Q_{2}$ to be the polyhedron

$Q_{1}+Q_{2}:=\\{w_{1}+w_{2};\ w_{1}\in Q_{1},w_{2}\in Q_{2}\\}.$

In the same way one may define the Minkowski sum of two arbitrary subsets in
${\mathbb{Q}}^{d}$.

Any polyhedron allows a Minkowski sum decomposition $Q=P+\sigma$, where $P$ is
a polytope and $\sigma$ is a cone in ${\mathbb{Q}}^{d}$. In this
decomposition, the _tail cone_ $\sigma$ is unique; it is given by

$\sigma=\\{w\in{\mathbb{Q}}^{d};\ w^{\prime}+tw\in Q\ \text{for all}\
w^{\prime}\in Q,t\in{\mathbb{Q}}_{\geqslant 0}\\}.$

A polyhedron with the tail cone $\sigma$ is called a _$\sigma$ -polyhedron_.
For example, polytopes are precisely $\sigma$-polyhedra with $\sigma=\\{0\\}$.

It is easy to check that for a fixed cone $\sigma$ the set of all
$\sigma$-polyhedra forms a commutative semigroup with respect to the Minkowski
sum.

Let us recall that with any $\sigma$-polyhedron $Q$ in ${\mathbb{Q}}^{d}$ one
may associate the normal fan $\mathcal{N}(Q)$: any point $v\in Q$ defines the
cone $\tau(v)$ consisting of all linear functions on ${\mathbb{Q}}^{d}$ which
reach their maximal value on $Q$ at the point $v$. The collection of cones
$\mathcal{N}(Q)=\\{\tau(v)\,|\,v\in Q\\}$ is finite and it is a fan in a sense
that a face of any cone in $\mathcal{N}(Q)$ is contained in $\mathcal{N}(Q)$
and the intersection of any two cones in $\mathcal{N}(Q)$ is a face of each of
them. Moreover, all cones in $\mathcal{N}(Q)$ are pointed if and only if $Q$
has full dimension in ${\mathbb{Q}}^{d}$.

The _support_ of a fan $\mathcal{N}$ is the union of all cones in
$\mathcal{N}$. The support of the normal fan $\mathcal{N}(Q)$ equals the dual
cone

$\sigma^{\lor}:=\\{l\in({\mathbb{Q}}^{d})^{*}\ |\ l|_{\sigma}\geqslant 0\\}.$

We say that a fan $\mathcal{N}_{1}$ _refines_ a fan $\mathcal{N}_{2}$, if
every cone in $\mathcal{N}_{1}$ is contained in some cone in
$\mathcal{N}_{2}$. Let $\mathcal{N}_{1}$ and $\mathcal{N}_{2}$ be two fans
with the same support. The _coarsest common refinement_ of $\mathcal{N}_{1}$
and $\mathcal{N}_{2}$ is the fan $\mathcal{N}$ with the same support, whose
cones are $\tau_{1}\cap\tau_{2}$, where $\tau_{1}\in\mathcal{N}_{1}$ and
$\tau_{2}\in\mathcal{N}_{2}$.

It is well-known that for normal fans $\mathcal{N}(Q_{1})$ and
$\mathcal{N}(Q_{2})$ of two $\sigma$-polyhedra $Q_{1}$ and $Q_{2}$ the
coarsest common refinement $\mathcal{N}$ coincides with the normal fan
$\mathcal{N}(Q_{1}+Q_{2})$.

Now let us consider a surjective homomorphism of lattices
$\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$ and the induced linear map of
vector spaces $\pi\colon{\mathbb{Q}}^{n}\to{\mathbb{Q}}^{m}$. We denote by
${\mathbb{Q}}^{n}_{\geqslant 0}$ the cone of vectors in ${\mathbb{Q}}^{n}$
with non-negative coordinates and let $C:=\pi({\mathbb{Q}}^{n}_{\geqslant
0})\subseteq{\mathbb{Q}}^{m}$. Consider the cone
$\sigma:=\pi^{-1}(0)\cap{\mathbb{Q}}^{n}_{\geqslant 0}$. With any point $u\in
C$ one associates the polyhedron
$P(u):=\pi^{-1}(u)\cap{\mathbb{Q}}^{n}_{\geqslant 0}$.

######  Lemma 1.

For any $u\in C$ the polyhedron $P(u)$ is a $\sigma$-polyhedron.

###### Proof.

For a vector $w\in{\mathbb{Q}}^{n}$, the condition $w^{\prime}+tw\in P(u)$ for
all $w^{\prime}\in P(u)$ and all $t\in{\mathbb{Q}}_{\geqslant 0}$ means that
$w\in{\mathbb{Q}}^{n}_{\geqslant 0}$ and
$u=\pi(w^{\prime}+tw)=\pi(w^{\prime})+\pi(tw)=u+t\pi(w)$. It is equivalent to
$w\in{\mathbb{Q}}^{n}_{\geqslant 0}$ and $\pi(w)=0$, or $w\in\sigma$. ∎

Clearly, for every $u\in C$ there is a positive integer $r$ such that $P(ru)$
is a lattice polyhedron. Also it is easy to check that $P(u_{1})+P(u_{2})$ is
contained in $P(u_{1}+u_{2})$ for all $u_{1},u_{2}\in C$.

We are interested in the following three properties of a pair $(u_{1},u_{2})$
with $u_{1},u_{2}\in C$:

(P1) $P(u_{1})+P(u_{2})=P(u_{1}+u_{2})$;

(P2)
$(P(u_{1})\cap{\mathbb{Z}}^{n})+(P(u_{2})\cap{\mathbb{Z}}^{n})=P(u_{1}+u_{2})\cap{\mathbb{Z}}^{n}$;

(P3) there exists $k\in{\mathbb{Z}}_{>0}$ such that for any
$s\in{\mathbb{Z}}_{>0}$ we have

$(P(sku_{1})\cap{\mathbb{Z}}^{n})+(P(sku_{2})\cap{\mathbb{Z}}^{n})=P(sk(u_{1}+u_{2}))\cap{\mathbb{Z}}^{n}.$

##  4\. Graded algebras

In this section we introduce an algebraic interpretation of the objects
discussed above. The projection $\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$
gives rise to an effective ${\mathbb{Z}}^{m}$-grading on the polynomial
algebra $A:={\mathbb{K}}[x_{1},\ldots,x_{n}]$. Namely, we put
$\deg(x_{i})=\pi(e_{i})$, where $e_{1},\ldots,e_{n}$ is the standard basis of
the lattice ${\mathbb{Z}}^{n}$. For further purposes we assume the ground
field ${\mathbb{K}}$ to be an algebraically closed field of characteristic
zero.

######  Remark 1.

The Linearization Problem [7] claims that up to automorphism any effective
${\mathbb{Z}}^{m}$-grading on ${\mathbb{K}}[x_{1},\ldots,x_{n}]$ is obtained
this way.

Below we follow the presentation given in [1] in a somewhat more general
situation. Let $A$ be an associative, commutative, integral, finitely
generated algebra with unit over ${\mathbb{K}}$. Suppose that $A$ is graded by
the lattice ${\mathbb{Z}}^{m}$, i.e., we have

$\displaystyle A$ $\displaystyle=$
$\displaystyle\bigoplus_{u\in{\mathbb{Z}}^{m}}A_{u}\ \ \ \ \text{with}\ \ \ \
A_{u_{1}}\cdot A_{u_{2}}\subseteq A_{u_{1}+u_{2}}.$

By the _weight cone_ of $A$ we mean the cone $C(A)\subseteq{\mathbb{Q}}^{m}$
generated by all $u\in{\mathbb{Z}}^{m}$ with $A_{u}\neq 0$. We investigate the
following problem: given $u_{1},u_{2}\in C(A)\cap{\mathbb{Z}}^{m}$, does there
exist an integer $k>0$ such that for any $s>0$ the multiplication map

$A_{sku_{1}}\otimes_{\mathbb{K}}A_{sku_{2}}\ \to\ A_{sk(u_{1}+u_{2})},\qquad
f\otimes g\ \mapsto\ fg$

is surjective? We call a pair $u_{1},u_{2}\in C(A)\cap{\mathbb{Z}}^{m}$
_generating_ if it has this property. If $A$ is a polynomial algebra with
${\mathbb{Z}}^{m}$-grading given by the projection $\pi$, this is precisely
property (P3) for a pair $(u_{1},u_{2})$ of lattice points in $C$.

Let us recall from [2] the concept of the GIT-fan associated to a graded
algebra. A ${\mathbb{Z}}^{m}$-grading on $A$ defines an action of the torus
$T:=\operatorname{Spec}({\mathbb{K}}[{\mathbb{Z}}^{m}])$ on
$X:=\operatorname{Spec}(A)$ such that for any $u\in{\mathbb{Z}}^{m}$, the
elements $f\in A_{u}$ are precisely the semiinvariants of the character
$\chi^{u}\colon T\to{\mathbb{K}}^{*}$, i.e., each $f\in A_{u}$ satisfies

$\displaystyle f(t\\!\cdot\\!x)$ $\displaystyle:=$
$\displaystyle\chi^{u}(t)f(x).$

The _orbit cone_ of a (closed) point $x\in X$ is the cone
$\omega(x)\subseteq{\mathbb{Q}}^{m}$ generated by all $u\in C(A)$ admitting an
$f\in A_{u}$ with $f(x)\neq 0$. The collection of orbit cones is finite, and
thus one may associate to any element $u\in C(A)$ its _GIT-cone_ :

$\displaystyle\lambda(u)$ $\displaystyle:=$
$\displaystyle\bigcap_{\begin{subarray}{c}x\in X,\\\
u\in\omega(x)\end{subarray}}\omega(x).$

These GIT-cones cover the weight cone $C(A)$ and, by [2, Theorem 3.11], the
collection $\Lambda(A)$ of all GIT-cones is a fan in the sense that if
$\lambda\in\Lambda(A)$ then also every face of $\lambda$ belongs to
$\Lambda(A)$, and for $\tau,\lambda\in\Lambda(A)$ the intersection
$\tau\cap\lambda$ is a face of both $\lambda$ and $\tau$. Note that we allow
here a fan to have cones containing lines.

######  Theorem 2.

[1, Theorem 1.1]

  1. (1)

If $u_{1},u_{2}\in C(A)\cap{\mathbb{Z}}^{m}$ is a generating pair, then the
weights $u_{1},u_{2}$ lie in a common GIT-cone $\lambda\in\Lambda(A)$.

  2. (2)

If $u_{1},u_{2}\in C(A)\cap{\mathbb{Z}}^{m}$ lie in a common GIT-cone
$\lambda\in\Lambda(A)$ and $u_{1}$ belongs to the relative interior
$\lambda^{\circ}$, then $(u_{1},u_{2})$ is a generating pair.

If two weights $u_{1},u_{2}\in C(A)\cap{\mathbb{Z}}^{m}$ lie on the boundary
of a common GIT-cone $\lambda\in\Lambda(A)$, then no general statement in
terms of the GIT-fan is possible: it may happen that $u_{1},u_{2}$ is
generating, and also it may happen that $u_{1},u_{2}$ is not generating. For
the first case there are obvious examples, and for the latter we present the
following one.

######  Example 1.

[1, Example 1.2] Consider the polynomial ring
$A:={\mathbb{K}}[x_{1},x_{2},x_{3},x_{4}]$. Then one may define a
${\mathbb{Z}}^{2}$-grading on $A$ by setting

$\deg(x_{1})\ :=\ (4,1),\quad\deg(x_{2})\ :=\ (2,1),\quad\deg(x_{3})\ :=\
(1,2),\quad\deg(x_{4})\ :=\ (1,3).$

The pair $u_{1}:=(2,1)$ and $u_{2}:=(1,2)$ is contained in a common GIT-cone
but it is not generating: one checks directly that the monomials
$x_{1}x_{2}^{s-2}x_{3}^{s-1}x_{4}\in A_{s(u_{1}+u_{2})}$ can never be obtained
by multiplying elements from $A_{su_{1}}$ and $A_{su_{2}}$.

######  Remark 2.

In [1, Theorem 1.5], a criterion for a pair of weights $(u_{1},u_{2})$ in one
GIT-cone to be generating is given in terms of normality of the image of a
morphism between certain quotient spaces.

The proof of [1, Theorem 1.1] is based on several propositions. A modification
of one of them will be used below. To formulate this modification, we need
some more notions from the theory of graded algebras.

A subalgebra $B$ of a graded algebra $A=\oplus_{u\in{\mathbb{Z}}^{m}}A_{u}$ is
_homogeneous_ if $B$ is the direct sum of intersections of $B$ with
homogeneous components of $A$. Every homogeneous subalgebra in $A$ inherits a
${\mathbb{Z}}^{m}$-grading.

With any weight $u\in{\mathbb{Z}}^{m}$ one associates a homogeneous subalgebra
$A(u):=\oplus_{r\geqslant 0}A_{ru}$. Note that the subalgebra $A(u)$ is
${\mathbb{Z}}_{\geqslant 0}$-graded.

Further, with any weights $u_{1}$ and $u_{2}$ one associates a homogeneous
subalgebra $A(u_{1},u_{2})$ in $A(u_{1}+u_{2})$ defined as

$A(u_{1},u_{2}):=\bigoplus_{r\geqslant 0}A_{ru_{1}}\cdot A_{ru_{2}}.$

We say that a homogeneous subalgebra $B$ of a ${\mathbb{Z}}_{\geqslant
0}$-graded algebra $A$ is _big_ , if the radical of the ideal
$B_{+}:=\oplus_{r>0}B_{r}$ coincides with $A_{+}:=\oplus_{r>0}A_{r}$.

Let $A$ be the polynomial algebra ${\mathbb{K}}[x_{1},\ldots,x_{n}]$ with
${\mathbb{Z}}^{m}$-grading given by a projection
$\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$. For every
$v\in{\mathbb{Z}}^{n}_{\geqslant 0}$ we denote by $x^{v}$ the monomial
$x_{1}^{v_{1}}\ldots x_{n}^{v_{n}}$.

######  Proposition 2.

Condition (P1) on a pair of weights $(u_{1},u_{2})$ is equivalent to each of
the conditions:

  1. (1)

the weights $u_{1},u_{2}$ lie in a common GIT-cone;

  2. (2)

the subalgebra $A(u_{1},u_{2})$ is big in $A(u_{1}+u_{2})$.

###### Proof.

The equivalence of conditions (1) and (2) is proved in [1, Proposition 2.1].

Let us prove that (P1) implies (2). Assume that
$P(u_{1}+u_{2})=P(u_{1})+P(u_{2})$ and take a monomial $x^{v}\in
A(u_{1}+u_{2})$. We have to prove that $x^{v}$ is contained in the radical of
the ideal $A(u_{1},u_{2})_{+}$, i.e., there is a positive integer $q$ such
that $x^{qv}\in A(u_{1},u_{2})_{+}$. By assumption, we have
$v=v^{\prime}+v^{\prime\prime}$ with $v^{\prime}\in P(u_{1})$ and
$v^{\prime\prime}\in P(u_{2})$. Then there is $q\in{\mathbb{Z}}_{>0}$ such
that $qv=qv^{\prime}+qv^{\prime\prime}$ with $qv^{\prime}\in
P(u_{1})\cap{\mathbb{Z}}^{n}$ and $qv^{\prime\prime}\in
P(u_{2})\cap{\mathbb{Z}}^{n}$. This proves the assertion.

Now we come to implication (2) $\Rightarrow$ (P1). Note that $v\in
P(u_{1}+u_{2})$ if and only if there is $s\in{\mathbb{Z}}_{>0}$ such that
$sv\in P(s(u_{1}+u_{2}))\cap{\mathbb{Z}}^{n}$ or, equivalently, the monomial
$x^{sv}$ lies in $A(u_{1}+u_{2})_{+}$. By assumption, this implies that
$x^{sv}$ is contained in the radical of $A(u_{1},u_{2})_{+}$, or there exists
$t\in{\mathbb{Z}}_{>0}$ such that $x^{tsv}\in A(u_{1},u_{2})_{+}$. The later
condition means that there are $v^{\prime}\in P(tsu_{1})\cap{\mathbb{Z}}^{n}$
and $v^{\prime\prime}\in P(tsu_{2})\cap{\mathbb{Z}}^{n}$ such that
$tsv=v^{\prime}+v^{\prime\prime}$. This condition implies $v\in
P(u_{1})+P(u_{2})$. ∎

##  5\. Positive results on normal location

We keep the notation introduced in the previous section. In particular, every
surjective homomorphism of lattices
$\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$ gives rise to a
${\mathbb{Z}}^{m}$-grading on the algebra ${\mathbb{K}}[x_{1},\ldots,x_{n}]$,
and we speak about the weight cone, the orbit cones and the GIT-cones
corresponding to this grading. In this situation, the weight cone coincides
with $C=\pi({\mathbb{Q}}^{n}_{\geqslant 0})$ and the orbits cones are
precisely the cones generated by all subsets of the set
$\\{w_{1},\ldots,w_{n}\\}$, where $w_{i}:=\pi(e_{i})$.

Let $\sigma$ be a pointed cone in ${\mathbb{Q}}^{d}$. Taking an appropriate
basis in ${\mathbb{Q}}^{d}$ we may assume that $\sigma$ is contained in
${\mathbb{Q}}^{d}_{\geqslant 0}$. Moreover, to any $\sigma$-polyhedra $Q_{1}$
and $Q_{2}$ we may apply a parallel translation and assume that $Q_{1}$ and
$Q_{2}$ are contained in the open octant ${\mathbb{Q}}^{d}_{>0}$.

######  Proposition 3.

Let $Q_{1}$ and $Q_{2}$ be two $\sigma$-polyhedra in ${\mathbb{Q}}^{d}$ of
full dimension. Then there are positive integers $n$ and $m$ with $d=n-m$ and
a surjective homomorphism $\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$ such
that $Q_{1}=P(u_{1})$ and $Q_{2}=P(u_{2})$ for some points $u_{1},u_{2}\in
C\cap{\mathbb{Z}}^{m}$ lying in a common GIT-cone.

###### Proof.

Let $\mathcal{N}(Q_{1})$ and $\mathcal{N}(Q_{2})$ be the normal fans of the
polyhedra $Q_{1}$ and $Q_{2}$, respectively. Denote by $\mathcal{N}$ the
coarsest common refinement of the fans $\mathcal{N}(Q_{1})$ and
$\mathcal{N}(Q_{2})$.

Let $m$ be the number of rays of the fan $\mathcal{N}$ and
$l_{1},\ldots,l_{m}$ be linear functions on ${\mathbb{Q}}^{d}$ generating the
rays of $\mathcal{N}$. Denote by $a_{i}$ (resp. by $b_{i}$) the maximal value
of $l_{i}$ on $Q_{1}$ (resp. on $Q_{2})$. Then the polyhedron $Q_{1}$ (resp.
$Q_{2}$) is given by inequalities $l_{i}\leqslant a_{i}$ (resp.
$l_{i}\leqslant b_{i}$) with $i=1,\ldots,m$. The fan $\mathcal{N}$ is the
normal fan of the polyhedron $Q_{1}+Q_{2}$, so the polyhedron $Q_{1}+Q_{2}$ is
given by inequalities $l_{i}\leqslant a_{i}+b_{i}$, $i=1,\ldots,m$.

We may assume that the linear functions $l_{1},\ldots,l_{m}$ have integer
coefficients. Let $n=d+m$ and consider the projection
$\pi\colon{\mathbb{Z}}^{n}\to{\mathbb{Z}}^{m}$ given by

$(x_{1},\ldots,x_{d},y_{1},\ldots,y_{m})\to(y_{1}+l_{1},\ldots,y_{m}+l_{m}).$

Let $u_{1}=(a_{1},\ldots,a_{m})$. Then the polyhedron $P(u_{1})$ is given by
conditions

$x_{1}\geqslant 0,\ldots,x_{d}\geqslant 0,\ y_{1}\geqslant
0,\ldots,y_{m}\geqslant 0,\ y_{1}+l_{1}=a_{1},\ldots,y_{m}+l_{m}=a_{m}.$

This system is equivalent to

$x_{1}\geqslant 0,\ldots,x_{d}\geqslant 0,\ l_{1}\leqslant
a_{1},\ldots,l_{m}\leqslant a_{m}.$

It proves that $P(u_{1})=Q_{1}$. The same arguments show that

$P(u_{2})=Q_{2}\quad\text{and}\quad P(u_{1}+u_{2})=Q_{1}+Q_{2}.$

Finally, by Proposition 2 the condition $P(u_{1})+P(u_{2})=P(u_{1}+u_{2})$
implies that $u_{1}$ and $u_{2}$ lie in a common GIT-cone. ∎

Let us denote by $\mathcal{N}_{1}$ (resp. $\mathcal{N}_{2}$) the normal fan of
the polyhedron $P(u_{1})$ (resp. $P(u_{2})$) living in the space
${\mathbb{Q}}^{d}$.

######  Proposition 4.

The fan $\mathcal{N}_{1}$ refines the fan $\mathcal{N}_{2}$ if and only if
$u_{1}$ is an interior point of a GIT-cone containing $u_{2}$.

###### Proof.

For any point $c=(c_{1},\ldots,c_{n})\in{\mathbb{Q}}^{n}$, we define a subset
in $\\{1,\ldots,n\\}$ as

$Z(c)=\\{i\ |\ c_{i}=0\\}.$

The condition that $u_{1}$ is an interior point of a GIT-cone containing
$u_{2}$ means that every orbit cone, which contains $u_{1}$, contains $u_{2}$
as well. Note that the coordinates of a point in $P(u_{1})$ (resp. $P(u_{2})$)
may be considered as coefficients of a linear combination of the vectors
$w_{1},\ldots,w_{n}$ that is equal to $u_{1}$ (resp. $u_{2}$). Taking this
into account, we may reformulate the above condition as: for every point
$v_{1}\in P(u_{1})$ there is a point $v_{2}\in P(u_{2})$ such that
$Z(v_{1})\subseteq Z(v_{2})$.

Since we assume that the polyhedra are contained in the open octant
${\mathbb{Q}}^{d}_{>0}$, the coordinates $x_{1},\ldots,x_{d}$ are positive on
$P(u_{1})$ and $P(u_{2})$. At the same time, for $y_{1},\ldots,y_{m}$ the
coordinate $y_{i}$ is zero at a point $v_{1}$ (resp. $v_{2}$) if and only if
the linear function $l_{i}$ reaches its maximum $a_{i}$ on $P(u_{1})$ at
$v_{1}$ (resp. $b_{i}$ on $P(u_{2})$ at $v_{2}$). The condition
$Z(v_{1})\subseteq Z(v_{2})$ means that every function $l_{i}$ which is
contained in the cone $\tau(v_{1})$ of the normal fan $\mathcal{N}_{1}$ (see
Section 3) is also contained in the cone $\tau(v_{2})$ of the normal fan
$\mathcal{N}_{2}$. Since the rays of the fans $\mathcal{N}_{1}$ and
$\mathcal{N}_{2}$ are generated by some of the vectors $l_{i}$ and every cone
is uniquely determined by the set of rays it contains, the last condition
means that every cone in $\mathcal{N}_{1}$ is contained in a cone of
$\mathcal{N}_{2}$ or, equivalently, the fan $\mathcal{N}_{1}$ refines the fan
$\mathcal{N}_{2}$. ∎

The next theorem generalizes [6, Theorem 1.1] from dimension $2$ to higher
dimensions.

######  Theorem 3.

Let $Q_{1}$ and $Q_{2}$ be two $\sigma$-polyhedra in ${\mathbb{Q}}^{d}$ of
full dimension. Assume that the normal fan $\mathcal{N}(Q_{1})$ refines the
normal fan $\mathcal{N}(Q_{2})$. Then there exists a positive integer $k$ such
that for every positive integer $s$ the polyhedra $skQ_{1}$ and $skQ_{2}$ are
normally located.

###### Proof.

Following Proposition 3, we realize $Q_{1}$ (resp. $Q_{2}$) as $P(u_{1})$
(resp. $P(u_{2})$). By Theorem 2, it suffices to show that the point $u_{1}$
lies in the relative interior of a common GIT-cone of the points $u_{1}$ and
$u_{2}$. It follows from Proposition 4. ∎

######  Remark 3.

The construction of two triangles presented in the proof of Proposition 1 is
extracted from Example 1: in notation of this example, we take the triangles
$P=P(u_{1})$ and $Q=P(u_{2})$ with respect to the projection
$\pi\colon{\mathbb{Z}}^{4}\to{\mathbb{Z}}^{2}$.

## References

  * [1] Ivan Arzhantsev and Jürgen Hausen. On the multiplication map of a multugraded algebra. Math. Res. Lett. 14 (2007), no. 1, 129-136
  * [2] Florian Berchtold and Jürgen Hausen. GIT-equivalence beyond the ample cone. Michigan Math. J. 54 (2006) no. 3, 483-516
  * [3] Winfried Bruns, Joseph Gubeladze, and Ngô Viêt Trung. Normal polytopes, triangulations, and Koszul algebras. J. Reine Angew. Math. 485 (1997), 123-160
  * [4] Günter Ewald and Uwe Wessels. On the ampleness of invertible sheaves in complete projective toric varieties. Results Math. 19 (1991), 275-278
  * [5] Joseph Gubeladze. Normal polytopes: Between discrete, continuous, and random. J. Pure Appl. Algebra 227 (2023), 107187
  * [6] Christian Haase, Benjamin Nill, Andreas Paffenholz, and Francisco Santos. Lattice points in Minkowski sums. Electron. J. Combin. 15 (2008), N11
  * [7] Mariusz Koras and Peter Russell. Linearization problems. In: Algebraic Group Actions and Quotients, pp. 91-107. Hindawi, Cairo, 2004
  * [8] Tadao Oda. Problems on Minkowski sums of convex lattice polytopes. Abstract submitted at the Oberwolfach Conference ‘‘Combinatorial Convexity and Algebraic Geometry’’ 26.10–01.11, 1997; see also arXiv:0812.1418, 7 pages


# Beyond “I don’t know”:
Overcoming Uncertainty in Multimodal Reasoning

Tejas Srinivasan1,2 Jack Hessel3 Tanmay Gupta2 Bill Yuchen Lin2
Yejin Choi2,4 Jesse Thomason1 Khyathi Raghavi Chandu2
1University of Southern California 2Allen Institute for Artificial
Intelligence
3Samaya AI 4University of Washington
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Selective “Selective Prediction" for Vision-Language Reasoning
by Knowing When Not to Abstain

Tejas Srinivasan1,2 Jack Hessel3 Tanmay Gupta2 Bill Yuchen Lin2
Yejin Choi2,4 Jesse Thomason1 Khyathi Raghavi Chandu2
1University of Southern California 2Allen Institute for Artificial
Intelligence
3Samaya AI 4University of Washington
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Selective “Selective Prediction" for Vision-Language Reasoning
by Inferring When Not to Abstain

Tejas Srinivasan1,2 Jack Hessel3 Tanmay Gupta2 Bill Yuchen Lin2
Yejin Choi2,4 Jesse Thomason1 Khyathi Raghavi Chandu2
1University of Southern California 2Allen Institute for Artificial
Intelligence
3Samaya AI 4University of Washington
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Selective “Selective Prediction”:
Reducing Unnecessary Abstention in Vision-Language Reasoning

Tejas Srinivasan1,2 Jack Hessel3 Tanmay Gupta2 Bill Yuchen Lin2
Yejin Choi2,4 Jesse Thomason1 Khyathi Raghavi Chandu2
1University of Southern California 2Allen Institute for Artificial
Intelligence
3Samaya AI 4University of Washington
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Prior work on selective prediction minimizes incorrect predictions from
vision-language models (VLMs) by allowing them to abstain from answering when
uncertain. However, when deploying a vision-language system with low tolerance
for inaccurate predictions, selective prediction may be over-cautious and
abstain too frequently, even on many correct predictions. We introduce
ReCoVERR, an inference-time algorithm to reduce the over-abstention of a
selective vision-language system without decreasing prediction accuracy. When
the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR
tries to find relevant clues in the image that provide additional evidence for
the prediction. ReCoVERR uses an LLM to pose related questions to the VLM,
collects high-confidence evidences, and if enough evidence confirms the
prediction the system makes a prediction instead of abstaining. ReCoVERR
enables two VLMs, BLIP2 and InstructBLIP, to answer up to 20% more questions
on the A-OKVQA task than vanilla selective prediction without decreasing
system accuracy, thus improving overall system reliability.

Selective “Selective Prediction”:
Reducing Unnecessary Abstention in Vision-Language Reasoning

Tejas Srinivasan1,2 Jack Hessel3 Tanmay Gupta2 Bill Yuchen Lin2 Yejin Choi2,4
Jesse Thomason1 Khyathi Raghavi Chandu2 1University of Southern California
2Allen Institute for Artificial Intelligence 3Samaya AI 4University of
Washington<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Illustration of ReCoVERR. The VLM predicts that the floor has two
tile colors with low confidence. Instead of abstaining, ReCoVERR collects
reliable and relevant visual evidences related to the question. ReCoVERR makes
salient the evidence that the floor tiles are red and white, helping to verify
the VLM’s original answer.

Instruction-tuned vision-language models (VLMs) Dai et al. (2023); Liu et al.
(2023); Laurençon et al. (2023); Bai et al. (2023) have achieved strong
accuracy on reasoning benchmarks, which typically require VLMs to produce an
answer for each instance. For downstream use by people, however, these systems
should abstain from answering when uncertain (e.g., by saying “I don’t know")
Rajpurkar et al. (2018). Selective prediction systems De Stefano et al.
(2000); El-Yaniv et al. (2010); Geifman and El-Yaniv (2017) aim to balance the
number of predictions made (coverage) and the error rate on instances which do
receive a prediction (risk).

However, a vanilla selective prediction system with low tolerance for
incorrect predictions will abstain too frequently to be practical, even when
the model answer may be correct Whitehead et al. (2022). For example, if a
user specifies that the BLIP2 Li et al. (2023) predictions should be right at
least 90% of the time, vanilla selective prediction will make a prediction for
just 4% of A-OKVQA Schwenk et al. (2022) questions, with 94% of the correct
predictions being abstained on.

We introduce ReCoVERR (Reason by Collecting Visual Evidences that are Reliable
and Relevant), an algorithm that increases the number of questions that a
selective VLM system can answer confidently while adhering to a specified risk
tolerance, _without any additional training_. When the VLM is uncertain about
its prediction for a given question, instead of abstaining outright, ReCoVERR
tries to verify the prediction by recovering reliable supporting (or
contradicting) evidence. This ability is predicated on two characteristics of
VLMs. First, VLMs can produce well-calibrated confidence estimates (§ 4.2.1).
Second, VLMs can often correctly and confidently recover information in the
image that entails a low-confidence initial prediction. For instance, in
Figure 1, when asked about the number of floor tile colors, a VLM (here,
BLIP2) correctly answers “two” but with low confidence: simple threshold-based
abstention would abstain on this instance. But, when asked to identify the
colors of the floor tiles, the model confidently answers “red and white”.
While it is obvious that “red and white” entails “two colors”, the VLM was
unable to confidently make that inference. Based on these two insights,
ReCoVERR searches for additional visual evidence by iteratively posing
relevant questions to the VLM, and collecting answers as visual evidence if
they are: a) _reliable_ , i.e, the VLM is highly confident in its answer, and
b) _relevant_ to the question the VLM is trying to answer in the first place.

Our experiments on the A-OKVQA reasoning task using two VLMs (BLIP2 and
InstructBLIP) demonstrate that ReCoVERR substantially increases the number of
questions that a selective VLM system can answer, while keeping system risk
under the specified risk tolerance, thus improving overall reliability
(Section 5). ReCoVERR is particularly helpful for BLIP2, which has not been
trained on the target task, by improving coverage by 20% and recall by 25-30%.
Our analysis reveals that the ability to give accurate confidence estimates,
and high estimates for correct predictions, is crucial to ReCoVERR’s
performance. Further experiments demonstrate the importance of ensuring
evidences are both reliable and relevant (Section 5.1), and that ReCoVERR
tuned for a single task can be directly applied to new tasks without further
tuning (Section 5.3). Our findings suggest that ReCoVERR is a promising
solution towards building more reliable multimodal reasoning systems.

##  2 Multimodal Selective Prediction

Figure 2: The ReCoVERR algorithm. If the VLM is uncertain in its prediction
(1), ReCoVERR tries to verify the VLM hypothesis by collecting evidences.
ReCoVERR undertakes multiple turns of evidence collection, which involves
generating visual evidences by using an LLM to ask questions to the VLM (2),
retaining the reliable (3) and relevant (4) evidences, and checking whether
the collected evidence entails the hypothesis (5).

We consider the task of answering a textual question about an image by drawing
inferences from what is visually observed. A vision-language model (VLM) is
given an input $x=(I,Q)\in\mathcal{X}$ consisting of an image $I$ and a
question $Q$. It aims to produce an answer $a\in A$ from a closed or open set
of possible answers.

Different from popular benchmarks of this form which require models to make a
prediction for each instance, we evaluate models on the selective prediction
setting, where abstention on individual instances is allowed De Stefano et al.
(2000). A decision function $g$, which has access to the image, question, VLM
prediction, and associated confidence111For brevity, we note arguments for $g$
when relevant. determines whether the system produces an answer or abstains
(denoted by $\varnothing$). The selective VLM system
$\mathcal{S}_{\texttt{VLM}}:\mathcal{X}\rightarrow\mathcal{A}\cup\\{\varnothing\\}$
is defined as:

$\displaystyle a$ $\displaystyle=\mathcal{M}_{\texttt{VLM}}(x)$
$\displaystyle\mathcal{S}_{\texttt{VLM}}(x)$
$\displaystyle=\begin{cases}a,&\text{if }g(a)=1\\\ \varnothing,&\text{if not
}g(a)=0\end{cases}$

Prior work (Whitehead et al., 2022) employs confidence-based selection, where
instead of assuming access to a VLM which has been trained to abstain with
explicit supervision, we assume a VLM that can produce both an answer, as well
as a confidence score for that answer
$\pi_{\texttt{{VLM}}}:\mathcal{A}\rightarrow[0,1]$ that estimates
$\pi_{\texttt{{VLM}}}(a)=P(a|Q,I;\mathcal{M}_{\texttt{VLM}})$. Confidence-
based selection relies on the confidence $\pi_{\texttt{{VLM}}}(a)$ being
higher for correct answers than incorrect ones, on average. The decision
function $g$ is a simple thresholding function, with a threshold $\gamma$:
$g(a;\gamma)=\mathbbm{1}\\{\pi_{\texttt{{VLM}}}(a)\geq\gamma\\}$. We refer to
this method as vanilla selective prediction.

###  2.1 Evaluating Selective Predictors

We evaluate selective prediction systems via coverage and risk El-Yaniv et al.
(2010). Given a labeled evaluation dataset
$\mathcal{D}=\\{(I_{i},Q_{i},a_{i})\\}_{i=1}^{N}$. _Coverage_ is the the
percentage of questions in $\mathcal{D}$ where the system chooses to make a
prediction. _Risk_ is the error rate on the questions where a prediction is
made. For a $\gamma$-threshold selective prediction system, these metrics can
be computed as:

$\displaystyle\mathcal{C}(\gamma)=\frac{\sum_{x_{i}\in\mathcal{D}}g(a_{i};\gamma)}{\mid\mathcal{D}\mid}$
(1)
$\displaystyle\mathcal{R}(\gamma)=\frac{\sum_{x_{i}\in\mathcal{D}}(1-\text{Acc}(a_{i}))\cdot
g(a_{i};\gamma)}{\sum_{x_{i}\in\mathcal{D}}g(a_{i};\gamma)}$ (2)

where a lower $\gamma$ trades off increased coverage for increased risk. In
practice, when deploying a selective VLM system with an application-specific
risk tolerance $r\in[0,1]$, the system designer would determine an appropriate
setting for the hyperparameters of $g$ using a calibration set. For example,
if $g$ is the simple $\gamma$-thresholding function, one would choose
$\gamma_{@r}$ to maximize coverage while being below the risk threshold on the
calibration set:

$\displaystyle\gamma_{@r}=\operatorname*{arg\,min}_{\gamma\in[0,1]}R(\gamma)\leq
r$

##  3 ReCoVERR

In practice, reasonable risk tolerance thresholds often lead to untenable
coverage, with many accurate predictions being discarded. A threshold-
selective BLIP2 Li et al. (2023) system, for example, answers fewer than 4% of
questions in A-OKVQA at 10% risk, but these represent only 6% of questions for
which the model prediction was correct.

We introduce ReCoVERR (Reason by Collecting Visual Evidences that are Reliable
and Relevant), an algorithm that increases a selective VLM system’s coverage
while remaining under the given risk tolerance (Figure 2). For each question
$Q$ for which the VLM predicts an answer $a$ with
$\pi_{\texttt{{VLM}}}(a)<\gamma_{@r}$, instead of abstaining, ReCoVERR uses
large language models to generate follow-up questions about the image. These
questions are answered by the VLM, and the resulting QA-pair is added as
evidence if it is both reliable—the VLM confidence in the answer is high—and
relevant—the introduction of the new evidence affects the downstream
confidence in the hypothesized answer $a$ to $Q$. If sufficient evidences are
collected to confidently entail the hypothesis, ReCoVERR elects to make a
prediction instead of abstaining.

Let $\mathcal{D}_{\varnothing}$ be the subset of test set $\mathcal{D}$ where
the selective prediction system would have abstained, based on the confidence
threshold $\gamma_{@r}$, and $\mathcal{D}_{S}$ be the set of questions that
were answered.

$\displaystyle\mathcal{D}_{\varnothing}=\\{x_{i}\in\mathcal{D};\pi_{\texttt{{VLM}}}(a_{i})<\gamma_{@r}\\};\mathcal{D}_{S}=\mathcal{D}\backslash\mathcal{D}_{\varnothing}$

ReCoVERR aims to answer additional instances
$\mathcal{D}_{\texttt{R}}\subseteq\mathcal{D}_{\varnothing}$, while keeping
the combined risk of $\mathcal{D}_{S}\cup\mathcal{D}_{\texttt{R}}$ under $r$.
Since risk of $\mathcal{D}_{S}$ is already estimated to be $r$, the risk of
$\mathcal{D}_{\texttt{R}}$ must also be $\leq r$. For each instance in
$\mathcal{D}_{\varnothing}$, ReCoVERR makes an online decision of whether to
answer the question, independent of other instances in
$\mathcal{D}_{\varnothing}$. Therefore, the expected risk of any instance
$x_{i}\in\mathcal{D}_{\texttt{R}}$ should be, at most, $r$; in other words,
the likelihood of each instance in $\mathcal{D}_{\texttt{R}}$ being correct
should be $\geq 1-r$. For example, if our system has a specified risk
tolerance of 20%, ReCoVERR aims to answer additional questions of which 80%
are correct.

Selective prediction hyperparameters:

$r\in[0,1]$ : Risk tolerance of system

$\gamma_{@r}$ : VLM conf threshold corresponding to risk $r$

ReCoVERR Model-based Tools:

$\mathbb{M}_{\texttt{Vis}}$: Set of additional vision models that output
information about the image in text form

$\mathcal{M}_{\texttt{QGen}}$: Question generation model

$\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}$: LM to paraphrase QA pair
into sentence

$\mathcal{M}_{\texttt{NLI}}$: NLI model

ReCoVERR hyperparameters:

$\delta_{\texttt{min}}.$ : minimum relevance of “relevant” evidences

$\pi_{\texttt{NLImin}}$: Minimum entailment confidence to answer

$N$: Maximum number of turns of evidence collection

$K$: Questions generated at each turn

Inputs: Question $Q$, image $I$, VLM $\mathcal{M}_{\texttt{VLM}}$

1 $a,\pi_{\texttt{{VLM}}}(a)\leftarrow\mathcal{M}_{\texttt{VLM}}(I,Q$)

2 if  _$\pi_{\texttt{{VLM}}}(a)\geq\gamma_{@r}$_ then

3 return $a$

4$\mathcal{H}\leftarrow\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}(Q,a)$

5
$\mathcal{E}_{R},\mathcal{E}_{RR}\leftarrow$InitImageEvidences($I;\mathbb{M}_{\texttt{Vis}}$)

6 for  _$i=1$ to  $N$_ do

7 $e_{1...K}\leftarrow$CollectKEvidences($I,Q,\mathcal{E}_{R};$

8 $\mathcal{M}_{\texttt{QGen}},\mathcal{M}_{\texttt{VLM}}$)

9 for  _$j=1$ to  $K$_ do

10 if  _VLMConfidence($e_{j};\mathcal{M}_{\texttt{VLM}}$) $\geq 1-r$_ then

11 $\mathcal{E}_{R}\leftarrow\mathcal{E}_{R}+[e_{j}]$

12 if  _Relevance($e_{j};\mathcal{M}_{\texttt{NLI}}$)
$\geq\delta_{\texttt{min}}.$_ then

13 $\mathcal{E}_{RR}\leftarrow\mathcal{E}_{RR}+[e_{j}]$

14

15

16 if  _$P(\mathcal{E}_{RR}\text{ entails
}\mathcal{H};\mathcal{M}_{\texttt{NLI}})\geq\pi_{\texttt{NLIMin}}$_ then

17 return $a$

18

return $\varnothing$

Algorithm 1 ReCoVERR Pseudocode

Algorithm 1 describes ReCoVERR in detail, including all hyperparameters. Given
an image $I$ and a question $Q$, ReCoVERR begins by generating an answer $a$
using the VLM $\mathcal{M}_{\texttt{VLM}}$. The VLM also returns a confidence
score $\pi_{\texttt{VLM}}(a)\in[0,1]$. If the confidence is higher than our
confidence threshold for selective prediction $\gamma_{@r}$, we can simply
return the answer. If not, we use a model
$\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}$ that converts the question-
answer pair $(Q,a)$ into a hypothesis statement $\mathcal{H}$. ReCoVERR will
now try to verify this hypothesis by collecting reliable and relevant visual
evidences.

##### Initialize Evidences from Vision Tools:

We first gather some general information about the image by invoking a set of
vision models, $\mathbb{M}_{\texttt{Vis}}$, that capture visual information in
language form (the specific tools we use in our instantiation of ReCoVERR are
highlighted in Section 4.3). Information from these tools instantiate two
evidence sets: $\mathcal{E}_{R}$, a set of reliable evidences about the image,
and $\mathcal{E}_{RR}$, a set of reliable _and_ relevant evidences.

ReCoVERR performs up to $N$ turns of evidence collection. In each turn, $K$
new visual evidences are generated from the VLM, the reliable and relevant
ones are retained, and we check whether the collected evidences sufficiently
entail the hypothesis. If we are unable to verify the hypothesis at the end of
$N$ turns, we abstain from answering.

##### Generating Visual Evidences:

We prompt a question generation model $\mathcal{M}_{\texttt{QGen}}$ to
generate a set of $K$ sub-questions $q_{1...K}$, conditioned on the target
question $Q$ and the already-collected reliable evidences $\mathcal{E}_{R}$.
For each sub-question $q_{j},j\in\\{1...K\\}$, the VLM produces an answer
$a_{j}$ with confidence $\pi_{\texttt{{VLM}}}(a_{j})$. The paraphraser model
$\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}$ paraphrases the pair
$(q_{j},a_{j})$ to a declarative sentence $S_{j}$. Each 4-tuple
$\big{(}q_{i},a_{j},\pi_{\texttt{{VLM}}}(a_{j}),S_{j}\big{)}$ represents an
evidence $e_{j}$.

##### Check Evidence Reliability:

Since ReCoVERR decides to make a prediction based on the collected evidences,
the correctness likelihood of a given evidence is an upper bound for the
correctness likelihood of the prediction. Since ReCoVERR aims to make
predictions with correctness likelihood $\geq 1-r$, we only consider evidences
$e_{j}$ that satisfy $\pi_{\texttt{{VLM}}}(a_{j})\geq 1-r$ as reliable for
$\mathcal{E}_{R}$. ReCoVERR requires that the confidence estimate
$\pi_{\texttt{{VLM}}}(a)$ is calibrated. A confidence estimate being
calibrated means that for all predictions whose confidence is
$\alpha\in[0,1]$, $\alpha$% of the predictions are actually correct. In our
experiments, we first evaluate calibration of our VLMs’ confidence estimates
(§ 4.2.1).

##### Check Evidence Relevance:

To decide whether a reliable evidence $e_{j}$ is also relevant, we adopt a
defeasible reasoning approach Rudinger et al. (2020). An evidence $e_{j}$ is
considered relevant if its truth value affects the entailment probability of
the hypothesis. We measure the absolute difference between the entailment
probabilities of the hypothesis $\mathcal{H}$ conditioned on the evidence
premise $S_{j}$ and its negated counterfactual i.e. $\bar{S}_{j}$.

$\displaystyle\delta(e_{j})=\mid\pi_{\texttt{{NLI}}}(\mathcal{H}|S_{j})-\pi_{\texttt{{NLI}}}(\mathcal{H}|\bar{S}_{j})\mid$

Evidence $e_{j}$ is added to the relevant evidence set $\mathcal{E}_{RR}$ if
the relevance $\delta(e_{j})\geq\delta_{\texttt{min}}.$

##### Check Sufficiency of Collected Evidences:

After each round of evidence collection, we concatenate the $S_{j}$ for all
evidences $e_{j}\in\mathcal{E}_{RR}$ into a premise sentence $S_{RR}$ and
calculate the hypothesis’s entailment probability
$\pi_{\texttt{{NLI}}}(\mathcal{H}|S_{RR})$. If that probability meets
$\pi_{\texttt{NLImin}}$, we return prediction $a$, increasing coverage with
sufficient estimated confidence that risk will not increase above $r$ as a
result. Else, after $N$ rounds, we abstain and return $\varnothing$.

##  4 Experiments

We instantiate a selective prediction task using the A-OKVQA benchmark and
compare ReCoVERR to simple threshold-based selective prediction with two
different backbone VLMs with calibrated confidence estimates, using metrics
that capture aspects and tradeoffs of risk and coverage.

###  4.1 Vision-Language Reasoning Task: A-OKVQA

A-OKVQA Schwenk et al. (2022) is a VQA task designed to require reasoning over
external knowledge and commonsense alongside image-based information. We
evaluate on the validation set ($n=1,075$) and use LAVE${}_{\textsc{GPT-3.5}}$
Mañas et al. (2023) to measure the accuracy of predicted answers. LAVE uses a
large language model222We utilize the gpt-3.5-turbo-16k-0613 checkpoint. to
estimate the semantic similarity of each predicted answer to the 10
crowdsourced answers in the benchmark.

###  4.2 Vision-Language Models

Figure 3: Calibration curves for BLIP-2 and InstructBLIP on A-OKVQA questions.

We experiment with two VLMs: BLIP2 Li et al. (2023) and InstructBLIP Dai et
al. (2023), both using FlanT5-XL as the LLM
backbone.333https://github.com/salesforce/LAVIS InstructBLIP is instruction
tuned on A-OKVQA; BLIP2 has not been trained on the A-OKVQA task.

####  4.2.1 Calibrated VLM Confidence Estimates

Confidence estimates for generative VLMs are not well defined. Straightforward
estimates such as the product of answer token likelihoods can severely
underestimate model confidence due to factors like _surface form competition_
Holtzman et al. (2021). Inspired by Tian et al. (2023), we devise a Self-
Prompting technique for estimating $\pi_{\texttt{{VLM}}}(a)$ by prompting the
VLM to verify “yes” or “no” correctness of its predictions and examine the
resulting probability distribution (full details in Appendix A). We further
calibrated this confidence estimate using Platt scaling Platt (1999). We
divide the A-OKVQA training data into 12,000 examples are used to fit the
calibration model and 5,000 to evaluate calibration error. We find that off-
the-shelf BLIP2 tends to be overconfident, evidenced by prediction confidence
being consistently higher than accuracy (Figure 3). Platt scaling further
reduces calibration error for both VLMs.

###  4.3 ReCoVERR Implementation Details

ReCoVERR can be instantiated with different choices of models and system
hyperparameters.

####  4.3.1 ReCoVERR Model-based Tools

In our experiments, the vision tools $\mathbb{M}_{\texttt{Vis}}$ consists of
LVIS Gupta et al. (2019) for object detection, and Qwen-VL Bai et al. (2023)
for region captioning. We use FlanT5-XL as a zero-shot sentence paraphraser
$\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}$ and entailment model
$\mathcal{M}_{\texttt{NLI}}$. Since our VLMs use FlanT5-XL as the LLM
backbone, our ReCoVERR instantiation does not require additional models. We
use GPT-3.5 with temperature 1.0 for $\mathcal{M}_{\texttt{QGen}}$. See
Appendix B for prompts.

####  4.3.2 ReCoVERR Hyperparameters

We set $N=10$ rounds of evidence collection with $K=10$ evidences generated
per turn. To count an evidence $e$ as reliable, we set relevance threshold
$\delta_{\texttt{min}}=0.2$. The minimum final hypothesis entailment
confidence is $\pi_{\texttt{{NLImin}}}=0.9$.

###  4.4 Selective Prediction Baselines

We compare ReCoVERR to a Vanilla Selective Prediction baseline Whitehead et
al. (2022), where the model abstains if the VLM prediction confidence
$g(a)=\\{1\text{ if }\pi_{\texttt{{VLM}}}(a)\leq\gamma_{@r}\text{ else }0\\}$,
and to a Vision Tools baseline. For Vision Tools, image caption from the VLM,
objects detected by LVIS, and region captions from Qwen-VL are provided as
evidence to the NLI model $\mathcal{M}_{\texttt{NLI}}$ directly, with no
rounds of evidence collection, equivalent to lines 1-5 of Algorithm 1 followed
by lines 14-15.

###  4.5 Evaluation Metrics

We measure risk ($\mathcal{R}$) and coverage ($\mathcal{C}$) (Eqs 2, 1) across
system predictions and abstentions. Additionally, we calculate Effective
Reliability $\Phi_{c}$ Whitehead et al. (2022), a metric that rewards correct
predictions, assigns a penalty $c$ to incorrect predictions, and assigns zero
reward to abstentions. We assign a penalty of $c=1$ for incorrect answers
($\Phi_{1}$). Finally, we define Selective Prediction Recall
($\mathbb{R}_{\texttt{SP}}$), the percentage of correct answers
($\text{Acc}(a)=1$) in the dataset that were answered.

$\mathbb{R}_{\texttt{SP}}$
$\displaystyle=\frac{\sum_{x_{i}\in\mathcal{D}}g(a_{i})\cdot\mathbbm{1}\\{\text{Acc}(a_{i})=1\\}}{\sum_{x_{i}\in\mathcal{D}}\mathbbm{1}\\{\text{Acc}(a_{i})=1\\}}$

|  | Risk tolerance $r$ = 10% |  | Risk tolerance $r$ = 20%
---|---|---|---|---
VLM | Method |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathcal{C}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$) |  |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathcal{C}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$)
BLIP2 (Off-the-shelf) | Vanilla SelPred | 6.1 | 3.4 | 3.8 | 6.0 |  | 14.9 | 17.1 | 24.1 | 34.2
Vision Tools | 13.8 | 17.4 | 23.5 | 33.5 |  | 17.0 | 23.5 | 35.1 | 48.3
ReCoVERR | 14.3 | 18.8 | 26.0 | 37.1 |  | 21.7 | 27.1 | 47.3 | 61.5
BLIP2 (Calibrated) | Vanilla SelPred | 4.1 | 3.2 | 3.4 | 5.5 |  | 11.9 | 16.8 | 21.9 | 32.0
Vision Tools | 13.2 | 17.5 | 23.4 | 33.7 |  | 15.1 | 23.9 | 33.6 | 47.2
ReCoVERR | 14.0 | 17.3 | 23.6 | 33.6 |  | 16.1 | 25.3 | 36.7 | 51.0
InstructBLIP (Off-the-shelf) | Vanilla SelPred | 9.3 | 20.5 | 25.1 | 34.8 |  | 17.2 | 38.3 | 57.7 | 72.2
Vision Tools | 10.5 | 26.1 | 32.9 | 44.8 |  | 17.6 | 39.6 | 60.5 | 75.3
ReCoVERR | 10.9 | 26.3 | 33.5 | 45.2 |  | 17.9 | 41.3 | 63.7 | 78.9
InstructBLIP (Calibrated) | Vanilla SelPred | 8.5 | 22.0 | 26.4 | 36.9 |  | 17.5 | 37.2 | 56.6 | 70.5
Vision Tools | 10.4 | 27.2 | 34.1 | 46.5 |  | 17.8 | 38.8 | 59.7 | 74.2
ReCoVERR | 11.8 | 29.8 | 38.7 | 51.8 |  | 18.6 | 41.4 | 65.0 | 79.8
Table 1: Metric results as percentages on the A-OKVQA task at two risk
tolerance levels. We evaluate selective prediction methods on the overall
system risk ($\mathcal{R}$), effective reliability ($\Phi_{1}$), coverage
($\mathcal{C}$) and recall ($\mathbb{R}_{\texttt{SP}}$). System risks in red
exceeded tolerance. Measurements in blue indicate when ReCoVERR outperformed
both baselines.

We note that the only stochastic component in our instantiation of ReCoVERR is
the question generation model. We run three seeds for each ReCoVERR experiment
and report average metric results. Table 8 shows full results with standard
deviation.

##  5 Results and Analysis

We evaluate selective prediction methods on A-OKVQA at two specified risk
tolerances: 10% and 20% risk. Table 1 shows the results of ReCoVERR when
applied to InstructBLIP and BLIP2, both with and without Platt scaling
calibration.

At 20% risk tolerance, we find that ReCoVERR improves coverage, effective
reliability, and recall over both baselines while staying below or at the
specified risk tolerance. At 10% risk tolerance, ReCoVERR and the Vision Tools
baseline tend to slightly overshoot the risk tolerance (by 1-2% for
InstructBLIP and 3-4% for BLIP2). We further see that ReCoVERR shows large
improvements of up to 20% on coverage, up to 15% on effective reliability and
up to 33% on recall for off-the-shelf BLIP2 and calibrated InstructBLIP.

Figure 4: Recall vs risk for different selective prediction methods applied to
different VLMs, at two specified risk tolerances (10% and 20%).

Figure 4 visualizes how different selective prediction methods compare on risk
$\mathcal{R}$ and recall $\mathbb{R}_{\texttt{SP}}$. We highlight several
takeaways (T*). T1: For the same risk tolerance, vanilla selective prediction
has lower coverage and recall for BLIP2 than InstructBLIP. BLIP2 has not been
trained on A-OKVQA, unlike InstructBLIP, and therefore produces more uncertain
answers. We further see that ReCoVERR results in larger improvements for
BLIP2. T2: Off-the-shelf BLIP2 sees largest risk increase with ReCoVERR. The
calibration curves in Figure 3 indicate that off-the-shelf BLIP2 makes
overconfident estimates, leading to more unreliable evidences being collected
and resulting in a large risk increase for ReCoVERR. T3: At 10% risk, most of
the additional risk and recall is caused by Vision Tools, rather than
ReCoVERR’s evidence collection. This finding is especially true for calibrated
BLIP2 and off-the-shelf InstructBLIP. T4: At 20% risk tolerance, ReCoVERR’s
evidence collection leads to greatest recall improvement for off-the-shelf
BLIP2 and calibrated InstructBLIP.

T3&4 can be understood by examining the confidence distribution of the VLMs’
predictions (Figure 5). We see that off-the-shelf BLIP2 has more predictions
with greater than 80% confidence compared to calibrated BLIP2, with the trend
reversed for InstructBLIP. This means that ReCoVERR finds more reliable
evidences ($\pi_{\texttt{{VLM}}}(a)\geq 1-r$) with off-the-shelf BLIP2 and
calibrated InstructBLIP, resulting in higher recall for ReCoVERR with those
two VLMs, thus explaining T3&4. These findings indicate that ReCoVERR works
best when the VLM is _strongly confident_ on correct answers.

Figure 5: Distribution of VLM prediction confidences on A-OKVQA calibration
set.

###  5.1 Reliability and Relevance Ablations

Method |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$)
---|---|---|---
VLM : Calibrated BLIP2
ReCoVERR | 16.1 | 25.3 | 51.0
\- Reliability | 28.4 | 31.6 | 85.9
\- Relevance | 16.1 | 25.3 | 51.1
VLM : Calibrated InstructBLIP
ReCoVERR | 18.6 | 41.4 | 79.8
\- Reliability | 25.9 | 40.5 | 91.9
\- Relevance | 18.6 | 39.7 | 76.9
Table 2: Ablation of the reliability and relevance components of ReCoVERR, at risk tolerance of $20\%$. Calibrated VLM | Method |  $\mathcal{R}$($\downarrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$)
---|---|---|---
Task: OK-VQA (20% risk tolerance)
BLIP2 | Vanilla | 15.4 | 25.4
ReCoVERR | 25.7 | 45.0
InstructBLIP | Vanilla | 18.5 | 56.3
ReCoVERR | 22.8 | 70.8
Task: Sherlock (10% risk tolerance)
BLIP2 | Vanilla | 11.1 | 24.9
ReCoVERR | 15.7 | 25.8
InstructBLIP | Vanilla | 9.7 | 33.4
ReCoVERR | 10.9 | 35.5
Table 3: ReCoVERR using a VLM calibrated for A-OKVQA can be directly applied to new tasks. $\mathcal{M}_{\texttt{QGen}}$ |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathcal{C}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$)
---|---|---|---|---
VLM : Calibrated BLIP2
ChatGPT | 16.1 | 25.3 | 36.7 | 51.0
Mistral-7B | 16.9 | 25.4 | 37.9 | 52.2
Tulu-2-7B | 15.8 | 24.8 | 35.8 | 50.0
VLM : Calibrated InstructBLIP
ChatGPT | 18.6 | 41.4 | 65.0 | 79.8
Mistral-7B | 19.1 | 39.9 | 63.8 | 77.8
Tulu-2-7B | 18.0 | 40.9 | 63.3 | 78.2
Table 4: Effect of different question generation models on ReCoVERR performance, at 20% risk tolerance. |  | Evidences $\mathcal{E}_{RR}$ collected by ReCoVERR
---|---|---
Image | Initial prediction |  {LLM Question $q_{j}$} {VLM Answer $a_{j}$} [$\pi_{\texttt{{VLM}}}(a_{j})$]
|  Question: What kind of a person usually eats food like this?
G.T. answers: vegetarian, vegan, healthy
VLM: vegetarian (conf=0.57) |  Does the image depict a variety of plant-based ingredients? yes [0.95]
Does the food predominantly consist of fruits and vegetables? yes [0.80]
Are there any meat items in the image? no [0.86]
Result: Correctly predicted
|  Question: The colors on the bus match the colors on what flag?
G.T. answers: Sweden, Ukraine
VLM: U.K. (conf=0.53) |  Is there any blue color on the bus? yes [0.95]
What is the color of the bus? yellow and blue [0.93]

Result: Correctly abstained
Table 5: Examples from the A-OKVQA task where, given a low-confidence VLM
prediction, ReCoVERR collects a set of highly-confident evidences
$\mathcal{E}_{RR}$ that corroborate (example 1) or contradict (example 2) the
prediction.

We perform an ablation of the reliability and relevance components of ReCoVERR
for calibrated BLIP2 and InstructBLIP, at a risk tolerance of 20%. In the
ReCoVERR formulation, for an evidence $e_{j}$ to be considered reliable, the
VLM confidence $\pi_{\texttt{{VLM}}}(a_{j}|I,Q_{j})$ must be at least $1-r$,
and for it to be considered relevant, the defeasible relevance $\delta(e_{j})$
must be at least $\delta_{\texttt{min}}=0.2$. For the reliability ablation, we
lower the minimum evidence confidence to be $0.5$ instead of $1-r=0.8$. For
the relevance ablation, we set $\delta_{\texttt{min}}=0$.

Table 2 shows the results of the reliability and relevance ablation. We see
that loosening the evidence confidence bound to be $0.5$ instead of $1-r$
causes a marked increase in risk for both VLMs, with system risk crossing the
risk tolerance significantly. Ablating the relevance component results in a
decrease in recall and effective reliability for InstructBLIP, but virtually
no effect for BLIP2.

###  5.2 Question Generation Model Ablations

The specific instantiation of ReCoVERR involves a choice of LLM for the
question generation model, $\mathcal{M}_{\texttt{QGen}}$. While our
experiments so far have used GPT-3.5, we demonstrate that ReCoVERR works
similarly well with open-source LLMs as well. We compare ReCoVERR performance
with GPT-3.5 against two open-source LLMs: Mistral-7B-Instruct Jiang et al.
(2023) and Tulu-2-7B Ivison et al. (2023). Our results in Table 4 demonstrate
that the specific instantiation of $\mathcal{M}_{\texttt{QGen}}$ did not make
a significant difference to ReCoVERR’s performance, for both calibrated BLIP2
and InstructBLIP.

###  5.3 Generalizing to New Tasks

We examine whether our instantiation of ReCoVERR calibrated for the A-OKVQA
task can be directly applied to new tasks without additional tuning (Table 3).
When we apply ReCoVERR with A-OKVQA-calibrated VLMs to OK-VQA Marino et al.
(2019) at 20% risk tolerance, we observe strong improvements in recall
(15-20%), but also increased risk (2-5% above the risk tolerance). Applying
ReCoVERR to Sherlock *Hessel et al. (2022), an abductive reasoning task, at
10% risk we observe smaller improvements in recall. These results indicate
that ReCoVERR requires some degree of task-specific tuning. Appendix C
contains full task details.

###  5.4 Qualitative ReCoVERR Examples

We present some examples of evidences collected by ReCoVERR in Table 5. The
collected evidences $\mathcal{E}_{RR}$ can be used to either corroborate
correct VLM predictions, or contradict incorrect ones. In example 1, when the
VLM predicts that a vegetarian is likely to eat this meal, ReCoVERR collects
supporting evidences indicating that the food does not contain any meat
products. In example 2, when the VLM predicts that the colors of the bus match
those of the U.K. flag, ReCoVERR finds that the color of the bus is highly
likely to be yellow and blue, thus contradicting the VLM’s prediction (since
the U.K. flag does not contain yellow).

##  6 Conclusion and Related Works

We introduce ReCoVERR, an algorithm to improve the coverage of a selective VLM
system while respecting a user-specified risk tolerance level. ReCoVERR
verifies low-confidence VLM predictions by recovering high-confidence
evidences in the image that support the prediction. We instantiate a selective
prediction task on the A-OKVQA reasoning benchmark and demonstrate the
quantitative advantages of ReCoVERR for inference-time selective prediction
that holds across different backbone VLMs, choice of underlying LLM
generators, and even gains some ground when transferred to different
benchmarks without recalibration.

A large breadth of prior work has studied the ability of models to abstain
from answering Chow (1957); De Stefano et al. (2000); El-Yaniv et al. (2010).
This ability is especially important when the model’s prediction cannot be
trusted Jiang et al. (2018), particularly for OOD Kamath et al. (2020) and
adversarial Varshney et al. (2022) inputs.

In the multimodal domain, previous works have explored abstention for error
recovery Wu et al. (2023); Khan and Fu (2023). Whitehead et al. (2022)
establish the ReliableVQA framework, wherein VQA models have the option to
abstain from answering. Shukor et al. (2023) evaluates VQA models on their
ability to abstain when a question does not apply to the image. Dancette et
al. (2023) trains VQA models to abstain for out-of-distribution inputs. In
contrast to _learning to abstain_ , we _reduce abstention_ of multimodal
selective prediction while maintaining reliability.

Using LLMs to query a pool of vision experts, either through program
generation Gupta and Kembhavi (2023); Surís et al. (2023); Subramanian et al.
(2023), or iterative querying Zeng et al. (2022); Shen et al. (2023); You et
al. (2023); Yang et al. (2023), can decompose vision-language reasoning. We
use LLMs to query an image for visual evidence, similar to RepARe Prasad et
al. (2023) which extracts additional visual context to rephrase underspecified
VQA questions.

## Limitations

The plug-and-play capability of ReCoVERR, while facilitating adaptability,
does introduce a degree of engineering complexity due to the sequential
dependencies in predictions. While it opens up flexible choices in modules, it
also introduces an overhead on response time. Our primary method black-box
APIs, and we also tested it with open-source models, as detailed in the Table
4. We leave conducting comprehensive testing with various open-source models
for future work.

The curated benchmarks tailored for VLM evaluation, while being beneficial,
may necessitate further exploration into scenarios that better align with
ecological validity de Vries et al. (2020) and real-world applications for
abstaining. Notably, our focus on image understanding excludes considerations
of underspecified, ambiguous, or safety-related abstaining, opening avenues
for expanding the scope of our approach in the future.

Our experiments are restricted to English, a consequence of leveraging
pretrained VLMs and LLMs, offers opportunities for future advancements in
multilingual contexts.

Finally, we assume that the base VLMs are calibrated for confidence
estimation. Although this is effective to some extent, it leaves room for
future exploration into alternative methodologies for making the models well-
calibrated.

## References

  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_.
  * Chow (1957) C.K. Chow. 1957.  An optimum character recognition system using decision functions.  _IRE Transactions on Electronic Computers_.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_.
  * Dancette et al. (2023) Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. 2023.  Improving selective visual question answering by learning from your peers.  In _Computer Vision and Pattern Recognition CVPR_.
  * De Stefano et al. (2000) Claudio De Stefano, Carlo Sansone, and Mario Vento. 2000.  To reject or not to reject: that is the question-an answer in case of neural classifiers.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 30(1):84–94.
  * de Vries et al. (2020) Harm de Vries, Dzmitry Bahdanau, and Christopher D. Manning. 2020.  Towards ecologically valid research on language user interfaces.  _CoRR_.
  * El-Yaniv et al. (2010) Ran El-Yaniv et al. 2010.  On the foundations of noise-free selective classification.  _Journal of Machine Learning Research_ , 11(5).
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv. 2017.  Selective classification for deep neural networks.  _Advances in neural information processing systems_ , 30.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Gupta et al. (2019) Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019.  Lvis: A dataset for large vocabulary instance segmentation.  In _Computer Vision and Pattern Recognition (CVPR)_.
  * Gupta and Kembhavi (2023) Tanmay Gupta and Aniruddha Kembhavi. 2023.  Visual programming: Compositional visual reasoning without training.  In _Computer Vision and Pattern Recognition (CVPR)_.
  * *Hessel et al. (2022) Jack *Hessel, Jena D *Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022.  The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning.  In _European Conference on Computer Vision (ECCV)_.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021.  Surface form competition: Why the highest probability answer isn’t always right.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2018) Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018.  To trust or not to trust a classifier.  _Neural Information Processing Systems (NeurIPS)_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  In _Association for Computational Linguistics (ACL)_.
  * Khan and Fu (2023) Zaid Khan and Yun Fu. 2023.  Selective prediction for open-ended question answering in black-box vision-language models.  In _R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models_.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023.  Obelics: An open web-scale filtered dataset of interleaved image-text documents.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Liu et al. (2023) Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. 2023.  Llava-plus: Learning to use tools for creating multimodal agents.
  * Mañas et al. (2023) Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2023.  Improving automatic vqa evaluation using large language models.  _arXiv preprint arXiv:2310.02567_.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Computer Vision and Pattern Recognition (CVPR)_.
  * Platt (1999) John Platt. 1999.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.
  * Prasad et al. (2023) Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2023.  Rephrase, augment, reason: Visual grounding of questions for vision-language models.  _arXiv preprint arXiv:2310.05861_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789.
  * Rudinger et al. (2020) Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020.  Thinking like a skeptic: Defeasible inference in natural language.  In _Findings of Empirical Methods in Natural Language Processing (EMNLP)_.
  * Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _European Conference on Computer Vision (ECCV)_.
  * Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_.
  * Shukor et al. (2023) Mustafa Shukor, Alexandre Ramé, Corentin Dancette, and Matthieu Cord. 2023.  Beyond task performance: Evaluating and reducing the flaws of large multimodal models with in-context learning.  _CoRR_ , abs/2310.00647.
  * Subramanian et al. (2023) Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023.  Modular visual question answering via code generation.  _arXiv preprint arXiv:2306.05392_.
  * Surís et al. (2023) Dídac Surís, Sachit Menon, and Carl Vondrick. 2023.  Vipergpt: Visual inference via python execution for reasoning.  _Proceedings of IEEE International Conference on Computer Vision (ICCV)_.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings.  In _Findings of the Association for Computational Linguistics (ACL)_.
  * Whitehead et al. (2022) Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. 2022.  Reliable visual question answering: Abstain rather than answer incorrectly.  In _European Conference on Computer Vision (ECCV)_.
  * Wu et al. (2023) Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E. Gonzalez, and Trevor Darrell. 2023.  See, say, and segment: Teaching lmms to overcome false premises.  _CoRR_.
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv:2303.11381_.
  * You et al. (2023) Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. 2023.  Idealgpt: Iteratively decomposing vision and language reasoning via large language models.  _arXiv preprint arXiv:2305.14985_.
  * Zeng et al. (2022) Andy Zeng, Maria Attarian, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al. 2022.  Socratic models: Composing zero-shot multimodal reasoning with language.  In _International Conference on Learning Representations (ICLR)_.

##  Appendix A Self-Prompting for Producing Calibrated VLM Confidence
Estimates

In order to do selective prediction, it is important that we use a calibrated
confidence estimate i.e. the model confidence $g(x)$ reflects the true
probability of its output being correct Platt (1999). The calibration of a
model’s confidence measure is typically evaluated using the Expected
Calibration Error Guo et al. (2017) over a calibration set.

Calibration has typically been studied in discriminative models, using the
probability of the predicted class as the confidence estimate. However, it is
unclear what makes for a good confidence estimate for a generative VLM that
produces an answer $a$ with multiple tokens. One solution is to sum or average
all the log probabilities in the answer token sequence string; however, this
will be an underestimate of the model’s confidence due to surface form
competition Holtzman et al. (2021).

Instead, we experiment with a _self-prompting_ , approach where the VLM is
prompted to verify the correctness of its answer. Specifically, after
predicting an answer $a$ from the VLM for the question $Q$, we present the
following prompt prompt($Q,a$) to the VLM:

> Question: {$Q$}
> Answer: {$a$}
> Is the given answer correct for the question? Answer yes or no:

We look at the VLM’s next token probability distribution
$P(\cdot|\texttt{prompt},I)$, and compute confidence estimate by looking at
the next token probabilities for the yes and no tokens:

$\displaystyle
P_{\texttt{VLM}}(a|Q)=\frac{P(\texttt{yes}|\texttt{prompt},I)}{P(\texttt{yes}|\texttt{prompt},I)+P(\texttt{no}|\texttt{prompt},I)}$

###  A.1 Comparison Against Other Confidence Measures

Figure 6: Calibration curves and expected calibration error (ECE) for various
VLM confidence measures, using both BLIP2 and InstructBLIP, on A-OKVQA
questions.

We compare our Self-Prompting approach for extracting VLM confidences to
several other confidence measures that look at token probabilities of the
VLM’s prediction. Suppose, for a question $Q$ about an image $I$, the VLM
generates an answer $a=(a_{1},a_{2},...,a_{N})$ consisting of $N$ tokens. We
consider the following confidence measures: Product of Token Probabilities
$\Big{(}\prod_{i=1}^{N}P_{\texttt{VLM}}(a_{i}|I,Q,a_{1...i-1})\Big{)}$, Mean
of Token Probabilities
$\Big{(}\frac{1}{N}\sum_{i=1}^{N}P_{\texttt{VLM}}(a_{i}|I,Q,a_{1...i-1})\Big{)}$
and First Token Probability $\Big{(}P_{\texttt{VLM}}(a_{1}|I,Q)\Big{)}$.
Figure 6 shows calibration curves for the above three confidence measures, as
well as Self-Prompting. We find that Self-Prompting has lower calibration
error than the other three confidence measures. Further, looking at the
confidence distributions, we see that the other three confidence measures
result in the majority of predictions having very low confidence ($<0.5$), and
these under-confident predictions are actually more accurate than their
confidence indicates.

###  A.2 Platt Scaling

Platt scaling Platt (1999) is a technique for calibrating classifiers by
training a logistic regression model over the logits for the output classes.
To train a Platt scaling calibrator, for 12,000 examples in the A-OKVQA
training data, we use Self-Prompting as described above for the off-the-shelf
VLMs to extract logits for the yes and no tokens. We train a logistic
regression model over these logits to predict whether the VLM prediction was
correct or not. Figure 3 shows the effect of Platt scaling calibration on the
VLM confidence estimates.

##  Appendix B Prompts for Text-Only Language Models

Table 6 contains the prompts we used for the various language-only functions
of ReCoVERR.

Function | Model | Inputs | Prompt
---|---|---|---
Question generation |  $\mathcal{M}_{\texttt{QGen}}$ |  $Q$: The question the VLM is trying to confidently answer
$a$:The VLM prediction for above question
$\mathcal{E}_{R}=[e_{1},e_{2},...]$: A list of reliable evidences that have
been collected about the image so far
$K$: The number of questions to generate in this turn. |  You are an AI assistant who has rich visual commonsense knowledge and strong reasoning abilities. You will be provided with:
1\. A target question about an image that you are trying to answer.
2\. Although you won’t be able to directly view the image, you will receive a
general caption that might not be entirely precise but will provide an overall
description.
3\. You may receive some additional evidences about the image.
Your goal is: To effectively analyze the image and select the correct answer
for the question, you should break down the main question into several sub-
questions that address the key aspects of the image.

What you already know about the image:
$e_{1}$
$e_{2}$
…

Target question: $\\{Q\\}$. Generate $K$ sub-questions that might help you
confirm whether the answer to the target question is ’$\\{a\\}$’.
Here are the rules you should follow when listing the sub-questions:
1\. Ensure that each sub-question is independent. It means the latter sub-
questions shouldn’t mention previous sub-questions.
2\. The sub-questions should be separated by a newline character.
3\. Each sub-question should start with "What" or "Is".
4\. Each sub-question should be short (less than 10 words) and easy to
understand.
5\. The sub-question are necessary to distinguish the correct answer.
Sentence paraphrasing |  $\mathcal{M}_{\texttt{QA}\rightarrow\texttt{S}}$ |  $Q$, $a$: The question-answer pair that needs to be paraphrased into a sentence |  Rephrase the question and answer into a single statement.
The re-phrased statement should summarize the question and answer.
The re-phrased statement should not be a question.

Question: Is the dog herding or guiding the cows?
Answer: guiding
Statement: The dog is guiding the cows.

Question: Are there any other written numbers visible in the image?
Answer: no
Statement: There are no other written numbers visible in the image.

Question: Which color of clothing is unique to just one of the two people
here?
Answer: black
Statement: The color of clothing that is unique to just one of the two people
here is black.

Question: Does the picture on the screen involve any human subjects or
animals?
Answer: human
Statement: The picture on the screen involves human subjects.

Question: $\\{Q\\}$
Answer: $\\{a\\}$
Statement:
Checking evidence sufficiency |  $\mathcal{M}_{\texttt{NLI}}$ |  $\mathcal{H}$: Hypothesis statement
$\mathcal{E}_{RR}$: A list of reliable and relevant evidences [$e_{1},e_{2},...$] |  Premise: $\\{\text{evidence statements in }\mathcal{E}_{RR}\text{ concatenated together}\\}$

Hypothesis: $\\{\mathcal{H}\\}$
Can we infer the hypothesis from the premise? Options: yes, no. Answer:
Table 6: Prompts for text-only language models

##  Appendix C OK-VQA and Sherlock Task Details

We apply ReCoVERR using A-OKVQA-calibrated VLMs for two new tasks, OK-VQA and
Sherlock.

OK-VQA Marino et al. (2019) is a VQA task that, similar to A-OKVQA, requires
external knowledge and commonsense reasoning. We evaluate on the OK-VQA
validation set, consisting of 5,046 instances.

Sherlock *Hessel et al. (2022) is an abductive reasoning task, where images
and image regions are paired with inferences. The original task formulations
in Sherlock (retrieval, localization, human rating comparison) do not directly
apply to generative VLMs that take image and text inputs and produce text
outputs. We re-formulate the comparison task into a binary judgment task,
where the VLM must output whether a given inference is true for an image or
not. An image-region-inference triple is paired with two human ratings that
rate the inference as true, false, or neutral. We treat instances where both
humans agree that the inference is true as positive instances, and both humans
agree that it is false as negative instances. We end up with a dataset of 561
image-region-inference triples. Each triple is presented to the VLM by drawing
a region box onto the image, and forming a templated question in the following
format:

> Is the given inference true for the given image? {Inference} Options: yes,
> no

We observe poor calibration on these tasks, possibly due to the out-of-
distribution nature of images with region boxes drawn on, due to which we do
not perform ReCoVERR experiments by calibrating on these respective datasets.

| Risk tolerance = 10% |  | Risk tolerance = 20%
---|---|---|---
VLM |  ReCoVERR Risk |  ReCoVERR Coverage |  Vanilla SelPred Coverage @ ReCoVERR Risk |  |  ReCoVERR Risk |  ReCoVERR Coverage |  Vanilla SelPred Coverage @ ReCoVERR Risk
BLIP2 (Off-the-Shelf) | 14.3 | 26.0 | 21.0 |  | 21.7 | 47.3 | 44.9
BLIP2 (Calibrated) | 14.0 | 23.6 | 23.3 |  | 16.1 | 36.7 | 29.2
InstructBLIP (Off-the-Shelf) | 10.9 | 33.5 | 33.3 |  | 17.9 | 63.7 | 58.5
InstructBLIP (Calibrated) | 11.8 | 38.7 | 32.3 |  | 18.6 | 65.0 | 59.5
Table 7: Coverage of ReCoVERR versus vanilla selective prediction, at the
final resultant risk that ReCoVERR produces. Measurements in blue indicate
ReCoVERR’s coverage exceeding that of vanilla selective prediction, at the
same risk.

##  Appendix D Comparing ReCoVERR and Vanilla Selective Prediction at
ReCoVERR’s Risk

We observe in Table 1 that ReCoVERR has both higher coverage and higher risk
than vanilla selective prediction. However, one simple way to increase the
vanilla selective prediction coverage is to set the threshold $\gamma_{@r}$ at
a risk level that corresponds to ReCoVERR’s risk. Table 7 shows the coverage
of vanilla selective prediction and ReCoVERR at the same risk level as
ReCoVERR’s risk.

|  | Risk tolerance $r$ = 10% |  | Risk tolerance $r$ = 20%
---|---|---|---|---
VLM | Method |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathcal{C}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$) |  |  $\mathcal{R}$($\downarrow$) |  $\Phi_{1}$($\uparrow$) |  $\mathcal{C}$($\uparrow$) |  $\mathbb{R}_{\texttt{SP}}$($\uparrow$)
BLIP2 (Off-the-shelf) | Vanilla SelPred | 6.1 | 3.4 | 3.8 | 6.0 |  | 14.9 | 17.1 | 24.1 | 34.2
Vision Tools | 13.8 | 17.4 | 23.5 | 33.5 |  | 17.0 | 23.5 | 35.1 | 48.3
ReCoVERR | 14.3 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{18.8}}_{\pm 0.2}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{26.0}}_{\pm 0.5}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{37.1}}_{\pm 0.6}}$ |  | 21.7 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{27.1}}_{\pm 0.8}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{47.3}}_{\pm 0.3}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{61.5}}_{\pm 0.8}}$
BLIP2 (Calibrated) | Vanilla SelPred | 4.1 | 3.2 | 3.4 | 5.5 |  | 11.9 | 16.8 | 21.9 | 32.0
Vision Tools | 13.2 | 17.5 | 23.4 | 33.7 |  | 15.1 | 23.9 | 33.6 | 47.2
ReCoVERR | 14.0 | 17.3 | 23.6 | 33.6 |  | 16.1 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{25.3}}_{\pm 0.5}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{36.7}}_{\pm 0.2}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{51.0}}_{\pm 0.6}}$
InstructBLIP (Off-the-shelf) | Vanilla SelPred | 9.3 | 20.5 | 25.1 | 34.8 |  | 17.2 | 38.3 | 57.7 | 72.2
Vision Tools | 10.5 | 26.1 | 32.9 | 44.8 |  | 17.6 | 39.6 | 60.5 | 75.3
ReCoVERR | 10.9 | 26.3 | 33.5 | 45.2 |  | 17.9 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{41.3}}_{\pm 0.3}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{63.7}}_{\pm 0.3}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{78.9}}_{\pm 0.4}}$
InstructBLIP (Calibrated) | Vanilla SelPred | 8.5 | 22.0 | 26.4 | 36.9 |  | 17.5 | 37.2 | 56.6 | 70.5
Vision Tools | 10.4 | 27.2 | 34.1 | 46.5 |  | 17.8 | 38.8 | 59.7 | 74.2
ReCoVERR | 11.8 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{29.8}}_{\pm 0.8}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{38.7}}_{\pm 1.3}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{51.8}}_{\pm 1.4}}$ |  | 18.6 | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{41.4}}_{\pm 0.3}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{65.0}}_{\pm 0.7}}$ | ${\color[rgb]{0.1015625,0.26953125,0.5859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.26953125,0.5859375}{\textbf{79.8}}_{\pm 0.8}}$
Table 8: Metric results as percentages on the A-OKVQA task at two risk
tolerance levels. We evaluate selective prediction methods on the overall
system risk ($\mathcal{R}$), effective reliability ($\Phi_{1}$), coverage
($\mathcal{C}$) and recall ($\mathbb{R}_{\texttt{SP}}$). System risks in red
exceeded tolerance. Measurements in blue indicate when ReCoVERR outperformed
both baselines.