# Continuity of Monge-Ampère potentials with prescribed singularities

Quang-Tuan Dang  The Abdus Salam International Centre for Theoretical Physics
(ICTP), Str. Costiera, 11, 34151 Trieste, TS, Italy<EMAIL_ADDRESS>
(Date: August 28, 2024)

###### Abstract.

We study the continuity of solutions to complex Monge–Ampère equations with
prescribed singularity type. This generalizes the previous results of DiNezza-
Lu and the author [DL17, Dan22a]. As an application we can run the
pluripotential Monge-Ampère flows in [Dan22b] starting at a current with
prescribed singularities.

###### Key words and phrases:

Monge-Ampère operator, prescribed singularities

###### 2020 Mathematics Subject Classification:

32U15, 32Q15, 32W20

##  1\. Introduction

Complex Monge-Ampère equations play a crucial role in studying canonical
metrics in Kähler geometry, following Yau’s solution [Yau78] to the Calabi
conjecture. As evidenced by recent developments in connection with the Minimal
Model Program, it is thus desirable to construct canonical metrics on
varieties with mild singularities; see [EGZ09, BBE+19] and references therein.
They led one to the study of degenerate complex Monge-Ampère equations.

In a series of recent papers [DDL18, DDL21a], Darvas-DiNezza-Lu intensively
studied complex Monge-Ampère equations with prescribed singularities. They
proved the existence and uniqueness of solutions in the context of big
cohomology classes. However, the regularity is unknown.

To state our main result, let $(X,\omega)$ be a compact Kähler manifold of
complex dimension $n$ and $\theta$ be a smooth closed (1,1) form representing
a big cohomology class. We let $\operatorname{{\rm PSH}}(X,\theta)$ denote the
set of $\theta$-psh functions. Recall that the cohomology class $\\{\theta\\}$
is big if there exists $\varphi\in\operatorname{{\rm
PSH}}(X,\theta-\varepsilon\omega)$ with analytic singularities for some
$\varepsilon>0$. Its ample locus is denoted by $\textrm{Amp}(\theta)$.

Fixing $\psi\in\operatorname{{\rm PSH}}(X,\theta)$ and $0\leq f\in L^{p}(X)$,
$p>1$, we look for a solution $\varphi\in\operatorname{{\rm PSH}}(X,\theta)$
satisfying

(1.1) $\theta^{n}_{\varphi}=f\omega^{n},\quad|\varphi-\psi|\leq C,$

where $\theta_{\varphi}^{n}$ denotes the non-pluripolar product of $\varphi$,
introduced in [BEGZ10]. The last condition means that $\varphi$ and $\psi$ has
the same singularity type.

As mentioned in [DDL18], for the solvability of 1.1, one needs to impose the
necessary condition that $\chi$ has model type singularities, i.e.
$\int_{X}\theta_{\chi}^{n}>0$ and $\chi-P_{\theta}[\chi]$ is bounded on $X$
(see Sect. 2.2 for precise definition). It is crucial to emphasize that model
type singularities are natural and appear in many different contexts of
complex differential geometry. For instance, all analytic and minimal
singularities are of the model type.

We state the main result which generalizes the one of DiNezza-Lu [DL17] and of
the author [Dan22a].

######  Theorem 1.1 (see Theorem 3.2).

Assume that $\chi\in\operatorname{{\rm PSH}}(X,\theta)$ has analytic
singularities. Let $\mu=ge^{-u}dV_{X}$ be a positive measure such that
$\mu(X)=\int_{X}\theta_{\chi}^{n}>0$, where $g\in L^{p}(X)$, $p>1$, and $u$ is
a quasi-psh function. Then the unique solution
$\varphi\in\mathcal{E}(X,\theta,P_{\theta}[\chi])$ of the equation

$\theta_{\varphi}^{n}=\mu,\quad\sup_{X}\varphi=0,$

is continuous in $\textrm{Amp}(\theta)\setminus E_{1/q}(u)$, where $q$ is the
conjugate exponent of $p$, i.e., $\frac{1}{p}+\frac{1}{q}=1$, and
$E_{1/q}(u)=\\{x\in X:\nu(u,x)\geq 1/q\\}$ with $\nu(u,x)$ being the Lelong
number of $u$ at $x$.

The existence (and uniqueness) of solution in was shown in [DDL18, DDL21b].
The characterization of solutions belonging to weighted subspace was discussed
in [DV20, DDL23]. A celebrated result of Siu shows that $E_{1/q}(u)$ is a
closed analytic subset, which means that $\varphi$ is continuous in a Zariski
open set.

The case when $\chi$ has minimal singularities has been shown in [Dan22a]. If
we additionally assume that $u$ is bounded on $X$, then the solution $\varphi$
is Hölder continuous in the ample locus $\textrm{Amp}(\theta)$, thanks to
[DDG+14]. In case $\mu$ is a smooth volume form it is expected the solution
$\varphi$ is smooth where $\chi$ is. But even $\chi$ has minimal
singularities; this is a widely open question. According to to [BEGZ10], the
answer is affirmative under an extra assumption that $\\{\theta\\}$ is nef.

The strategy of the proof is the same as that of [DL17, Dan22a]. One can not
expect the Monge-Ampère potential $\varphi$ to be globally bounded in the
context of prescribed singularities. In the latter references, the proof
relied on a theory of the generalized method of Kołodziej [Koł98], which makes
use of a theory of generalized Monge-Ampère capacities, developed in [DL15],
also in [DDL18, DDL21a]. We extend the result and propose in this paper an
alternative proof by using the quasi-psh envelope technique, recently
developed by Guedj-Lu [GL21a, GL21b].

The result obtained in the context of elliptic equations allows us to have an
analogous one in the parabolic counterpart, continuing pluripotential Monge-
Ampère flows from degenerate initial data. This generalizes the one in
[Dan22b], is briefly discussed in Section 4.

##  2\. Preliminaries

In this section we recall some terminology and notation.

###  2.1. Non-pluripolar complex Monge–Ampère measures

We denote $(X,\omega)$ a compact Kähler manifold of dimension $n$ and fix
$\theta$ a smooth closed (1,1) form.

An upper semi-continuous function
$\varphi:X\rightarrow\mathbb{R}\cup\\{-\infty\\}$ is called quasi-plurisub-
harmonic (quasi-psh for short) if it is locally the sum of a smooth and a
plurisubharmonic (psh for short) function. We say that $\varphi$ is
$\theta$-plurisubharmonic ($\theta$-psh for short) if it is quasi-psh, and
$\theta+dd^{c}\varphi\geq 0$ in the sense of currents, where $d^{c}$ is
normalized so that $dd^{c}=\frac{i}{\pi}\partial\bar{\partial}$. We let
$\operatorname{{\rm PSH}}(X,\theta)$ denote the set of all $\theta$-psh
functions which are not identically $-\infty$. The cohomology class
$\\{\theta\\}$ is big if the set $\operatorname{{\rm
PSH}}(X,\theta-\varepsilon\omega)$ is not empty for some $\varepsilon>0$.

From now on, we assume that $\\{\theta\\}$ is big, unless specified
otherwise.. Following Demailly [Dem92], we can find a closed positive
$(1,1)$-current $T_{0}\in\\{\theta\\}$ such that

$T=\theta+dd^{c}\chi\geq 2\delta_{0}\omega$

for some $\delta_{0}>0$ with $\chi$ a quasi-psh function with _analytic
singularities_ , i.e. locally
$\chi=c\log\left[\sum_{j=1}^{N}|f_{j}|^{2}\right]+v,$ where $v$ is a bounded
function and the $f_{j}$’s are holomorphic functions. We see that on

$\Omega:=X\setminus\\{\chi=-\infty\\},$

$\chi$ is smooth. It moreover follows from [Bou04] that we can choose $\chi$
such that $\Omega$ is the largest Zariski open subset of all points $x\in X$
for which there exists a Kähler current $T_{x}\in\alpha$ with analytic
singularities such that $T_{x}$ is smooth in a neighborhood of $x$. This locus
is called the ample locus of $\theta$, also denoted by $\textrm{Amp}(\theta)$.

Given $\varphi,\psi\in\operatorname{{\rm PSH}}(X,\theta)$, we say that
$\varphi$ is less singular than $\psi$, and denote by $\psi\preceq\varphi$, if
there exists a constant $C$ such that $\psi\leq\varphi+C$ on $X$. We say that
$\varphi,\psi$ have the same singularity type, and denote by
$\varphi\simeq\psi$ if $\varphi\preceq\psi$ and $\psi\preceq\varphi$. This
defines an equivalence relation on $\operatorname{{\rm PSH}}(X,\theta)$, whose
equivalence classes are the singularity types $[\varphi]$. There is a natural
least singular potential in $\operatorname{{\rm PSH}}(X,\theta)$ given by

$\displaystyle V_{\theta}:=\sup\\{\varphi\in\operatorname{{\rm
PSH}}(X,\theta):\varphi\leq 0\\}.$

A function $\varphi$ is said to have minimal singularities if it has the same
singularity type as $V_{\theta}$. In particular, $V_{\theta}=0$ if $\theta$ is
semi-positive. Note aso that $V_{\theta}$ is locally bounded in the ample
locus.

Let $\theta^{1},\cdots,\theta^{n}$ closed smooth real (1,1) form representing
big cohomology classes and $\varphi_{j}\in\operatorname{{\rm
PSH}}(X,\theta^{j})$. Following the construction of Bedford-Taylor [BT87], it
has been shown in [BEGZ10] that for each $k\in\mathbb{N}$,

$\mathbf{1}_{\cap_{j}\\{\varphi_{j}>V_{\theta^{j}}-k\\}}\theta^{1}_{\max(\varphi_{1},V_{\theta^{1}}-k)}\wedge\cdots\wedge\theta^{n}_{\max(\varphi_{n},V_{\theta^{n}}-k)}$

is well-defined as a Borel positive measure with finite total mass. The
sequence of these measures is non-decreasing in $k$ and it converges weakly to
the so called Monge-Ampère product, denoteted by

$\theta^{1}_{\varphi_{1}}\wedge\cdots\wedge\theta^{n}_{\varphi_{n}}.$

This does not charge on pluripolar sets by definition. When
$\theta^{1}=\cdots=\theta^{n}=\theta$ and $\varphi_{1}=\cdots=\varphi_{n}$ we
obtain the non-pluripolar Monge-Ampère measure of $f$, denote by
$(\theta+dd^{c}\varphi)^{n}$ or simply by $\theta_{\varphi}^{n}$.

Let $\phi_{j}\operatorname{{\rm PSH}}(X,\theta^{j})$ be such that $\phi_{j}$
is less singular than $\varphi_{j}$. By [DDL18, Thm. 2.4] we have that

$\int_{X}\theta^{1}_{\varphi_{1}}\wedge\cdots\wedge\theta^{n}_{\varphi_{n}}\leq\int_{X}\theta^{1}_{\phi_{1}}\wedge\cdots\wedge\theta^{n}_{\phi_{n}}.$

We say that
$\theta^{1}_{\varphi_{1}}\wedge\cdots\wedge\theta^{n}_{\varphi_{n}}$ has full
mas with respect to
$\theta^{1}_{\phi_{1}}\wedge\cdots\wedge\theta^{n}_{\phi_{n}}$ if the equality
holds. We let
$\mathcal{E}(X,\theta^{1}_{\phi_{1}},\ldots,\theta^{n}_{\phi_{n}})$ denote the
set of such $n$-tuple $(\varphi_{1},\ldots,\varphi_{n})$. In the particular
case when the potentials involved are from the same cohomology class
$\\{\theta\\}$, and with $\phi$ less singular than $\varphi$ and
$\int_{X}\theta_{\varphi}^{n}=\int_{X}\theta_{\phi}^{n}$ then we simply write
$\varphi\in\mathcal{E}(X,\theta,\phi)$. Also, we simply write
$\mathcal{E}(X,\theta)$ when $\phi=V_{\theta}$.

We recall here the plurifine locality of the non-pluripolar Monge-Ampère
measure (see [BEGZ10, Sect. 1.2]) for later use.

######  Lemma 2.1.

Assume that $\varphi$, $\psi$ are $\theta$-psh function such that
$\varphi=\psi$ on an open set $U$ in the plurifine topology. Then

$\mathbf{1}_{U}\theta_{\varphi}^{n}=\mathbf{1}_{U}\theta_{\psi}^{n}.$

For practice, we stress that sets of the form $\\{u<v\\}$, where $u$, $v$ are
quasi-psh functions, are open in the plurifine topology.

We recall the following classical inequality (see e.g. [DDL21b, Lemma 4.5]).

######  Lemma 2.2.

Let $\varphi,\psi\in\operatorname{{\rm PSH}}(X,\theta)$ be such that
$\varphi\leq\psi$. Then

$\mathbf{1}_{\\{\varphi=\psi\\}}\theta^{n}_{\varphi}\leq\mathbf{1}_{\\{\varphi=\psi\\}}\theta^{n}_{\psi}.$

###### Proof.

For the reader’s convenience, we briefly give a proof here. If $\varphi$ and
$\psi$ are locally bounded, the result follows, due to Demailly (see e.g.
[GZ17, Theorem 3.23]). For general case, set
$\varphi^{t}:=\max(\varphi,V_{\theta}-t)$,
$\psi^{t}:=\max(\psi,V_{\theta}-t)$. Then $\varphi^{t}$ and $\psi^{t}$ are
locally bounded on $\Omega$, it follows that

$\mathbf{1}_{\\{\varphi>V_{\theta}-t\\}\cap\\{\psi>V_{\theta}-t\\}\cap\\{\varphi^{t}=\psi^{t}\\}}\theta^{n}_{\varphi^{t}}\leq\mathbf{1}_{\\{\varphi>V_{\theta}-t\\}\cap\\{\psi>V_{\theta}-t\\}\cap\\{\varphi^{t}=\psi^{t}\\}}\theta^{n}_{\psi^{t}}$

using plurifine locality. Letting $t\to+\infty$, the inequality follows. ∎

###  2.2. Quasi-plurisharmonic envelopes and model potentials

Given a measurable function $h:X\to\mathbb{R}$, we define the $\theta$-psh
envelope of $h$ by

$P_{\theta}(h):=(\sup\\{u\in\operatorname{{\rm PSH}}(X,\theta):u\leq
h\;\text{on}\,X\\})^{*}$

where the star means that we take the upper semi-continuous regularization.
Given a $\theta$-psh function $\phi$, J. Ross and D. Witt Nyström [RW14]
introduced the ”rooftop envelope”

$P_{\theta}[\phi](h)=\left(\lim_{C\to+\infty}P_{\theta}(\min(\phi+C,h))\right)^{*}.$

If $h=0$ we simply write $P_{\theta}[\phi]$. A potential
$\phi\in\operatorname{{\rm PSH}}(X,\theta)$ is called a model potential if
$\int_{X}\theta_{\phi}^{n}>0$ and $\phi=P_{\theta}[\phi]$.

######  Proposition 2.3.

Assume that $h=a\varphi-b\psi$, where $\varphi,\psi$ are quasi-psh functions,
and $a,b$ are positive constants. If $P_{\theta}(h)\not\equiv-\infty$ then
$(\theta+dd^{c}P(h))^{n}$ puts no mass on the contact set $\\{P(h)=h\\}$.

We note that $h=a\varphi-b\psi$ is well-defined in the complement of a
pluripolar set and by assumption $P_{\theta}(h)\in\operatorname{{\rm
PSH}}(X,\theta)$. Moreover, $P_{\theta}(h)\leq a\varphi-b\psi$ means that
$P_{\theta}(h)+b\psi\leq a\varphi$ on $X$. A generalized result is proved when
$h$ is merely quasi-continuous, c.f. [DDL23, Theorem 2.7].

###### Proof.

It is already well-known in some literature [GLZ19, DDL21b]. We just sketch
the proof here. We assume that $\varphi$, $\psi$ are $\omega$-psh. Thanks to
[Dem92], we can find $\varphi_{j}\in\operatorname{{\rm
PSH}}(X,\omega)\cap\mathcal{C}^{\infty}(X)$ be such that
$\varphi_{j}\searrow\varphi$. We set $u_{j}=P_{\theta}(a\varphi_{j}-b\psi)$
and $u:=P_{\theta}(h)$, so $u_{j}\searrow u$ (see [GLZ19, Prop. 2.2]). Since
$a\varphi_{j}-b\psi$ is lower semi-continuous, the set
$\\{u_{j}<a\varphi_{j}-b\psi\\}$ is open. It thus follows from a classical
balayage argument that for each $j$, $\theta^{n}_{u_{j}}$ vanishes on
$\\{u_{j}<a\varphi_{j}-b\psi\\}$. Also, $u_{j}\leq a\varphi_{j}-b\psi$, hence
we have that

$\int_{X}\min(a\varphi_{j}-b\psi-u_{j},1)\theta_{u_{j}}^{n}=0$

by [GLZ19, Prop. 2.5]. The functions $\min(a\varphi_{j}-b\psi-u_{j},1)$ are
uniformly bounded, are quasi-continuous and converge in capacity to
$\min(a\varphi-b\psi-u,1)$ (which is quasi-continuous and bounded on $X$). By
letting $j\to+\infty$, the conclusion follows from [DDL21b, Thm 2.3]. ∎

######  Proposition 2.4.

Let $\varphi,\psi\in\operatorname{{\rm PSH}}(X,\theta)$ be such that $\psi$ is
more singular than $P_{\theta}[\varphi]$. Then for any $b>0$,
$P_{\omega}(b\varphi-b\psi)$ is a $\omega$-psh function with full Monge-Ampère
mass.

###### Proof.

We adapt the argument in [Dan22a, Prop. 2.10] which goes back to [DDL21b]. We
assume that $\theta\leq A\omega$ for some $A>0$. For each $j\in\mathbb{N}$ we
set $\varphi_{j}:=\max(\varphi,\psi-j)$ and
$u_{j}:=P_{\omega}(b\varphi_{j}-b\psi)$. Then $(u_{j})$ is a decreasing
sequence of $\omega$-psh functions, and $u_{j}\geq-jb$. We will show that
$\lim_{j}\psi_{j}$ is not identically $-\infty$. We let for each $j$,
$D_{j}:=\\{u_{j}=b\varphi_{j}-b\psi\\}$ denote the contact set. Observe that
the sets $D_{j}$ are non-empty for $j$ large enough. Since $u_{j}+b\psi\leq
b\varphi_{j}$ it follows from the maximum principle and Proposition 2.3 that

$\omega_{u_{j}}^{n}\leq\mathbf{1}_{D_{j}}[\omega+dd^{c}u_{j}+b(A\omega+dd^{c}\psi)]^{n}\leq\mathbf{1}_{D_{j}}((Ab+1)\omega+dd^{c}b\varphi_{j})^{n}$

Set $\tilde{\omega}:=\left(\frac{1}{b}+A\right)\omega$. Fix $t>0$. Since
$\varphi_{j}=\varphi$ on $\\{\varphi>V_{\theta}-t/b\\}$ for $j>t/b$, by
plurifine locality we have for $j>t/b$,

$\int_{\\{\varphi\leq\psi-t/b\\}}\tilde{\omega}_{u_{j}}^{n}=\int_{X}\tilde{\omega}_{\varphi_{j}}^{n}-\int_{\\{\varphi>\psi-t/b\\}}\tilde{\omega}_{\varphi}^{n}.$

We see that

$\\{u_{j}\leq-t\\}\cap
D_{j}=\\{\varphi_{j}\leq\psi-t/b\\}\subset\\{\varphi\leq\psi-t/b\\}.$

From these things above, we have that

(2.1)
$\begin{split}\omega_{u_{j}}^{n}(u_{j}\leq-t)&\leq\mathbf{1}_{D_{j}}b^{n}\tilde{\omega}_{\varphi_{j}}^{n}(u_{j}\leq-t)\\\
&\leq b^{n}\tilde{\omega}_{\varphi_{j}}^{n}(\varphi\leq\psi-t/b)\\\ &\leq
b^{n}\left(\int_{X}\tilde{\omega}_{\varphi_{j}}^{n}-\int_{\\{\varphi>\psi-t/b\\}}\tilde{\omega}_{\varphi}^{n}\right)\end{split}$

Suppose by contradiction that $\sup_{X}u_{j}\rightarrow-\infty$ as
$j\to+\infty$. It thus follows that $\\{u_{j}\leq-t\\}=X$ for $j$ large
enough, $t$ being fixed. Hence, for $j>0$ large enough, (2.1) becomes

$\int_{X}\omega^{n}\leq
b^{n}\left(\int_{X}\tilde{\omega}_{\varphi_{j}}^{n}-\int_{\\{\varphi>\psi-t/b\\}}\tilde{\omega}_{\varphi}^{n}\right).$

Letting $j\rightarrow+\infty$, we obtain

(2.2) $\int_{X}\omega^{n}\leq
b^{n}\left(\int_{X}\tilde{\omega}_{\varphi_{j}}^{n}-\int_{\\{\varphi>\psi-t/b\\}}\tilde{\omega}_{\varphi}^{n}\right),$

where we have used that

$\tilde{\omega}_{\varphi_{j}}^{n}=\sum_{k=0}^{n}\binom{n}{k}(\tilde{\omega}-\theta)^{k}\wedge\theta_{\varphi_{j}}^{n-k}\to\tilde{\omega}_{\varphi}^{n}$

in the weak sense of measures on $X$, thanks to [DDL18, Thm. 2.3, Rmk. 2.5].
Indeed, since $\psi$ is more singular than $P_{\theta}[\varphi]$ hence
$\varphi\leq\varphi_{j}\leq P_{\theta}[\varphi]$. By monotonicity, for
$k=0,1,\ldots,n$,

$\int_{X}(\tilde{\omega}-\theta)^{k}\wedge\theta_{\varphi}^{n-k}=\int_{X}(\tilde{\omega}-\theta)^{k}\wedge\theta_{\varphi_{j}}^{n-k}=\int_{X}(\tilde{\omega}-\theta)^{k}\wedge\theta_{P_{\theta}[\varphi]}^{n-k}.$

Finally, letting $t\rightarrow+\infty$ in (2.2) we obtain a contradiction.
Consequently, $u_{j}$ decreases to a $\omega$-psh function, we infer that
$P_{\omega}(b\varphi-b\psi)$ is a $\omega$-psh function for any $b>0$.

We proceed exactly the same as [Dan22a, Prop. 2.10] to obtain that
$P_{\omega}(b\varphi-b\psi)\in\mathcal{E}(X,\omega)$. ∎

##  3\. Proof of the Main Theorem

According to [Bou04] we can find $\chi$ a quasi-psh function with analytic
singularities such that

$\theta+dd^{c}\chi\geq 2\delta_{0}\omega$

for some $\delta>0$ and moreover
$\textrm{Amp}(\theta)=X\setminus\\{\chi=-\infty\\}$. Let $\phi$ be a model
type singularity, i.e., $\phi=P_{\theta}[\phi]$. We assume in this section
that $\phi$ is less singular than $\chi$. We emphasize that the assumptions on
$\phi$ are truly natural, for instance, $\phi=V_{\theta}$ or
$\phi=P_{\theta}[\chi]$, as described in [DDL18, Remark 1.6].

Given a non-negative Radon measure $\mu$ whose total mass is
$\int_{X}\theta^{n}_{\phi}>0$, we consider the Monge-Ampère equation

(3.1) $\theta^{n}_{\varphi}=\mu.$

The systematic study of such equations with precribed singularities in big
cohomology classes has been initiated in [DDL18, DDL21a]. It has been shown
there that (3.1) admits a unique normalized solution
$\varphi\in\mathcal{E}(X,\theta,\phi)$ if and only if $\mu$ put no mass on
pluripolar sets. The characterization of solutions belonging to weighted
subspace was proved in [DV20, DDL23].

Our goal is to prove the following:

######  Theorem 3.1.

Assume $\phi$ is as above. Let $\varphi\in\mathcal{E}(X,\theta,\phi)$ be
normalized by $\sup_{X}\varphi=0$. Assume that

$\theta^{n}_{\varphi}\leq e^{-u}gdV,$

for some quasi-psh function $u$ on $X$, and $0\leq g\in L^{p}(dV)$, with
$p>1$. Assume that $u$ is locally bounded on an open set $U\subset{\rm
Amp}(\theta)$. Then $\varphi$ is continuous on $U$.

For a quasi-psh function $u$ and $c>0$ we set

$E_{c}(u):=\\{x\in X,\nu(u,x)\geq c\\},$

where $\nu(u,x)$ denote the Lelong number of $u$ at $x$. A well-known result
of Siu asserts that the Lelong super-level sets $E_{c}(\varphi)$ are closed
analytic subsets of $X$.

As a consequence of Theorem 3.1 we obtain the following theorem which implies
our theorem in the introductory part.

######  Theorem 3.2.

Assume $\phi$ is as above. Assume $\nu=gdV$ to be a Radon measure, with $0\leq
g\in L^{p}(dV)$ for some $p>1$. Let $\mu$ be a non-pluripolar measure such
that $\mu(X)=\int_{X}\theta_{\phi}^{n}$. Assume that $\mu=fd\nu$, with $f\leq
e^{-u}$ for some quasi-psh function $u$ on $X$. Let
$\varphi\in\mathcal{E}(X,\theta,\phi)$ be the unique normalized solution to
(3.1). Then $\varphi$ is continuous on $\textrm{Amp}(\theta)\setminus
E_{1/q}(u)$, where $q$ denotes the conjugate exponent of $p$.

###### Proof.

The proof relies on Demailly’s equsingular approximation theorem [Dem15]. We
argue the same as in [Dan22a, Theorem 3.1]. ∎

###### Proof of Theorem 3.1.

The proof relies on quasi-psh envelope technique developed in [GL21a]. This
can also be applied to give an alternative proof for [Dan22a, Theorem 3.1]. We
divide the proof in several steps

Step 1. We prove that $\varphi$ is locally bounded on $U$.

We pick $a>0$ so that $au$ is $\delta_{0}\omega$-psh. Set $\psi:=\chi+au$. We
thus have

$\theta+dd^{c}\psi\geq\delta_{0}\omega$

and $\psi$ is locally bounded on $U$. We also obtain $\psi\leq\phi+au$. We
claim that

$\varphi\geq\psi-A$

for a positive constant $A$ only depending on $\delta_{0}$, $p$, $dV$, and
$\int_{X}e^{-P_{\omega}(a^{-1}\varphi-a^{-1}\phi)}gdV$.

Set $\tilde{\varphi}:=P_{\delta_{0}\omega}(\varphi-\psi)$. By Proposition 2.4,
$\tilde{\varphi}$ is an $\omega$-psh function. One can show that
$\sup_{X}\tilde{\varphi}$ is uniformly bounded from above by applying the
argument in [GZ17]. Without loss of generality we can normalize
$\sup_{X}\psi=0$. The set $G:=\\{\psi>-1\\}$ is non empty plurifine open set.
We observe that $\tilde{\varphi}(x)\leq(\varphi-\psi)(x)\leq 1$ for $x\in G$,
hence

$\tilde{\varphi}(x)-1\leq V_{G,\omega}:=\sup\\{u\in\operatorname{{\rm
PSH}}(X,\omega):u|_{G}\leq 0\\}.$

By [GZ17, Thm . 9.17.1] we have that $\sup_{X}V_{G,\omega}<+\infty$ since $G$
is non-pluripolar, hence $\sup_{X}\tilde{\varphi}\leq
1+\sup_{X}V_{G,\omega}<+\infty$.

Since $\varphi-\psi$ is bounded from below and quasi-continuous, it follows
from Theorem that the Monge-Ampère meaure
$(\delta_{0}\omega+dd^{c}\tilde{\varphi})^{n}$ is concentrated on the contact
set $D:=\\{\tilde{\varphi}+\psi=\varphi\\}$. We thus get

$(\delta_{0}\omega+dd^{c}\tilde{\varphi})^{n}\leq\mathbf{1}_{D}(\theta+dd^{c}(\tilde{\varphi}+\psi))^{n}$

We also observe that $\tilde{\varphi}+\psi$ is an $\theta$-psh function such
that $\tilde{\varphi}+\psi\leq\varphi$ on $X$. It follows from Lemma 2.2 that

$\mathbf{1}_{D}(\theta+dd^{c}(\tilde{\varphi}+\psi))^{n}\leq\mathbf{1}_{D}(\theta+dd^{c}\varphi)^{n}.$

Therefore, we have

$\begin{split}(\delta_{0}\omega+dd^{c}\tilde{\varphi})^{n}&\leq\mathbf{1}_{D}(\theta+dd^{c}\varphi)^{n}\\\
&\leq\mathbf{1}_{D}e^{-u}dgV\\\
&=\mathbf{1_{D}}e^{a^{-1}\tilde{\varphi}}e^{-a^{-1}(\varphi-\chi)}gdV\\\
&\leq\mathbf{1_{D}}e^{a^{-1}\sup_{X}\tilde{\varphi}}e^{-P_{\omega}(a^{-1}\varphi-a^{-1}\phi)}gdV\end{split}$

It follows from the Hölder inequality that
$e^{-P_{\omega}(a^{-1}\varphi-a^{-1}\phi)}g$ belongs to $L^{r}(X,dV)$ for some
$r\in(1,p)$. By Kołodziej’s estimate [Koł98] (see also [GL21a] for alternative
one), we infer that $\tilde{\varphi}\geq-A$ is uniformly bounded from below.
This proves our claim.

Step 2. There exists a sequence of functions $\varphi_{j}\in\operatorname{{\rm
PSH}}(X,\theta)\cap\mathcal{C}^{0}(\textrm{Amp}(\theta))$ such that
$\phi\geq\varphi_{j}$ decreases to $\varphi$.

For convenience, we normalize $\varphi$ so that $\sup_{X}\varphi=-1$. Let
$0\geq h_{j}$ be a sequence of smooth functions decreasing to $\varphi$. Then
the sequence of $\theta$-psh functions $\varphi_{j}:=P_{\theta}[\chi](h_{j})$
decreases to $\varphi$ as $j\to+\infty$. Indeed, since the operator
$P_{\theta}$ is monotone, hence the sequence $\varphi_{j}$ is decreasing to a
$\theta$-psh function $v$. Note that $[\phi]\geq[\varphi]$ hence $\varphi$ is
a candidate defining $\varphi_{j}$ we thus have $\varphi_{j}\geq\varphi$ for
all $j$ so $v\geq\varphi$. Moreover, $v(x)\leq\varphi_{j}(x)\leq
h_{j}(x),\forall\;x\in X$, for all $j$, hence $v(x)\leq\varphi(x)$, as
claimed.

Moreover, it follows from [McC20, Theorem 1.1] that $\varphi_{j}$ is
continuous outside the singular locus of $\chi$, hence on
$\textrm{Amp}(\theta)$.

Step 3. Continuity of solutions. We finally adapt the arguments in [GL21b,
Theorem 3.7] to prove the continuity of the solution $\varphi$ on $U$.

Fix $\lambda\in(0,1)$. Since Proposition 2.4 ensures that for any $b>0$,
$P_{\omega}(b\varphi-b\phi)$ has zero Lelong number everywhere on $X$ it
follows from the Hölder inequality that $ge^{-P_{\omega}(b\varphi-b\phi)}\in
L^{r}(dV)$ for some $r>1$. It was shown (see e.g. [Koł98, EGZ09]) that there
exists a bounded $\delta_{0}\omega$-psh function solving

$(\delta_{0}\omega+dd^{c}u_{j,\lambda})^{n}=e^{u_{j,\lambda}}(g_{j,\lambda}e^{-P_{\omega}(b\varphi-b\phi)}+h)dV,$

where $h=\omega^{n}/dV$,
$g_{j,\lambda}=\lambda^{-n}\mathbf{1}_{\\{\varphi<\varphi_{j}-\lambda\\}}g$.
Moreover, $g_{j,\lambda}e^{-P_{\omega}(\varphi-\phi)}\to 0$ in $L^{r}$. It
thus follows from the stability property (see e.g. [GLZ18]) that
$u_{j,\lambda}$ uniformly converges to 0 as $j\to+\infty$. We consider

$v_{j,\lambda}:=(1-\lambda)\varphi_{j}+\lambda(\psi+u_{j,\lambda})-C\lambda$

for $C>0$ to be chosen hereafter. We observe that
$v_{j,\lambda}\leq\phi+\lambda au$ since $\chi\leq\phi$ and
$\varphi_{j}\leq\phi$ by Step 2. We compute

(3.2) $(\theta+dd^{c}v_{j,\lambda})^{n}\geq
e^{u_{j,\lambda}}\mathbf{1}_{\\{\varphi<\varphi_{j}-\lambda\\}}ge^{-P_{\omega}(b\varphi-b\phi)}dV.$

By previous step, we have $\varphi_{j}\geq\varphi\geq\psi-A$ for a positive
constant $A$. Hence on the set $\\{\varphi<v_{j,\lambda}\\}$,

$\varphi<\varphi_{j}-\lambda(\varphi_{j}-\psi)+\lambda\sup_{X}|u_{j,\lambda}|-C\lambda\leq\varphi_{j}-\lambda$

where we have chosen $C>1+\sup_{X}|u_{j,\lambda}|+A$. Moreover on this set we
have

(3.3) $\begin{split}e^{u_{j,\lambda}}ge^{-P_{\omega}(b\varphi-b\phi)}&\geq
e^{-\sup_{X}|u_{j,\lambda}|}ge^{-b\varphi+b\phi}\\\ &\geq
e^{-\sup_{X}|u_{j,\lambda}|}ge^{-bv_{j,\lambda}+b\phi}\\\ &\geq
ge^{Ca^{-1}-u}\end{split}$

since $v_{j,\lambda}\leq\phi+\lambda au$ with $b=(\lambda a)^{-1}$. Therefore,
it follows from (3.2) and (3.3) that

$e^{Ca^{-1}}(\theta+dd^{c}\varphi)^{n}\leq(\theta+dd^{c}v_{j,\lambda})^{n}\leq(\theta+dd^{c}\max(\varphi,v_{j,\lambda}))^{n}$

on the set $\\{\varphi<v_{j,\lambda}\\}$. Since
$\varphi\in\mathcal{E}(X,\theta,\phi)$ we can apply the comparison principle
(see [DDL21b, Lemma 2.3]) to get

$e^{Ca^{-1}}\int_{\\{\varphi<v_{j,\lambda}\\}}\theta_{\varphi}^{n}\leq\int_{\\{\varphi<v_{j,\lambda}\\}}\theta^{n}_{\max(\varphi,v_{j,\lambda})}\leq\int_{\\{\varphi<v_{j,\lambda}\\}}\theta_{\varphi}^{n}.$

It thus follows that $\varphi\geq\max(\varphi,v_{j,\lambda})\geq
v_{j,\lambda}$ a.e. with respect to the measure $\theta_{\varphi}^{n}$, hence
everywhere by the domination principle (see [DDL18, Proposition 3.11]).
Letting $j\to+\infty$ we obatain

$\liminf_{j\to+\infty}\inf_{K}(\varphi-\varphi_{j})\geq-(\sup_{K}|\psi|+C)\lambda$

for any compact set $K\Subset U$. Letting $\lambda\to 0$ we have that
$\varphi_{j}\to\varphi$ uniformly on $K$. This completes the proof. ∎

##  4\. Pluripotential Monge-Ampère flows through prescribed singularities

The results obtained in the previous section allow us to obtain analogous ones
in the parabolic counterpart. We investigate in this section the following
complex Monge-Ampère flow

(CMAF) $\displaystyle
dt\wedge(\omega_{t}+dd^{c}\varphi_{t})^{n}=e^{\dot{\varphi}_{t}+F(t,\cdot,\varphi_{t})}fdV\wedge
dt$

on $X_{T}$, where

  * •

$X_{T}:=(0,T)\times X$ with $T<+\infty$;

  * •

$0\leq f\in L^{p}(X,dV)$ for some $p>1$, and $f>0$ almost everywhere;

  * •

$(\omega_{t})_{t\in[0,T)}$ is a smooth family of closed $(1,1)$-forms on $X$
such that

$g(t)\theta\leq\omega_{t},\quad\forall\,t\in[0,T),$

with $g(t)$ an increasing smooth positive function on $[0,T]$.

  * •

$F:[0,T]\times X\times\mathbb{R}\rightarrow\mathbb{R}$ is continuous on
$[0,T]\times X\times\mathbb{R}$, increasing in $r$ and is uniformly Lipschitz,
convex in $(t,r)\in[0,T]\times\mathbb{R}$,

  * •

$\varphi:[0,T)\times X\to\mathbb{R}$ is the unknown function, with
$\varphi_{t}=\varphi(t,\cdot)$.

We consider an $\theta$-psh function $\phi$ as previous, i.e., $\phi$ is model
type and is less singular than $\chi$. We recall that from [DDL18, DDL21a]
there exists an $\theta$-psh function $\rho_{\phi}$ normalized by
$\sup_{X}\rho_{\phi}=0$ such that

$(\theta+dd^{c}\rho_{\phi})^{n}=2^{n}e^{c_{1}}fdV,\quad[\rho_{\phi}]=[\phi]$

where $c_{1}$ the normalizing constant such that
$\int_{X}\theta_{\phi}^{n}=\int_{X}2^{n}e^{c_{1}}fdV$.

Let now $\varphi_{0}$ be a $\omega_{0}$-psh function which is less singular
than $g(0)(\rho_{\rho}+\chi)/2$. Then there exists a constant $C_{0}>0$ such
that

$\displaystyle g(0)\frac{\rho_{\phi}+\chi}{2}-C_{0}\leq\varphi_{0}.$

######  Definition 4.1.

The set $\mathcal{P}(X_{T},\omega)$ of parabolic potentials consists of
functions $\varphi:X_{T}\rightarrow[-\infty,+\infty)$ such that

  * •

$\varphi$ is upper semi-continuous on $X_{T}$ and $\varphi\in L^{1}_{\rm
loc}(X_{T})$;

  * •

for each $t\in(0,T)$ fixed, the slice $\varphi_{t}:x\mapsto\varphi(t,x)$ is
$\omega_{t}$-psh on $X$;

  * •

for any compact subinterval $J\subset(0,T)$, there exists a positive constant
$\kappa=\kappa_{J}(\varphi)$ such that

(4.1) $\displaystyle\partial_{t}\varphi\leq\kappa-\kappa(\rho_{\phi}+\chi),$

in the sense of distributions on $J\times\Omega$.

######  Definition 4.2.

We say that a parabolic potential $\varphi\in\mathcal{P}(X_{T},\omega)$ is a
pluripotential subsolution to (CMAF) on $X_{T}$ if

  * •

for each $t\in(0,T)$ fixed, the $\omega_{t}$-psh function $\varphi(t,\cdot)$
is locally bounded in $\Omega$

  * •

the inequality

$\displaystyle(\omega_{t}+dd^{c}\varphi_{t})^{n}\wedge dt\geq
e^{\dot{\varphi}_{t}+F(t,\cdot,\varphi_{t})}fdV\wedge dt$

holds in the sense of measures in $(0,T)\times\Omega$.

######  Definition 4.3.

A Cauchy datum for (CMAF) is a $\omega_{0}$-psh function
$\varphi_{0}:X\rightarrow\mathbb{R}$ as above. We say
$\varphi\in\mathcal{P}(X_{T},\omega)$ is a subsolution to the Cauchy problem:

$\displaystyle(\omega_{t}+dd^{c}u_{t})^{n}=e^{\partial_{t}u_{t}+F(t,\cdot,u_{t})}fdV,\quad
u\big{|}_{\\{0\\}\times X}=\varphi_{0}$

if $\varphi$ is a pluripotential subsolution to (CMAF) such that
$\limsup_{t\rightarrow 0}\varphi(t,x)\leq\varphi_{0}(x)$ for all $x\in X$.

We let $\mathcal{S}_{\varphi_{0},f,F}(X_{T})$ denote the set of pluripotential
subsolutions to the Cauchy problem above.

######  Lemma 4.4.

The set $\mathcal{S}_{\varphi_{0},f,F}(X_{T})$ is non-empty, uniformly bounded
from above on $X_{T}$, and stable under finite maxima.

###### Proof.

We proceed the same arguments as in [Dan22b, Lemma 2.2] to obtain a
subsolution

(4.2) $\underline{u}(t,x):=g(t)\frac{\rho_{\phi}(x)+\chi(x)}{2}-C_{1}(t+1),$

and all subsolutions are uniformly bounded by $M_{0}$. ∎

######  Definition 4.5.

We let

$\displaystyle
U=U_{\varphi_{0},f,F,X_{T}}:=\sup\\{\varphi\in\mathcal{S}_{\varphi_{0},f,F}(X_{T}):\underline{u}\leq\varphi\leq
M_{0}\\}$

denote the upper envelope of all subsolutions.

In the same vain we obtain the regularity in time $t$ of the Perron upper
envelope.

######  Theorem 4.6.

There exists uniform constants $L_{U}>0$, $C_{U}>0$ such that for

(4.3) $\displaystyle t|\partial_{t}U(t,x)|\leq
L_{U}-L_{U}(\rho_{\phi}(x)+\chi(x));$ (4.4) $\displaystyle
t^{2}\partial_{t}^{2}U(t,x)\leq C_{U}-C_{U}(\rho_{\phi}(x)+\chi(x)),$

in the sense of distributions in $X_{T}$.

######  Theorem 4.7.

The upper envelope $U:=U_{\varphi_{0},f,F,X_{T}}$ is a pluripotential solution
to the Cauchy problem for the parabolic complex Monge-Ampère equation (CMAF)
in $X_{T}$. Moreover, $U$ is locally uniformly semi-concave in
$(0,T)\times\Omega$.

###### Proof.

The proof relies on a balayage process. We proceed exactly the same as in
[Dan22b, Theorem 3.1]. ∎

######  Theorem 4.8.

For each $t\in(0,T)$, $U_{t}$ is continuous in $\Omega$.

###### Proof.

We have seen that

$U_{t}\geq g(t)\frac{\rho_{\phi}+\chi}{2}-C_{1}(t+1)\geq g(t)\chi-C(t)$

for $C(t)$ a positive constant only depending on $t$. Next, we proceed the
same as Step 2, 3 in Theorem. This completes the proof. ∎

We can obtain the uniqueness result.

######  Theorem 4.9.

Let $\Phi$ be a pluripotential solution to the Cauchy problem for (CMAF) with
initial data $\varphi_{0}$. Assume that

  * •

$\Phi$ is locally uniformly semi-concave in $(0,T)$;

  * •

for each $t$, $\Phi_{t}$ and $U_{t}$ have the same singularities;

  * •

$\Phi_{t}$ is continuous in $\Omega$.

Then $\Phi=U$.

###### Proof.

We refer the reader to [Dan22b, Proposition3.4, Theorem 3.7]. ∎

## References

  * [BBE+19] R. J. Berman, S. Boucksom, P. Eyssidieux, V. Guedj, and A. Zeriahi.  Kähler-Einstein metrics and the Kähler-Ricci flow on log Fano varieties.  J. Reine Angew. Math., 751:27–89, 2019.
  * [BEGZ10] S. Boucksom, P. Eyssidieux, V. Guedj, and A. Zeriahi.  Monge-Ampère equations in big cohomology classes.  Acta Math., 205(2):199–262, 2010.
  * [Bou04] S. Boucksom.  Divisorial Zariski decompositions on compact complex manifolds.  Ann. Sci. École Norm. Sup. (4), 37(1):45–76, 2004.
  * [BT87] E. Bedford and B. A. Taylor.  Fine topology, Šilov boundary, and $(dd^{c})^{n}$.  J. Funct. Anal., 72(2):225–251, 1987.
  * [Dan22a] Q.-T. Dang.  Continuity of Monge-Ampère potentials in big cohomology classes.  Int. Math. Res. Not. IMRN, (14):11180–11201, 2022.
  * [Dan22b] Q.-T. Dang.  Pluripotential Monge-Ampère flows in big cohomology classes.  J. Funct. Anal., 282(6):Paper No. 109373, 65, 2022.
  * [DDG+14] J.-P. Demailly, S. Dinew, V. Guedj, H. H. Pham, S. Kołodziej, and A. Zeriahi.  Hölder continuous solutions to Monge-Ampère equations.  J. Eur. Math. Soc. (JEMS), 16(4):619–647, 2014.
  * [DDL18] T. Darvas, E. Di Nezza, and C. H. Lu.  Monotonicity of nonpluripolar products and complex Monge-Ampère equations with prescribed singularity.  Anal. PDE, 11(8):2049–2087, 2018.
  * [DDL21a] T. Darvas, E. Di Nezza, and C. H. Lu.  Log-concavity of volume and complex Monge-Ampère equations with prescribed singularity.  Math. Ann., 379(1-2):95–132, 2021.
  * [DDL21b] T. Darvas, E. Di Nezza, and C. H. Lu.  The metric geometry of singularity types.  J. Reine Angew. Math., 771:137–170, 2021.
  * [DDL23] T. Darvas, E. Di Nezza, and C. H. Lu.  Relative pluripotential theory on compact Kähler manifolds.  arXiv:2303.11584, 2023\.
  * [Dem92] J.-P. Demailly.  Regularization of closed positive currents and intersection theory.  J. Algebraic Geom., 1(3):361–409, 1992.
  * [Dem15] J.-P. Demailly.  On the cohomology of pseudoeffective line bundles.  In Complex geometry and dynamics, volume 10 of Abel Symp., pages 51–99. Springer, Cham, 2015.
  * [DL15] E. Di Nezza and C. H. Lu.  Generalized Monge-Ampère capacities.  Int. Math. Res. Not. IMRN, (16):7287–7322, 2015.
  * [DL17] E. Di Nezza and C. H. Lu.  Complex Monge-Ampère equations on quasi-projective varieties.  J. Reine Angew. Math., 727:145–167, 2017.
  * [DV20] D.-T. Do and D.-V. Vu.  Complex Monge-Ampere equations with solutions in finite energy.  arXiv:2010.08619, to appear in Math. Res. Letters, 2020.
  * [EGZ09] P. Eyssidieux, V. Guedj, and A. Zeriahi.  Singular Kähler-Einstein metrics.  J. Amer. Math. Soc., 22(3):607–639, 2009.
  * [GL21a] V. Guedj and C. H. Lu.  Quasi-plurisubharmonic envelopes 1: Uniform estimates on Kähler manifolds.  arXiv:2106.04273, 2021\.
  * [GL21b] V. Guedj and C. H. Lu.  Quasi-plurisubharmonic envelopes 3: Solving Monge-Ampère equations on hermitian manifolds.  arXiv:2107.01938, 2021\.
  * [GLZ18] V. Guedj, C. H. Lu, and A. Zeriahi.  Stability of solutions to complex Monge-Ampère flows.  Ann. Inst. Fourier (Grenoble), 68(7):2819–2836, 2018.
  * [GLZ19] V. Guedj, C. H. Lu, and A. Zeriahi.  Plurisubharmonic envelopes and supersolutions.  J. Differential Geom., 113(2):273–313, 2019.
  * [GZ17] V. Guedj and A. Zeriahi.  Degenerate complex Monge-Ampère equations, volume 26 of EMS Tracts in Mathematics.  European Mathematical Society (EMS), Zürich, 2017.
  * [Koł98] S. Kołodziej.  The complex Monge-Ampère equation.  Acta Math., 180(1):69–117, 1998.
  * [McC20] Nicholas McCleerey.  Envelopes with prescribed singularities.  J. Geom. Anal., 30(4):3716–3741, 2020.
  * [RW14] J. Ross and D. WittNyström.  Analytic test configurations and geodesic rays.  J. Symplectic Geom., 12(1):125–169, 2014.
  * [Yau78] S. T. Yau.  On the Ricci curvature of a compact Kähler manifold and the complex Monge-Ampère equation. I.  Comm. Pure Appl. Math., 31(3):339–411, 1978.


11institutetext:  Microsoft 11email<EMAIL_ADDRESS>22institutetext:
KU Leuven, Belgium
33institutetext: Contract.fit, Brussels, Belgium 44institutetext: L3i - La
Rochelle Université, La Rochelle, France

# Multimodal Adaptive Inference for Document Image Classification with Anytime
Early Exiting

Omar Hamed 11 Souhail Bakkali 44 0000-0001-9383-3842 Matthew Blaschko 22
0000-0002-2640-181X Sien Moens 22 0000-0002-3732-9323 Jordy Van Landeghem 2233
0000-0002-9838-3024

###### Abstract

This work addresses the need for a balanced approach between performance and
efficiency in scalable production environments for visually-rich document
understanding (VDU) tasks. Currently, there is a reliance on large document
foundation models that offer advanced capabilities but come with a heavy
computational burden. In this paper, we propose a multimodal early exit (EE)
model design that incorporates various training strategies, exit layer types
and placements111Code is available at https://github.com/Jordy-VL/multi-modal-
early-exit. Our goal is to achieve a Pareto-optimal balance between predictive
performance and efficiency for multimodal document image classification.
Through a comprehensive set of experiments, we compare our approach with
traditional exit policies and showcase an improved performance-efficiency
trade-off. Our multimodal EE design preserves the model’s predictive
capabilities, enhancing both speed and latency. This is achieved through a
reduction of over 20% in latency, while fully retaining the baseline accuracy.
This research represents the first exploration of multimodal EE design within
the VDU community, highlighting as well the effectiveness of calibration in
improving confidence scores for exiting at different layers. Overall, our
findings contribute to practical VDU applications by enhancing both
performance and efficiency.

###### Keywords:

Multimodal Document Image Classification Adaptive Inference Anytime Early
Exits

##  1 Introduction

Thanks to the advancements in multimodal deep learning (DL) systems and
hardware design, along with abundant data availability, VDU models are
becoming increasingly deeper, larger, and more powerful. However, it is
crucial to acknowledge that such advancements entail substantial demands on
both workload and memory resources. For production scalability, it is
quintessential to investigate the performance-efficiency trade-offs for such
multimodal models. Contemporary state-of-the-art (SOTA) frameworks for VDU
predominantly involve multimodal models that offer advanced capabilities
through the conventional approach of ‘pretrain-then-finetune’ in tasks such as
document image classification, named entity recognition, and document visual
question answering [18, 7, 10, 1, 22, 5, 15]. Nevertheless, substantial
computational resources are necessary to fine-tune and deploy in production, a
factor to consider when evaluating practical VDU applications. Improving the
predictive capabilities of multimodal models requires large network parameters
and computing power, albeit at the cost of efficiency.

Conversely, to improve the efficiency of a multimodal VDU system, a compromise
has to be made in its predictive capabilities. Rather than (only) chasing SOTA
on task-specific performance metrics, Pareto efficiency [16] —as illustrated
in Figure 1— is proposed as an objective better aligned with VDU real-world
practices, where both performance and efficiency are paramount. Pareto
efficiency allows comparing the significance of improvements on both criteria
and informs threshold selection among different trade-offs and compromises
that arise in scalable production settings.

When deployed in real-time industrial applications, predictive models
encounter challenges in making instantaneous decisions, as evident across
various applications. For instance, in the context of an information retrieval
system that classifies images to facilitate efficient content indexing, the
system should swiftly provide accurate classifications to support rapid and
relevant information retrieval. The system must adapt to varying processing
demands, enabling rapid decisions in contexts where quick identification of
essential information within a document is necessary. Therefore, deploying
image classification models on resource-constrained devices presents its own
set of challenges [17]. To the best of our knowledge, the research community
has not sufficiently explored multimodal models for multimodal document image
classification that can deliver high-quality results with low resource and
computational requirements.

é

Figure 1: Performance-efficiency trade-offs that allow comparing models for
when both criteria are important, which makes any point on the Pareto frontier
to be considered Pareto-efficient vs. all below the frontier.

As such, in this paper, we thoroughly investigate the use of early exiting in
multimodal models for document image classification, presenting a novel
empirical and intuition-based thresholding scheme. We explore adaptive
inference with various designs of EE strategies with a multi-exit
LayoutLMv3Base architecture [10]. We assess the efficacy of our methods across
a diverse set of language and vision-centric examples, by comparing the
baseline with both established and proposed EE strategies. The study’s
experiments are aimed at delving into the intricate relationship between the
complexity of documents and the feasibility of EE design to adapt inference
according to instance difficulty. Based on the above, we consider the
following research question: What EE architecture design would achieve SOTA
Pareto-efficiency for multimodal document image classification?

To address this question, we focus on LayoutLMv3Base [10], which has proven to
be effective in various VDU tasks. During fine-tuning, we investigate
different aspects of EE design such as training strategies, exit placements,
and exit layer types. At inference time, we experiment with post-hoc
calibration to adjust the confidence scores of the exit layers, and propose a
novel heuristic EE policy for optimal thresholding to compare with both global
thresholding and multi-exit thresholding. The reported results demonstrate
that our multi-exit design yields substantial improvements in terms of speed
and latency, while maintaining the majority of the model’s predictive
capabilities. This underscores the practicality and effectiveness of our
proposed methods in achieving notable efficiency gains without compromising
predictive accuracy.

##  2 Related work

##### Dynamic Neural Networks with Adaptive Early-Exit Inference.

Dynamic Neural Networks [26, 25, 29] are an enhanced type of neural networks
that can dynamically adjust their computational path based on the input,
resulting in significant improvements in inference time. Multiple research
efforts investigate how to effectively design early-exit networks. DeeBERT
[21] updates BERT architecture by inserting intermediate classifiers between
the encoder layers, then jointly fine-tunes them with the original
classification layer while freezing the model backbone. This method achieves
up to 40% inference time savings with minimal loss of accuracy on GLUE tasks.
In [6], RomeBERT improved on DeeBERT by introducing gradient regularization
and self distillation. Gradient regularization allows the training of the
model backbone whereas self-distillation allows the use of the predictions of
the final layer as soft targets for the off-ramps, and minimizes the KL-
divergence between them to fix the significant drop in intermediate-
classifiers performance. BERxiT [24], improves on the existing EE
architectures and extends the early exiting design for regression tasks by
introducing a learning-to-exit (LTE) module. The LTE module and the
intermediate classifiers are set up simultaneously and shared by all layers.
The whole model, along with the LTE module, is then trained end-to-end.

##### Training Strategies for Dynamic Inference.

Different training strategies have shown their effectiveness in training
early-exit networks. In [13], the authors discuss two strategies. The first
strategy is end-to-end training, which minimizes a single loss function that
combines the losses of the intermediate classifiers and the final classifier.
The second strategy is training of intermediate classifiers, which consists of
two stages: in the first stage, the backbone model is trained without any
extra classifiers; in the second stage, the backbone weights are frozen, and
the intermediate classifiers are added and trained separately. Additionally,
[24] presents a novel fine-tuning strategy named Alternating. It alternates
between two objectives for each iteration, based on whether the iteration is
odd or even. For odd iterations, it only updates the weights of the backbone
model, while during even iterations, it updates only the intermediate
classifiers.

##### Inference Strategies.

There are different inference strategies that determine the model’s exiting
behavior. Inference strategies are composed of two main points: exit criterion
and exit policy. The exit criterion includes confidence scoring functions
(CSFs) that allow the model to exit early based on the input difficulty and
the quality of the predictions. Meanwhile, the exit policy is associated with
thresholding the early exiting branches. One of the common CSFs is maximum
confidence [13, 9], where the model exits at the first branch that satisfies a
pre-defined confidence threshold. Another approach is the entropy-based
criterion [20], where the model exits at the branch with the least entropy
value. DynamicDet [19] proposes an adaptive router for image classification
tasks, using an exit criterion based on the detection losses of each exit
classifier, which are employed to compute a reward signal for the adaptive
router. DVT [23] presents an architecture with a series of Transformers with
varying token numbers. These are activated one by one at inference time,
depending on the confidence score of each classifier in the series.
Additionally, Zhou et al.[29] propose EE when multiple consecutive classifiers
agree on the same prediction. Regarding the exit policy, the majority of
research adopts a greedy approach, utilizing a single threshold for all exit
classifiers [3, 4, 14]. In contrast, alternative methods aim to determine
thresholds more effectively. For example, EENET [11] employs a scheduling
architecture to learn an optimal early-exit policy. Similarly, [28] utilizes
genetic algorithms to learn optimal thresholds, while [12] applies Bayesian
optimization for the same purpose.

While prior research has focused on unimodal architectures, this paper breaks
new ground by introducing EE network design for multimodal DU tasks. The most
comparable work is DynMM [27]; however, it employs a different approach,
incorporating data fusion from diverse modalities. Our work analyzes how
different exit classifiers (such as gates and ramps), diverse training
strategies, various exit placements, and different exit policies impact the
multimodal EE architecture. The study also explores the substantial
performance enhancements achieved through the calibration of exit classifiers.
Moreover, we present a novel and empirical exit policy heuristic that
surpasses traditional methods without adding any extra computational burden
for finding optimal thresholds.

##  3 Methodology

This work focuses on optimizing the use of LayoutLMv3Base across modalities
through an EE network design. This design allows premature termination from
the text-modality layer, visual modality layer, or their combined output,
after modality fusion or different encodings. In multimodal architectures,
dedicated layers for each modality facilitate tailored EEs, optimizing
inference based on efficiency. In LayoutLMv3Base, exiting from the vision
modality is particularly efficient, avoiding the costly OCR step for text.
Drawing on the information bottleneck theory [21], deeper layers capture more
information, enabling the model to learn intricate patterns for accurate
predictions. Firstly, we investigate various exit placements by adding
intermediate classifiers at different parts of the network, and exploring both
gates [24] and ramps [6] as exit layer types. Secondly, to optimize a multi-
exit network, we explore various fine-tuning strategies such as subgraph
weighting,exit entropy regularization. Finally, we compare different exiting
and inference policies, such as global thresholding and guided random multi-
exit threshold, and propose a new heuristic exit policy.

###  3.1 EE Design

Given a multimodal input-output pair $(x,y)$ and a pre-trained multimodal
architecture with $L$ layers, organized according to a computational graph $G$
(allowing for potential parallel computation), the goal is to obtain a
parameterized mapping $f(\cdot;\theta):\mathcal{X}\to\mathcal{Y}$, where
$\mathcal{Y}=[K]$ for $K$ mutually exclusive classes. This is typically
obtained by minimizing a loss function (e.g. , cross-entropy (CE)) over the
last layer output (logits), $\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta_{L})$,
where $\theta_{L}$ holds all (trainable) parameters until the $L$-th layer. An
EE model extends $f$ by introducing $B$ additional intermediate classifiers,
which (for layer $l\in[L]$) can be either a gate $g:f(X;\theta_{l-1}\to{0,1})$
passing on the intermediate representation to the final classifier, or a ramp
$r(\cdot;\theta):f(X;\theta_{l-1})\to[K]$ directly making a final prediction.

###  3.2 Fine-tuning Strategies

####  3.2.1 Training Configuration.

Exploring the trade-offs in EE design, we implement three distinct training
strategies that offer different approaches for updating the weights of $B$
subgraphs in a multi-exit network, with each subgraph $G_{b}\subset G$
incorporating all parameters ($\theta_{G_{b}}$) used up to the corresponding
exit branch. The most straightforward strategy, known as multi-exit loss,
involves combining the losses for each subgraph with the classifier loss:

$\displaystyle\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta))+\sum_{b=1}^{B}\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta_{G_{b}}))$
(1)

This objective assumes uniform weights for all $B$ loss terms and makes no
assumptions about the importance of exits compared to the final classifier.
Other training strategies enhance Equation 1 by i) restricting loss updates to
parameters within the scope of the exit branch, ii) incorporating a
hyperparameter ($\gamma\in[0,1]$) to regulate the influence of all exits
versus the classifier [6], and iii) introducing an entropy regularization term
over all exit losses to assign weights based on normalized uncertainty
relative to other exits.

  * •

Subgraph Weighting: This training strategy implements $\gamma$ to tradeoff
between exit classifiers and the final classifier. Additionally, each exit
weight is based on the relative number of parameters used, with the inverse
weighting reinforcing the importance of earlier exits.

$\displaystyle(1-\gamma)\underbrace{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta))}_{\text{Final
classifier
loss}}+\gamma\underbrace{\sum_{b=1}^{B}w_{b}\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta_{G_{b}}))}_{\text{Exit
classifiers loss}}$ (2)

where $w_{b}$ is the weight assigned to the b-th exit (e.g. using 5% of
parameters would give a weight of 20), and $\theta_{G_{b}}$ is the set of
parameters up to the b-th layer.

  * •

Exit Entropy Regularization: This strategy relies on an entropy regularization
term [2] to discourage branches with high uncertainty. Intuitively, exits with
high entropy respective to all other exits will receive less importance than
exits with low entropy, which exhibit high certainty in their prediction. The
entropy regularization term measures the mutual information between each
branch and its target output, and imposes a penalty on branches with lower
mutual information.

$\displaystyle(1-\gamma)\underbrace{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta))}_{\text{Final
classifier
loss}}+\gamma\underbrace{\sum_{b=1}^{B}\lambda_{b}\mathcal{L}_{\mathrm{CE}}(Y,f(X;\theta_{G_{b}}))}_{\text{Exit
classifiers loss}}$ (3)

where $\lambda_{b}$ is the regularization coefficient for the b-th exit, which
arises from the calculation of the exit layer entropy $H(x,\cdot)$, normalized
over all other exits, and inversed,
$\mathbf{\lambda}=1-\mathrm{softmax}(H(Y;f(X;\theta_{G})))$.

  * •

Weighted - Entropy Regularization: This strategy integrates both subgraph
weighting and entropy regularization strategies, by using a weighted loss
function and an entropy regularization term for each exit. When applying this
joint strategy, we set $\gamma=0.5$ to ensure weighting stability.

####  3.2.2 Exits Placement Configuration.

We conduct experiments with diverse exit branch configurations throughout the
network to assess their impact on model performance. Given the exponential
increase in possible combinations with network depth, we choose six exit
placement configurations capable of capturing distinct model aspects.
Subsequently, each of these placement configurations undergoes training using
the aforementioned strategies. The exit configurations are as follows:

  * •

Independent-All: Each modality (vision and language) has its own exit branch,
along with an exit branch at every encoder layer. This results in 14 exits in
total.

  * •

Concat-All: A single exit branch is positioned after the concatenation layer,
along with an exit branch at every encoder layer. This results in 13 exits in
total.

  * •

Concat-Single: A single exit branch is positioned after the concatenation
layer, and another exit branch is positioned after the 6th encoder layer. This
results in 2 exits in total.

  * •

Concat-Quarter: Four exit branches are placed at the 1st, 4th, 8th, and 12th
encoder layers, in addition to the exit branch after the concatenation layer.
This results in 5 exits in total.

  * •

Concat-Alternate: Four exit branches are located at the 2nd, 5th, 9th, and
11th encoder layers, in addition to the exit branch after the concatenation
layer. This also results in 5 exits in total.

####  3.2.3 Exit Classifiers Type.

Moreover, we investigate the impact of various exit classifiers on model
performance. For each combination of a training strategy and an exit placement
configuration, we employ either ramps (i.e. softmax activation) or gates (i.e.
sigmoid activation) as the exit classifier, resulting in a base total of 32
experiments (without any hyperparameter tuning). A gate takes the intermediate
representation $f(X;\theta_{b-1})$ as input and produces after a sigmoid
activation a probability $p_{b,i}$ for exiting. The gate is trained by
minimizing a binary cross-entropy loss function, expressed as follows:

$\displaystyle\mathcal{L}_{\mathrm{gate}}(Y,f(X;\theta_{b-1}))=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\log
p_{b,i}+(1-y_{i})\log(1-p_{b,i})$ (4)

Where N represents the number of samples, $y_{i}$ denotes the ground truth
label (0 or 1) for the i-th sample, and $p_{b,i}$ represents the output
probability of the gate for the i-th sample at exit layer $b$.

Conversely, a ramp serves as a multi-class classifier producing a probability
distribution across $K$ classes to indicate the predicted class label for the
input. It involves a softmax function, which takes the intermediate
representation $f(X;\theta_{b-1})$ as input and generates a vector $p_{b,i}$
containing probabilities for each class. The training of a ramp is
accomplished by minimizing a cross-entropy loss function, as follows:

$\displaystyle\mathcal{L}_{\mathrm{ramp}}(Y,f(X;\theta_{b-1}))=-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K}y_{i,k}\log
p_{b,i,k}$ (5)

Where N represents the number of samples, $y_{i,k}$ denotes the ground truth
label (0 or 1) for the i-th sample and the k-th class, and $p_{b,i,k}$
represents the output probability of the ramp for the i-th sample and the k-th
class at exit layer $b$.

###  3.3 Inference Strategies

####  3.3.1 Exit criterion.

In this work, we opt for the Maximum Softmax Probability (MSP), shown in
Equation 6 as the exit criterion. We were motivated by the minimal difference
observed compared to using alternative strategies such as entropy.
Additionally, MSP, being an L2 metric, produces values within the range of 0
to 1, simplifying the determination of exit thresholds.

$\displaystyle\displaystyle
p_{b}\approx\mathrm{MSP}(x;\theta_{G_{b}})=\max_{k\in
K}\frac{\exp(z_{b,k})}{\sum_{c=1}^{K}{\exp{(z_{b,c})}}},\text{ }b\in[1,B-1],$
(6)

where $z_{b}$ denotes the raw logits at layer $b$. The exit criterion should
indicate the ’probability’ of correctly exiting, as the MSP should be an
approximation of.

####  3.3.2 Calibration.

Initiating our evaluation, we calibrate the exit logits of each layer using a
validation set. Employing temperature scaling, we aim to determine the optimal
temperature $T$ that minimizes cross-entropy. Subsequently, this optimal
temperature value is applied to smoothen the logits $z$ at each exit
classifier, as outlined in Equation 7:

$\displaystyle
p_{b,k}=\frac{\exp({\frac{z_{b,k}}{T}})}{\sum_{c=1}^{K}{\exp{(\frac{z_{b,c}}{T})}}},\text{
}b\in[1,B-1],$ (7)

where $T$ represents the temperature scaling factor, and $p_{b,k}$ represents
the calibrated probabilities at exit layer $b$ for class $k$.

####  3.3.3 Global Thresholding.

Exits are activated when their confidence value exceeds a certain threshold.
However, the challenge comes in choosing this threshold for each exit. A
common approach is to assign a single global threshold for all exits as shown
in Equation 8. Despite that the global threshold approach could be a simple-
to-setup solution that can achieve some computational reduction gains, it is
still suboptimal.

$\displaystyle\hat{y}=\begin{cases}f_{b}(x;\theta)&\text{if
}p_{b}\geq\tau,b\in[1,B-1]\land\tau\in]0,1[\\\
f_{L}(x;\theta)&\text{otherwise}\end{cases}$ (8)

where $x$ is the input, $\hat{y}$ is the output prediction, $f_{b}$ and
$p_{b}$ are the output prediction and output probability of the b-th
intermediate classifier respectively, $\tau$ is the global threshold, and $B$
is the number of exit classifiers.

####  3.3.4 Multi-exit Thresholding.

exploits more degrees of freedom where each layer has its own threshold as
shown in Equation 9. The best thresholds would be found by a brute force
method, but this is impractical because the number of possible thresholds
grows exponentially with the number of exits. We use a heuristic random
subsampling method and limit our experiments to $10e+6$ random combinations of
thresholds for each exit. To find the potential thresholds for each exit, this
method applies a simple heuristic to obtain data-driven thresholds: it
computes the MSPs for each exit and then selects the $i^{th}$ percentile of
the MSP vector as a potential threshold. Different 10th percentiles are
randomly selected per exit and combined with those of other exits to create
diverse multi-threshold sets.

$\displaystyle\hat{y}=\begin{cases}f_{b}(x;\theta)&\text{if
}p_{b}\geq\tau_{b}\text{ where }b\in[1,B-1]\land\tau_{b}\in]0,1[\\\
f_{L}(x;\theta)&\text{otherwise}\end{cases}$ (9)

where $\tau_{b}$ is the selected threshold for a layer $b$.

####  3.3.5 Proposed Thresholding Heuristic.

Given the shortcomings of both the global and randomized multi-exit
thresholding, we propose a dynamic thresholding heuristic to serve as a well-
rounded approximation. The proposed formula sets a different threshold for
each layer, depending on the tradeoff between accuracy and calibration error
at that layer. A lower threshold value indicates a better balance between
accuracy and calibration, while a higher value indicates a worse balance. A
threshold value of 0 means that the accuracy and calibration error are equal,
implying perfect calibration. A threshold value of 1 means that the accuracy
is zero, implying complete inaccuracy. The proposed heuristic can be expressed
as follows:

$\displaystyle T_{b}=1-\frac{\mathrm{ACC}_{b}}{\mathrm{ECE}_{b}}\text{ where
}b\in[1,B-1]$ (10)

where $\mathrm{ACC}_{b}$ and $ECE_{b}$ are the post-calibration accuracy and
expected calibration error at exit layer $b$ respectively.

To measure the discrepancy between the average confidence and the average
accuracy of the predictions, we use the expected calibration error (ECE),
which is computed over $J$ bins as follows:

$\displaystyle\text{ECE}=\sum_{j}^{J}\frac{\mathcal{B}_{j}}{N}|{\mathrm{ACC}(\mathcal{B}_{j})-\mathrm{MSP}(\mathcal{B}_{j})|}$
(11)

where $\mathrm{ACC}(\mathcal{B}_{j})$ and $\mathrm{MSP}(\mathcal{B}_{j})$
represent the average probability and the expected confidence for bin $j$,
respectively.

A min-max normalization is applied to the potential thresholds for each layer.
A tuneable parameter $\varepsilon$ is introduced to the min-max normalization,
which avoids the thresholds from being exactly 0 or 1 and controls the gap
between them. The normalization step is as in Equation 12:

$\displaystyle\overrightarrow{\tau}_{normalized}=\frac{{(\overrightarrow{\tau})}-(\min(\overrightarrow{\tau})-\varepsilon)}{(\max(\overrightarrow{\tau})+\varepsilon)-(\min(\overrightarrow{\tau})-\varepsilon)}$
(12)

where $\overrightarrow{\tau}$ is a vector for all layers thresholds
[$\tau_{1}$, $\tau_{2}$, …, $\tau_{B-1}$] and $\varepsilon$ is a tunable
parameter to control the absolute min and max values.

##  4 Experiments and Results

##### Experimental Setup.

To evaluate the effectiveness of our proposed EE design, we use a
subset222https://huggingface.co/datasets/jordyvl/rvl_cdip_100_examples_per_class
of the Ryerson Vision Lab Complex Document Information Processing (RVL-CDIP)
benchmark dataset [8]. It consists of 800 samples for training, 400 samples
for validation, and 400 samples for test sets. The samples are balanced over
the 16 classes : advertisement, budget, email, file folder, form, handwritten,
invoice, letter, memo, news article, presentation, questionnaire, resume,
scientific publication, scientific report, specification. The motivation for
limiting experiments on a subset of data comes on the one hand, from the need
to explore the exponential size EE design space, while on the other hand,
real-word settings typically do not provide 25K samples per document type,
thus imposing real-world constraints.

We apply easyOCR to extract the text within document images along with the 2D
bounding box coordinates. We use LayoutLMv3Base as the baseline model and
evaluate its performance based on both accuracy and efficiency, with latency
as a measure. Latency is then calculated as a function of the exit location
relative to the final classifier, assuming identical parameters and complexity
for all exit classifiers. We start our experiments by re-implementing
LayoutLMv3Base on the RVL-CDIP subset., and denote the result as
LayoutLMv3${}_{Base}^{\dagger}$. LayoutLMv3${}_{Base}^{\dagger}$ yields and
accuracy of 80.75% on document image classification (i.e. with no exits),
which is used as a benchmark for the conducted experiments. For fine-tuning,
AdamW is used as an optimizer with a batch size of 2 and a learning rate of
1e-4 for 20 epochs.

Figure 2: Effect of calibration for Ramp exits and training strategy.

##### Effect of Calibration for Ramp Exits.

In Figure 2, the comparison between calibrated and uncalibrated exits for
ramps is depicted, focusing on the tradeoffs between performance and
efficiency. The x-axis illustrates latency reduction, while the y-axis
represents accuracy. The graphs span thresholds from 0 to 1 with a step of
0.05, revealing the impact of calibration on the tradeoff between performance
and efficiency across various training methods and the ramp exit classifier in
global thresholding. The results show that calibration facilitates more
adaptable thresholding in most scenarios, whereas the uncalibrated version
exhibits constraints, with latency improvements leveling off at relatively low
thresholds compared to the calibrated version. For example, we observe that
the calibrated versions of Weighted - Ramp and Entropyreg - Ramp have better
performance-efficiency tradeoffs than the uncalibrated versions for all
models. The Weighted-entropyreg - Ramp performs worse than other training
strategies, requiring better tuning of the multiple weightings.

Figure 3: Effect of calibration for Gate exits and training strategy

##### Effect of Calibration for Gate Exits.

In contrast to the ramp exit type, the gate exit type exhibits a distinct
Pareto-efficiency trend, as depicted in Figure 3. Relative to ramps, gates
demonstrate less pronounced latency reduction at lower thresholds. Despite
this, calibration consistently enhances trade-offs in most scenarios compared
to their uncalibrated counterparts. Interestingly, specific configurations
perform better with ramps than with gates, as seen in the case of the Concat-
single configuration, which exhibits limited latency reduction with gates but
higher efficiency with ramps. In addition, different training methods impact
how well a specific model configuration performs. The weighted training method
seems to achieve a higher Pareto-efficiency tradeoff, especially for the
calibrated version, than the other training methods. For instance, the Concat-
All variant can balance performance and efficiency better with the weighted
method than the weighted-entropyreg method.

##### Exit Policies Comparison.

Figure 4 compares the performance of different exit policies (global threshold
uncalibrated, global threshold calibrated, multi-exit thresholding, and
proposed thresholding heuristic) for a subset of model configurations. These
configurations are denoted as Weighted—Concat-alternate—Ramp, Weighted—Concat-
quarter—Ramp, Weighted—Independent-all—Ramp, and
Entropyreg— Concat-alternate —Gate. The proposed heuristic can achieve a
similar or better tradeoff than global or multi thresholding on calibrated
exits without requiring extra tuning. For example, the heuristic method has
higher accuracy and comparable latency reduction to global thresholding in the
weighted — Independent-all — Ramp configuration. However, the heuristic method
is less effective than global thresholding in the Entropyreg — Concat —
alternate — Gate configuration in terms of latency improvement, yet it yields
better accuracy. Nevertheless, it is evident that the heuristic method can
offer a competitive tradeoff without requiring tuning or searching in the
multi-threshold space. The optimal inference strategy depends on the use case
and the preference for computation efficiency or accuracy. Some may take
advantage of the many options of multi-thresholding, while others may simply
resort to the heuristic method.
Taking the weighted — Concat-quarter — Ramp as an example, it demonstrates
better performance than the baseline model. In the heuristic approach, with an
accuracy of 80.75%, the associated latency reduction is 20%. Comparatively,
employing global thresholding showcases varying latencies for different
accuracy levels: at 81%, the latency is 16.16%; at 81.25%, it drops to 13.29%,
and further decreases to 9% at an accuracy of 81.75%. This analysis
underscores the nuanced trade-offs between accuracy and latency under
different thresholding strategies.

Figure 4: A comparison between the different exit policies, varying
calibration. Figure 5: Exit patterns for Weighted — Concat-quarter — Ramp
configuration on RVL-CDIP dataset classes, showing the exit behaviour for
different intermediate classifiers per class

##### Exit Patterns.

Furthermore, we carried out a qualitative analysis to identify patterns about
which classes tend to be predicted correctly and allow the model to exit at a
relatively early stage compared to the final top classifier. We used the
Weighted — Concat-quarter — Ramp model as an example, as shown in Figure 5.
The figure shows log-scaled normalized counts of how often each class is
exited at a given intermediate classifier. It is clear from Figure 5 that some
classes, such as file_folder, specification, and scientific publication, are
more likely to be exited correctly at relatively earlier stages, indicating
that these classes are somewhat easier to classify than others. On the other
hand, classes such as news_article and scientific_report are more likely to be
exited at relatively later stages or even at the top classifier, indicating
that they require more complex and rich features for the model to learn to be
accurately predicted. We note that the exact exit classifier may differ
depending on the model configuration, due to changes in exit placements and
the number of classifiers. However, interestingly, the exit patterns for
classes are remarkably consistent across different models. Figure 6
illustrates different samples that are correctly classified at different
stages of the network. As for the file_folder category, it is the easiest to
identify among the other classes because it primarily comprises a white page
with few texts located at the bottom right corner of the document image. This
type of content does not require processing through all encoders to be
accurately predicted. In contrast, the specification category poses a slightly
greater challenge due to its larger amount of text and intricate layout. This
includes the presence of substantial structured tables, a characteristic
shared with other classes such as form, questionnaire, and
scientific_publication. These variations contribute to the complexity of
inter-class variability, which presents a notable challenge in multimodal
document image classification. Moreover, the news_article category poses the
greatest challenge in classification, as it is frequently mistaken for the
advertisement category. This confusion arises because both categories often
feature similar visual elements, including headlines, images, and text blocks.
Consequently, classification models struggle to distinguish between them based
solely on visual or text features. As a result, document images typically need
to be processed through more layers of the network to achieve accurate
predictions.

Figure 6: Samples for different classes with different difficulty and exited
at different places in the network

##  5 Conclusion

This research represents the first exploration of multimodal early exiting in
the document understanding community. The research studies how to achieve a
controllable balance between efficiency and performance in VDU tasks. The EE
design incorporates various training strategies, exit layer types, and exit
placements to achieve Pareto-optimality. Our experiments demonstrate the
importance of calibrating the exit branches to significantly improve the
possible tradeoffs for flexible tuning based on the use case. Additionally,
different exit thresholding policies were compared, in which we observed
improvements in performance-efficiency trade-offs with the proposed
thresholding approaches, with a reduction of over 20% in latency while
maintaining a minimal loss in accuracy when weighted subgraphs and ramps are
used as training strategy and exit type, respectively. Overall, the proposed
methodology opens avenues for further research in adaptive inference for
multimodal tasks, offering a promising direction for scalable VDU
applications.

##### Limitations.

While our research has yielded promising results, it is essential to
acknowledge its limitations. Our experiments were focused on a classification
task that is discriminatively modeled, i.e. a limited output space respective
to the number of classes, whereas EE design for generative models remains to
be explored, e.g. large language models with outputs requiring autoregressive
decoding and early exiting at different timesteps. Additionally, we did not
study the effect of combining multiple inference optimization approaches such
as distillation, quantization, and pruning with the current EE design. As the
experiments were conducted on a smaller subset, it would be interesting to see
if the results scale with larger training data or with larger architectures.
These unexplored aspects present opportunities for future research to enhance
the comprehensiveness of our findings and provide a more nuanced understanding
of the proposed approach.

## References

  * [1] Appalaraju, S., Tang, P., Dong, Q., Sankaran, N., Zhou, Y., Manmatha, R.: Docformerv2: Local features for document understanding. arXiv preprint arXiv:2306.01733 (2023)
  * [2] Araujo, V., Hurtado, J., Soto, A., Moens, M.F.: Entropy-based stability-plasticity for lifelong learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3721–3728 (2022)
  * [3] Baccarelli, E., Scardapane, S., Scarpiniti, M., Momenzadeh, A., Uncini, A.: Optimized training and scalable implementation of conditional deep neural networks with early exits for fog-supported iot applications. Information Sciences 521, 107–143 (2020)
  * [4] Berestizshevsky, K., Even, G.: Dynamically sacrificing accuracy for reduced computation: Cascaded inference based on softmax confidence. In: International Conference on Artificial Neural Networks. pp. 306–320. Springer (2019)
  * [5] Blau, T., Fogel, S., Ronen, R., Golts, A., Ganz, R., Avraham, E.B., Aberdam, A., Tsiper, S., Litman, R.: Gram: Global reasoning for multi-page vqa. arXiv preprint arXiv:2401.03411 (2024)
  * [6] Geng, S., Gao, P., Fu, Z., Zhang, Y.: Romebert: Robust training of multi-exit bert (2021)
  * [7] Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Barmpalios, N., Nenkova, A., Sun, T.: Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems 34, 39–50 (2021)
  * [8] Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). pp. 991–995. IEEE (2015)
  * [9] Huang, G., Chen, D., Li, T., Wu, F., Van Der Maaten, L., Weinberger, K.Q.: Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844 (2017)
  * [10] Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: Pre-Training for Document AI With Unified Text and Image Masking. ACM International Conference on Multimedia pp. 4083–4091 (2022)
  * [11] Ilhan, F., Chow, K.H., Hu, S., Huang, T., Tekin, S., Wei, W., Wu, Y., Lee, M., Kompella, R., Latapie, H., et al.: Adaptive deep neural network inference optimization with eenet. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1373–1382 (2024)
  * [12] Jayakodi, N.K., Chatterjee, A., Choi, W., Doppa, J.R., Pande, P.P.: Trading-off accuracy and energy of deep inference on embedded systems: A co-design approach. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37(11), 2881–2893 (2018)
  * [13] Laskaridis, S., Kouris, A., Lane, N.D.: Adaptive inference through early-exit networks: Design, challenges and directions. In: Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning. pp. 1–6 (2021)
  * [14] Li, H., Zhang, H., Qi, X., Yang, R., Huang, G.: Improved techniques for training adaptive deep networks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1891–1900 (2019)
  * [15] Li, P., Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Manjunatha, V., Liu, H.: Selfdoc: Self-supervised document representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5652–5660 (2021)
  * [16] Liu, X., Sun, T., He, J., Wu, J., Wu, L., Zhang, X., Jiang, H., Cao, Z., Huang, X., Qiu, X.: Towards efficient NLP: A standard evaluation and a strong baseline. In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (eds.) Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3288–3303. Association for Computational Linguistics, Seattle, United States (Jul 2022)
  * [17] Matsubara, Y., Levorato, M., Restuccia, F.: Split computing and early exiting for deep learning applications: Survey and research challenges. ACM Computing Surveys 55(5), 1–30 (2022)
  * [18] Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., Pałka, G.: Going full-tilt boogie on document understanding with text-image-layout transformer. In: International Conference on Document Analysis and Recognition. pp. 732–747. Springer (2021)
  * [19] Tang, S., Wang, Y., Kong, Z., Zhang, T., Li, Y., Ding, C., Wang, Y., Liang, Y., Xu, D.: You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10781–10791 (2023)
  * [20] Teerapittayanon, S., McDanel, B., Kung, H.T.: Branchynet: Fast inference via early exiting from deep neural networks. In: 2016 23rd international conference on pattern recognition (ICPR). pp. 2464–2469. IEEE (2016)
  * [21] Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle. In: 2015 ieee information theory workshop (itw). pp. 1–5. IEEE (2015)
  * [22] Wang, D., Raman, N., Sibue, M., Ma, Z., Babkin, P., Kaur, S., Pei, Y., Nourbakhsh, A., Liu, X.: Docllm: A layout-aware generative language model for multimodal document understanding (2023)
  * [23] Wang, Y., Huang, R., Song, S., Huang, Z., Huang, G.: Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems 34, 11960–11973 (2021)
  * [24] Xin, J., Tang, R., Yu, Y., Lin, J.: Berxit: Early exiting for bert with better fine-tuning and extension to regression. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: Main Volume. pp. 91–104 (2021)
  * [25] Xing, Q., Xu, M., Li, T., Guan, Z.: Early exit or not: Resource-efficient blind quality enhancement for compressed images. In: European Conference on Computer Vision. pp. 275–292. Springer (2020)
  * [26] Xu, C., McAuley, J.: A survey on dynamic neural networks for natural language processing. arXiv preprint arXiv:2202.07101 (2022)
  * [27] Xue, Z., Marculescu, R.: Dynamic multimodal fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2574–2583 (2023)
  * [28] Zhang, L., Tan, Z., Song, J., Chen, J., Bao, C., Ma, K.: Scan: A scalable neural networks framework towards compact and efficient models. Advances in Neural Information Processing Systems 32 (2019)
  * [29] Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., Wei, F.: Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems 33, 18330–18341 (2020)